How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?

Authors: jiahua  dong
wenqi  liang
hongliu  li
duzhen  zhang
meng  cao
henghui  ding
salman  khan
shahbaz  fahad
  khan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-05 07:48:14

Abstract: Custom diffusion models (CDMs) have attracted widespread attention due to their astonishing generative ability for personalized concepts. However, most existing CDMs unreasonably assume that personalized concepts are fixed and cannot change over time. Moreover, they heavily suffer from catastrophic forgetting and concept neglect on old personalized concepts when continually learning a series of new concepts. To address these challenges, we propose a novel Concept-Incremental text-to-image Diffusion Model (CIDM), which can resolve catastrophic forgetting and concept neglect to learn new customization tasks in a concept-incremental manner. Specifically, to surmount the catastrophic forgetting of old concepts, we develop a concept consolidation loss and an elastic weight aggregation module. They can explore task-specific and task-shared knowledge during training, and aggregate all low-rank weights of old concepts based on their contributions during inference. Moreover, in order to address concept neglect, we devise a context-controllable synthesis strategy that leverages expressive region features and noise estimation to control the contexts of generated images according to user conditions. Experiments validate that our CIDM surpasses existing custom diffusion models. The source codes are available at https://github.com/JiahuaDong/CIFC.

Main paper: 
 introduction: 
 Latent diffusion models (LDMs) [38, 33, 66, 19] have demonstrated unprecedented capabilities in generating high-fidelity images by incorporating large-scale collections of image-text pairs in the latent feature space. Until now, LDMs [4, 30] have achieved remarkable progress in various application fields, including image editing [29, 26] , art creation [8, 54] , and reconstruction of fMRI brain scans [48] . In order to synthesize some personalized concepts according to user preferences, custom diffusion models (CDMs) [40, 60, 11] rely on low-rank adaptation (LoRA) [14] to finetune the large-scale LDMs for multi-concept customization [22] . They extend the vision-language dictionary of pretrained LDMs to bind personalized concepts with specific subjects users need to synthesize.Generally, most existing CDMs [60, 68, 3] assume that users' personalized concepts are fixed and cannot incrementally increase over time. However, this assumption is unrealistic in real-world applications, where users want to continually synthesize a series of new personalized concepts from their own lives. To address this setting, CDMs [3, 57, 2] typically require storing all image-text training pairs of old concepts to finetune the pretrained LDMs via LoRA [14, 59] . Nevertheless, the high computation costs and privacy concerns [52] may render CDMs impractical as the number of old personalized concepts consecutively increases. If the above CDMs retain all low-rank weights associated with old concepts that are obtained in previous customization tasks and then merge them to learn new personalized concepts continually [59, 65] , they may experience significant loss of individual attributes on old personalized concepts (i.e., catastrophic forgetting [36, 6] ) for versatile customization. Moreover, in real-world scenarios, users may wish to control the contexts and objects associated with multiple old concepts in synthesized images according to the conditions they provide (e.g., scribble or bounding box [24, 44] ). It forces CDMs [60] to heavily suffer from the challenge of concept neglect [1] (i.e., some old concepts are missing during multi-concept composition).To handle the above real-world scenarios, in this paper, we propose a new practical problem named Concept-Incremental Flexible Customization (CIFC). In the CIFC setting, as shown in Fig. 1(a) , CDMs can consecutively synthesize a sequence of new personalized concepts in a concept-incremental manner for versatile customization (e.g., multi-concept generation [21] , style transfer [56] and image editing [29] ). Additionally, users can control the context and objects of the generated images based on the specific conditions they provide. As aforementioned, the CIFC problem faces two main challenges for versatile concept customization in this paper: catastrophic forgetting of old personalized concepts when learning new concepts consecutively under a concept-incremental manner, and concept neglect when performing multi-concept composition according to users-provided conditions.To resolve the challenges in CIFC, we develop a novel Concept-Incremental text-to-image Diffusion Model (CIDM), which can effectively address catastrophic forgetting and concept neglect. On one hand, to mitigate the catastrophic forgetting of old personalized concepts, we propose a novel concept consolidation loss for training and devise an elastic weight aggregation (EWA) module for inference. This loss employs learnable layer-wise concept tokens and an orthogonal subspace regularizer to explore task-specific knowledge (i.e., unique attributes of personalized concepts), while learning layer-wise common subspaces across different tasks to capture task-shared knowledge. Additionally, the EWA module utilizes learnable layer-wise concept tokens to merge all low-rank weights of old personalized concepts, based on their contributions to versatile concept customization. On the other hand, we develop a context-controllable synthesis strategy to tackle concept neglect for multi-concept composition. It leverages layer-wise textual embeddings to enhance the expressive ability of region features and relies on region noise estimation to control the contexts of generated image, conforming to users-provided conditions. Comprehensive experiments illustrate the effectiveness of our proposed CIDM in addressing the CIFC problem. The main contributions of this paper are listed below:• We propose a new practical problem named Concept-Incremental Flexible Customization (CIFC), where the main challenges are catastrophic forgetting and concept neglect. To address the challenges in the CIFC problem, we develop a novel Concept-Incremental text-to-image Diffusion Model (CIDM), which can learn new personalized concepts continuously for versatile concept customization.• We devise a concept consolidation loss and an elastic weight aggregation module to mitigate the catastrophic forgetting of old personalized concepts, by exploring task-specific/task-shared knowledge and aggregating all low-rank weights of old concepts based on their contributions in the CIFC.• We develop a context-controllable synthesis strategy to tackle concept neglect. This strategy controls the contexts of synthesized images according to user conditions by enhancing the expressive ability of region features with layer-wise textual embeddings and incorporating region noise estimation. 
related work: 
 Incremental Learning [25, 49, 20] accumulates previous experience to incrementally learn new tasks without the need for retraining from scratch. To prevent catastrophic forgetting of old tasks, most incremental or continual learning models mainly employ knowledge distillation between old and new tasks [63, 36, 5] , replay some images from old tasks [16, 43] , or dynamically expand network architecture to encode new knowledge [61, 15, 7] . Nevertheless, these methods [16, 61, 49, 36] are primarily designed for classifying new object categories consecutively, which cannot be directly applied to tackle continual concept customization tasks without catastrophic forgetting. [11, 40, 22] focuses on extending large-scale diffusion model [31, 46, 64] to synthesize personalized concepts for users. After [40] proposes to tackle the subject-driven generation by finetuning all network parameters of the pretrained diffusion model [48] on personalized concepts, some works use textual inversion [9, 53] to learn word embeddings of personalized concepts [50] . For multi-concept customization, [22] can jointly train multiple concepts or combine different diffusion models by optimizing a few parameters in the cross-attention layers, while [27] aims to capture different clusters of concept neurons. Motivated by [22] , Han et al. [12] finetune the singular values of latent encoding weights, thereby improving the efficiency for concept customization. [42, 57, 18] perform efficient test-time customization by training concept-specific encoders. Besides, [59, 11] fuse multiple low-rank weights to resolve multi-concept customization. In order to tackle continual text-to-image synthesis tasks, Sun et al. [47] devise a lifelong diffusion model to accumulate concept information. Unfortunately, it cannot control the contexts of synthesized images and suffers from concept neglect in multi-concept composition [62, 58] . To address the issue of missing concepts, [66, 30] utilize spatial conditions (e.g., sketch and pose) for composition. However, these custom diffusion models [59, 47, 51, 58] cannot consecutively learn a sequence of new concepts to tackle the CIFC problem, as they face challenges related to catastrophic forgetting and concept neglect. 
preliminary and problem definition: 
 Preliminary: Latent diffusion models (LDMs) [41, 10] rely on some conditional inputs (e.g., text prompt [33, 37] or image [17, 32] ) to control the contexts of synthesized images. They use an encoder E(•) and a decoder D(•) to perform image synthesis in the latent space [33] . Custom diffusion models (CDMs) [12, 9, 28] utilize low-rank adaptation (LoRA) [14, 55] to learn new personalized concepts by finetuning the pretrained LDMs [1, 38] . Given a pair of personalized image x and its text prompt p, the encoder E(•) maps x to a latent feature z, and z t denotes the noisy latent feature at the t-th(t = 1, • • • , T ) timestep.After the text encoder Γ(•) (e.g., pretrained CLIP [35] ) maps p to the textual embedding c = Γ(p), the objective to learn personalized concept {x, p} at the t-th timestep is:EQUATIONwhere ϵ θ ′ (•) denotes the denoising UNet proposed in [38, 33] , and it can gradually denoise z t by predicting the noisy estimation ϵ θ ′ (z t |c, t) as Gaussian noise ϵ ∼ N (0, I). In the paper, θ ′ = θ 0 +△θ consists of the pretrained parameter θ 0 = {W l 0 } L l=1 in LDMs [39, 33] and low-rank parameter △θ = {△W l } L l=1 updated by LoRA [22, 11] . W l 0 , △W l ∈ R a×b denote the pretrained weight and low-rank weight in the l-th (l = 1, • • • , L) transfromer layer of θ ′ , respectively. a and b are the row and column of matrices. As introduced in [40, 68] , △W l = A l B l can be decomposed as two low-rank factors A l ∈ R a×r and B l ∈ R r×b , where r ≪ min(a, b) denotes the rank. However, most CDMs [11, 12, 9, 28] assume that the number of users' personalized concepts remains constant over time. This assumption is unrealistic in real-world applications, where users wish to consecutively generate a series of new personalized concepts based on their preferences. More importantly, they significantly suffer from catastrophic forgetting [36] of old personalized concepts and concept neglect when performing versatile customization in a concept-incremental manner.Problem Definition: To address the above challenges, we propose a new practical problem named Concept-Incremental Flexible Customization (CIFC). In the CIFC, there is a series of consecutive textguided concept customization tasks T = {T g } G g=1 , where G denotes the task quantity. According to users' personal preferences, the g-th taskT g = {x k g , p k g , y k g } ng k=1, which includes n g triplets of image x k g , text prompt p k g , and its concept tokens y k g ∈ Y g , belongs to one of the versatile customization categories: multi-concept generation [69] , style transfer [67] and image editing [3] . Here, p k g indicates the textual description of x k g (e.g., photo of a [V * ] [V dog ]), whereas y k g denotes the concept tokens (e.g., [V * ] [V dog ]) in p k g . Y g is concept space of the g-th task, and it comprises C g new personalized concepts y g = ∪ ng k=1 y k g in the g-th task. Particularly, the concept spaces between any two tasks have no overlap:Y g ∩ (∪ g-1 i=1 Y i ) = ∅.It implies that C g new concepts in the g-th task are different from g-1 i=1 C i old personalized concepts from g-1 old tasks under the CIFC setting. Considering the practicality of the CIFC setting, we don't allocate any memory storage to store or replay the training data of all tasks {T g } G g=1 , ensuring that all concept customization tasks are learned in a concept-incremental manner. The CIFC setting can continually learn new personalized concepts in a concept-incremental manner for versatile customization while tackling the forgetting on old concepts.  
concept consolidation loss: 
 In order to learn the g-th text-guided concept customization task T g , we use the LoRA [14, 38] to finetune the pretrained denoising UNet ϵ θ0 (•) on personalized data {x k g , p k g , y k g } ng k=1 by optimizing Eq. (1), and then obtain an updated model ϵ θ ′ g (•), where θ ′ g = θ 0 + △θ g , △θ g = {△W l g } L l=1 , and △W l g = A l g B l g ∈ R a×b is the updated low-rank weight in the l-th layer. A l g ∈ R a×r and B l g ∈ R r×b are low-rank factors. As introduced in [14, 60] , △θ g can encode most of C g personalized concept identity within the g-th task. To address the CIFC problem, a trivial solution for learning the g-th task is to store the updated low-rank weights of all tasks {T i } g i=1 learned so far, and then linearly merge them by evaluating their contributions [59, 65] during training. However, it may result in substantial loss of individual characteristics within some personalized concepts, when learning new concepts continually in the CIFC setting. This phenomenon is referred to as catastrophic forgetting on old personalized concepts. To mitigate catastrophic forgetting during training, as show in Fig. 1(a) , we develop a concept consolidation loss L CCL to explore task-specific and task-shared knowledge.Task-Specific Knowledge indicates distinctive characteristics of personalized concepts within each concept customization task. To explore this knowledge, we introduce learnable layer-wise concept tokens to better preserve unique attributes of personalized concepts in the synthesized images. It is significantly different from existing textual inversion methods [9, 67, 53] that inject a unified text prompt into all transformer layers of ϵ θ ′ g (•). For a triplet {x k g , p k g , y k g } in the g-th task, we define L layer-wise text prompts as {p k,l g } L l=1 , where p k,l g has its own learnable layer-wise concept tokens y k,l g in the l-th transformer layer. For example, given a textual descriptionp k g (photo of a [V * ] [V dog ]) of image x k g , its text prompt of the l-th layer is defined as p k,l g (photo of a [V l * ] [V l dog ]), and [V l * ] [V l dog ] indicates learnable concept tokens y k,lg in the l-th layer. After using p k g to initialize {p k,l g } L l=1 , we inject the textual embedding c k,l g = Γ(p k,l g ) encoded via the text encoder Γ(•) into the l-th layer of ϵ θ ′ g (•). When we train ϵ θ ′ g (•) via Eq. ( 1), the learnable layer-wise concept tokens can capture unique characteristics of old personalized concepts from different layers to surmount catastrophic forgetting.However, the discriminative ability of task-specific knowledge to distinguish different personalized concepts can significantly deteriorate as the number of concept customization tasks gradually increases under the CIFC settings. To tackle this issue, we devise an orthogonal subspace regularizer to constrain the low-rank weights of different customization tasks. It can enhance the discriminative ability of task-specific knowledge by ensuring the orthogonality of concept subspaces across different tasks. Given the low-rank weight △θ g = {△W l g } L l=1 in the g-th task, △W l g = A l g B l g can be regarded as consisting of the low-rank concept subspace A l g = [a l,1 g , • • • , a l,r g ] ∈ R a×r and its linear weighting matrixB l g = [b l,1 g , • • • , b l,r g ] ⊤ ∈ R r×b , where b l,i g ∈ R b denotes the linear weighting coefficients of a l,i g ∈ R a (i = 1, • • • , r).In the g-th task, we perform the orthogonal subspace regularizer on the low-rank concept subspaces of different tasks:g-1 i=1 L l=1 A l i (A l g ) ⊤ = 0.Since the orthogonal constraint is not differentiable, we propose an alternative optimization strategy that minimizes the absolute value of the inner product between different subspaces:R 1 = g-1 i=1 L l=1 A l i (A l g ) ⊤ .Task-Shared Knowledge represents the shared semantic information across different tasks with semantically similar concepts, which is beneficial to address catastrophic forgetting on old personalized concepts. To capture task-shared knowledge, we propose to learn a layer-wise common subspace W l * ∈ R a×b shared across different tasks in the l-th layer. Given the low-rank weights {△θ i } g i=1 learned so far, the learnable projection matrixH l i ∈ R a×a can encode common semantic information of {△θ i } g i=1 into W l * via R 2 = g i L l ∥△W l i -H l i W l * ∥ 2 F .Therefore, in the g-th task, the concept consolidation loss L CCL to learn both task-specific and task-shared knowledge is defined as:EQUATIONwhere c k g = {c k,l g } L l=1 indicates L layer-wise textual embeddings, γ 1 and γ 2 are balance parameters. Two-Step Optimization: To train the proposed CIDM via Eq. ( 2) in the g-th task, we devise a two-step optimization strategy in each training batch. Firstly, to capture task-specific information, we utilize Eq. ( 2) to update the learnable layer-wise concept tokens and low-rank weight △θ g = {△W l g } L l=1 , when fixing H l i and W l * . Secondly, to explore task-shared knowledge, we fix △θ g and the learnable layer-wise concept tokens, and then only use R 2 in Eq. ( 2) to update H l i and W l * respectively. Notably, the detailed optimization procedure is shown in the appendix section (Sec. A.1). After utilizing the two-step optimization strategy to learn the g-th concept customization task, we can obtain the task learner Z g = {△W l g , y l g } L l=1 , where y l g = { y l,i g } Cg i=1 , and y l,i g is the i-th (i = 1, • • • , C g ) concept token learned at the l-th transformer layer through Eq. (2).To tackle catastrophic forgetting on old personalized concepts during inference, as shown in Fig. 1(b) , we store g task learners {Z i } g i=1 learned so far (g ≥ 2), and develop an elastic weight aggregation (EWA) module to adaptively merge them for versatile concept customization. Note that the memory storage of storing Z i (i = 1, • • • , g) only accounts for 0.25% of the pretrained model θ 0 , making it negligible in practical applications. Specifically, given a text prompt p for inference, we use it to initialize layer-wise text prompts { p l } L l=1 and extract their textual embeddingsc = { c l ∈ R ne×d } L l=1 via text encoder Γ(•),where n e and d denote the token number and feature dimension, respectively. Given the stored task learners {Z i } g i=1 , we can collect all concept tokens {Φ l } L l=1 learned so far, whereΦ l = ∪ g i=1 y l i ∈ R nc consists of n c = g i=1 C i concept tokens learned in the l-th layer. After Γ(•) encodes {Φ l } L l=1 to latent embeddings {e l ∈ R nc×d } L l=1, we average those latent embeddings belonging to the same task to obtain new embeddings { e l ∈ R g×d } L l=1 (g is number of tasks learned so far). Subsequently, we compute semantic relations M ∈ R g between c l and e l , and use M to adaptively merge low-rank weights {△W l i } g i=1 of all learned tasks in the l-th layer. Therefore, the merged low-rank weight △ W l in the l-th transformer layer can be formulated as follows:EQUATIONwhere max(•) denotes the maximization function along the row axis.ψ(M) = M 2 /∥M 2 ∥ F ∈ R gis used to normalize the semantic relations M, and ψ(M) i is the i-th element of ψ(M).Inference: After employing Eq. ( 3) to merge all low-rank weights learned so far, we obtain a new denoising UNet ϵ θ ′ * (•) for inference, where θ ′ * = θ 0 + △θ * , and△θ * = {△ W l } L l=1 . Notably, θ ′ *has encoded substantial distinctive attributes of all personalized concepts learned so far, which can effectively mitigate the catastrophic forgetting of old concepts during inference. When we directly use ϵ θ ′ * (•) obtained via Sec. 4.2 to perform multi-concept customization under the CIFC setting, it cannot generate high-fidelity images according to users-provided conditions (e.g., scribble or bounding box [24] ), and heavily suffers from the challenge of concept neglect [1] (i.e., some concepts are missing in the synthesized images). Thus, as shown in Fig. 1(c ), we devise a context-controllable synthesis strategy to address the conditional generation and concept neglect.Conditional Generation: Besides the initial text prompt p defined in Sec. 4.2, users can provide U pairs of region conditions { p u , s u } U u=1 , where s u is the bounding box to synthesize concepts related to the u-th region text prompt p u . Then we use Γ(•) to extract layer-wise textual embeddingsc u = { c l u ∈ R ne×d } L l=1 for p u .Given the initial text prompt p, we can use the new denoising UNet ϵ θ ′ * (•) in Sec. 4.2 to obtain its feature map f l ∈ R h l ×w l ×d in the l-th transformer layer encoded by textual embedding c l , where h l and w l are height and width of f l . Different from [22] , we perform layer-wise regional cross-attention between textual embedding c l u and f l to obtain the u-th region feature f l u ∈ R h l u ×w l u ×d in the l-th layer, where h l u and w l u are height and width of bounding box s u . Specifically,f l u = σ(Q l (K l u ) ⊤ / √ d) • V l u , where σ(•) is sigmoid function, Q l = Ω(f l w q ⊙ m l u ) ∈ R h l u ×w l u ×d , K l u = c l u w k ∈ R ne×d and V l u = c l u w v ∈ R ne×d . m l u ∈ R h l ×w lis the binary region mask in the l-th layer, where the values inside the bounding box s u are set to 1. Ω(•) can retain only Figure 3 : Some qualitative comparisons of multi-concept customization generated by SDXL [33] , where ITP indicates the initial text prompt, and RTP denotes the region text prompt. the features inside s u , and ⊙ is the Hardmard product. w q , w k , w v ∈ R d×d are mapping matrices in the new denoising UNet ϵ θ ′ * (•). Then, the values of f l inside the bounding boxes { s u } U u=1 are respectively replaced with the corresponding region features {f l u } U u=1 to obtain a new feature map f l . We apply the layer-wise regional cross-attention to all layers of ϵ θ ′ * (•) for conditional generation. Multi-Concept Composition: After integrating the above conditional generation into the new denoising UNet ϵ θ ′ * (•), inspired by [13] , we can obtain the noise estimationE t = ϵ θ ′ * (z t |t) + s • (ϵ θ ′ * (z t | c, t) -ϵ θ ′ * (z t |t)) ∈ R h L ×w L ×d L for the initial text prompt p, where E t is the output of the L-th transformer layer in ϵ θ ′ * (•) at the t-th timestep. h L , w L , d L are the height, width and channel of E t , respectively. ϵ θ ′ * (z t |t) is the unconditional noise estimation, ϵ θ ′ * (z t | c, t) is the conditional noise estimation based on the textual embeddings c, and s = 7.5 denotes the scale. Therefore, the noise estimation E t u ∈ R h L ×w L ×d L for the u-th region condition { p u , s u } can be formulated as follows:EQUATIONwhereϵ θ ′ * (z t |[ c u , s u ], t) is the conditional noise estimation based on [ c u , s u ].For multi-concept customization in the CIFC, we aggregate U region noise estimations to address concept neglect:EQUATIONwhere m L u ∈ R h L ×w L is the binary region mask of the bounding box s u in the L-th layer. α is the balance weight. Following [38, 33] , we forward E t * to the denoising process for image synthesis. Benchmark Dataset: Motivated by [47, 11, 45] , in this paper, we construct a new challenging concept-incremental learning (CIL) dataset including ten continuous textguided concept customization tasks to illustrate the effectiveness of our model under the CIFC setting. In the CIL dataset, seven customization tasks have different object concepts (i.e., V1 dog, V2 duck toy, V3 cat, V4 backpack, V5 teddy bear, V7 dog and V9 cat) from [40, 22] , and the remaining three tasks have different style concepts (i.e., V6, V8 and V10 styles) collected from website. Considering the practicality of the CIFC setting, we set about 3 ∼ 5 text-image pairs for each task. Particularly, we introduce some semantically similar concepts (e.g., V1 and V7 dogs, V3 and V9 cats), making the CIL dataset more challenging under the CIFC setting. Implementation Details: We utilize two popular diffusion models: Stable Diffusion (SD-1.5) [38] and SDXL [33] as the pretrained models to conduct comparison experiments. For fair comparisons, we train all SOTA comparison methods and our model using the same backbone and Adam optimizer, where the initial learning rate is 1.0 × 10 -3 to update textual embeddings, and 1.0 × 10 -4 to optimize the denoising UNet. For the low-rank matrices, we follow [11] to set r = 4. We empirically set γ 1 = 0.1, γ 2 = 1.0 in Eq. ( 2), α = 0.1 in Eq. ( 5), and the training steps are 800.Evaluation Metrics: After learning the final concept customization task under the CIFC setting, we conduct both the qualitative and quantitative evaluations on versatile generation tasks: single/multi-concept customization, custom image editing, and custom style transfer. For the quantitative evaluation, we follow [22] to use textalignment (TA) and image-alignment (IA) as metrics. Specifically, for image-alignment (IA), we use the image encoder of CLIP [34] to evaluate the feature similarity between the synthesized image and original sample. For text-alignment (TA), we utilize the text encoder of CLIP [34] to compute the text-image similarity between the synthesized image and its corresponding prompt. 
qualitative comparisons: 
 To verify the superiority of our model under the CIFC setting, we introduce extensive qualitative comparisons, including single/multi-concept customization (see Figs. 2 3 ), custom image editing (see Fig. 4 ), and custom style transfer (see Fig. 5 ). 1) As presented in Fig. 2 , the proposed model achieves the best performance for single-concept customization by addressing catastrophic forgetting of old personalized concepts and preserving superior attributes of each learned concept. 2) For multi-concept customization, we incorporate the regionally controllable sampling module proposed in [11] into existing comparison methods to fairly compare them with our model. As shown in Fig. 3 , all comparison methods significantly suffer from the challenge of concept neglect, and they are difficult to generate multiple object concepts. In contrast, our proposed model can effectively tackle the challenge of concept neglect via the proposed context-controllable synthesis strategy. 3) To perform custom image editing, we introduce Anydoor [3] as a plug-in for all comparison methods in Fig. 4 . The qualitative comparisons in Fig. 4 reflect the effectiveness of our model in custom image editing. Such substantial improvement benefits from the superior performance of our concept consolidation loss to preserve unique identity of each learned concept under the CIFC setting. 4) Fig. 5 shows qualitative comparisons of custom style transfer. Our proposed model presents the best performance for image-to-image style transfer, since the elastic weight aggregation can preserve style integrity in the CIFC setting. The above qualitative comparisons in Figs. 2 3 4 5 verify that the proposed model can significantly surpass existing latent diffusion models to tackle the CIFC problem. This subsection analyzes the effectiveness of each module in our model: elastic weight aggregation (EWA), contextcontrollable synthesis (CCS), task-specific knowledge (TSP) and task-shared knowledge (TSH) in the concept consolidation loss (CCL). Tab. 3 presents the ablation studies of single-concept customization in terms of IA. When compared with Baseline, the performance of our model improves by 0.2% ∼ 3.3% in terms of IA, after we add the proposed TSP, TSH and EWA modules. It demonstrates the effectiveness of our model in resolving the CIFC problem by addressing the catastrophic forgetting and concept neglect. As shown in Fig. 6 , we analyze the effectiveness of CCS in multi-concept customization, where CCS includes the layer-wise regional cross-attention (RCA) and the region noise estimation (RNE) modules. In Fig. 6 , the performance of our model decreases substantially when removing the RCA and RNE modules. It verifies the effectiveness of our CCS startegy in addressing conditional generation and concept neglect. Moreover, Fig. 7 shows ablation analysis about TSP and TSH. It illustrates that our model can capture task-specific information within each customization task and explore task-shared knowledge across different tasks to tackle the CIFC problem via optimizing Eq. ( 2). 
conclusion: 
 In this paper, we propose a novel Concept-Incremental text-to-image Diffusion Model (CIDM) to address a practical Concept-Incremental Flexible Customization (CIFC) problem, where the CIFC problem has two major challenges: catastrophic forgetting and concept neglect. Specifically, we devise a concept consolidation loss and an elastic weight aggregation module to respectively resolve catastrophic forgetting during training and inference. They can capture task-specific/task-shared knowledge and aggregate all low-rank weights of old concepts according to their contributions in the CIFC. To address concept neglect, we propose a new context-controllable synthesis strategy, which can control the contexts of synthesized images according to users-provided conditions. Extensive experiments on versatile customization tasks (single/multi-concept customization, custom image editing and style transfer) show the superior performance of our CIDM in tackling the CIFC problem compared to SOTA methods. In the future, we will leverage multi-modal large language models to address the CIFC problem and apply the proposed model to personalized video generation. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: Please check the source codes at https://github.com/JiahuaDong/CIFC for the descriptions of safeguards.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Please refer to Sec. 5 for the licenses of existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: As shown in Algorithm 1, we introduce the detailed algorithm pipeline of our proposed CIDM to tackle the CIFC problem. For the first text-guided concept customization task (g = 1) in the training process, we only utilize Eq. ( 1) to optimize the textual embeddings of layer-wise concept tokens and low-rank weight △θ g = {△W l g } L l=1 . When g ≥ 2, we devise a two-step optimization strategy in each training batch to efficiently train our CIDM via Eq. ( 2). It can continually learn new personalized concepts under the CIFC setting, while overcoming catastrophic forgetting of old personalized concepts by exploring both task-specific and task-shared knowledge during training.1) In the first step, we fix H l i and W l * to update learnable layer-wise concept tokens and low-rank weight △θ g = {△W l g } L l=1 via minimizing the loss L CCL in Eq. ( 2). Particularly, the loss L CCL can capture task-specific knowledge (i.e. unique concept attributes within each customization task).2) In the second step, in order to explore task-shared knowledge across different tasks, we fix the learnable layer-wise concept tokens and low-rank weight △θ g to update H l i and W l * via minimizing the regulaizerR 2 = g i=1 L l=1 ∥△W l i -H l i W l * ∥ 2 F .As a result, the gradients of R 2 with respect to H l i and W l * are formulated as follows:EQUATIONGiven the gradients ∂R 2 /∂H l i and ∂R 2 /∂W l * in Eq. ( 6), we can iteratively update H l i and W l * via the following objective:EQUATIONwhere η denotes the learning rate to update H l i and W l * . In this paper, we set the value of η to be the same as the learning rate of the Adam optimizer.This subsection introduces more details about dataset, comparison methods and evaluation metrics.Dataset: Inspired by [47, 11, 45] , we construct a new challenging concept-incremental learning (CIL) dataset including ten continuous text-guided concept customization tasks to verify the effectiveness of our model under the CIFC setting. In the CIL dataset, as shown in Fig. 8 , seven tasks have different object concepts (i.e., V1 dog, V2 duck toy, V3 cat, V4 backpack, V5 teddy bear, V7 dog and V9 cat) from [40, 22] , and the remaining three tasks have different style concepts (i.e., V6, V8 and V10 styles) collected from website. Considering the practicality of the CIFC setting, we set about 3 ∼ 5 text-image pairs for each task, where the text prompts are extracted by BLIP [23] . Particularly, different from L2DM [47] and Mix-of-Show [11] , we introduce some semantically similar concepts (e.g., V1 and V7 dogs, V3 and V9 cats), making the CIL dataset more challenging in the CIFC setting. tackle the CIFC setting. CLoRA [45] proposes a self-regularized low-rank adaption to continually learn new personalized concepts. L2DM [47] builds a long-term memory bank to reconstruct old personalized concepts, and performs knowledge distillation to mitigate catastrophic forgetting. LoRA-M [69] equally amalgamates all LoRA layers to retrain the latent diffusion model. LoRA-C [69] focuses on exploring contributions of different LoRA layers for multi-concept composition.Evaluation Metrics: After learning the final concept customization task under the CIFC setting, we conduct both the qualitative and quantitative evaluations on versatile generation tasks: single/multiconcept customization, custom image editing, and custom style transfer. For the quantitative evalua- tion, we follow [22] to use text-alignment (TA) and image-alignment (IA) as metrics. Specifically, for image-alignment (IA), we use the image encoder of CLIP [34] to evaluate the feature similarity between the synthesized image and original sample. For text-alignment (TA), we utilize the text encoder of CLIP [34] to compute the text-image similarity between the synthesized image and its corresponding prompt. As shown in Fig. 9 , we roughly categorize prompts into three categories according to the learned concepts. To analyze quantitative comparisons between our model and SOTA methods, inspired by [11, 22, 45, 47] , we input an evaluation prompt and a unify negative prompt to sample 50 images. Therefore, given 20 evaluation prompts in this paper, we can generate 1,000 images for each concept to evaluate performance in terms of TA and IA.In this paper, we use two popular diffusion models: Stable Diffusion (SD-1.5) [38] and SDXL [33] as the pretrained models to conduct comparison experiments. For fair comparisons, we train all SOTA comparison methods and our model using the same diffusion model and Adam optimizer, where the initial learning rate is 1.0 × 10 -3 to update textual embeddings, and 1.0 × 10 -4 to optimize the denoising UNet. For the low-rank matrices, we follow [11] to set r = 4. Moreover, we empirically set γ 1 = 0.1, γ 2 = 1.0 in Eq. ( 2), α = 0.1 in Eq. ( 5), and the training steps are 800. In this paper, we train our model on two NVIDIA RTX 4090 GPUs. In each text-guided concept customization task, we finetune both the textual embeddings of layer-wise concept tokens and low-rank weights via the proposed two-step optimization strategy in Sec. A.1. For example, given a textual description p k g (photo of a [V * ] [V dog ]) of image x k g in the g-th task, its text prompt of the l-th layer is defined asp k,l g (photo of a [V l * ] [V l dog ]). After using [V * ] [V dog ] to initialize [V l * ] [V l dog ]in the l-th transformer layer, we only update the layer-wise concept tokens (e.g., [V l * ] [V l dog ]). For fair comparisons in the CIFC setting, all comparison methods and our model use the same prompt augmentation and image augmentation during the training phase. Additionally, we set the random seed as 0 for all comparison experiments. More importantly, we have no prior knowledge about task order, task quantity and data distributions in the CIFC problem. Single-Concept Customization: As presented in Figs. 10 11 , we conduct extensive qualitative comparisons of single-concept customization, when using Stable-Diffusion (SD-1.5) [38] and SDXL [33] as the pretrained denoising UNet. In Fig. 11 , given a text prompt (e.g., "V1 dog riding a bike, wearing a wizard hat") for inference, we observe that our model can effectively follow the prompt instruction to synthesize images that preserve the superior identity of the personalized concept "V1 dog". In contrast, existing comparison methods suffer from significant catastrophic forgetting, generating unpleasant images with distorted objects and unmatched identity.Multi-Concept Customization: Fig. 12 presents comprehensive comparison experiments of multiconcept customization. For fair comparisons with existing methods, we apply the region-aware cross-attention module proposed in Mix-of-Show [11] into all comparison methods. The quanlitative comparisons in Fig. 12 demonstrate the superior performance of our model in multi-concept customization, when continually learning new personalized concepts under the CIFC setting. Particularly, given an initial text prompt (ITP) and some region text prompts (RTP) with users-provided bounding boxes, all comparison methods exhibit significant concept neglect and they are difficult to perform multi-concept customization. On the one hand, it is evident that all comparison methods fail to preserve the identity of learned concepts due to catastrophic forgetting. On the other hand, these methods neglect some important object concepts during the synthesis process. For example, users may want to generate an image of a V1 dog and a V7 dog drinking a cup of beer, but none of the comparison methods can generate the V1 and V7 dogs, as shown in Fig. 12 .Custom Style Transfer: As shown in Fig. 13 , we introduce some comparisons of custom style transfer to evaluate the effectiveness of our proposed CIDM in addressing the CIFC problem. From Fig. 13 , we conclude that our model achieves the best generation performance according to userprovided style concepts, compared to other existing methods. It verifies that the designed elastic weight aggregation module plays a crucial role in preserving distinctive style attributes in the CIFC setting. In contrast, existing comparison methods struggle to explore the identity of different style concepts, due to the catastrophic forgetting and concept neglect in the CIFC setting. Societal Impact: To tackle the practical concept-incremental flexible customization (CIFC) problem, our proposed concept-incremental text-to-image diffusion model (CIDM) aims to continually learn new personalized concepts in a concept-incremental manner for versatile customization (e.g., single/multi-concept customization, custom image editing and custom style transfer), while tackling catastrophic forgetting and concept neglect on old concepts. In particular, it allows users to continually generate a sequence of images using their new personalized concepts. Additionally, users can control the contexts of synthesized images according to their own preferences.Generally, the proposed CIFC setting can enable the creation of highly personalized content for various applications such as marketing, entertainment, and education. This can lead to more engaging and relevant experiences for users. More importantly, artists, designers, and content creators can benefit from tools that adapt to their unique styles and preferences over time. This can foster innovation and creativity by providing customized suggestions and automating repetitive tasks. Additionally, it can be tailored to meet the needs of different user groups, including those with disabilities. For example, generating personalized educational materials can cater to diverse learning styles and needs. Consequently, the CIFC problem proposed In this paper is worth being studied. More importantly, our proposed CIDM can achieve state-of-the-art performance in addressing the CIFC problem, highlighting the importance of this work in promoting the development of text-toimage diffusion models. In the CIFC, the use of personalized data to train our model may raise privacy issues. However, this is a common concern for all latent diffusion models (LDMs). Ensuring that user data is handled securely and ethically is paramount to prevent misuse or unauthorized access, safeguarding both privacy and trust.Limitations: Although we construct a concept-incremental dataset containing four semantically similar concepts to verify the effectiveness of our model, it will still suffer from catastrophic forgetting and concept neglect as the number of semantically similar concepts increases. The main limitation of this work is that our model struggle to continually learn large-scale semantically similar concepts provided by users. Thus, we will explore how to increase the scalability of our model in the future.Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper makes a significant contribution to the field of continual learning in generative models, with a particular focus on personalized text-to-image generation. The proposed CIDM method introduces innovative solutions to the challenges of catastrophic forgetting and concept neglect, addressing a problem of high relevance and complexity. The paper demonstrates a solid level of novelty and technical rigor, supported by a well-executed experimental evaluation. Based on the strength of the contributions, the clarity of presentation, and the comprehensive coverage of experiments, the review committee has reached a consensus to recommend this paper for acceptance.

Reviews: review: paper_summary: paper_summary: This paper tackles the problem of continually adapting text-to-image diffusion customization models.  The proposed method employs a novel concept consolidation loss, elastic weight aggregation module, and context-controllable synthesis strategy. Extensive experiments demonstrate that the proposed method performs strongly compared to SOTA baseline methods.
strength_weakness: strength_weakness: 1. Please include a table that compares the training, parameter, and memory costs of your method against other methods. This comparison is crucial for a comprehensive evaluation of your method.

2. Although the results are clear and detailed, they lack confidence intervals or similar statistical measures in the quantitative metrics. Running the experiments with multiple random seeds would ensure robustness and provide a more reliable assessment.

3. The dataset used is quite limited. To fully justify your setting, you should explore more diverse datasets and longer task sequences. For instance, with only 10 simple concepts, it would be feasible to store a LoRA version of Custom Diffusion or even the Custom Diffusion deltas in memory and merge them on the fly for the target task, achieving zero forgetting in a practical manner. To truly demonstrate the impact of your method and setting, beyond what can be trivially solved by adapters or state-of-the-art LoRA-based merging methods, you should consider long task sequences or challenging benchmarks such as celebrity faces.
questions: questions: Please see my weaknesses section. I think 1 and 2 are crucial for acceptance (would raise my score to WA if done thoroughly). Part 3 is important, too, but more difficult to address in a short period of time.
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces the Concept-Incremental text-to-image Diffusion Model (CIDM), which addresses the Concept-Incremental Flexible Customization (CIFC) problem. This approach represents one of the first explorations into learning new customization tasks incrementally, effectively navigating the dual challenges of catastrophic forgetting and concept neglect. To combat catastrophic forgetting, the paper proposes a new concept consolidation loss coupled with an elastic weight aggregation module, which together helps preserve knowledge of previously learned personalized concepts. Additionally, the development of a novel context-controllable synthesis strategy specifically addresses the challenge of concept neglect, ensuring that new concepts are integrated without overshadowing existing knowledge. The effectiveness of the CIDM is validated through comprehensive experiments across a range of text-to-image generation tasks, demonstrating its capability to learn new customization tasks consecutively with notable efficacy.
strength_weakness: strength_weakness: 1. A critical concern arises regarding the scalability of the model, specifically the accumulated memory load after learning each task. This is particularly pertinent in continual learning scenarios where efficiency is crucial. 

2. The paper lacks detailed implementation information on critical tasks such as custom image editing and style transfer. Providing these details would significantly enhance the clarity and replicability of the research.

3. The concept of balancing task-specific and task-shared knowledge is intriguing and seems to be central to effectively addressing the CIFC problem. To substantiate the claims of its effectiveness, the authors should include ablation studies that isolate and quantify the impact of these components. Additionally, visualizations comparing the results with and without these strategies would provide a clearer, more direct illustration of their value.
questions: questions: 1. I recommend the authors include details on memory consumption or the number of stored parameters after each task during the rebuttal process.

2. The authors should incorporate a comprehensive description of custom image editing and style transfer processes, including specific parameters and settings used, to allow for a better understanding and evaluation of the proposed methods' effectiveness in practical applications.
limitations: limitations: There are no potential negative societal impacts.
paper_summary: paper_summary: This paper explores a novel and practically significant problem, namely how custom diffusion models can continuously learn new personalized concepts while avoiding catastrophic forgetting and concept neglect. The authors developed a  concept consolidation loss and an elastic weight aggregation module to explore task-specific knowledge and task-shared knowledge, and aggregated all low-rank weights of old concepts based on the contributions of old concepts. To address the problem of concept neglect, the authors propose a context-controllable synthesis strategy that can utilize expressive regional features and noise estimation to control the generated contexts. Experimental results confirm the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. It would be better to mention the comparison methods of experiments in the related work，such as LoRA-M, LoRA-C, CLoRA and L2DM, and emphasize the differences between them and the proposed method.
2. This paper attempts to address two challenges in the Concept-Incremental text-to-image Diffusion Model: catastrophic forgetting and concept neglect. However, from the text, I did not understand the relationship between continual learning and concept neglect. I would like to know if concept neglect is a challenge introduced by continual learning, and what impact continual learning has on original concept neglect?
3. The baseline choices for continual learning such as EWC and LWF are too outdated. Why not choose new and more effective continual learning algorithms? Intuitively, more advanced algorithms should be able to better mitigate catastrophic forgetting.
4. Due to the past focus of continual learning mainly on classification tasks, I hope to see more discussion on the background of continual generation models. For example, in this setting, how severe are catastrophic forgetting and concept neglect specifically, and whether some more objective evaluation criteria can be introduced.
5. How much do different weighting schemes between LoRAs affect the quality of generation, compared to some intuitive weighting baselines?
6. Does the value of the rank in LoRA and its placement in the model (e.g., which layer) have an impact on the results?
questions: questions: Please check the above concerns in the weakness points. And If my concerns are resolved, I will consider increasing my score.
limitations: limitations: See the weaknesses above.
paper_summary: paper_summary: This paper introduces a novel approach called CIDM to address the challenge of Concept-Incremental Flexible Customization in text-to-image generation. The authors identify two key issues in CIFC: catastrophic forgetting of previously learned concepts and concept neglect during multi-concept composition. The proposed CIDM tackles these problems through several innovative components, including a concept consolidation loss, elastic weight aggregation, and a context-controllable synthesis strategy. The paper provides experiments on various customization tasks to demonstrate the effectiveness of their approach compared to existing methods.
strength_weakness: strength_weakness: 1.  The symbol definitions are too complex which are hard to follow.
2. The paper introduces several new hyperparameters. How sensitive is the model to these?
3. How well does the model perform across different application domains and with varied types of personalized concepts?
questions: questions: See weakness.
limitations: limitations: Yes

Comments: title: Thanks for your constructive comments.
Thank you for your valuable comments!
Thank you for your kind response and support.
Thank you for your generous support!
Thank you for your follow-up comments!
comment: Dear Reviewer Th4Q:

Thank you for supporting our work. We sincerely appreciate your insightful comments that help improve our paper. We will take them into account when making the final revisions.

Best regards,

Authors of Paper #1919
I appreciate the authors' hard work and have raised my score. I hope this paper will be accepted.
Dear Reviewer SpKd,

We are pleased that we have addressed some of your concerns. We are very grateful to you for dedicating your time and effort to evaluating our paper. 

Best regards,

Authors of Paper #1919
Thanks for the authors' responses. Some of my concerns have been solved.
Dear Reviewer 5EBF,

We are pleased that we have addressed most of your concerns. Thanks for your insightful comments and approval of our work.

Best regards,

Authors of Paper #1919
Thank you for your responses! My concerns are addressed. Hence, I increase my score.
Dear Reviewer YpEo,

Glad to hear that your concerns have been addressed well. Thank you for your great efforts in reviewing and for the good questions. 

Best regards,

Authors of Paper #1919
Thank you for your clear response. I have no further questions.
Thank you for your follow-up comments!

**1)** As for the application of continual learning in the generation community, our proposed model can help users continually synthesize a series of new personalized concepts for versatile customization based on their provided conditions, while addressing catastrophic forgetting of old personalized concepts. In contrast, existing state-of-the-art (SOTA) diffusion models in the generation community may experience significant loss of individual attributes in old personalized concepts (i.e., catastrophic forgetting) during versatile customization when continually learning new personalized concepts. The experiment results shown in Tabs. 1-2 and Figs. 2-5 of the submitted manuscript also illustrate the effectiveness of our model in addressing catastrophic forgetting and continually learning new concepts, compared to other state-of-the-art (SOTA) methods.

**2) Advantages of Our model Over Multi-LoRA Composition:**

**a)** Compared with SOTA multi-LoRA composition methods such as LoRA-M [64] and LoRA-C [64], our proposed elastic weight aggregation module utilizes learnable layer-wise concept tokens to merge all low-rank weights of old personalized concepts, based on their contributions to versatile concept customization. This module ensures that our model is more effective to resolve longer task sequences for versatile concept customization than other SOTA multi-LoRA composition methods such as LoRA-M [64] and LoRA-C [64]. As shown in the following results,  we use a challenging benchmark dataset (i.e., celebrity faces) to construct thirty continuous text-guided concept customization tasks. Compared with our model, current multi-LoRA composition methods (LoRA-M [64] and LoRA-C [64]) suffer from significant performance degradation, due to the catastrophic forgetting caused by the composition of longer sequence tasks.

| Methods               | | Parameters (M) | | Memeory Costs (M) | |   IA (SD-1.5)       |  
|-                      |-|:-:             |-|:-:                |-|:-:             |            
| EWC [19]              | |    0.80        | |  6.87              | | 72.7$\pm$0.14  |               
| LoRA-M [64]           | |    0.40        | |  1.72             | | 72.0$\pm$0.17  |     
| LoRA-C [64]           | |    0.40        | |  1.72             | | 72.3$\pm$0.08  |          
| CLoRA [43]            | |    0.40        | |  1.72             | | 74.1$\pm$0.13  |         
| L2DM [46]             | |    0.80        | |  3.43             | |73.4$\pm$0.11  |        
| **CIDM (Ours)**       | |    0.41        | |  1.74             | | **75.7$\pm$0.09**  |         
---

**b)** Mix-of-Show [10] proposes a gradient fusion strategy to train a composed LoRA weight that mimics the predictions of individual LoRAs. However, it requires retraining the entire diffusion model and CLIP when learning new personalized concepts continually, which can substantially increase computational complexity and memory cost. As shown in the following results, compared with our model, Mix-of-Show [10] requires larger training parameters and memory costs to learn a new concept customization task, making it impractical to handle longer task sequences under the CIFC setting. Moreover, to ensure fair comparisons with existing methods, we apply the regional control techniques proposed in Mix-of-Show [10] to all comparison methods. The qualitative comparisons in Figs. 3 and 12 of the submitted manuscript demonstrate the superior performance of our model in multi-concept customization when continually learning new personalized concepts under the CIFC setting. 

| Methods               | | Parameters (M) | | Memeory Costs (M) |
|-                      |-|:-:             |-|:-:                |
| Mix-of-Show [10]      | |     983.65   | |     3930.46         | 
| **CIDM (Ours)**       | |         **0.41**   | |     **1.74**          |
---

Refs:

[10] Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models.

[64] Multi-LoRA Composition for Image Generation.
Thanks for the author's response and I have another question. In the open source community of diffusion models, the generation of multi-concept images can be achieved through multi-lora composition and some regional control techniques. How can we further understand the application of continual learning in the generation community and the advantages over multi-lora composition, for example, [1][2]?

[1] Multi-LoRA Composition for Image Generation

[2] Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models