Importance-aware Shared Parameter Subspace Learning for Domain Incremental Learning

Authors: shiye  wang
changsheng  li
jialin  tang
ye  yuan
guoren  wang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 04:34:38

Abstract: Parameter-Efficient-Tuning (PET) for pre-trained deep models (e.g., transformer) hold significant potential for domain increment learning (DIL). Recent prevailing approaches resort to  prompt learning, which typically involves learning a small number of prompts for each domain to avoid the issue of catastrophic forgetting. However, previous studies have pointed out prompt-based methods are often challenging to optimize, and their performance may vary non-monotonically with trainable parameters. In contrast to previous prompt-based DIL methods, we put forward an importance-aware shared parameter subspace learning for domain incremental learning, on the basis of low-rank adaption (LoRA). Specifically, we propose to incrementally learn a domain-specific and domain-shared low-rank parameter subspace for each domain, in order to effectively decouple the parameter space and capture  shared information across different domains. Meanwhile, we present a momentum update strategy for learning the domain-shared subspace, allowing for the smoothly accumulation of knowledge in the current domain while mitigating the risk of forgetting the knowledge acquired from previous domains. Moreover, given that domain-shared  information might hold varying degrees of importance across different domains, we design an importance-aware mechanism that adaptively assigns an importance weight to the domain-shared subspace for the corresponding domain. Finally, we devise a cross-domain contrastive constraint to encourage domain-specific subspaces to capture distinctive information within each domain effectively, and enforce orthogonality between domain-shared and domain-specific subspaces to minimize interference between them. Extensive experiments on image domain incremental datasets demonstrate the effectiveness of the proposed method  in comparison to the related state-of-the-art methods.

Main paper: 
 introduction: 
 Domain incremental learning (DIL) has attracted increasing attention in multimedia and machine learning communities [34, 47] . In DIL, it's assumed that domains are constantly evolving and often exhibit significant variation in sequence, with domain indexes being not provided for inference. Due to the dynamically changing characteristic of data distributions in DIL, traditional deep models face a significant challenge: simply fine-tuning them on new domains often results in a substantial performance drop on previous domains, a phenomenon known as "catastrophic forgetting" [29] . To this end, Parameter-Efficient-Tuning (PET) offers a promising approach for DIL, which adapts pre-trained deep models (e.g., transformer) to different domains using significantly fewer learnable parameters and resources.In recent years, many PET approaches have been proposed for DIL [42, 47, 50] , where one popular strategy is the adoption of prompt learning, an emerging paradigm derived from NLP. Prompt learning enables pre-trained language models to be repurposed for various tasks without the need for retraining. Among these prompt-based methods, they typically learn a small number of prompts for each domain to avoid the issue of catastrophic forgetting. For instance, L2P [50] is the pioneering work that establishes a prompt pool, dynamically guiding a pre-trained model to learn from domains sequentially. S-Prompts [47] attempts to independently learn prompts across domains with pre-trained transformers, where S-Prompts only necessitates a single cross-entropy loss during training and a straightforward K-NN operation for domain identification during inference. CODA-Prompt [42] replaces the prompt pool with a decomposed prompt, which consists of a weighted sum of learnable prompt components. This decomposition enhances prompting capacity by expanding into a new dimension. While these methods have achieved promising performance, previous studies have pointed out that prompt-based methods are often challenging to optimize, and their performance may vary non-monotonically with trainable parameters [13] .In contrast to prompt-based DIL methods, we put forward a new PET method for DIL in this paper, inspired by the low-rank adaption (LoRA) technique. LoRA was originally proposed for large language models, which freezes the pre-trained model weights and integrates two trainable low-rank matrices into each layer of the Transformer architecture, significantly diminishing the number of trainable parameters for downstream tasks. Because of its impressive performance, LoRA has been widely applied to various tasks, such as text understanding [54] , image compression [28] , multitask learning [45] , etc. However, as of now, there is few study on employing LoRA for domain-incremental learning. A straightforward approach for DIL is to apply LoRA to acquire two low-rank matrices for each newly coming domain. However, such a strategy overlooks the potential to capture relations among multiple domains and uncover shared information across different domains, leading to suboptimal outcomes.In light of these, we propose to learn an importance-aware shared parameter subspace learning for DIL. Specifically, we attempt to decouple each parameter matrix into three distinct components: a domain-specific matrix tailored to each domain, a domain-shared matrix holding relevance across all domains, and a low-rank matrix serving as the coefficient matrix. Through this approach, we can effectively capture shared information spanning diverse domains. To mitigate the issue of catastrophic forgetting, we present a momentum update strategy to learn the domain-shared subspace, facilitating the smooth accumulation of knowledge within the current domain. Moreover, considering that domain-shared information may hold varying degrees of importance to different domains (which has been verifies in Figure 2 ), we design a dynamic importance allocation mechanism that adaptively assigns an importance weight to the domain-shared subspace specific to each domain. In addition, we devise a cross-domain contrastive constraint to effectively encourage domain-specific subspaces to capture distinctive information within each domain. Finally, we impose another constraint ensuring orthogonality between domain-shared and domain-specific subspaces, with the goal of minimizing interference between them.Our contributions are summarized as follows:• We propose an shared parameter subspace learning approach for domain incremental learning, building upon LoRA. We incrementally learn a domain-specific and domain-shared low-rank parameter subspace for each domain in a momentum update manner, which can effectively capture shared information across different domains. • We design an importance-aware mechanism to adaptively weight the domain-shared subspace for each domain, thereby further improving the performance. Moreover, we present two constraints on domain-specific and domain-shared subspaces to enhance their representation capabilities. • Extensive experiments on three image domain incremental datasets demonstrate the effectiveness of our proposed method in comparison with the state-of-the-art approaches. 
related work: 
 In this section, we review some related works, including continual learning and parameter-efficient tuning methods.Continual learning (CL) aims to enable the model to adapt to new class, new tasks or domains without suffering from catastrophic forgetting [38, 52, 55] . There are three common CL setups: taskincremental learning (TIL) [14, 32, 36] , class incremental learning (CIL) [1, 37, 50] , and domain incremental learning (DIL) [5, 40, 47] .In fact, knowing task identity during inference in TIL limits its practical utility. While in CIL, classes usually stem from the same domain, thus partially alleviating the challenge. This paper delves into DIL, where classes remain constant but domains vary significantly in sequence, without task indexes provided for inference.In recent years, many DIL methods have been proposed, such as replay-based methods [2, 40, 44] and rehearsal-free methods [42, 47, 50] . In this work, we focus on rehearsal-free methods due to their increased practical applicability. The representative rehearsalfree methods primarily rely on prompt learning including L2P, S-Prompts, and CODA-Prompt [42, 47, 50] . These prompt-based methods fall under the umbrella of Parameter-Efficient-Tuning (PET), commonly characterized by learning a limited number of prompts for each domain. These approaches circumvent the necessity of retraining pre-trained deep models and helps alleviate the issue of catastrophic forgetting. However, as aforementioned, optimizing prompt-based methods can often be challenging, and their performance may exhibit non-monotonic variations with trainable parameters. In contrast to these prompt-based DIL methods, we attempt to learn a shared parameter subspace across various domains to capture common information among them for improving the performance of DIL.In addition, there is a category of methods loosely related to DIL, called domain adaptation (DA) [4, 21, 25, 35] . The main difference between DA and DIL lies in their respective objectives. DA primarily aims to enhance the accuracy of target domains, whereas DIL focuses on minimizing the cumulative error across all domains, alongside implementing measures to mitigate forgetting on earlier domains. Notably, DA methods often require access to target domain data for distribution matching, making them not directly applicable to DIL.As deep learning models continuously grow in size, it becomes crucial to study on parameter efficient methods. In recent years, there has been a significant increase in research focused on Parameter-Efficiency-Tuning (PET) [13] . The representative methods include Adapter based methods [18, 33, 43 Orthogonality ⊥ ⊥ ⊥ ! $ " # Pre-trained ViT ⊥ $ % Domain-shared subspace $ & ' Domain-specific subspace L ! L " L ℓ Importance & ) & $ * Inter-domain Contrastive Constraint & " * & ) & ! * & )Figure 1 : Illustration of our domain incremental learning framework. To exploit the interdependencies among sequential domains, we attempt to incrementally learn a subspace 𝐴 𝑠 𝑖 specific to each newly coming domain 𝑖, and a subspace 𝐴 𝑐 shared by all domains. Meanwhile, we present a momentum update strategy on 𝐴 𝑐 to facilitate the smooth accumulation of knowledge in the current domain while mitigating the risk of forgetting knowledge acquired from previous domains. Moreover, given that 𝐴 𝑐 might hold varying degrees of importance to different domains, we design an importance-aware mechanism that adaptively assigns an importance weight to 𝐴 𝑐 for the corresponding domain. In addition, we impose an cross-domain contrastive constraint between the current domain-specific subspace 𝐴 𝑠 𝑖 and the historical domain-specific subspace 𝐴 𝑠 𝑗 , 𝑗 < 𝑖, as to amplify the distinctions across different domains. Finally, we enforce orthogonality between 𝐴 𝑐 and each 𝐴 𝑠 𝑖 to further enhance their representation capacities. 41, 48] , and LoRA-based methods [6, 17, 31] . Adapter [12] is initially introduced to incorporate efficient and learnable modules for adapting a fixed pre-trained transformer to various new textual tasks in the NLP field. Prompt tuning [19] and Prefix tuning [22] insert learnable tokens to the input or hidden tokens, enabling the adaptation of the transformer to new tasks. LoRA [13] injects trainable low-rank adaption modules to the frozen pre-trained transformer, greatly reducing the number of trainable parameters for downstream tasks. Due to the substantial advantages of LoRA, [46] extends LoRA for language model continual learning. These PET methods have been developed for various tasks, including visual transfer learning [3, 15] , video action recognition [51] , image generation [27] , and visual continual learning [9, 47] , etc. In this study, we attempt to explore the utilization of LoRA tailored for addressing the domain-incremental learning challenge. 
proposed method: 
 In this section, we will elaborate our proposed method. We will begin by presenting the problem formulation and then provide a detailed introduction to our method.We concentrate on continual learning with incremental domains, i.e., domain incremental learning (DIL). In DIL, a model sequentially absorbs 𝑀 incoming domains In this paper, we aim to explore LoRA-based Parameter-Efficient Tuning (PET) for DIL, where we exploit a small number of learnable parameters to effectively fine-tuning the pre-trained model. We let the feature extractor 𝑊 consist of the fixed pre-trained model 𝑊 0 and the learnable PET module Δ𝑊 . Inspired by LoRA [13] , we present a general objective function for DIL by minimizing the following cross-entropy loss as:D := {D 1 , D 2 , • • • , D 𝑀 },EQUATION)where 𝑦 is the ground truth of sample 𝑥 in the current domain D 𝑖 . Note that the domain identity of samples are not provided during inference. 𝜃 𝑖 can be optimized as most of DIL methods [42, 50] . Note that we utilize the Vision Transformer (ViT) [7] as our pre-trained model 𝑊 0 , and we keep 𝑊 0 to be frozen during learning all domains.Here we can omit it for convenience.In order to efficiently optimize Δ𝑊 𝑖 , a naive method is to directly apply LoRA to acquire two low-rank matrices for each newly coming domain. As aforementioned, this strategy fails to harness the potential of capturing cross-domain relations and discovering shared information across diverse domains, ultimately resulting in suboptimal outcomes. Therefore, our objective is to incrementally learn shared parameter subspace to capture the common information across different domains for DIL.We propose an importance-aware shared parameter subspace learning network for DIL, as illustrated in Figure 1 . Our method attempts to model the relations across the domains in sequence while mitigating the issue of forgetting. As depicted in Figure 1 , we first attempt to decouple each parameter matrix Δ𝑊 𝑖 into three distinct components: a domain-specific matrix 𝐴 𝑠 𝑖 tailored to each domain 𝑖, a domain-shared matrix 𝐴 𝑐 holding relevance across all domains, and a low-rank matrix 𝐵 𝑖 serving as the coefficient matrix. To ensure that the shared matrix 𝐴 𝑐 can accumulate crucial common information from each domain and alleviate forgetting, we present a momentum update strategy on 𝐴 𝑐 to smoothly accumulate of knowledge of all domains. Moreover, we devise an importance mechanism to weight 𝐴 𝑐 for each domain, given that 𝐴 𝑐 has varying degrees of importance to different domains. Moreover, we impose a cross-domain contrastive constraint on different domain-specific subspaces 𝐴 𝑠 𝑖 to capture distinctive information within each domain. Finally, we present another orthogonality constraint on domain-shared subspace 𝐴 𝑐 and each domain-specific subspace 𝐴 𝑠 𝑖 , so as to minimize the interference between them.Here we first introduce how to decouple each parameter matrix Δ𝑊 𝑖 into three distinct components: a domain-specific matrix 𝐴 𝑠 𝑖 , a domain-shared matrix 𝐴 𝑐 , and a low-rank coefficient matrix 𝐵 𝑖 .Parameter Subspace Decomposition for DIL. In fact, we can view the low-rank parameter matrix Δ𝑊 𝑖 ∈ R 𝑑 1 ×𝑑 2 for domain 𝑖 as an inner product between two low-rank matrices 𝐴 𝑖 and 𝐵 𝑖 as:EQUATIONwhere 𝐴 𝑖 ∈ R 𝑟 ×𝑑 2 and 𝐵 𝑖 ∈ R 𝑑 1 ×𝑟 . 𝑟 ≪ min(𝑑 1 , 𝑑 2 ). The parameter subspace 𝐴 𝑖 can be regarded as a base matrix and the parameter subspace 𝐵 𝑖 can be regarded as a coefficient representation matrix. By low-rank matrix decomposition, the most significant information within 𝑊 𝑖 can be captured by 𝐴 𝑖 and 𝐵 𝑖 . 𝐴 𝑖 and 𝐵 𝑖 can be spanned by 𝑟 row vectors, respectively:EQUATIONHowever, Eq. ( 2) does not explicitly model the relationships among different domains, leading to suboptimal performance. To this end, we further decompose the low-rank parameter matrix 𝐴 𝑖 of each domain 𝑖 into two components: 𝐴 𝑠 𝑖 that is specific to each domain 𝑖, and 𝐴 𝑐 that is shared by all 𝑀 domains. The formulation is as follow:EQUATIONwhere the matrices 𝐴 𝑠 𝑖 and 𝐴 𝑐 can be represented by 𝑟 row vectors, denoted as:𝐴 𝑠 𝑖 = span{𝑎 𝑠,1 𝑖 , 𝑎 𝑠,2 𝑖 , • • • , 𝑎 𝑠,𝑟 𝑖 } 𝐴 𝑐 = span{𝑎 𝑐,1 , 𝑎 𝑐,2 , • • • , 𝑎 𝑐,𝑟 }. (5)Therefore, given a frozen pre-trained model 𝑊 0 , we can efficiently train the model for the current domain 𝑖 by:EQUATIONNote that Eq. ( 6) is applied to each layer of the pre-trained model. For the sake of convenience, we omit the index of the layer. By Eq. ( 6), we can decouple the parameter subspace of each domain into a domain-specific parameter subspace and a domain-shared parameter subspace.After that, we can plug Eq. ( 6) into Eq. ( 1), and obtain a new loss function as:EQUATION3.3.2 Momentum Update on Domain-shared Subspace. As domains arrive sequentially, direct optimization of the shared subspace 𝐴 𝑐 for each newly arriving domain leads to updates primarily focused on learning the current domain. However, this excessive emphasis on the current domain may lead to the knowledge associated with previous domains being overwritten, potentially resulting in forgetting the knowledge of earlier domains. To alleviate this issue, we introduce a momentum update strategy to optimize the domain-shared subspace for each domain, facilitating the seamless integration of learned knowledge across all historical domains, inspired by [10] . By leveraging the momentum update strategy to optimize the shared subspace 𝐴 𝑐 , we can preserve the direction and velocity of updates from previous domains. This smoothing update trajectory can reduce excessive adjustments to previous domains, thus alleviating the issue of forgetting. Formally, we update 𝐴 𝑐 by exploiting the smooth copies of previous domain-specific subspaces 𝐴 𝑠 𝑖 in every update step 𝑡 as:EQUATIONwhere 𝜂 ∈ (0, 1) is a large (i.e., close to 1) momentum coefficient. Note that 𝐴 𝑐 (𝑡) is updated at every step 𝑡 as new domains continuously arrive. The domain-specific parameter subspace 𝐴 𝑠 𝑖 (𝑡 -1) is updated by back-propagation.In this section, we will introduce how to further enhance the capacity of the decomposed parameter subspaces, so as to improve the performance of DIL.Given that domain-shared parameter subspace 𝐴 𝑐 holds varying degrees of importance to different domains (which has been verified in Figure 2 ), thus it is crucial to design a dynamic importance allocation mechanism to adaptively assign a weight to 𝐴 𝑐 for each domain. To achieve this, we introduce a hyper-parameter 𝜌 𝑖 (𝑡) to dynamically weight the domain-shared parameter subspace at every training step 𝑡 as:EQUATIONTo automatically learn the hyperparameter 𝜌 𝑖 (𝑡), we design an importance allocation mechanism to dynamically learn 𝜌 𝑖 (𝑡) for each domain based on their sensitivity in the training dynamics. As pointed out in [24, 53] , the sensitivity of parameters essentially approximates the parameter change in loss: If the removal of a parameter causes a large influence on the loss, then the model is highly sensitive to the parameter, indicating that the parameter is extremely important. Motivated by this, we attempt to leverage the sensitivity of parameter subspaces to calculate the importance scores for each domain. An intuitive idea is that the higher the sensitivity of a subspace, the higher the corresponding importance score should be allocated to that subspace. Note that in order write conveniently, we will omit the symbol 𝑡.To this end, we consider the average sensitivity of each parameter within subspace 𝐴 𝑠 𝑖 as the overall contribution 𝑄 𝑖 of 𝐴 𝑠 𝑖 to the model performance. To ensure comparability across 𝑛 subspaces for all domains, we normalize𝑄 = {𝑄 1 𝑖 , 𝑄 2 𝑖 , • • • , 𝑄 𝑛 𝑖 }of 𝑛 subspaces into a standard Gaussian distribution. Then we impose a sigmoid function to obtain the importance score 𝜌 𝑖 ∈ (0, 1) for each subspace 𝐴 𝑠 𝑖 . Therefore, we can calculate the importance score of subspaces 𝐴 𝑠 𝑖 :𝜌 𝑖 = 1/(1 + exp 𝐸 (𝑄) -𝑄 𝑖 √︁ 𝑉 𝑎𝑟 (𝑄)),EQUATIONwhere ℎ = 𝑟 × 𝑑 2 denotes the number of parameters, and 𝑔 𝑖 = 𝐹𝑙𝑎(𝐴 𝑠 𝑖 ) denotes that we flatten the matrix 𝐴 𝑠 𝑖 into a long vector 𝑔 𝑖 . 𝑔 𝑗 𝑖 is the 𝑗-th element in the vector 𝑔 𝑖 . 𝑆 (•) denotes a sensitivity function for each parameter. 𝐸 (𝑄) and √︁ 𝑉 𝑎𝑟 (𝑄) denote the mean and standard deviation of the contribution of 𝑛 After that, we define the sensitivity of an individual parameter as the absolute value of the product between the gradient and the parameter [24, 30] . We can calculate the sensitivity of the 𝑗 𝑡ℎ parameter in subspace 𝐴 𝑠 𝑖 as:EQUATIONwhere Eq.( 11) approximates the change in loss when a parameter of subspace is zeroed out. It indicates that if the exclusion of a parameter from 𝐴 𝑠 𝑖 has a significant influence, then the model is highly sensitive to that parameter. Consequently, it is crucial to assign a high level of importance to that particular parameter.In fact, the sensitivity function 𝑆 calculates the loss changes reflected by an individual batch of samples. To mitigate the evaluation error caused by the sensitivity of a single batch, we can employ exponential moving average to smooth 𝑆 (𝑔EQUATIONwhere 𝜆 1 ∈ (0, 1) is the hyperparameter that controls the proportion of historical records and the current batch in the moving average. Given the intricate training dynamics may incur high variability and significant uncertainty, we introduce an additional computation of the uncertainty for sensitivity. The uncertainty captures the local temporal changes in sensitivity and is defined as:EQUATIONAfter that, we also smooth uncertainty by exponential moving average as:EQUATIONFinally, we adopt the the product between the sensitivity and uncertainty as the enhanced sensitivity as:EQUATIONWe dynamically allocate importance to different parameter subspaces by calculating their respective importance scores using the enhanced sensitivity measures, as defined in Eq. ( 15).Cross-domain Contrastive on Domain-specific Subspaces. In order to enable domain-specific subspaces 𝐴 𝑠 𝑖 to learn more discriminative knowledge specific to the current domain compared to other domains, we impose an cross-domain contrastive constraint on these domain-specific subspaces, thereby amplifying the distinctions between them. The idea is as follows: we expect that the probability of the current domain's data belonging to the true class predicted by the 𝑖 𝑡ℎ adaptation model 𝑓 (•; Δ𝑊 𝑖 , 𝜃 𝑖 ) should be higher than the highest probability using the adaptation models 𝑓 (•; Δ𝑊 𝑗 , 𝜃 𝑗 ), where 𝑗 < 𝑖, for each historical domain 𝑗. We compute the positive discrepancy probability 𝑝 𝑙 𝑖 of sample 𝑥 𝑙 ∈ D 𝑖 from the 𝑖 𝑡ℎ adaptation module on the correct class 𝑦 𝑙 as:EQUATIONwhere the output of 𝑓 (•; Δ𝑊 𝑖 , 𝜃 𝑖 ) has 𝑐 dimensions, and 𝑝 𝑙 𝑖 denotes the probability belonging to the true class 𝑦 𝑙 . We get the negative discrepancy probability 𝑝 𝑙 𝑗 of sample 𝑥 𝑙 from the 𝑗 𝑡ℎ adaptation module trained on the historical domain 𝑗 (𝑗 < 𝑖) as:EQUATIONThen we can formulate the cross-domain contrastive loss based on these discrepancy probabilities as follow:EQUATIONwhere the 𝑖 𝑡ℎ domain has |D 𝑖 | samples in total, and 𝜏 is a temperature parameter. Note that we only update the domain-specific subspaces 𝐴 𝑠 𝑖 and its weighting coefficients 𝐵 𝑖 when adapting to current domain 𝑖, making the historical adaption matrices 𝐴 𝑠 𝑗 , 𝐵 𝑗 for 𝑗 = 1, • • • , 𝑖 -1 and the domain-shared subspace 𝐴 𝑐 frozen.By the cross-domain contrastive constraint, we can encourage the model to effectively capture distinctive information specific to each domain while maintaining a clear separation between them.Orthogonality on Parameter Subspaces. Moreover, to minimize the interference between the domain-shared subspace and domain-specific subspaces, we propose enforcing orthogonality between each parameter vector 𝑎 𝑠 𝑖 in the domain-specific subspace 𝐴 𝑠 𝑖 and each parameter vector 𝑎 𝑐 of the domain-shared subspace 𝐴 𝑐 . This approach reduces parameter redundancy and enhances their representation capabilities. We formulate the orthogonality on these two kinds of subspaces as:EQUATIONwhere 𝑎 𝑠 𝑖 , 𝑎 𝑐 ∈ R 𝑑 2 . We can achieve this by exploiting an orthogonality penalty loss as:EQUATION)After introducing all the components of our work, we now give the final objective function based on ( 7), ( 18) and ( 20), which can be expressed as:EQUATIONwhere 𝛼 and 𝛽 are two trade-off hyperparameters.During training, we optimize our model by minimizing the loss in Eq.( 21) as the domains continuously arrive. We provide the overall optimization algorithm in Algorithm 1 of the Appendix A. At inference time, as the domain identifier is not available, we employ an existing method to infer the domain identifier, similar to the approach described in [47] . Specifically, we extract the features of each domain using the pre-trained model combined with the first low-rank adaptation model. Subsequently, we apply the k-means algorithm to these features to obtain a set of 𝑘 prototype vectors that effectively represent the information of current domain. For inference the domain identifier of the coming samples, we utilize the same pre-trained model with the first low-rank adaptation model to extract features of the input samples. We then estimate the nearest cluster center based on these features and use the index of the selected nearest cluster center as the domain identifier, denoted as 𝑞. Consequently, we exploit the 𝑞 𝑡ℎ low-rank module combined with the pre-trained model to make prediction. 
experiments 4.1 experimental settings: 
 4.1.1 Datasets. We conduct experiments on three standard DIL benchmark datasets: CDDB [20] , DomainNet [35] , and CORe50 [26] . CDDB is a continual deepfake detection dataset, which aims to identify real and fake images across various domains. We choose the most challenging Hard track as in [47] . We denote it as CDDB-Hard. It involves 5 sequential deepfake detection domains with roughly 27,000 images, i.e., GauGAN, BigGAN, WildDeepfake, WhichFac-eReal, and SAN. DomainNet has 6 distinct domains with roughly 600,000 images, and each domain containing 345 categories. We adopt the same DIL setup as DomainNet in [47] . CORe50 is a publicly available dataset for continual object recognition. It consists of 11 different domains, where each domain contains 50 categories. We follow [8, 50] to utilize 8 domains with 120,000 images for incremental training, and leave the rest 3 unseen domains for testing.For CDDB-Hard, we report the average forward detection accuracy across all domains, similar to [20] . For DomainNet and CORe50, we report the average forward classification accuracy of all domains, similar to [47] and [50] .4.1.2 Baselines. Since our method belongs to the category of rehearsal-free domain incremental learning (DIL) approaches, we compare our method with three state-of-the-art rehearsal-free DIL methods, i.e., prompt-based methods including L2P [50] , S-Prompts [47] , and CODA-Prompt [42] . In addition, we compare our method with two representative parameter-efficient-tuning (PET) methods for continual learning, including one prompt-based method, DualPrompt [49] and a hybrid PET method, LAE [9] . Moreover, we also take one regularization-based method, EWC [16] , and one distillation-based method, LwF [23] , as our baselines. Finally, we apply LoRA to each domain, and acquire the low-rank matrices independently across different domains, denoted as DiLoRA. We take DiLoRA as another baseline. To ensure a fair comparison, we evaluate all PET based methods using the same model, ViT-B/16 [7] , pre-trained on the ImageNet21k [39] dataset. 4.1.3 Implementation Details. We conduct experiments using the GeForce RTX 3090 Ti GPU. Input images are randomly resized to 224 × 224 resolution and normalized by standard deviation, following S-Prompts [47] . All baselines employ the same data augmentation strategy as ours for fair comparisons. Optimization across all experiments is performed using the Adam optimizer. We set the learning rate to 0.003 and utilize the batch size of 50 for three datasets. For the CDDB-Hard dataset, we set the hyper-parameter 𝛼 to 0.5 and the hyper-parameter 𝛽 to 0.5. For the DomainNet dataset, we set 𝛼 to 0.1 and 𝛽 to 0.5. For the CORe50 dataset, we set 𝛼 to 4 and set 𝛽 to 2. We train each domain 13 epochs, 5 epochs, and 3 epochs for CDDB-Hard, DomainNet, and CORe50, respectively. For the three datasets, we set the exponential moving average hyperparameters 𝜆 1 and 𝜆 2 to 0.85 throughout the experiment. The momentum coefficient 𝜂 mentioned in Eq.( 8) is empirically set to 0.9999. During inference, we set the number of clusters to 𝑘 = 5, as recommended by S-Prompts [47] . We repeat every experiment three times to report the average result.We conduct experiments on the three datasets, and test two kinds of different numbers of domains for each dataset: the first three domains and all five domains for CDDB-Hard, the first three domains and all six domains for Do-mainNet, and the first four domains and all eight domains for CORe50. The average results are reported in Table 1 . It can be observed that our method consistently outperformed all other competing approaches. Particularly, our method outperforms all four Prompting-based CL methods, demonstrating the effectiveness of our parameter subspace learning approach for DIL. when compared with DiLoRA, our method achieves better performance. This demonstrates that by leveraging the shared information among different domains, we are able to enhance the overall performance of DIL. Moreover, our method outperforms LAE in a large margin. This may be because LAE aims to equally accumulate historical knowledge among all past domains. However, not all the information in these domains is helpful. Thus, it is necessary to learn importance-aware parameter subspace for DIL.We design several variants of our method to analyse the effect of different components. The results are listed in Table 2 . "Ours-w.o.-𝐴 𝑐 " and "Ours-w.o.-𝐴 𝑠 𝑖 " refer to our method without utilizing the domain-shared subspace and domain-specific subspaces, respectively. "Ours-w.o.-importance" denotes the equal contribution between domain-shared and domain-specific subspaces for each domain. Our method achieves better performance than "Ours-w.o.-𝐴 𝑐 ", illustrating that it is necessary to capture common information among different domains for DIL. Our method performs better than "Ours-w.o.-importance", indicating that it is important to assign different weights to the domain-shared parameter subspace for each domain. Moreover, when removing the domainspecific subspace for each domain, "Ours-w.o.-𝐴 𝑠 𝑖 " achieves worse performance, demonstrating domain-specific information is very useful for representing each domain.We also verify the loss terms in our final objective function in (21) . We first perform the experiments by removing both the L 𝑜 term and the L 𝑐 loss term, denoted as "Ours-w.-cls-only". "Oursw.o.-o" means that we remove the orthogonality constraint on the domain-shared and domain-specific subspaces by setting the hyperparameter 𝛽 = 0. Moreover, we test the effectiveness of the crossdomain contrastive loss by setting 𝛼 = 0, denoted as "Ours-w.o.-c". The results are listed in Table 3 . It is evident that each loss term in our method contributes to the final performance of our method. More ablation studies can be found in Appendix B.1.Effect of Different Importance Weighting. We conduct an experiment on the CDDB-Hard dataset to illustrate that domainshared parameter subspace holds varying degrees of importance to different domains. Specifically, we use the first loss term L in (21) to acquire domain-shared subspace and domain-specific subspaces in a momentum update manner. After that, we manually introduce 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0. varying proportions of importance on domain-shared subspace and domain-specific subspaces, ranging from {0, 0.1, 0.2, • • • , 1}. The results are shown in Figure 2 . As illustrated in Figure 2 , we can make the following observations: (1) There is a notable variation in the importance of domain-shared information across each domain. (2) The utilization of domain-shared information enhances performance for each domain in the CDDB-Hard dataset (When the proportion is set to zero, it means that we do not use domainshared information). (3) The optimal proportion of domain-shared information significantly varies across different domains. Therefore, we can draw the conclusion that dynamically assigning an importance weight to the domain-shared subspace benefits the final performance of DIL.Recall that our domain-shared and domain-specific subspaces consist of 𝑟 row vectors when decomposing the parameter matrix.Here we investigate the effect of the varying number of vectors on the three datasets. As illustrated in Figure 3 (a), we find that for the CORe50 dataset, setting the number of vectors within the subspace to 2 is sufficient to achieve good results. The performance of having a larger number of vectors drop slightly. This can be attributed to the relatively simple content in the image, where lower rank matrices can capture sufficient information. For the Domain-Net and CDDB-Hard datasets, we observe that setting a relatively Overall, the performance of all incremental domains initially increases as the number of the inserted blocks increases, and then levels off. We observe that by attaching the decomposed subspaces to the first 10 transformer blocks, we can achieve the best performance for all the three datasets. Impact for Hyper-parameters. Finally, we analyze the impact of the involved main hyper-parameters in our method on the performance. We examine the parameter sensitivity of 𝛼 and 𝛽 introduced in Eq.( 21) on the CDDB-Hard dataset. For evaluation, we keep all other hyperparameters to be fixed except for the one being tested. We analyse 𝛼 and 𝛽 varying from {0.05, 0.1, 0.5, 1, 5} respectively, and report the results in Figure 5 (a) and (b). we can see the performance of our method is relatively stable in a relatively wide range. Thus it is easy to set in practical applications. More analysis on the hyper-parameter sensitivity can be found in Appendix B.3. 
conclusion: 
 In this paper, we proposed an importance-aware shared parameter subspace learning framework for domain incremental learning (DIL). Our approach focused on incrementally learning a domainshared low-rank parameter subspace across all domains and a domain-shared low-rank parameter subspace for each domain. Moreover, we presented a momentum update strategy to learn the domain-shared subspace, which can smoothly capture shared information across different incremental domains to mitigate forgetting. Recognizing that the importance of the domain-shared subspace may vary across domains, we proposed an importance-aware mechanism to adaptively learn importance weight for each domain. Furthermore, we imposed two constraints on the domain-specific and domain-shared subspaces, so as to enhance their representation capabilities. Extensive experiments on three publicly available DIL datasets verified the effectiveness of the proposed method, compared to the state-of-the-arts. 
 
 Appendix:  


Decision: This paper proposed a novel method to efficiently fine-tuning model parameters in domain incremental task. The proposed framework tries to decouple the learnable parameters with regard to being domain-specific or domain shared parameters; Thus enhancing the overall performance. 

We thank the authors for their great work and the comprehensive rebuttal discussion provided. We think this work is impactful and beneficial and definitely suitable for ACM MM.

The paper addresses an important aspect. The experiments and the results are solid. The novelty of the method is original. We think this paper is useful and impactful.

Reviews: review: paper_summary: paper_summary: In this paper, the authors proposed an importance-aware shared parameter subspace learning framework for domain incremental learning
(DIL), which focused on incrementally learning a domain-shared low-rank parameter subspace across all domains and a
domain-shared low-rank parameter subspace for each domain. Extensive experiments validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Novel method and interesting idea. The idea of decomposing the LoRA-like parameter into domain-shared part and domain-specific part is novel and inspiring to me.
2. Extensive experiments validate the effectiveness of the proposed method.
3. The paper is well-written and easy to follow.
limitations: limitations: 1. Could the proposed method be integrated with CNNs？
2. Could the method be further extended to variants of LoRA (i.e. ConvLoRA[1])?

It would be ok to answer 'No', overall, I'm satisfied with the paper.

[1] CONVOLUTION MEETS LORA: PARAMETER EFFICIENT FINETUNING FOR SEGMENT ANYTHING MODEL, ICLR 2024
paper_summary: paper_summary: The authors propose a novel parameter-efficient tuning (PET) approach for domain incremental learning (DIL) based on low-rank adaptation (LoRA). The key contributions include learning domain-specific and shared low-rank subspaces, using momentum updates and importance weighting for the shared subspace, and imposing contrastive and orthogonality constraints. Experiments on three DIL datasets show improvements over several state-of-the-art baselines.
strength_weakness: strength_weakness: 1. The paper addresses the challenging and practically relevant problem of domain incremental learning using a novel LoRA-based PET approach. This is a timely contribution given the growing interest in parameter-efficient tuning methods. The technical approach is well-motivated and grounded in sound principles.
2. The experimental evaluation is quite comprehensive, covering three diverse DIL datasets and comparing against a good selection of 8 recent state-of-the-art baselines.
limitations: limitations: 1. While the overall results are strong, the performance gains are not uniform across the datasets. On DomainNet and CORe50, the improvements over some baselines are quite marginal. 
2. The approach introduces several hyperparameters ($\alpha$, $\beta$, $r$, momentum coefficient, block insertion strategy, etc.). While the sensitivity to some of these is analyzed, it's unclear how a practitioner should set these for a new dataset. 
3. The finding that the optimal importance of shared information varies widely across domains is intriguing but not sufficiently explained. What characteristics of the domains cause this? How does the proposed importance weighting mechanism capture this? 
4. There are some missing details that would aid in reproducibility, such as the specific rank $r$ used for the subspaces and the layers where they are inserted.
paper_summary: paper_summary: This papre puts forward an importance-aware shared parameter subspace learning for domain incremental learning, on the basis of low-rank adaption (LoRA). It focused on incrementally learning a domainshared low-rank parameter subspace across all domains and a domain-shared low-rank parameter subspace for each domain.  Extensive experiments on image domain incremental datasets demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The writting is clear.
2. The subspaces as special domain and shared domain are reasonable, and an interesting idea.
3. The proposed importance allocation mechanism is to dynamically learn the hyperparameter in each domain, which is soundable.
limitations: limitations: 1. The reason of adopting Lora is not clear,.
2.  Some formula symbols are not explained clearly, like (11) the loss function L is L_final in (15)?  
3. Since too many parameters in this paper, it is better to annotate clear which one is learnable.
paper_summary: paper_summary: In this paper, the authors proposed an importance-aware shared parameter subspace learning for domain incremental learning, on the basis of low-rank adaption (LoRA), which incrementally learn a domain-specific and domain-shared low-rank parameter subspace for each domain, in order to effectively decouple the parameter space and capture shared information across different domains. Moreover, given that domain-shared information might hold varying degrees of importance across different domains, the authors designed an importance-aware mechanism that adaptively assigns an importance weight to the domain-shared subspace for the corresponding domain. Experiments on image domain incremental datasets demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The various modules in the entire method are relatively mature. The authors designed an importance-aware mechanism that adaptively assigns an importance weight to the domain-shared subspace
for the corresponding domain.
limitations: limitations: 1. The motivation discussed in the paper is not very clear. The authors claim that: 'incrementally learn a domain-specific and domain-shared low-rank parameter subspace for each domain in a momentum update manner, which can effectively capture shared information across different domains.'. However, the theoretical justifications lack. Could the authors elaborate on the theoretical justification for the effectiveness, especially regarding the role of shared information across different domains? 

2. Emprical results mainly focus on visual classification tasks, are there more results on other visual tasks?