Learning Expandable and Adaptable Representations for Continual Learning
Authors: Ruilong Yu
Mingyan Liu
Fei Ye
Adrian G. Bors
Rongyao Hu
Jingling sun
shijie zhou
Venue: NeurIPS 2025 poster ()
Keywords: Mixture model
Continual Learning
Primary area: optimization

Abstract: Extant studies predominantly address catastrophic forgetting within a simplified continual learning paradigm, typically confined to a singular data domain. Conversely, real-world applications frequently encompass multiple, evolving data domains, wherein models often struggle to retain many critical past information, thereby leading to performance degradation. This paper addresses this complex scenario by introducing a novel dynamic expansion approach called Learning Expandable and Adaptable Representations (LEAR). This framework orchestrates a collaborative backbone structure, comprising global and local backbones, designed to capture both general and task-specific representations. Leveraging this collaborative backbone, the proposed framework dynamically create a lightweight expert to delineate decision boundaries for each novel task, thereby facilitating the prediction process. To enhance new task learning, we introduce a novel Mutual Information-Based Prediction Alignment approach, which incrementally optimizes the global backbone via a mutual information metric, ensuring consistency in the prediction patterns of historical experts throughout the optimization phase. To mitigate network forgetting, we propose a Kullback–Leibler (KL) Divergence-Based Feature Alignment approach, which employs a probabilistic distance measure to prevent significant shifts in critical local representations. Furthermore, we introduce a novel Hilbert-Schmidt Independence Criterion (HSIC)-Based Collaborative Optimization approach, which encourages the local and global backbones to capture distinct semantic information in a collaborative manner, thereby mitigating information redundancy and enhancing model performance. Moreover, to accelerate new task learning, we propose a novel Expert Selection Mechanism that automatically identifies the most relevant expert based on data characteristics. This selected expert is then utilized to initialize a new expert, thereby fostering positive knowledge transfer. This approach also enables expert selection during the testing phase without requring any task information. Empirical results demonstrate that the proposed framework achieves state-of-the-art performance.

Learning Expandable and Adaptable Representations for Continual Learning
Ruilong Yu', Mingyan Liu', Fei Ye'; Adrian G. Bors Rongyao Hu', Jingling Sun', and Shijie Zhou'
University of Electronic Science and Technology of China, Chengdu, China Harbin Institute of Technology, Shenzhen, China
University of York, York, U.K.
yr1666@outlook.o com, 2023312903dstu, hit.edu. cn, feiyeQuestc. setreesstt.edu...e.. cn, adrian www.aao..ororroo.ooooooooo...... ac. ac. uk, ryhu@uestc edu edu.cn, en,
jingling. sun9 1 00gmail.........mm.....mmm...1 com, sjzhouQuestc.edu. edwaesttc.eu......
Abstract
Extant studies predominantly address catastrophic forgetting within a simplified continual learning paradigm, typically confined to singular data domain. Con- versely, real-world applications frequently encompass multiple, evolving data domains, wherein models often struggle retain many critical past information, thereby leading to performance degradation. This paper addresses this complex scenario by introducing novel dynamic expansion approach called Learning Expandable and Adaptable Representations (LEAR). This framework orchestrates collaborative backbone structure, comprising global and local backbones, de- signed to capture both general and task-specific representations. Leveraging this collaborative backbone, the proposed framework dynamically creates a lightweight expert to delineate decision boundaries for each novel task, thereby facilitating the prediction process. To enhance new task learning, we introduce novel Mutual Information-Basse Prediction Alignment approach, which incrementally optimizes the global backbone via mutual information metric, ensuring consistency in the prediction patterns of historical experts throughout the optimization phase. To miti- gate network forgetting, we propose Kullback-Leibler (KL) Divergence-Based Feature Alignment approach, which employs probabilistic distance measure prevent significant shifts in critical local representations. Furthermore, we intro- duce novel Hilbert-Schmine Independence Criterion (HSIC)-Based Collaborative Optimization approach, which encourages the local and global backbones to cap- ture distinct semantic information in collaborative manner, thereby mitigating information redundancy and enhancing model performance. Morcover, to accel- erate new task learning, we propose novel Expert Selection Mechanism that automatically identifies the most relevant expert based on data characteristics. This selected expert is then utilized initialize knowledge transfer. This approach also enables expert selection during the testing phase without requiring any task information. Empirical results demonstrate that the proposed framework achieves state-of-the-art performance. Code is available at wtttt:://// /ate::///tttbbb thub. ccw../yrluussst//eer////2//2////
expert, thereby fostering positive
"Corresponding author: feiye uest..uues....
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Modern deep learning frameworks have shown exceptional effectiveness across various visual tasks However, the high performance of these approaches largely depends large datasets, which are frequently unavailable in settings marked by constant change. This approach to learning is known Continual/Lifeein Lifelong Learning (CL), which aims to create model that can continuously integrate new information while preserving all previously learned knowledge. Catastrophic forgetting significant issue that hinders the model's 's performance on earlier tasks [32] arising when the model tries to adjust its parameters to learn new tasks.
11 [9]
Among the various approaches to mitigate catastrophic forgetting in continual learning [1]. Expansion- Capansion-Based Methods (EBMs) have emerged as leading and highly effective strategies. The core idea is to dynamically expand the model's internal structure by adding task-specific modules to allocate dedicated capacity for each new task. However, current EBMs primarily focus on Class- Incremental Learning (CIL) 37 within single domain, neglecting the scenario of learning across multiple domains, known as have investigated DIL, their evaluated domains (e.g., Aireraft 22 MNIST (27) have achieved near-perfect accuracy with pre-trained ViTs [00, making these benchmarks inadequate for assessing genuine continual learning capabilities. Therefore, we establish a more challenging and more realistic Multi-domain Continual learning (MDCL) scenario, where the task sequence comprises not only complex domains with large discrepancies but also mixture of domains with underlying similarities. In this study, we aim to improve the model's performance in MDCL by considering three aspects including plasticity, stability and efficiency. To implement this goal, we propose a novel approach called LEAR and its core idea is to fully explore the stable and dynamic representations extracted by the pre-trained ViT backbones to achieve fast adaptation while adaptively optimizing the backbones to maintain all previously learned information.
Learning (DIL). Although studies 44 [1] 21)
(1) Plasticity. Existing EBMs improve downstream task performance by integrating task-specific prompts [46] (36] or adapters (30, 49] into fixed pretrained backbone. However, these methods focus on exploring representations from single pre-trained backbone, which fails to address more challenging data domains such as CUB-200 [39] and ImageNet-R [17. Thus, to improve plasticity in challenging MDCL scenario, we introduce a novel collaborative backbone architecture for LEAR, comprising global and a local backbone, designed to capture general and task-specific information across all tasks. Leveraging this collaborative backbone structure, the proposed LEAR framework dynamically generates lightweight expert to learn the decision boundary for each new task, thereby achieving commendable performance. The results presented in Tab. [ a 2 demonstrate that our method achieves superior performance on most individual datasets in the MDCL scenario, which also validates that EBMs with frozen pretrained backbones cannot provide sufficient plasticity in MDCL.
(2) Stability. Many EBMs have been shown to achieve excellent stability in CIL. However, the excellent stability is usually achieved by freezing all parameters of the pre-trained models during the training, which may lead to forgetting of historical tasks in MDCL, especially when facing the severe domain shifts (c.g. ChestX [43] + ImageNet-R) in long task sequences. To address this limitation, we propose a unified optimization function to regulate the optimization behaviour of the collaborative backbone structure. This function consists of a Mutual Information- -aasad Prediction Alignment (MIBPA) loss and Kullback-Leible former dynamically optimizes the global backbone while preventing negative knowledge transfer at the prediction level, and the latter aligns historical and current representation distributions at the feature level. Such a design enables LEAR to achieve rehearsal continual learning by actively consolidating historical knowledge at both the prediction and feature levels when fine-tuning the collaborative backbones with new task data, rather than freezing parameters passively. Such a design has not been explored in the existing CL field. Furthermore, mitigate optimization interference and information redundancy between the collaborative backbones, we propose a novel Hilbert- Schmidt Independence Criterion- Based Collaborative Optimization (HSICBCO) strategy to encourage two backbones to capture different semantic information, thus promoting effective complementary learning of MDCL tasks. The experimental results demonstrate that LEAR significantly outperforms all baseline methods in terms of overall average accuracy in three MDCL scenarios.
Divergence-Bast
Feature Alignment (KLDBFA) loss. The
(3) Efficiency. Many existing EBMs usually ignore the task relevance and do not explore the previously learned parameter information to accelerate the new task learning. As a result, these methods optimize each new expert from scratch, which may result in considerable computational
2

---

costs and parameter redundancy when dealing with MDCL that contains analogous data domains. To address this issue, we aim to promote the efficient learning process of LEAR by proposing a novel Expert Selection Mechanism (ESM) that selectively transfers the parameter information learned by a selected expert into the new expert construction process. Specifically, the proposed ESM models each expert's knowledge as Gaussian memory distribution and only preserve its critical statistical information. For each new task, the proposed ESM selects the most relevant expert by minimizing the Mahalanobis distance between stored distributions and incoming data, and reuses its parameters to facilitate new task learning. During the testing phase, ESM autonomously routes testing samples to the most suitable expert in agskstgnottic manner.
The principal contributions of this research are enumerated as follows: (1) This paper explores the challenging MDCL scenarios by proposing novel approach called Learning Expandable and Adaptable Representations (LEEAR) that optimizes and manages collaborative backbone structure, comprising a global backbone and a local backbone, respectively. This design can help capture general and task-specific representations, which achieve excellent performance in MDCL. (2) novel MIBPA approach is proposed to optimize the global backbone via mutual information measure that ensures the consisteney of the prediction pattern of each history expert when adjusting the parameters of the global backbone. (3) A novel KLDBFA approach is proposed to regulate the optimization behaviour of the local backbone by preventing significant changes in many critical local representations. Such design can preserve task-specific representation information and prevent significant negative knowledge transfer effects. (4) A novel HSIBCO strategy is proposed to enforce the disentanglement between global and local representations, which avoids information redundancy and improves the model's 's performance. (5) A novel ESM is proposed to select the most relevant expert according the data's characteristics, which is used in the training phase promote the positive knowledge transfer process and in the testing phase to implement the expert selection procedure. The results from an extensive suite of experiments demonstrate that our proposed approach significantly outperforms existing baselines across all experimental configurations.
2 Related Work
Rehearsal- based techniques represent a widely adopted strategy for mitigating forgetting by dynam- ically incorporating a limited number of historical examples into the memory buffer (5)6]. These memory samples are leveraged alongside new training instances to enhance model performance during the new task learning. Thus, the quality of the memorized samples is paramount within the rehearsal-bass -ased optimization framework [14. Moreover, rehearsal- based approaches can be augmented through the integration of regularization techniques, with the objective of further elevating the overall efficacy of the model [2] 92 [0] 45). In addition, memory studies have proposed to train the generative models to implement the memory system, which can provide infinite generative replay
samples 1 33
35
50 23)
Prompt-based techniques leverage frozen pre-trained models like Vision Transformers (ViT) 10 as feature extractors, adapting them to sequential tasks through task-specific learnable prompt parameters. Current approaches employ diverse prompt management strategies including L2P [47hss s shared prompt pool with query-key retrieval mechanism, DualPrompt [46]'s 's separation of task-agnostic (G-Prompt) and task-specific (E-Prompt) components, and PODD--Prompt (36] 's irrentioo-weighhee cross-task prompt expansion. HiDeePPrompl [40] further advances performance by hierarchically decomposing
learning objectives for optimized task adaptation.
Expansion- based methods represent robust approach to mitigating network forgetting continual learning [8) Such an approach dynamically expands the network architecture to enhance the learning ability of the new task 22 38.. Beyond convolutional neural networks, expansion-base
techniques have also been explored to leverage the capabilities of ViTs the foundational network. These methods usually create self-attention blocks with the task-specific classifier to adapt to the new task learning [T] 48 30 49). However, these methodologies typically involve freezing the pre-trained model, which limits their adaptability to complex and unknown data domains. We provide additional information on the related work in Appendix-A from Supplementary Materials (SM).
3 Methodology
3.1 Problem Definition
CL seeks to develop a model capable of acquiring knowledge across multiple sequences tasks while retaining previously acquired information. This paper addresses more pragmatic learning context in
3

---

Shep L: Breeldgee presevvtioon
Adapheble ligreertotens (LEAR)
Learnty Sierada
Rackleee
Trail activlted buciburs Ser Wile)fid)
Mediel
Dapicete and freeze
Roaen Lee
Notare
Gibel Bucibced
Step 2: Frecer the traned expert
Rarend and
Lanz?: Maithhh the conlihercy Nittarial prodectin p-ple (0r:)))
Gesancts globoll/
export
represetctient
Task Awri
Local Backbene
Step 3: Gentnat nemoer dittibation
Nataral
Algrang dstribatios post end present represetation
Loosk: Capture divtect regestantotions "00
Seopl:
Frouee Locel
Generate stoti represuttttion
Suzey 1212 1PN2
Training hailar
Begh phone of hark jr1
Trainin thone task
phose of
Figure 1: The training framework of the proposed LEAR. The data samples from new tasks processed through a collaborative backbone structure to learn task- shared, task-specific and backbone- distinct representations via the proposed MIBPA, KLDBFA and HSICBCO, respectively. ESM constructs memory distributions and selects relevant experts for network expansion and test evaluation.
which each task encompasses previously unseen challenging data domains. Let D {x,, y
and DJ = Y " denote the i-th training and testing datasets, respectively. In class- incremental learning scenario [3) a data split procedure is performed to divide the training dataset DE into C₁ subsets [ Di,c;) according to the category, where each task T, is associated with a training dataset D., formed by samples from several adjacent classes. the context of specific task learning Ty. the model is restricted to utilizing only the training dataset D. with all preceding datasets {D, ,D. being inaccessible. In DIL, each task is conceptualized as distinct data domain, denoted as {D$ .... D.). with n representing the total number of tasks. In contrast to these two CL scenarios which posit that each task comprises non-overlapping and heterogeneous data samples, a new task within our MDCL encompasses data samples that exhibit not only similar semantic characteristics with previously seen tasks but also significant domain shifts. Consequently, it becomes imperative to leverage existing parameters to facilitate learning these analogous tasks, thereby accelerating the training process and minimizing resource consumption. Once the final task learning is finished, the model's efficacy is assessed across all testing datasets (DF ... D through the lens of classification accuracy.
3.2 Collaborative Backbone Structure
Recent investigations in CL have assessed the efficacy of leveraging a pre-trained ViT [0] to enhance model performance. These methodologies typically incorporate the pre-trained ViT as the primary backbone, facilitating the expert construction process while concurrently freezing its parameters mitigate catastrophic forgetting. Nevertheless, this architectural design constrains the model's 's capacity for learning in the context of novel tasks, particularly when the incoming data exhibits divergent domain characteristics. This paper addresses this limitation by introducing a novel collaborative backbone architecture, comprising global and local backbones, each instantiated via pre-trained ViT to facilitate rapid adaptation. Specifically, the global backbone incrementally updates its parameters throughout the optimization phase, with the objective of generating task-shared representation applicable across tasks. Conversely, the local backbone engineered dynamically adapt to new tasks through parameter adjustments. We propose to optimize only the final three layers of the global and local backbone mitigate computational expenses.
Let Foz: : X' -> Z' denote a global backbone, implemented using a pre-trained ViT, which receives a data sample over the data space and returns feature vector z' over the feature space 2'. Similarly, let Fo:: : X' + z' denote a local backbone, which has the same input-output pattern as the global backbone. For a given data sample x, we can obtain its feature representations extracted by the global and local backbones, expressed as:
Zgg = Foo 21 For (x),
(I)

---

By using Eq. ① the proposed framework dynamically creates a lightweight expert (E₂) consisting of a simple feature transformation module Fzg; Z' Z and linear classifier Fag: 2 - y, aiming
to learn decision boundary for specific task. Fuf receives the local representation z and returns feature vector over the feature space Z, which is concatenate with global representation 2, and fed into the linear classifier Poy make the prediction over the space y The subscript j denotes the expert index, and @ denotes the concatenation operation that combines two representations into a single feature vector. The prediction process of the j-th expert is expressed as
FclE,,x) x) = Fag (Fo (x) + F Fof (F))
By integrating representations derived by global and local backbones, the expert & in Eq. (2) can improve its generalization performance for a given data sample x.
3.3 Mutual Information- Prediction Alignment
The global backbone's objective is to furnish a unified representation across all observed tasks. Consequently, optimization of the global backbone is susceptible to catastrophic forgetting, impacting all historical experts. To mitigate this, we introduce novel Mutual Information- Based Prediction Alignment (MIBPA) methodology, designed to maintain the consistency of predictions of all historical experts when changing the parameters of the global backbone during the acquisition of new tasks. Specifically, construct parameter-shared auxiliary model Fig by replicating and freezing the global backbone's final three layers, then connecting them in parallel with intermediate features from the backbone's 's preceding layers. This auxiliary model subsequently guides the global backbone's optimization, producing two distinct prediction sets through the i-th expert, formulated as:
Y = -fycl = (Fo (xx) @ Fof (xx)))
.b. 8 = FF (x.) @ F of (For 1, .0).
FoF
where b denotes the size of the data batch X = {x1, }6} and x. denotes the c-th data sample of X. Let P(Y fr) denote joint distribution, where P(Y*) and P(Y+) represent the marginal distributions of Y and Y respectively. Let Y and ÿ denote two random variables over the joint distribution P(Y Yr)). The proposed MIBPA approach minimizes the mutual information between yi and V expressed as:
- E {Ever (P(Y', (') 8*) log P(Y:, P (*) p P(Y' ((P) }}
T());)= = Eyerr
(4)
where P(Y )r' y²) signifies the probability density function of P(Y Y*). The mutual infor- mation term (YY;; (Y"))")) as defined in Eq. 4 evaluates the distance of the prediction made by the i-th expert built on the previously and currently learned global backbones. A small mutual information term 1(Y;; yri) indicates that updating the global backbone can still maintain the prediction pattern of the i-th expert. Finally, the final MIBPA regularization loss function the j-th task learning is defined as
1 E i- AM = j - - (1()))))}
(5)
3.4 Kullback-Leible (KL) Divergence-Bast Based Feature Alignment
The iterative updating of the pre-trained backbones facilitates the temporal capture of local rep- resentations, thereby potentially enhancing the acquisition of novel tasks. However, this process risks inducing adverse knowledge transfer and performance degradation across historical experts. Regularization methods like EWC (24) and MAS 44, which typically impose constraints on param- eter updates, are not desirable to capture the complex distributional shifts across domains, while knowledge distillation methods like LWF 28 and iCaRL [d require maintaining additional teacher networks that become computationally prohibitive the number of tasks grows. To address these limitations, we propose Kullback-Leeibe Divergence-Band -aseeed Feature Alignment (KIDBFA), designed to preserve crucial historical parameters during the optimization of the local backbone.
5

---

Our design rationale for selecting KL divergence stems from two key considerations: Firstly, modern generative evaluation metrics (e.g., FID [8], Kernel MMD [3] operate on the Gaussian distribution assumption in high-dimensional feature spaces. This motivates us to model backbone features as Gaussian distributions. Fig. in Appendix-C from the SM also provides additional empirical valida- tion for the Gaussian distribution assumption. Secondly, KI. divergence offers unique advantages over alternative distributional metrics: (1) Its directional property enables targeted constraint of current features toward historical distributions, unlike symmetric metrics (e.g., -aasen-Shannon divergence); (2) It maintains computational efficiency compared to expensive metrics like MMD or Wasserstein distance. These characteristics make KL divergence ideally suited for continual learning scenarios requiring efficient knowledge preservation.
Specifically, upon each task transition, the proposed KLDBFA approach duplicates and immobilizes the local backbone Far as frozen model Fg. following a similar procedure MIBPA. Fgi serves to regulate the optimization dynamics of the local backhone. For given data batch X = (x,,""',,a two distinct sets of feature vectors are derived utilizing Fgr and FØ.- respectively, as follows:
z' = Zze Ze = For (Xo), c 1. .b) Z' {zel z Foi (xe), c 1..,
(6)
Building upon the Gaussian assumption stated above, we model two Gaussian distributions P(Z') (H1, E1) and P(Z')- = (N Σ2), through calculating the mean vectors 1 22 and covariance matrix E,. Σn of Z and Z', respectively. We propose to employ the KL divergence to evaluate the discrepancy between P(Z') and P(Z') as regularization loss term, expressed as:
1 (KI(P(Z)))P((Z)) 2
log (et((22)
- d+ (r(Eg'E)) + (u2 - -m1) -m) E2 (u2 - mi)
(e,(
CKLDBEA (xz(P(Z)) ((
where D denotes the KL divergence. det(.), d, and tr(.) represent the determinant, dimension, and trace of a matrix, respectively.
3.5 HSIC-Based Collaborative Optimization
The global and local backbones are designed capture distinct feature representations, thereby poten- tially improving model efficacy. To further facilitate the disentanglement between these backbones, we introduce novel Hilbert-Schmidt Independence Criterion (HSIC)-Basee Collaborative Optimiza- tion (HSICBCO) methodology. This approach leverages an independence criterion to maximize the divergence of knowledge between the global and local backbones. Specifically, employ the HSIC measure [12), given its property of ranging from 0 to infinity, with 0 signifying statistical indepen- dence. Consequently, minimizing the HSIC term allows for enhanced disentanglement between the global and local backbones, which can be easily added to the primary loss function.
Let 799 and Z' denote two distinct domains, and let I2z z represent joint distribution from which sample pair (zg. z1) is drawn using global and local backbones across Zo x Z The primary objective of HSIC, as delineated in [12,, within the framework of Reproducing Kernel Hilbert Space (RKHS), 42]. is to quantify the dependency between the domains of the variables z, and z by assessing the norm of the cross-covariance operator over the domain 79 Z:. Let Q and B be the RKHSs defined on 7 and Z', respectively, and let fq: Z + Q, and fB: Z - B denote their respective feature mappings. The associated reproducing kernels are defined as k(Zz. z/z) = and 1(21, 2 = (fn(z)), fn()))) where gg 2 E 29 and 21, 2 E Z!. The cross-covariance operator between fo and fO defined as follows:
Oz -Ez (fo(zz) - Ez, (fo(z))) @ (fa(z) - Ez, (f (f(())))
(8)
where @ is the tensor product. HSIC is defined as the square of the Hilbert-Schmid norm of Cz
11 = En, =b [k( zz, M2z z|)] CHSIO(Q,B,u B, 2 = |C2,, 11
+ Ez x2 [k( : 2 [,, a
x/ [1(,, z1) - 2E [Exg lk(zz. : 2 E [[|z zf)]],
where Ez,,2, 21 2,, represents the expectation over samples {zg. 2 and (z/z z drawn from Fzz ..

---

3.6 Expert Selection Mechanism
In scenarios where analogous data domains are encountered subsequent tasks, the reuse of pertinent parameters and information becomes imperative for the efficient learning of such domains, thereby accelerating the training process of a novel task. To this end, this study introduces a novel Expert Selection Mechanism (ESM), designed to identify the most relevant expert for given new task, facilitating the reuse of existing parameters to initialize a new expert, which, in turn, can engender positive knowledge transfer effects.
The memory distribution. Specifically, we utilize a frozen pre-trained VIT as feature extractor, denoted as FØs: X' + 2", to generate static data representations. where of represents the fixed parameters of the ViT backbone. To mitigate parameter redundancy, we duplicate and freeze the local backbone's s final three layers as in KLDBFA in the first task. Upon the completion of specific task learning phase (T₁), a subset of training samples {xx) is randomly selected from Dj and processed by For to extract the class token representation z = For (xx) These extracted features are subsequently propagated through the fully connected layer of the current expert E,, yielding transformed features:
2 = F
(10)
By using the transformed features, we obtain the empirical mean μe, and covariance matrix E, by
Mi = Σ }
Σ
1 m
{(2 H2) - -M,)
(11)
Subsequently, a multivariate Gaussian distribution N₁ NN Eg) constructed preserve the statistical information about the j-th task. We call N2 as the memory distribution for the expert E which is always fixed during the subsequent learning.
The expert selection process. When processed by the frozen backbone For to obtain zi = For (xi). For the I-th representation z"' we employ the feature transformation modules {F of For) of all existing experts ..,,... Eg}
task Ty+1 begins, its training samples (x) Yr" first
to generate set of transformed features zf F P. (z)), Wc 1,... .j. Based on the transformed features, the most relevant expert e.. with the minimum average Mahalanobis distance at the new task learning (T}+1) is selected by
= argmin
Σ Vls - Mc) '2 1e))
(12)
where c* denotes the index of the selected expert. Finally, new expert Ej+1 is initialized using the parameters of the selected expert E.r, particularly inheriting its feature transformation module F Fzy and linear classifier F The new expert is then fine-tuned on the subset of incoming task samples to adapt to the new domain. During inference, test samples are assigned to the most suitable expert for prediction using the same selection strategy. The reason for choosing the Mahalanobis distance is that it accounts for the scale and correlation of variables, making it suitable for datasets where features are correlated or have different units. Other distance measures are analyzed Appendix-D from SM. This Mahalanobis distance-basee expert selection strategy enables the model to dynamically identify the most semantically related expert for knowledge transfer, thereby accelerating convergence and reducing parameter overhead during continual learning.
3.7 Algorithm Implementation
The learning procedure of the proposed LEAR (illustrated in Fig. consists of three stages:
Step 1: Collaborative backbone initialization. We initialize both global and local backbones using pre-trained VIT models. These networks serve as the feature extractors for all tasks throughout the Continual Learning process. For given input x, obtain its feature representations extracted by both backbones using Eq. (T).
Dynamic expert creation and selection. During the training of the j-th task (T,), we dynamically create lightweight expert tailored to this domain. To ensure effective knowledge transfer, we select the most relevant historical expert based on the Mahalanobis distance computed
Step 2:
7

---

Table 1: The classification accuracy (%) of all testing datasets after learning the CDM task sequence.
Methods
C10
Disease
MNIST
RESISC45
EuroSAT
Timg
C100
ChestX
ImgR
CUB200
Avg
(Ee++RR)
22.78
34.00
11.33
8.24
18.18
7.66
31.79
25.99
54.04
78.23
29.22
CLS-ER
22.58
27.22
12.50
14.32
24.17
14.14
34.23
25.99
52.46
75.78
30.34
RanPAC
87.17
96.76
87.45
84.20
92.53
71.46
51.44
39.63
43.30
56.18
71.01
MoE
92.74
32.44
91.87
54.02
41.85
6.12
78.98
19.60
78.43
77.24
57.33
L2P
20.66
11.15
14.08
4.50
13.65
11.67
31.76
21.52
58.26
81.30
26.85
DAP
8.83
2.78
18.94
3.36
18.19
11.29
16.34
60.68
80.37
22.38
D- D--ormpt
25.99
9.08
16.57
4.13
7.30
22.73
38.36
17.33
59.06
82.44
28.30
C-Proopt
13.57
2.33
9.10
1.90
14.18
0.68
3.40
14.35
4.14
60.55
12,42
Ours
95.44
98.46
96.59
92.04
95.00
81.24
85.10
45.95
70.47
85.80
84.61
Table 2: The classification accuracy (%) of all testing datasets after learning the ETI task sequence.
Methods
EuroSAT
Tlmg ImgR CUB200
C100 MNIST RESISC45
ChestX
C10
Disease
Avg
DER++(R))
51.82
36.25
2.32
6.20
23.08
65.97
40.45
27.41
82.69
97.77
43.40
CLS-ER
45.76
29.33
16.19
33.08
30.74
67.10
46.44
30.26
80.29
97.62
47.68
RanPAC
92.64
70.87
43.75
56.13
51.83
88.08
83.51
40.34
86.74
96.88
71.08
MoE
43.01
0.94
55.48
25.16
74.22
97.67
84.57
33.38
96.88
99.90
61.12
L2P
10.88
1.45
0.97
4.04
14.55
12.17
22.48
9.16
89.69
98.62
26.40
DAP
9.93
1.07
1.84
17.65
15.74
15.44
22.19
16.34
86.20
98.44
28.48
D-Prompt
10.61
1.31
1,44
3.09
17.03
23.83
25.03
14.77
93.42
99.33
28.99
C-Prompt
11.66
0.67
0.62
0.33
1.72
13.04
6.14
16.62
21.74
96.21
16.88
Ours
95.89
81.25
69.57
84.12
85.30
98.56
92.92
45.45
96.60
99.30
84.90
over previous feature distributions through Eq. 122. initialize new expert.
As a result, the selected expert employed to
Step 3: Interactive optimization with alignment constraints. Once the new expert & created, we perform joint optimization incorporating mutual infornation-based prediction alignment via Eq. (5) and KL-divergence feature alignment using Eq. 7 Furthermore, we calculate the HSIC regularization losses to encourage disentanglement between global and local backbones using Eq. (9) All regularization terms are incorporated into the main objective function for expert optimization
= = -' E {volt) log(F x.) AL
Cfnall (X)
(13)
+ >2LKIIDBFA + >aLniscc P,
where xc)(1) denotes the predicted probability class for sample x ^1, X2. 0g are trade-off hyperparameters balancing different loss components. The model parameters {0o, 0., s 18} is updated using Eq. (13). The detailed algorithm is summarized in Appendix-B from SM.
4 Experiment
4.1 Experimental Setup
Metrics. In the context of the MDCL scenarios, we assess and compare model efficacy at the final task through two key metrics: the classification accuracy of a single domain (e.g., C10 or CUB200) and the overall performance across all domains (Avg).
Datasets. The datasets used our experiment can be logically categorized into three primary fields according 211) Natural Domains include CIFAR-10 25 (C10), TinyImageNet (26
(TImg), CUB-200 (39] MNIST 27 and ImageNet-R [17] (ImgR), covering a range of tasks from basic image classification to fine-grained recognition and robustness testing across various visual styles. Aerial Domains comprise EuroSAT [16] and RESISC45 [7]. focusing satellite imagery for land cover classification and environmental monitoring. Medical Domains consist of CropDiseases [31] (Disease) and ChestX [43], specialized for identifying plant discases and diagnosing medical conditions through radiographic images, respectively. Then, randomly shuffle these datasets to construct three highly challenging MDCL scenarios (CDM, ETI and TRC which are derived from the initial letters of the first three domains). Detailed experimental configurations are provided in Appendix-C from S.M.

---

Table 3: Comparison of Baselines in terms of parameter and computational efficiency (in CDM).
Methods
Train Params|
Iter/st
GPU Avgl
GPU Max
CPU Avgl
CPU Max.
DRR++RRe
42.84M
1.04
12919.37MB
12919.37MB
9756.08MB
16805.70MB
CLS-ER
42.84M
2.13
6199.39MB
6199.39MB
9658.21MB
16700.56MB
RanPAC
1.19M
4.27
3065.27MB
3087.97MB
9823.56MB
17465.18MB 18MB
MoE
4.03M
1.77
11098. 88MB
11098. 88MB
14203.6 6M203.61MB
17262.77MB
L2P
0.20M
5.08
3420.06MB
3420.06MB
9654.75MB
16798.61MB
DAP
0.51M
2.76
3686.95MB
3686.95MB
9911.54MB
04958.32MB
D-Prompt
0.41M
3.11
6456.64MB
3556. 64MB
9866.40MB
17040.50MB
C-Prompt
3.99M
4.91
4775.65MB
4775.65MB
9869.61MB
16941. 20MB
Ours
42.54M
3.06
2926.20MB
2926.20MB
9525.63MB
16681.54MB
Table 4: Impact of individual and combined components model performance in ETI.
Methods
EuroSAT
Tlmg ImgR
CUB200
C100 MNIST RESISC45 ChestX
C10 Disease
Avg
CB
19.59
8.37
11.81
12.29
36.92
49.88
61.14
30.04
92.79
99.00
42.18
SBE
85.38
70.49
55.42
73.45
77.05
90.05
85.93
37.73
93.65
99.13
76.83
CBE
92.38
76.49
62.42
78.45
82.05
95.05
90.93
38.73
94.17
99.04
80.97
CBE+MI
95.08
80.93
68.19
82.49
84.68
97.15
91.29
39.56
96.18
98.28
83.38
CBE+KL
92.18
78.02 66.13
78.97
82.50
92.24
90.54
44.58
95.81
99.13
82.01
CBE+HSIC
93.65
76.97 64.37
78.56
82.35
91.88
91.29
45.53
95.49
99.06
81.92
CBE+MI+KI.
95.35
81.61
68.30
83.56
85.54
97.43
92.28
46.44
96.22
99.01
84.57
CBE+MI+HSIC
95.87
80.85 68.74
83.15
85.52
98.03
92.08
42.43
96.33
98.91
84.19
CBE+KL+HSIC
93.24
78.22
66.45
80.49
82.55
92.21
91.39
45.26
95.59
99.08
82.45
LEAR
95.89
81.25
69.57
84.12
85.30
98.56
92.92
45.45
96.60
99.30
84.90
LEAR w/o ESM
3.72
1.25
2.36
83.95
4.91
14.63
1.05
17.85
99.15
23.71
4.2 Experimental Results
Results in Multi-Domain Continual Learning. Our comprehensive evaluation compares LEAR against state-of ohette-rrt approaches across three distinct domain sequences (Tables and 2}. As shown in Table T. LEAR achieves an outstanding average accuracy 84.61% % the CDM scenario. Specifically, LEAR outperforms the replay- based DER++ (Refresh) by 55.39%, highlighting its superior ability to mitigate catastrophic forgetting without requiring memory buffers. When compared to expansion- based methods, LEAR shows substantial advantages over both RanPAC (71.01%) and MoE-adapters (57.33%) particularly in challenging domains like TinyImageNet and ChestX, while maintaining consistent performance across all evaluated domains.
The reshuffled domain sequence in Table 2 further validates LEAR's adaptability, where achieves an even higher average accuracy of 84.90% the ETI scenario. outperforms dual-branch method CLSS ER (47.68%) by 37.22%, with especially large gap on ImageNet-R. Moreover, LEAR demonstrates over 55% higher average accuracy than the domain-incremental method DAP (28.48%) and other listed prompt-based approaches. These results highlight LEAR's 's ability to effectively learn and retain knowledge regardless of the domain order.
Moreover, as shown in Figure 2 (a), the proposed LEAR achieves the lowest forgetting rate in CDM scenario compared to alternative methods. As the number of tasks increases, LEAR consistently maintains stable and superior performance across domains with various fields and different complex- ities, effectively addressing the catastrophic forgetting prevalent in existing approaches. Detailed results for the TRC scenario are provided in the Appendix-D from SM.
4.3 Ablation Studies
The impact of components in LEAR. Table 4 provides empirical validation for the theoretical contributions each proposed module. Where "CB" denotes using only the collaborative backbone with single shared expert network across all data domains; "CBE" extends CB with task-specific expert network expansion and ESM expert selection; "SBE" denotes the configuration where "CBE"'s dual backbone architecture is replaced with single backbone; and "CBE-MIIKLL/SSI" represents CBE augmented with individual components or combination components. "LEAR w/o ESM" denotes randomly selecting experts during the beginning and testing phase of each task. This table

---

1101
(a) The forgetting curve of CDM.
(b) Visualization of distributions.
(c) Clustering results of features.
Figure 2: (a): The comparison of the forgetting curves between LEAR and other baseline methods after learning the CDM sequence. (b): Visualization of memory distributions from the ETI sequence after PCA dimensionality reduction, showing mean positions and covariance ellipsoids. (c): Feature visualization of global backbone and local backbone with HSICBCO regularization.
demonstrates that the collaborative backbone design ("SBE" and the dynamically expanded expert network "CB"->"CBE" significantly enhance model performance in ETI. In addition, each regularization term and combinations contribute to varying degrees of performance improvement over "CBE" on these two sequences. Furthermore, the model's performance will significantly drop when inappropriate experts are chosen ("L.EAR"->"LA
w/o ESM"), thereby demonstrating the necessity of ESM. These evaluation results align well with our methodological design.
The analysis of parameter and computational efficiency. As shown in Table LEAR achieves the lowest GPU and CPU utilization among all baseline methods. While baselines including prompt-based approaches (e.g., DualPrompt) and adapter- based variants (e.g.. Addaterss indeed contain fewer trainable parameters (0.5%-5% per ViT block), they inevitably require complete backpropagation through all VIT blocks, necessitating the storage of intermediate activations throughout the entire backhone due to chain rule dependencies, which maintain extensive computation graphs. Conversely, LEAR's innovative design strategically fine-tunes only the last three VIT layers and terminates backpropagation after the third-last layer, thereby significantly reducing the computation graph and GPU usage.
Visualization of the Expert Selection Mechanism. Figure 2b shows the first five ESM-preserved distributions from the ETI scenario, visualized in 3D space after PCA reduction. ESM computes Mahalanobis distances between these distributions and incoming task samples to select experts for either network expansion or test evaluation.
Visualization of the HSICBCO approach, Both the global and local backbones are initialized with identical pretrained weights. Under the guidance of MIBPA and KLDBFA, they learn task-general and task-specific representations, respectively. However, their feature representations still exhibit strong correlations. As illustrated in Figure the proposed HSICBCO module effectively decouples these representations, demonstrating its capability to promote distinct feature learning. Additional ablation results are provided in the Appendix-D from SM.
5 Conclusion and Limitation
In this paper, we propose LEAR, novel framework for Multi-Domain Continual Learning that simultaneously addresses stability plasticity and efficiency. Specifically, built on collaborative backbone structure, we introduce MIBPA and KLDBFA to maintain historical prediction consistency and task-specific feature alignment during model updates, while HSICBCO ensures disentangled and complementary representations. Additionally, ESM dynamically selects relevant experts for efficient network expansion and task-agnostic prediction. The empirical results demonstrate the effectiveness of the proposed approach. The primary limitation of this paper is that the proposed approach would contain considerable number of parameters. To address this issue, we will propose a novel expert merging technology with self-distillation for effective model compression.
10

---

Acknowledgements
This work was supported by the Sichuan Provincial Natural Science Foundation Project (No.20252NSPSC05100)) and National Natural Science Foundation of China (Grant No: 62506067)
References
[1] A. Achille, T. Eccles, L. Matthey, C. Burgess, N. Watters, A. Lerchner, and I. Higgins. Life-long disentangled representation learning with cross-domain latent homologies. In Advances in Neural Information Processing Systems (NeurIPS), pages 9873-9883, 2018.
[2] Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty continual learning with adaptive regularization. In Advances in Neural Information Processing Systems, pages 4394-4404, 2019.
[3] Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition, pages 11254-11263 2019.
[4] Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
1 11254-13 11263, 2019.
[5] Jihwan Bang. Hyunseo Koh, Seulki Park, Hwanjun Song, Jung-Wor Woo Ha, and Jonghyun Choi. Online continual learning on a contaminated data stream with blurry task boundaries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9275-9284, June 2022.
[6] Hyuntak Cha, Jacho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9516-9525, 2021.
[7] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the Proceedings of the IEEE, 105(10):1855-188, 2017.
[8] C. Cortes, X. Gonzalvo, Kuznetsov, M. Mohri, and Yang. Adanet: Adaptive structural learning of artificial neural networks. In Proc. of Int. Conf. Machine Learning (ICML), vol. PMLR 70, pages 874 883, 2017.
[9] Danruo Deng, Guangyong Chen, Jianye Hao, Qiong Wang. and Pheng-Ann Heng, Flattening sharpness for dynamic gradient projection memory benefits continual learning. Advances in Neural Information Processing Systems, 14:18710-18721 2021.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint vrXiv:2010.11923 2020.
[11] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Trans- formers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9285-9295, 2022.
[12] A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf. Measuring statistical dependence with Hilbert-Schmidt norms. In Proc. Int. Conf. on Algorithmic Learning Theory, vol. Lecture Notes Artif. Intell. (LNAI) 3734, pages 63-77, 2005.
[13] Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kemel method for the two-sample-prooles Advances neural information processing systems, 19, 2006.
[14] Yiduo Guo, Bing Liu, and Dongyan Zhao. Online continual learning through mutual information maximization. International Conference on Machine Learning, pages 8109-8126. PMLR, 2022.
11

---

[15] K. He. X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[16] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-222)) 2019.
[17] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021.
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[19] G. E. Hinton, S. Osindero, and Y. W. Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18(7):157-1155,, 2006.
[20] Saurav Jha, Dong Gong, He Zhao, and Lina Yao. Npel: Neural processes for uncertainty-ant continual learning. Advances in Neural Information Processing Systems, 36, 2024.
[21] Dahuin Jung, Dongyoon Han, Jihwan Bang. and Hwanjun Song. Generating instance-level prompts for rehearsal-free continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11847-11857, 2023.
[22] Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Husegawa-Johnson, Sung Ju Hwang, and Chang D Yoo. Forget-frre continual learning with winning subnetworks. In International Conference on Machine Learning, pages 10734-10750. PMLR, 2022.
[23] Junsu Kim, Hoseong Cho, Jihyeon Kim, Yihalem Yimolal Tiruneh, and Seungryul Baek. Sddgr: Stable diffusion-based deep generative replay for class incremental object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 28772-28781, 2024.
[24] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Gtrbska-BBrwinnse D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. Proc. of the National Academy of Sciences (PNAS), 114(13)3551-3522 2017.
[25] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Univ. of Toronto, 2009.
[26] Ya Le and Xuan Yang. Tiny imageNet visual recognition challenge. Technical report, Univ. of Stanford, 2015.
[27] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proc. of the IEEE, 86(1 01-22278-2224 1998.
[28] Z. Li and D. Hoiem. Learning without forgetting, IEEE Trans. on Pattern Analysis and Machine Intelligence, 40 40(12):2935-2947, 2017.
[29] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aireraft. arXiv preprint arXiv: 106.55,, 2013.
[30] Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 36, 2024.
[31] Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers in plant science, 7:215232, 2016.
[32] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
12

---

[33] J. Ramapuram, M. Gregorova, and A. Kalousis. Lifelong generative modeling. In Proc. Int. Conf. on Learning Representations (ICLR), arXiv preprint arXiv:1705.09847. 2017.
[34] -ylves Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph Lampert. iCaRL: Incremental classifier and representation learning. In Proc. of the IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), pages 2001-2010, 2017.
[35] H. Shin, K. Lee, Kim, and J. Kim. Continual learning with deep generative replay. In Advances in Neural Inf. Proc. Systems (NIPS), pages 2990-2999999 2017.
[36] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Bescante--onilla Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11909-11919, 2023.
[37] Gido M van de Ven and Andreas S Tolias. Three continual learning scenarios. In NeurIPS Continual Learning Workshop, volume 1, 2018.
[38] Vinay Kumar Verma, Kevin Liang, Nikhil Mehta, Piyush Rai, and Lawrence Carin. Efficient feature transformations for discriminative and generative continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13865-13875, 2021.
[39] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- uesd birds-200-2011 dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011.
[40] Liyuan Wang, Jingyi Xie, Xingxing Zhang. Mingyi Huang, Hang Su, and Jun Zhu. Hierarchi- cal decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 36:69054 69076, 2023.
[41] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, 46(8):5362-5383, 2024.
[42] T. Wang and W. Li. Kernl learning and optimization with Hilheer--chmidt independence criterion. Int. Jour of Machine Learning and Cyber,, 1700 2018.
[43] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks weakly- supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097-2106, 2017.
[44] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained trans- formers: An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5665
2022.
[45] Zhenyi Wang, Yan Li, Li Shen, and Heng Huang. A unified and general framework for continual learning. arXiv preprint 2rXiv:2403.13222 2024.
[46] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su. Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal- continual learning. In European conference on computer vision, pages 631-648. Springer, 2022.
[47] Zifeng Wang. Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 1399-19,99 2022.
[48] Mengqi Xue, Haofei Zhang, Jie Song, and Mingli Song. Meta-attention for vit-backed continual learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 150-159, 2022.
13

---

[49] Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang. Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 23219-23230, 2024.
[50] M. Zhai, L. Chen, Tung, J He, M. Nawhal, and G. Mori. Lifelong GAN: Continual learning for conditional image generation. In Proc. of the IEEE/CVF Int. Conf. Computer Vision (ICCV), pages 2759-2768, 2019.
[51] Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation continual learning of vision-language models. In Proceedings of the IEEF/CVF international conference on computer vision, pages 19125-19136, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss the limitations of our paper in the conclusion and Appendix-E from SM.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiaiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the algorithm implementation in Section 3.7 and Appendix-B from SM, along with the source code. The detailed experimental setup is documented in Appendix-C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the algorithm implementation in Section 3.7 and Appendix-B from SM, along with the source code. The detailed experimental setup is documented in Appendix-C.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
haboliic/uiiees/dddelbbbimssssss Psllcll for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/mipss.cc/pulli/guidde
wwtsssssssiiiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the algorithm implementation in Section 3.7 and Appendix-B from SM, along with the source code. The detailed experimental setup is documented in Appendix-C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Detailed experimental results are provided in Appendix-D from SM.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the detailed experimental setup (include compute resources) in Appendix-C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tt.p://ee..iss.ccocooo///
cc http://wwwpppp.cooooo..../ttttta
Answer: [Yes]
Justification: The research conducted in our paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the societal impacts of the work in Appendix-E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21


Reviews:
[{"review_id": "plHLsFlFSg", "reviewer": "Reviewer_a1tB", "summary": "The paper introduces LEAR, a buffer-free framework for multi-domain continual learning (MDCL). Specifically, to prevent forgetting while adapting to new tasks, LEAR (i) matches old vs new expert logits with a mutual-info loss (MIBPA) for the global side, (ii) constrains drift in the local feature space with a KL-divergence loss (KL-DBFA), and (iii) pushes the two branches to carry different feature representations with an HSIC independence loss. After each task, it stores only the mean vector and covariance matrix of that expert’s transformed features, computed in a fully frozen ViT feature space; when the next task arrives, it picks the closest old expert by Mahalanobis distance and clones its weights to start learning from a good initial point. Results show the proposed LEAR boosts average accuracy and decreases forgetting on three challenging domain-shuffled image benchmarks compared to prior works.", "questions": "1.My primary concern is that it remains unclear why the proposed method achieves such significant accuracy improvements over prior continual learning approaches.\n\n1.1 Although the authors provide an ablation study examining the three designed regularization terms, it is difficult to discern their individual contributions. In particular, the feature separation term (HSIC) does not appear to yield a meaningful accuracy gain.\n\n1.2 Rather than focusing solely on the regularization terms, it would be more informative to demonstrate the effectiveness of the collaborative backbone design, which is the key contribution of this work.\n\n1.3 It is not well explained why the proposed MIBPA approach mitigates catastrophic forgetting, nor how effectively it accomplishes this. \n\n2.In line 175, what does the \"concentration operation\" stand for? \n\n3.Which layers are frozen in the auxiliary model? Lines 185–186 state that the auxiliary model freezes the final three layers of the global backbone. However, it appears that in the rest of the backbone, all layers except the last three are frozen. Could you clarify this point? Additionally, why are the last three layers chosen specifically, rather than a different number of layers?\n\n4.What is the exact model architecture of the lightweight expert?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This work addresses a more practical and challenging scenario, multi-domain continual learning, as opposed to the class-incremental learning that most prior methods have focused on.\n\n2. The experimental results demonstrate significant accuracy improvements over existing methods.\n\nWeakness:\n1. The quality of the writing needs improvement. The paper makes it difficult to grasp the overall continual learning process, and Figure 1 is hard to understand until reaching the final subsection (Section 3.7) of the Methodology.\n2. While the proposed method achieves notable accuracy gains over previous approaches, the underlying reasons for its success remain unclear. Please refer to the detailed comments under “Questions” below.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns by 1) justifying the proposed method compared to prior CL methods, and 2) conducting ablation studies to show the effectiveness of each proposed component. However, I believe the paper still requires significant revisions to clarify the differences from prior works better and to demonstrate the effectiveness of each proposed component more clearly. In this case, I maintain my current score.", "rebuttal": "**We appreciate the valuable comments by Reviewer a1tB.**\n\n**Q1: Concerns regarding writing quality.**\n\n**A1:** We have enhanced the paper's readability and improved Fig.1 to more clearly depict the main process of LEAR.\n\n**Q2: Reason for significant accuracy improvements.**\n\n**A2:** The proposed LEAR have two primary advantages over the existing continual learning methods: \n\n(1) By emulating the brain's memory structure through CLS theory, the proposed LEAR manages a collaborative backbone architecture to provide more adaptive and semantically rich representation information when compared to the existing continual learning methods, including L2P, DAP, DualPrompt, MoEAdapter that usually employ a single fixed backbone network.\n\n(2) Compared to the existing continual learning methods, the proposed LEAR integrates three mechanisms, including the Mutual Information-Based Prediction Alignment (MIBPA), Kullback–Leibler (KL) Divergence-Based Feature Alignment (KLDBFA) and HSIC-Based Collaborative Optimization (HSICBCO), aiming to maintain a good plasticity-stability trade-off and reduce information redundancy. Such a design can deal with a long sequence of complex tasks and achieve excellent performance. We evaluate the performance of the proposed approach by introducing a more challenging Multi-Domain Continual Learning (MDCL) setting that incorporates not only highly difficult datasets (such as ImageNet-R, TinyImageNet, and ChestX) but also features scenarios with both similar feature distributions between datasets (e.g., EuroSAT and RESISC45) and severe domain shifts (e.g., CIFAR100→CropDisease), as well as fine-grained datasets (CUB200) with extended task sequences. Consequently, as evidenced in the forgetting curves in Fig. 2 of Appendix-D.1 from the Supplementary Materials (SM), many baseline methods exhibit significant performance degradation in later stages of the task sequence, demonstrating their inability to learn and capture complex datasets.\n\n**Q3: Individual contributions of three regularization terms, particularly HSICBCO.**\n\n**A3:** To better demonstrate the individual contributions of each module in LEAR, we conducted additional ablation studies on two dataset sequences (ETI and TRC), as presented in the following tables:\n### Table 1: Results of ablation studies on ETI and TRC\n|Methods|EuroSAT|TImg|ImgR|CUB200|C100|MNIST|RESISC45|ChestX|C10|Disease|Avg|Var|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|CBE|92.38|76.49|62.42|78.45|82.05|95.05|90.93|38.73|94.17|99.04|80.97|308\n|CBE+MIBPA|95.08|80.93|68.19|82.49|84.68|97.15|91.29|39.56|96.18|98.28|83.38|293\n|CBE+KLDBFA|92.18|78.02|66.13|78.97|82.50|92.24|90.54|44.58|95.81|99.13|82.01|244\n|CBE+HSICBCO|93.65|76.97|64.37|78.56|82.35|91.88|91.29|45.53|95.49|99.06|81.92|247\n\n|Methods|TImg|RESISC45|CUB200|ChestX|ImgR|EuroSAT|MNIST|C10|Disease|C100|Avg|Var|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|CBE|79.01|89.72|80.68|30.26|64.06|92.34|94.15|94.44|97.05|86.19|80.79|370\n|CBE+MIBPA|81.24|90.76|82.51|40.13|67.98|95.19|97.68|95.64|98.03|85.77|83.49|288\n|CBE+KLDBFA|78.77|90.36|80.22|40.21|66.52|94.67|97.01|95.27|98.40|87.02|82.85|291\n|CBE+HSICBCO|79.14|90.21|80.54|42.12|66.21|93.26|94.24|94.48|97.25|85.41|82.29|259\n\nWhere \"CBE\" denotes the configuration using solely the collaborative backbone with task-specific network expansion and ESM expert selection, and \"CBE+MIBPA/KLDBFA/HSICBCO\" represents CBE augmented with individual components. The results demonstrate that each regularization term contributes to varying degrees of performance improvement over CBE across both ETI and TRC. \n\nSpecifically, the contribution of HSICBCO can be further reflected through the variation of accuracy.  As shown in the \"Var\" columns of Table 1, \"CBE+HSICBCO\" achieves significantly reduced accuracy variance compared to \"CBE\" while maintaining improved average accuracy (Avg). This demonstrates that HSICBCO narrows the performance gaps across datasets, and enhances the performance on challenging datasets (e.g., ChestX, ImgR), which collectively validates HSICBCO's contribution to LEAR.\n\nIn addition, we present a performance comparison of the model after learning the first three domains of these two sequences:\n### Table 2: Performance after learning the 3th dataset of ETI and TRC\n|Methods|EuroSAT|TImg|ImgR|Avg|\n|-|-|-|-|-|\n|CBE|93.18|80.79|70.75|81.57|\n|CBE+HSICBCO|94.08|81.21|71.14|82.14|\n\n|Methods|TImg|RESISC45|CUB200|Avg|\n|-|-|-|-|-|\n|CBE|81.57|91.23|84.38|85.73|\n|CBE+HSICBCO|82.45|91.53|84.84|86.27|\n\nThis table shows that HSICBCO effectively mitigates severe domain shifts (e.g., RESISC->CUB). This is because HSICBCO effectively encourages the global and local backbones to capture distinct knowledge, as further illustrated in Figure 3 of Appendix D.4 from SM, where it balances the model's generalization capability across dissimilar domains with its plasticity toward the current task. \n\n**Q4:  The effectiveness of the collaborative backbone.**\n\n**A4:** To validate the necessity of the collaborative backbone for adapting MDCL, we conducted a comparative analysis between \"SBE\" (Single Backbone with task-adaptive network expansion) and \"CBE\":\n\n|Methods|EuroSAT|TImg|ImgR|CUB200|C100|MNIST|RESISC45|ChestX|C10|Disease|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|SBE|85.38|70.49|55.42|73.45|77.05|90.05|85.93|37.73|93.65|99.13|76.83\n|CBE|92.38|76.49|62.42|78.45|82.05|95.05|90.93|38.73|94.17|99.04|80.97\n\n|Methods|TImg|RESISC45|CUB200|ChestX|ImgR|EuroSAT|MNIST|C10|Disease|C100|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|SBE|72.58|81.94|70.12|32.93|57.85|91.19|89.80|89.66|93.96|84.59|76.46\n|CBE|79.01|89.72|80.68|30.26|64.06|92.34|94.15|94.44|97.05|86.19|80.79\n\nAs evidenced by the experimental results, the collaborative design yields significantly better performance than the single-backbone. Numerous current dynamic expansion frameworks utilize a single pre-trained ViT to enhance continual learning performance. Nevertheless, this approach yields only a uniform feature representation for each novel task, thereby constraining the model’s capacity to accommodate extended sequences of heterogeneous tasks. To overcome the inherent limitations of a single pre-trained ViT, we introduce a collaborative backbone architecture comprising both local and global backbones, designed to extract task-invariant and task-adaptive representations, respectively. To further refine this architecture, we propose a novel Hilbert-Schmidt Independence Criterion-Based Collaborative Optimization (HSICBCO) method, which incentivizes the local and global backbones to learn complementary semantic features. This strategy is particularly effective for handling complex and diverse datasets. In the revised version, we have provided a more comprehensive rationale and detailed exposition of the collaborative backbone architecture.\n\n**Q5: Why and how MIBPA mitigates catastrophic forgetting.**\n\n**A5:** The central concept of the proposed MIBPA is to preserve the established predictive patterns of each historical expert when optimizing the parameters of the global backbone. This is accomplished by generating both the prior and current prediction distributions for each expert, utilizing the previously updated and currently optimized backbones, respectively. MIBPA incorporates a mutual information-based regularization loss to align these historical and current prediction distributions, thereby mitigating catastrophic forgetting within the network.\n\n**Q7: Typo: concentration operation.**\n\n**A7:** That is a typo and we have corrected it to ''concatenation''. \n\n**Q8: The reason for freezing three Layers and auxiliary models.**\n\n**A8:** The auxiliary model is designed to retain the parameter knowledge acquired by the global backbone following each task transition. Consequently, all layers of the auxiliary model are frozen, and its fixed representations are leveraged to guide the optimization behaviour of the global backbone. To minimize parameter overhead, the auxiliary model shares the majority of its parameters with the global backbone across all but the final three layers. We have enhanced the clarity and coherence of this explanation in the revised version.\n\nThe rationale for selecting the final three layers lies in the fact that high-level representation layers capture semantically enriched features, which are advantageous for a variety of downstream applications [1]. [2] also find that deeper layers are disproportionately the source of forgetting. Empirical evidence presented below further substantiates that utilizing the last three trainable layers of the global backbone yields robust performance with limited parameter growth compared to other choices (FTX denotes fine-tune X layers of backbones).\n\n|Methods|TImg|RESISC45|CUB200|ChestX|ImgR|EuroSAT|MNIST|C10|Disease|C100|Avg|TrainParms\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|FT1|80.42|90.02|82.68|37.14|65.76|95.21|97.17|95.62|98.54|84.17|82.67|14.26M\n|FT2|81.06|90.54|82.53|38.92|67.63|95.37|98.17|95.94|98.90|85.41|83.45|28.43M\n|FT3|80.30|92.41|83.49|44.67|69.28|96.62|98.53|95.84|99.16|86.22|84.65|42.54M\n|FT4|81.65|91.76|82.87|44.89|69.94|96.18|97.63|96.43|99.14|86.80|84.73|56.80M\n\n|Methods|EuroSAT|TImg|ImgR|CUB200|C100|MNIST|RESISC45|ChestX|C10|Disease|Avg|TrainParms\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|FT1|94.52|79.83|65.42|80.92|82.89|97.29|90.51|40.90|95.37|98.29|82.60|14.26M\n|FT2|94.90|80.94|67.05|83.21|84.67|98.16|90.63|44.31|95.78|98.82|83.85|28.43M\n|FT3|95.89|81.25|69.57|84.12|85.30|98.56|92.92|45.45|96.60|99.30|84.90|42.54M\n|FT4|96.01|81.96|69.44|83.80|86.05|98.49|92.98|47.86|96.34|99.22|85.22|56.80M\n\n**Q9: Arch of expert.**\n\n**A9:** Each expert is composed of two distinct modules: the first is a fully connected layer comprising 500 hidden units while the second is a linear classifier realized through a fully connected layer with 1,268 hidden units.\n\n## References:\n\n[1] Action bank: A high-level representation of activity in video. ICCV 2022.\n\n[2] Anatomy of catastrophic forgetting: Hidden representations and task semantics. arXiv 2020."}, {"review_id": "X7iCL3uhYN", "reviewer": "Reviewer_WD1L", "summary": "The paper addresses the limitations of existing class-incremental learning (CIL) methods, which often operate within a single domain and fail to reflect real-world scenarios. To overcome this, the authors introduce a new Multi-Domain Continual Learning (MDCL) setting, where each task corresponds to a distinct dataset, introducing both class and domain shift. To tackle MDCL, they propose a collaborative architecture consisting of a global backbone and a local backbone to capture general and task-specific information. To enhance performance and reduce forgetting, the approach integrates several components such as Mutual Information-Based Prediction Alignment (MIBPA), Kullback–Leibler Divergence-Based Feature Alignment (KLDBFA), Hilbert-Schmidt Independence Criterion-Based Collaborative Optimization (HSICCO), and Expert Selection Mechanism (ESM).", "questions": "Questions:\n1. Could you clarify what is stored for each task after training, specifically regarding the parts of the local backbone that need to be stored?\n2. What would be the total number of parameters of this pipeline, not just the trainable parameters, but all the parameters that need to be stored, including those from the experts, task-specific covariances, etc.?\n3. Could you please provide the performance of the collaborative architecture with the global and local backbones and without any other components?\n4. Could you also provide the performance on standard single-domain CIL benchmarks for comparison, such as CIFAR-100, CUB, or ImageNet-R with 10-task splits?", "limitations": "Yes, but a more thorough analysis of the number of parameters could have been provided.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The MDCL setting that the paper presents more accurately represents the real-world CL scenario and can introduce a more challenging benchmark for the community to pursue, given many CIL methods already achieve commendable results on a single domain.\n2. The experiments show that many current methods fail to handle MDCL, yet the proposed method shows great performance, surpassing state-of-the-art baselines.\n\nWeaknesses:\n1. The paper is hard to follow due to the overwhelming introduction of multiple components without sufficient context. It would be clearer if the authors provided more in-depth discussion before each component explaining the motivation. For example, for mitigating forgetting, the authors say they introduce KLDBFA—it would have been better to briefly reference options like regularization and distillation. By making this connection to previous work, they could make it easier to understand and then mention why they decided to go with KL. The same approach could have been applied to each component.\n2.  Before the MDCL setting, it would be better to also compare the proposed approach on the standard single-domain CIL setting. The proposed method should also work on a single domain, so having that comparison helps establish the method’s performance, as all literature reports results in that scenario. It would be easier to compare, and it also strengthens the work by showing it can handle a single domain. You could then move to MDCL to show that current methods fail on that setting.\n3. The ablation studies on different model components should be more thorough. There is w/o MI, KL, and HSIC, which show method performance without each of these components. However, ESM is not discussed (there is a visualization in Table 2b, but accuracy without ESM is not given). Additionally, HSIC in Table 2 shows no meaningful improvement—the model is only 0.11% worse when removing it, which is not statistically significant, so given its complexity, its inclusion may not be justified. The results reported in Table 2 are much better than any other baselines, but it is unclear where these improvements originate from. To provide more insight, the authors should first present results for the collaborative architecture with only the global and local backbones, then progressively add a component or a combination of components to demonstrate their contributions and possible interactions. Having the joint fine-tuning results, where the model is fine-tuned in a non-continual setting on all the data, would also serve as a valuable upper bound for comparison.\n\nI would reconsider my evaluation if my concerns regarding the experiments are addressed. I would also like to review the comments from other reviewers.", "was_revised": true, "final_justification": "I sincerely thank the authors for their rebuttal. After carefully considering the other reviews and the rebuttal, I believe this is a good paper that outperforms many strong baselines. However, the presentation of the method was not very clear, making it difficult to fully understand how the approach works or achieves these results. The additional ablation study provided in the rebuttal partially addressed my concerns, and I have decided to raise my score. I still encourage the authors to further improve the writing and presentation of the method in order to maximize the impact of this work.", "rebuttal": "We sincerely appreciate the reviewer's constructive comments and recognition of the (1) the practical significance of our proposed Multi-Domain Continual Learning (MDCL) compared to conventional CIL learning settings, and (2) the demonstrated superior performance of LEAR over prior methods in this challenging scenario.\n\nFor weaknesses and questions, we provide our feedback as follows.\n\n**Q1: The discussion of motivation before introduction of components.**\n\n**A1:** We sincerely appreciate your invaluable suggestions and guidance. Due to space constraints in the main text, it's challenging to provide a detailed elaboration on the motivation of each module. We have thoroughly revised the introduction of the KLDBFA module to provide clearer motivation and methodological justification. The revised content is presented as follows:\n\nThe iterative updating of the pre-trained backbones facilitates the temporal capture of local representations, thereby potentially enhancing the acquisition of novel tasks. However, this process risks inducing adverse knowledge transfer and performance degradation across historical experts. Regularization methods like EWC [1] and MAS [2], which typically impose constraints on parameter updates, are not desirable to capture the complex distributional shifts across domains, while knowledge distillation methods like LWF [3] and iCaRL [4] require maintaining additional teacher networks that become computationally prohibitive as the number of tasks grows. To address these limitations, we propose Kullback-Leibler Divergence-Based Feature Alignment (KLDBFA), designed to preserve crucial historical parameters during the optimization of the local backbone.\n\nOur design rationale for selecting KL divergence stems from two key considerations: Firstly, modern generative evaluation metrics (e.g., FID, Kernel MMD) operate on the Gaussian distribution assumption in high-dimensional feature spaces. This motivates us to model backbone features as Gaussian distributions. Fig. 1 in Appendix-C from the SM also provides additional empirical validation for the Gaussian distribution assumption. Secondly, KL divergence offers unique advantages over alternative distributional metrics: (1) Its directional property enables targeted constraint of current features toward historical distributions, unlike symmetric metrics (e.g., Jensen-Shannon divergence); (2) It maintains computational efficiency compared to expensive metrics like MMD or Wasserstein distance. These characteristics make KL divergence ideally suited for continual learning scenarios requiring efficient knowledge preservation.\n\n**Q2: Performance comparison on CIL settings.**\n\n**A2:** We sincerely agree with your comment. As MDCL is indeed significantly more challenging than standard CIL, comparing with existing methods under the CIL setting would further demonstrate and strengthen the superiority of LEAR. Therefore, we have further supplemented the comparison of LEAR with relevant methods on CIL performance over CIFAR-100, CUB-200, ImageNet-R, and TinyImageNet, each partitioned into 10 tasks:\n### Table 1: Results of CIL\n|Method|C100|CUB|ImgR|TinyImg|\n|-|-|-|-|-|\n|RanPAC|92.23|**90.32**|78.11|72.89|\n|MoE|85.21|82.26|76.77|80.23|\n|L2P|82.76|79.23|73.73|76.37|\n|DualPrompt|84.12|83.21|**78.47**|81.38|\n|CODAPrompt|86.33|83.36|74.45|82.80|\n|LEAR|**95.80**|88.38|77.67|**85.86**|\n\nThe results demonstrate LEAR's excellent adaptability to CIL settings, where it achieves competitive performance. We have incorporated this comparative analysis, along with detailed discussions, prior to presenting the MDCL results in the revised paper.\n\n**Q3: Suggestions of ablation studies and performance upper bound.**\n\n**A3:** We note that all four reviewers raise concerns in the individual contributions of LEAR's components and have provided consistent suggestions for improving our ablation studies. Recognizing the importance of this issue, we have conducted extensive additional experiments on ETI and TRC sequences during the rebuttal phase and reorganized the ablation study tables accordingly:\n### Table 2: Results of ablation studies\n|Methods|EuroSAT|TImg|ImgR|CUB200|C100|MNIST|RESISC45|ChestX|C10|Disease|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|UpperBound|96.71|82.77|72.81|85.08|86.39|98.97|93.72|50.14|96.49|99.35|86.24|\n|CB|19.59|8.37|11.81|12.29|36.92|49.88|61.14|30.04|92.79|99.00|42.18|\n|CBE|92.38|76.49|62.42|78.45|82.05|95.05|90.93|38.73|94.17|99.04|80.97|\n|CBE+MI|95.08|80.93|68.19|82.49|84.68|97.15|91.29|39.56|96.18|98.28|83.38|\n|CBE+KL|92.18|78.02|66.13|78.97|82.50|92.24|90.54|44.58|95.81|99.13|82.01|\n|CBE+HSIC|93.65|76.97|64.37|78.56|82.35|91.88|91.29|45.53|95.49|99.06|81.92|\n|CBE+MI+KL|95.35|81.61|68.30|83.56|85.54|97.43|92.28|46.44|96.22|99.01|84.57|\n|CBE+MI+HSIC|95.87|80.85|68.74|83.15|85.52|98.03|92.08|42.43|96.33|98.91|84.19|\n|CBE+KL+HSIC|93.24|78.22|66.45|80.49|82.55|92.21|91.39|45.26|95.59|99.08|82.45|\n|LEAR|95.89|81.25|69.57|84.12|85.30|98.56|92.92|45.45|96.60|99.30|84.90|\n|LEAR w/o ESM|3.72|1.25|2.36|83.95|8.24|4.91|14.63|1.05|17.85|99.15|23.71|\n\n\n|Methods|TImg|RESISC45|CUB200|ChestX|ImgR|EuroSAT|MNIST|C10|Disease|C100|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|CB|8.43|22.67|39.45|23.51|25.67|42.74|60.95|64.50|93.45|84.87|46.62|\n|CBE|79.01|89.72|80.68|30.26|64.06|92.34|94.15|94.44|97.05|86.19|80.79|\n|CBE+MI|81.24|90.76|82.51|40.13|67.98|95.19|97.68|95.64|98.03|85.77|83.49|\n|CBE+KL|78.77|90.36|80.22|40.21|66.52|94.67|97.01|95.27|98.40|87.02|82.85|\n|CBE+HSIC|79.14|90.21|80.54|42.12|66.21|93.26|94.24|94.48|97.25|85.41|82.29|\n|CBE+MI+KL|81.24|90.39|82.60|44.11|68.58|96.01|98.37|96.22|98.86|86.17|84.25|\n|CBE+MI+HSIC|82.21|90.43|83.05|44.74|68.41|95.18|95.86|96.09|98.91|85.85|84.07|\n|CBE+KL+HSIC|79.67|89.88|81.44|42.47|66.27|95.07|96.23|95.54|98.91|86.74|83.22|\n|LEAR|80.30|92.41|83.49|44.67|69.28|96.62|98.53|95.84|99.16|86.22|84.65|\n|LEAR w/o ESM|4.27|4.83|68.49|9.42|3.76|14.59|1.98|6.33|1.64|9.25|12.46|\n\nWhere ''CB'' denotes using only the collaborative backbone with a single shared expert network across all datasets; ''CBE'' extends CB with task-specific expert network expansion and ESM expert selection; and \"CBE+MI/KL/HSIC\" represents CBE augmented with individual components or a combination of components. ''UpperBound'' denotes the results in a non-continual setting.\n\nThis table demonstrates that the dynamically expanded expert network ('CB'->'CBE') significantly enhances model performance on both ETI and TRC. Furthermore, each regularization component and their respective combinations yield distinct enhancements in performance relative to CBE across both sequences.\n\n\nFurthermore, the ablation study for ESM was initially omitted because ESM serves as the core component of LEAR. Without ESM, LEAR would lose its capability to dynamically select appropriate experts for domain-specific evaluation under task-agnostic conditions. To address this omission, we further conduct experiments with randomly selected experts during the beginning and testing phase of each task, which is denoted as ''LEAR w/o ESM''. The results demonstrate that the model's performance will significantly drop when inappropriate experts are chosen, thereby demonstrating the necessity of our ESM.\n\n**Q4: Storage at the end of the task.**\n\n**A4:** Following the network's input-output pipeline, we preserve the following components per task: (1) The last three layers of both global and local backbones, which are saved and frozen as Frozen-Global and Frozen-Local backbones (overwrite their corresponding components saved from the previous task); (2) Task-specific mean vectors and covariance matrices which computed from the features extracted by the Static backbone for a random subset of training samples.\n\n**Q5: Total parameter count.**\n\n**A5:** The total parameter count reaches approximately 241.7M, with the following detailed composition: (1) Two complete ViT backbones (Global and Local) comprise 172M (86M\\*2) parameters; (2) Three backbone variants (Frozen-Global, Frozen-Local, and Static) collectively contribute 63.6M (21.2M\\*3) parameters; (3) 10 task-specific expert networks account for about 4.8M (0.38M\\*10 for 10 fc layers + 1M for all classifiers) parameters; and (4) Mean vectors and covariance matrices for 10 tasks sum to 1.3M (0.13M\\* 10 for 10 mu \\& sigma) parameters.\n\n**Q6: Performance of solely collaborative architecture.**\n\n**A6:** Please refer to the results of ''CBE'' in Table 2. However, if you are referring to the results after excluding the task-specific expert network expansion, please refer to those associated with ''CB''.\n\n**Q7: Performance on CIL setting.**\n\n**A7:** Please refer to ''**A2**''.\n\n### References:\n\n[1] Overcoming catastrophic forgetting in neural networks//PNAS17\n\n[2] Task-free continual learning//CVPR19\n\n[3] Learning without forgetting//TPAMI17\n\n[4] iCaRL: Incremental Classifier and Representation Learning//CVPR17"}, {"review_id": "pBsiGsX0yT", "reviewer": "Reviewer_NZTA", "summary": "This paper proposes LEAR to address multi-domain continual learning that jointly addresses the stability–plasticity trade-off. The approach incorporates a collaborative backbone along with three key components: MIBPA and KLDBFA for maintaining historical knowledge consistency and task-specific feature alignment, and HSICBCO for learning disentangled representations. An expert selection module (ESM) dynamically activates relevant experts to enable scalable network expansion and efficient inference. Experimental results validate the effectiveness of the proposed method.", "questions": "Please refer to the weaknesses.", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work in Section 5.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n+ The paper addresses a new and relatively unexplored problem setup.\n+ The proposed method appears to outperform prior methods significantly under the new setting.\n+ The paper is well-structured and includes a detailed empirical study.\n\n### Weaknesses:\n+ Despite the claim of proposing several new techniques, many of the components seem to be straightforward applications of existing techniques in continual learning. For instance:\n   + The use of a dual-branch architecture is a well-established strategy in the continual learning literature.\n   + The proposed LEAR module, which learns task-specific experts or adaptors, closely resembles techniques used to build expandable representations across tasks.\n   + The idea behind MIBPA (maintaining consistency with historical predictions) is a well-known strategy for mitigating forgetting in continual learning. In this work, the authors adopt mutual information as the consistency metric instead of the more common cross-entropy. However, it's unclear whether this choice offers any real advantage. Since mutual information is symmetric and generally less sensitive to low-confidence outputs than cross-entropy, it's not obvious why it would be preferable in this context. A theoretical or empirical comparison between the two would help clarify whether this design decision is justified.\n+ Based on the discussion above, I encourage the authors to clearly delineate which components are truly novel and which are adaptations of known techniques.\n\n+ Equation (7) treats the outputs of the two branches as random variables, but the formulation lacks detail. What assumptions are made about their distributions? Is there a specific probabilistic model behind this formulation?\n+ The task definition needs further clarification. Does each task correspond to a specific domain or dataset? Or, is the number of classes per task fixed? If the former, I would also be interested in how the method performs under the latter setup, which arguably reflects a more practical scenario.\n+ The ablation in Table 1 could be more informative. Instead of removing components in parallel, it would be better to add or remove them step by step. This would make it clearer how each module contributes to the final performance. The current setup doesn’t fully show the relative importance of each part.\n+ Although the authors append code in the supplementary materials, the provided code cannot run in its current version , which undermines reproducibility.", "was_revised": true, "final_justification": "I will maintain my original score. I believe that this work requires substantial revisions before it is ready for publication. In particular, the paper should place more emphasis on the truly novel components and reduce the focus on the more straightforward applications of existing techniques.", "rebuttal": "We appreciate the valuable comments by Reviewer NZTA.\n\n**Q1: The use of dual-branch architecture in CL.**\n\n**A1:** We clarify that our method differs fundamentally from dual-branch approaches in both architecture and learning dynamics. Specifically, dual-branch methods like DualNet and CLS-ER rely on buffer to preserve historical knowledge, while LEAR is Rehearsal-Free. Additionally, these methods fully fine-tune the entire pretrained models, whereas LEAR only fine-tunes the last three layers of the backbone. Furthermore, existing approaches usually lack or even omit the feature interaction between the two branches, while LEAR employs MIBPA, KLDBFA, and HSICBCO to establish robust global-local knowledge consolidation and adaptation. \n\n**Q2: The differences from the existing dynamic expandable models.**\n\n**A2:** Conventional approaches typically utilize a single pre-trained ViT as the foundational backbone to facilitate adapter integration. In contrast, the proposed LEAR framework optimizes a collaborative backbone architecture, incorporating both global and local backbones to deliver task-agnostic and adaptive feature representations. Leveraging this collaborative structure, LEAR executes expert construction via a comprehensive feature integration process. This architecture effectively accommodates heterogeneous data domains, representing a novel direction within the continual learning landscape. Furthermore, while baseline methods rely on routers or dynamic weighting schemes for expert aggregation, LEAR introduces an innovative Expert Selection Mechanism (ESM) that autonomously identifies the most suitable expert and leverages its parameter information for constructing new experts. This mechanism enhances the learning of novel tasks and constitutes an unexplored strategy in current continual learning research. To further demonstrate LEAR's effectiveness compared to other expansion-based approaches in MDCL,We have supplemented the experimental comparison between LEAR and three baseline methods: EASE[1], SEMA[2], and the LoRA-based approach InfLoRA [3], on both ETI and TRC sequences. Experimental results in Table 1 confirm LEAR's significant performance advantage over these baselines. \n\n### Table1: Supplementary performance comparison between relevant methods and LEAR\n|Methods|EuroSAT|TImg|ImgR|CUB200|C100|MNIST|RESISC45|ChestX|C10|Disease|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|EASE|4.36|35.32|22.78|60.98|40.56|58.24|34.72|10.42|94.38|90.84|45.26|\n|InfLoRA|76.79|66.25|33.74|71.45|77.23|92.15|77.56|32.35|87.70|93.66|70.89|\n|SEMA|10.47|67.76|47.05|70.35|55.94|69.87|89.03|25.62|89.23|96.96|62.23|\n|SLCA|85.78|0.37|0.52|0.79|1.55|10.34|2.15|9.59|10.17|91.35|21.26|\n|LEAR|95.89|81.25|69.57|84.12|85.30|98.56|92.92|45.45|96.60|99.30|84.90|\n\n|Methods|TImg|RESISC45|CUB200|ChestX|ImgR|EuroSAT|MNIST|C10|Disease|C100|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|EASE|15.22|48.51|50.98|5.17|32.78|54.72|81.14|67.46|79.84|81.84|51.77|\n|InfLoRA|32.63|68.98|75.75|21.38|51.74|73.69|86.02|86.17|83.45|82.49|66.23|\n|SEMA|56.24|11.43|46.35|8.45|55.93|67.65|80.41|82.23|91.11|80.36|58.02|\n|SLCA|63.67|0.78|0.67|14.84|0.62|11.31|11.20|13.70|8.21|51.18|17.62|\n|LEAR|80.30|92.41|83.49|44.67|69.28|96.62|98.53|95.84|99.16|86.22|84.65|\n\n**Q3: The novelty of MIBPA and the motivation of MI rather than CE.**\n\n**A3:** Ensuring consistency with historical predictions has been extensively adopted in regularization-based approaches. Typically, these methods formulate the current active model (classifier) as a student module and duplicate it as a teacher module following each task transition. Catastrophic forgetting is mitigated by minimizing the divergence between the outputs of the student and teacher modules. While this framework offers robust stability, it often compromises adaptability when confronted with complex datasets. In contrast to conventional regularization-based techniques, the proposed MIBPA introduces two key innovations:\n\n(1) Rather than duplicating the entire classifier as a teacher module after each task switch, MIBPA incorporates a parameter-sharing auxiliary model that retains select parameters from the global backbone (specifically, the final three layers) and leverages this auxiliary model to guide the optimization of the global backbone. This strategy effectively reduces the overall parameter count; \n\n(2) Whereas existing methods enforce output alignment between the current classifier and the teacher module during training, MIBPA focuses on preserving the prediction consistency of each historical expert during global backbone optimization, while simultaneously encouraging the current expert to assimilate data from the ongoing task (MI is symmetric). This approach utilizes supervised signals from all previously acquired experts to control the optimization process of the global backbone and is entirely different from the existing regularization-based methodologies. \n\nExceptionally, SLCA [4] models the class-wise distributions of historical tasks and aligning the classification layers in a post-hoc fashion. However, the experimental results in Table 1 demonstrate that this modeling and alignment approach struggles to adapt to the complex task prediction distributions in MDCL.\n\nAdditionally, we performed ablation studies substituting the mutual information component in MIBPA with cross-entropy (CE) loss. The experimental results ('CBE+CE' in Table 3) demonstrate a substantial performance degradation when employing asymmetric CE. This is because the model over-emphasizes the historical prediction distributions while neglecting to learn the current task. This empirical evidence motivated our choice of mutual information.\n\n**Q4: Truly novel components in LEAR.**\n\n**A4:** We claim that the expert construction process is a known technology that was adopted in this study. In contrast, the proposed collaborative backbone structure, Mutual Information-Based Prediction Alignment, and Kullback–Leibler (KL) Divergence-Based Feature Alignment are novel technologies, which are not explored in the existing continual learning studies.\n\n**Q5: The distribution modeling in Eq7.**\n\n**A5:** We clarify that no probabilistic models were employed in this process. We sincerely invite you to refer to our response to Q1 in the rebuttal to reviewer tTuR, as we are constrained by word limits and unable to answer this question.\n\n**Q6: The task definition.**\n\n**A6:** We clarify that in the MDCL setting, each individual task corresponds to a complete, undivided benchmark dataset. Regarding the second setting you mentioned, we designed three distinct data sequences, each comprising two datasets and further partitioned into 10 tasks per dataset (20 tasks total). Each task is trained only 1 epoch. The experimental results demonstrate that LEAR still shows robust adaptability to domain shift and outperforms all three compared methods in terms of average task performance.\n\n### Table2 \n|Method|CUB-ImgR||TinyImg-ImgR||ImgR-CUB||\n|-|-|-|-|-|-|-|\n||Avg|Last|Avg|Last|Avg|Last|\n|DER++(Re)|60.85|83.71|51.18|83.93|60.50|82.76|\n|MoEAdapter|12.61|23.53|25.60|17.57|43.83|75.85|\n|DualPrompt|52.34|67.44|60.67|42.17|57.08|79.98|\n|LEAR|78.74|75.34|72.36|69.47|77.82|81.39|\n\n**Q7: Improvement of ablation study.**\n\n**A7:** we have conducted extensive additional experiments on ETI and TRC sequences during the rebuttal phase and reorganized the ablation study tables accordingly:\n\n### Table3: Results of revised ablation studies\n|Methods|EuroSAT|TImg|ImgR|CUB200|C100|MNIST|RESISC45|ChestX|C10|Disease|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|CBE|92.38|76.49|62.42|78.45|82.05|95.05|90.93|38.73|94.17|99.04|80.97|\n|CBE+MI|95.08|80.93|68.19|82.49|84.68|97.15|91.29|39.56|96.18|98.28|83.38|\n|CBE+CE|90.25|36.96|21.21|10.69|29.69|48.42|51.80|42.83|75.59|70.50|47.79|\n|CBE+KL|92.18|78.02|66.13|78.97|82.50|92.24|90.54|44.58|95.81|99.13|82.01|\n|CBE+HSIC|93.65|76.97|64.37|78.56|82.35|91.88|91.29|45.53|95.49|99.06|81.92|\n|CBE+MI+KL|95.35|81.61|68.30|83.56|85.54|97.43|92.28|46.44|96.22|99.01|84.57|\n|CBE+MI+HSIC|95.87|80.85|68.74|83.15|85.52|98.03|92.08|42.43|96.33|98.91|84.19|\n|CBE+KL+HSIC|93.24|78.22|66.45|80.49|82.55|92.21|91.39|45.26|95.59|99.08|82.45|\n|LEAR|95.89|81.25|69.57|84.12|85.30|98.56|92.92|45.45|96.60|99.30|84.90|\n\n|Methods|TImg|RESISC45|CUB200|ChestX|ImgR|EuroSAT|MNIST|C10|Disease|C100|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|CBE|79.01|89.72|80.68|30.26|64.06|92.34|94.15|94.44|97.05|86.19|80.79|\n|CBE+MI|81.24|90.76|82.51|40.13|67.98|95.19|97.68|95.64|98.03|85.77|83.49|\n|CBE+CE|79.60|71.17|26.80|35.65|43.22|93.35|91.79|94.81|96.78|73.19|70.64|\n|CBE+KL|78.77|90.36|80.22|40.21|66.52|94.67|97.01|95.27|98.40|87.02|82.85|\n|CBE+HSIC|79.14|90.21|80.54|42.12|66.21|93.26|94.24|94.48|97.25|85.41|82.29|\n|CBE+MI+KL|81.24|90.39|82.60|44.11|68.58|96.01|98.37|96.22|98.86|86.17|84.25|\n|CBE+MI+HSIC|82.21|90.43|83.05|44.74|68.41|95.18|95.86|96.09|98.91|85.85|84.07|\n|CBE+KL+HSIC|79.67|89.88|81.44|42.47|66.27|95.07|96.23|95.54|98.91|86.74|83.22|\n|LEAR|80.30|92.41|83.49|44.67|69.28|96.62|98.53|95.84|99.16|86.22|84.65|\n\nWhere \"CBE\" denotes the configuration using solely the collaborative backbone with task-specific network expansion and ESM expert selection, and \"CBE+MIBPA/KLDBFA/HSICBCO\" represents CBE augmented with individual components or combination of components. The results demonstrate that each regularization term contributes to varying degrees of performance improvement over CBE across both ETI and TRC. \n\n**Q8: Code implement.**\n\n**A8:** Should our paper be accepted, we will definitely release the full codebase with comprehensive documentation.\n\n### References:\n\n[1]Expandable subspace ensemble for pre-trained model-based class-incremental learning//CVPR24\n\n[2]Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning//CVPR25\n\n[3]Inflora: Interference-free low-rank adaptation for continual learning//CVPR24\n\n[4]SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model//ICCV23"}, {"review_id": "qBmbrm78IC", "reviewer": "Reviewer_tTuR", "summary": "This work focuses on addressing catastrophic forgetting on evolving data domains of continual learning, which is multi-domain continual learning. This work introduces dynamic expansion approach comprising global and local backbones. The global backbone is trained using a mutual information metric. To mitigate forgetting, HSIC-based method is used. Also, this framework includes expert selection to identify the most relevant expert according to the input distribution.", "questions": "1. Which pre-trained ViT is used in the proposed framework, does the comparing methods using the same pre-trained model?\n\n2. In Table 3, the number of training parameters of the proposed framework is among the largest, but how does the GPU utilization is the least? The section that refers to Table 3 is only reiterating the superiority of the framework. I don’t see an explanation of this in the text.\n\n3. Also, how would it be that the proposed framework has the almost comparable number of training parameters with DER++ and CLS-ER? From my understanding, the proposed framework only trains the lightweight experts and the last layers of the backbones.\n\n4. On line 175, what is the “concentration operation”? Isn’t it “concatenation”? If it is concatenation, the symbol should be $\\bigoplus$.", "limitations": "See weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1.\tThe writing is clear. \n2.\tThe motivation of the global and local backbones seems acceptable.\n\nWeaknesses\n\n1.\tIn Section 3.4, the KL divergence is applied between two feature distributions. I don’t think it is a good idea to use KL divergence as a metric between two feature distributions, because the KL divergence is usually used on predictions (or logits, probabilities). Features are in high dimensions, I don’t know how you get the distributions of the features exactly (Actually I went through the code in the supplementary material, and did not find the specific implementation), but if you just take the softmax on the features to get the probabilities, most of the values would be close to zero. This provides little information. I think this section needs more elaboration.\n2.\tThe results in Table 1 shows that the framework achieve better performance than other methods by a large margin. But as I see the results on the last three rows, I wonder if there are results of another baseline method that does not use any of the three alignments. Because it seems the three alignments are not too significant in terms of achieving the competitive results. This leads to another concern that is the main contribution of the improvement comes from the extra pre-trained backbone, the larger pre-trained backbone or the alignments?", "was_revised": true, "final_justification": "The authors' rebuttal addresses most of my concerns. However, it seems irresponsible to omit a key part of the code implementation in the supplementary material when the code is provided, even though the authors have clarified the implementation in the rebuttal.\nI have raised my rating accordingly.", "rebuttal": "We sincerely appreciate your positive feedback regarding the clarity of our writing and the validity of our collaborative backbone design motivation.\n\nFor weaknesses and questions, we provide our feedback as follows.\n\n**Q1: Implement of KLDBFA.**\n\n**A1:** We clarify that we approximate the feature distributions from both the Local backbone and Frozen Local backbone as multivariate Gaussian distributions, for which we derive closed-form solutions of their KL divergence. Actually, many modern generative model evaluation metrics (FID, Kernel MMD) assume that real and generated data can be approximated by Gaussian distributions in a high-dimensional feature space. Therefore, we can model the feature distributions from both backbones for a data batch as Gaussian distributions.  The rationale for choosing KL divergence over alternative distributional metrics is because: (1) Unlike symmetric metrics such as Jensen-Shannon divergence, KL divergence's directional nature allows us to specifically constrain the current feature distribution to remain close to the historical distribution; (2) The computational overhead of distances like MMD and Wasserstein distance is relatively high.\n\nIn code implementation, we separately compute the mean vectors and covariance matrices of the feature vectors output by the Local Backbone and the Frozen Local Backbone, and then calculate the KL divergence between these two distributions $P({\\bf Z}^l)=\\mathcal{N}(\\boldsymbol{\\mu}_1, \\boldsymbol{\\Sigma}_1)$ and $P({{\\hat{\\bf Z}}}^l)=\\mathcal{N}(\\boldsymbol{\\mu}_2, \\boldsymbol{\\Sigma}_2)$ ($\\boldsymbol{\\mu}$ and $\\boldsymbol{\\Sigma}$ denotes mean vectors and covariance matrices, respectively) using the following equation:\n\n$$D_{\\mathrm{KL}}(P({\\bf Z}^l) \\,||\\,  P({\\hat{\\bf Z}}^l) ) = \\frac{1}{2} \\left[\\log \\left( \\frac{\\det \\boldsymbol{\\Sigma}_2}{\\det \\boldsymbol{\\Sigma}_1} \\right)- d + \\mathrm{tr}(\\boldsymbol{\\Sigma}_2^{-1} \\boldsymbol{\\Sigma}_1)+ (\\boldsymbol{\\mu}_2 - \\boldsymbol{\\mu}_1)^\\top \\boldsymbol{\\Sigma}_2^{-1} (\\boldsymbol{\\mu}_2 - \\boldsymbol{\\mu}_1)\\right]$$\n\nwhere 'det', 'd', and 'tr' denote the determinant, dimension, and trace of a matrix, respectively.\n\n**Q2: The source of LEAR performance improvement and collaborative backbone design independent performance.**\n\n**A2:**  We clarify that we did not employ larger pre-trained backbones compared to baseline methods to enhance performance. Specifically, the performance improvement stems from three synergistic components: (1) the collaborative global-local backbone structure, (2) the task-specific dynamically expandable expert networks, and (3) the proposed three regularizations terms.\n\nWe note that all four reviewers raise concerns in the individual contributions of LEAR's components and have provided consistent suggestions for improving our ablation studies. Recognizing the importance of this issue, we have conducted extensive additional experiments on ETI and TRC sequences during the rebuttal phase and reorganized the ablation study tables accordingly:\n\n### Table 1: Results of revised ablation studies\n|Methods|EuroSAT|TImg|ImgR|CUB200|C100|MNIST|RESISC45|ChestX|C10|Disease|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|CB|19.59|8.37|11.81|12.29|36.92|49.88|61.14|30.04|92.79|99.00|42.18|\n|SBE|85.38|70.49|55.42|73.45|77.05|90.05|85.93|37.73|93.65|99.13|76.83\n|CBE|92.38|76.49|62.42|78.45|82.05|95.05|90.93|38.73|94.17|99.04|80.97|\n|CBE+MI|95.08|80.93|68.19|82.49|84.68|97.15|91.29|39.56|96.18|98.28|83.38|\n|CBE+KL|92.18|78.02|66.13|78.97|82.50|92.24|90.54|44.58|95.81|99.13|82.01|\n|CBE+HSIC|93.65|76.97|64.37|78.56|82.35|91.88|91.29|45.53|95.49|99.06|81.92|\n|CBE+MI+KL|95.35|81.61|68.30|83.56|85.54|97.43|92.28|46.44|96.22|99.01|84.57|\n|CBE+MI+HSIC|95.87|80.85|68.74|83.15|85.52|98.03|92.08|42.43|96.33|98.91|84.19|\n|CBE+KL+HSIC|93.24|78.22|66.45|80.49|82.55|92.21|91.39|45.26|95.59|99.08|82.45|\n|LEAR|95.89|81.25|69.57|84.12|85.30|98.56|92.92|45.45|96.60|99.30|84.90|\n\n\n|Methods|TImg|RESISC45|CUB200|ChestX|ImgR|EuroSAT|MNIST|C10|Disease|C100|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|CB|8.43|22.67|39.45|23.51|25.67|42.74|60.95|64.50|93.45|84.87|46.62|\n|SBE|72.58|81.94|70.12|32.93|57.85|91.19|89.80|89.66|93.96|84.59|76.46\n|CBE|79.01|89.72|80.68|30.26|64.06|92.34|94.15|94.44|97.05|86.19|80.79|\n|CBE+MI|81.24|90.76|82.51|40.13|67.98|95.19|97.68|95.64|98.03|85.77|83.49|\n|CBE+KL|78.77|90.36|80.22|40.21|66.52|94.67|97.01|95.27|98.40|87.02|82.85|\n|CBE+HSIC|79.14|90.21|80.54|42.12|66.21|93.26|94.24|94.48|97.25|85.41|82.29|\n|CBE+MI+KL|81.24|90.39|82.60|44.11|68.58|96.01|98.37|96.22|98.86|86.17|84.25|\n|CBE+MI+HSIC|82.21|90.43|83.05|44.74|68.41|95.18|95.86|96.09|98.91|85.85|84.07|\n|CBE+KL+HSIC|79.67|89.88|81.44|42.47|66.27|95.07|96.23|95.54|98.91|86.74|83.22|\n|LEAR|80.30|92.41|83.49|44.67|69.28|96.62|98.53|95.84|99.16|86.22|84.65|\n\nWhere ''CB'' denotes using only the collaborative backbone with a single shared expert network across all datasets; ''CBE'' extends CB with task-specific expert network expansion and ESM expert selection; ''SBE'' denotes the configuration where ''CBE'''s dual backbone architecture is replaced with a single backbone; and \"CBE+MI/KL/HSIC\" represents CBE augmented with individual components or combination of components.\n\nThis table demonstrates that the collaborative backbone design (''SBE''->''CBE'') and the dynamically expanded expert network (''CB''->''CBE'') significantly enhances model performance on both ETI and TRC. In addition, each regularization term and their combinations contribute to varying degrees of performance improvement over ''CBE'' on these two sequences. \n\n**Q3: The use of pre-trained ViT.**\n\n**A3:** We initialize both the global backbone and the local backbone using a ViT-B/16 model pre-trained on ImageNet-21K. All comparison methods used the same pre-trained model, except for the MoEAdapter which adopted the CLIP ViT-B/16.\n\n**Q4: The relation between the number training parameters and the GPU usage.**\n\n**A4:** We would like to clarify that GPU memory consumption in our experiment setting is not solely determined by the number of trainable parameters; it can stem from multiple sources, including the size of the pre-trained model used, additional components such as adapters or prompts introduced for continual learning, and the use of memory buffers. For instance, although MoE Adapter has a relatively small number of trainable parameters, it employs the CLIP model as its backbone. Furthermore, the majority of LEAR's trainable parameters originate from fine-tuned layers in the ViT backbone, resulting in substantial GPU usage overlap with the original frozen ViT. Additionally, gradient information is not preserved during training. Therefore, the low GPU memory usage of LEAR is both reasonable and expected.\n\n**Q5: Why LEAR has the almost comparable number of training parameters with DER++ and CLS-ER.**\n\n**A5:** We clarify that both DER++(Rresh) [1] and CLS-ER [2] were originally implemented using a single ResNet-18 backbone without frozen parameters. To ensure fair comparison when migrating these baselines to ViT, we adopt the consistent backbone configurations between LEAR (keep last three layers of both backbones trainable) and other baselines (DER++(Rresh) and CLS-ER). This results in comparable scales of trainable parameters across all three methods.\n\nIn practice, we have also conducted comparative experiments by activating the last three layers of ViT backbones for other baselines, yielding the following results:\n\n|Methods|EuroSAT|TImg|ImgR|CUB200|C100|MNIST|RESISC45|ChestX|C10|Disease|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|RanPAC+FT3|97.83|44.49|21.44|16.76|22.05|87.94|78.94|40.06|74.41|95.84|57.97|\n|L2P+FT3|11.15|2.06|1.50|1.90|14.97|21.12|31.94|11.72|89.01|98.32|28.37|\n|DAP+FT3|14.86|0.83|2.02|4.66|18.48|30.66|27.34|15.91|90.01|98.91|30.37|\n|DualP+FT3|12.69|0.69|0.30|0.69|0.93|14.53|3.09|15.84|25.55|96.72|17.10|\n|CODAP+FT3|14.16|0.51|1.02|0.64|2.30|13.02|8.80|9.52|39.79|95.61|18.54|\n|LEAR+FT3|95.89|81.25|69.57|84.12|85.30|98.56|92.92|45.45|96.60|99.30|84.90|\n\nwhere ''FTX'' denotes that we activate the final ''X'' layers of the ViT backbone. These results indicate that merely scaling up model parameters cannot guarantee performance gains in MDCL scenario, which necessitates the incorporation of regularization constraints during backbone adaptation to mitigate adverse parameter optimization caused by domain shift.\n\nIn addition, we further considered how to reduce the number of training parameters. We tried fine-tuning only the MHSA and MLP sublayers in the last three layers of the VIT backbone:\n\n|Methods|TImg|RESISC45|CUB200|ChestX|ImgR|EuroSAT|MNIST|C10|Disease|C100|Avg|TrainParms\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|FT1|80.42|90.02|82.68|37.14|65.76|95.21|97.17|95.62|98.54|84.17|82.67|14.26M\n|FT2|81.06|90.54|82.53|38.92|67.63|95.37|98.17|95.94|98.90|85.41|83.45|28.43M\n|FT3|80.30|92.41|83.49|44.67|69.28|96.62|98.53|95.84|99.16|86.22|84.65|42.54M\n|FT3MHSA|81.04|92.36|83.22|44.18|69.03|96.95|98.45|96.13|99.18|85.70|84.62|14.27M\n|FT3MLP|80.75|90.94|82.68|42.40|68.97|96.32|97.53|95.88|98.91|85.65|84.00|28.43M\n|FT4|81.65|91.76|82.87|44.89|69.94|96.18|97.63|96.43|99.14|86.80|84.73|56.80M\n\nThese results demonstrate that fine-tuning only the MHSA achieves comparable or even better performance than fine-tuning the entire block. This move reduced the total number of parameters to 14.27M, a significant reduction of 28.27M compared to the original.\n\n**Q6: Typo: concentration operation.**\n\n**A6:** Thank you for highlighting this oversight. It is indeed a concatenation operation, we have corrected the word and symbol in the revised paper.\n\n### References:\n\n[1]A unified and general framework for continual learning//ICLR24\n\n[2]Learning fast, learning slow: A general continual learning method based on complementary learning system//ICLR22"}]

Author rebuttal:

Meta-review: This paper proposes LEAR, a framework for continual learning that adaptively expands representations while reusing and consolidating prior knowledge through module selection and regularization. Reviewers found the method well-motivated and appreciated its ability to balance plasticity and stability via dynamic expansion and adaptive reuse, supported by extensive experiments on multiple benchmarks. Concerns focused on the clarity of the expansion mechanism, the computational overhead of maintaining multiple modules, and limited theoretical grounding for the reuse strategy, but the rebuttal provided additional ablations, efficiency analyses, and improved explanations that alleviated these issues. Some reviewers noted that comparisons to certain recent baselines could be expanded, but overall empirical results were strong and robust. The consensus is that the paper presents a novel and practical contribution to continual learning with solid technical grounding and convincing evaluation. I urge the authors to include the extended discussion with the reviewers (as much as the space permits) in the camera-ready version of the paper.
Decision: Accept (poster)