Learning Multi-Source and Robust Representations for Continual Learning
Authors: Fei Ye
Yongcheng Zhong
Qihe Liu
Adrian G. Bors
Jingling sun
Rongyao Hu
shijie zhou
Venue: NeurIPS 2025 poster ()
Keywords: Continual Learning
Pre-trained Models
Dynamic Expansion Models
Primary area: machine_learning_for_sciences

Abstract: Plasticity and stability denote the ability to assimilate new tasks while preserving previously acquired knowledge, representing two important concepts in continual learning. Recent research addresses stability by leveraging pre-trained models to provide informative representations, yet the efficacy of these methods is highly reliant on the choice of the pre-trained backbone, which may not yield optimal plasticity. This paper addresses this limitation by introducing a streamlined and potent framework that orchestrates multiple different pre-trained backbones to derive semantically rich multi-source representations. We propose an innovative Multi-Scale Interaction and Dynamic Fusion (MSIDF) technique to process and selectively capture the most relevant parts of multi-source features through a series of learnable attention modules, thereby helping to learn better decision boundaries to boost performance. Furthermore, we introduce a novel Multi-Level Representation Optimization (MLRO) strategy to adaptively refine the representation networks, offering adaptive representations that enhance plasticity. To mitigate over-regularization issues, we propose a novel Adaptive Regularization Optimization (ARO) method to manage and optimize a switch vector that selectively governs the updating process of each representation layer, which promotes the new task learning. The proposed MLRO and ARO approaches are collectively optimized within a unified optimization framework to achieve an optimal trade-off between plasticity and stability. Our extensive experimental evaluations reveal that the proposed framework attains state-of-the-art performance. The source code of our algorithm is available at https://github.com/CL-Coder236/LMSRR.

Learning Multi-Source and Robust Representations for Continual Learning
Fei Ye', YongCheng Zhong', Qihe Liu ; Adrian G. Bors?, JingLing Sun', Rong Hu', ShiJie Zhou'
Schooo of Information and Software Engineering,
University of Electronic Science and Technology of China
Department of Computer Science, University of York [feiye@uestee edu.en, 202422090410@stth uest..eett.eduuco gtheliu@uester edu. adrian.bors @ york.ac.uk, jlsun @ uestc.ede uestc.edu.cn. ryhu
sjzhou uestc.edu.en)
Abstract
Plasticity and stability denote the ability to assimilate new tasks while preserving previously acquired knowledge, representing two important concepts in continual learning. Recent research addresses stability by leveraging pre-trained models provide informative representations, yet the efficacy of these methods is highly reliant on the choice of the pre-trained backbone, which may not yield optimal plasticity. This paper addresses this limitation by introducing streamlined and potent framework that orchestrates multiple different pre-trained backbones derive semantically rich multi-source representations. We propose an innovative Multi-Scale Interaction and Dynamic Fusion (MSIDF) technique to process and selectively capture the most relevant parts of multi-source features through a series learnable attention modules, thereby helping to learn better decision boundaries to boost performance. Furthermore, we introduce novel Multi-Level Repre- sentation Optimization (MLRO) strategy to adaptively refine the representation networks, offering adaptive representations that enhance plasticity. To mitigate over-regularization issues, we propose a novel Adaptive Regularization Optimiza- tion (ARO) method manage and optimize switch vector that selectively governs the updating process each representation layer, which promotes the new task learning. The proposed MLRO and ARO approaches within a unified optimization framework to achieve an optimal trade-off between plasticity and stability. Our extensive experimental evaluations reveal that the proposed framework attains state-of-the-an performance. The source code of our http://itntbnoooo//-/ooo////////
collectively optimized
algorithm available
Introduction
To thrive in natural environments, advanced intelligent entities must possess a robust ability to assimilate new information while retaining previously acquired critical knowledge [[7]. This ability, known continual learning (CL), is also pivotal in artificial intelligence systems, facilitating the deployment of numerous real-time applications such as autonomous driving and robotic navigation. Despite the impressive performance of contemporary deep learning models static datasets (21). they experience substantial performance degradation continual learning scenarios due to catastrophic forgetting 44 This phenomenon occurs when the neural network overwrites its parameters to accommodate new task learning, leading to network forgetting.
*correpponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent research has expanded beyond the issue of catastrophic forgetting to introduce two pivotal concepts in evaluating a model's efficacy in continual learning plasticity. which refers to the model's capacity to assimilate new tasks, and stability, which denotes its ability to retain previously acquired knowledge [28]. Most existing studies mainly focus on enhancing stability by developing several methods, which can be divided into three primary categories: Rehearsal-basee techniques [00 4 which utilize and refine a memory system to retain select historical examples; dynamic expansion-based methods [3 (4), which focus on dynamically constructing and integrating new sub- networks within a cohesive framework to accommodate new information; and regularization-bar bbadddd strategies [30 42], which seek fine-tune and adjust the model's 's parameters by imposing penalties on substantial alterations to critical parameters. Among these strategies, leveraging a memory system is an effective means of maintaining stability, though its efficacy diminishes significantly when the memory buffer size is constrained [60]. Conversely, dynamic expansion methods are suitable for handling extended task sequences, maintaining robust performance on historical tasks by freezing all previously trained network parameters [61]. Nonetheless, freezing the majority of the model's parameters can prevent the new task learning and thus adversely affect plasticity.
To balance stability and plasticity in continual learning. recent studies have explored pre-trained models by either extracting robust features or dynamically constructing new sub-networks based on these foundational architectures [40 [5 43). Nonetheless, the effectiveness of these approaches largely relies the selection of the pre-trained backbone, which would fail to achieve optimal plasticity, particularly when confronted with novel data domains. In this study, we tackle this challenge by introducing innovative framework named Learning Multi-Source and Robust Representations (LMSRR). This framework orchestrates several different pre-trained Vision Transformer (ViT) backhones as representation networks, delivering robust feature information to enhance performance. Specifically, we propose a novel Multi-Scale Interaction and Dynamic Fusion (MSIDF) method to proficiently amalgamate multi-source features from diverse representation networks into an augmented representation. This method captures the most important parts of the representation in response incoming samples through several learnable attention modules, thereby enhancing plasticity. Furthermore, the proposed MSIDF approach incorporates an adaptive weighting mechanism to autonomously determine the significance of each attention module, facilitating the interaction among multi-scale features and aiding in uncovering the intricate underlying structure of the data, which further improves the model's performance.
On the other hand, numerous existing studies usually freeze the representation network to ensure stability, which inadvertently diminishes the model's capacity to learn new tasks due to the limited number of activation parameters. In this paper, we address this challenge by introducing an innovative Multi-Level Representation Optimization (MLRO) strategy. This approach incorporates penalty term in the primary objective function, which minimizes the divergence between all previously acquired and currently activated representations, thereby maintaining stability during the new task learning. Furthermore, we propose novel Adaptive Regularization Optimization (ARO) strategy, designed to selectively penalize parameter changes within each representation layer. Specifically, the proposed ARO approach introduces a learnable switch vector, which is dynamically optimized and continuously generates differentiable variables to selectively regulate the optimization process of each representation layer during training. Such an approach effectively relieves reu-leggulaiization issues while preserving robust plasticity. Unlike prior multi-model fusion approaches such as CoFiMA 44 and Model Soup (58), which either average independently trained models or expand architectures with task-specific modules, our LMSRR framework dynamically aggregates multiple pre-trained backbones through unified feature-space fusion mechanism. This design enables LMSRR to adapt efficiently across tasks in continual learning scenarios without introducing additional task-specific parameters.
We conducted an extensive suite of experiments in continual learning, and the empirical findings reveal that the proposed approach attains state-of-te oftt--f-the-att performance. The principal contributions of this research are delineated as follows:
We propose novel LMSRR framework to explore multi-source representations from several different pre-trained VIT backbones to boost the model's performance in continual learning.
We propose a novel MSIDF approach to effectively integrate multi-source features into a compact and semantically rich representation, which can maintain good plasticity.
2

---

We propose a novel MLRO approach to automatically regulate the optimization process of each representation layer, which can maintain stability during the new task learning.
We propose a novel ARO approach to optimize a learnable switch vector that selectively penalizes the change in the parameters of each representation network, which can avoid over-regularization issues.
2
Related Work
Rehearsal-based techniques represent widely adopted strategy for mitigating forgetting by dynam- ically incorporating a limited number of historical examples into the memory buffer [5] 9]. These memory samples are leveraged alongside new training instances enhance model performance during the new task learning. Thus, the quality of the memorized samples is paramount within the rehearsal-baat -based optimization framework 20 Moreover, rehearsal based approaches can be augmented through the integration of regularization techniques, with the objective of further elevating the overall efficacy of the model [2 14, 26]. In addition, memory studies have proposed train the generative models to implement the memory system, which can provide infinite generative replay
samples [
52 64 31).
Knowledge distillation (KD) techniques were initially developed for model compression. The funda- mental concept of the KD framework involves establishing teacher-student architecture, wherein a loss function is employed to align the predictions of the teacher and student models. This process aims facilitate the transfer of knowledge from the complex teacher model to the simpler student model KD has found extensive applications in deep learning, yielding substantial results. Given its
[18 233 advantageous properties and performance, KD has also been utilized to mitigate network forgetting in continual learning scenarios. The primary objective of integrating KD within continual learning to minimize the divergence between the predictions of the student and teacher models during task learning, outlined in Learning Without Forgetting (LWF) (37] Moreover, rehearsal- approaches can be synergistically combined with KD to form a unified learning framework, which has demonstrated enhanced model performance, as illustrated in [48). Additionally, the self-KD
approach has been proposed to maintain previously acquired representations, thereby alleviating network forgetting, discussed in [9].
Dynamic network architectures represent a robust approach to mitigating network forgetting in con- tinual learning [3]. Such approaches dynamically expand the network capacity to enhance the learning ability for new tasks [29] 53). Beyond convolutional neural networks, dynamic expan- sion techniques have also been explored to leverage the capabilities of Vision Transformers (ViT) as the foundational backbone. These methods typically create self-attention blocks combined Additionally, another investigation 46 proposes a dual learning framework that integrates a VIT with a multimodal large language model, introducing a Miscs-Fisher Outlier Detection and Interaction (VFF-ODI) strategy to enhance inter-model communication. However, these methodologies often involve freezing large portions of the pre-trained backbone, which limits adaptability to complex and unseen domains. Moreover, recent architecture- basee methods such as RPSNet (5) alleviate forgetting by selecting task-specific subnetworks within shared backbone, enabling partial parameter reuse across tasks. In contrast, LMSRR maintains a fixed architecture and performs semantic-level fusion across multiple pretrained backbones, achieving task- -annssgc adaptability without subnetwork selection.
15
with task-specific classifiers to adapt new tasks [6 59
43
3 Methodology
3.1 Problem Statement
continual learning (CL), models face the limitation of being unable to access the entire training dataset. The training for each task is restricted to data samples pertinent to the current task, and data from previous tasks is inaccessible. prominent scenario in this domain is Incremeanlllttll Learning (TIL), where the training dataset D {(x,. v;) [j = 1..... ... No} is divided into multiple task- specific subsets (Di,- ... D ). each corresponding to an individual task T,. During the learning of specific task J. the model is confined to data samples from the relevant training subset Dj. while
In
all prior subsets (Di,... ... D } remain inaccessible. In each task, the model learns to discriminate
3

---

Frozce
Learnable
Hadamard product
Malkiply
Cambel-Soffmar
Pre-trained Vit
Stack
ANO Oltpeciive Funeliun
Task
(o) Faig q
Prozen Layer
Learnable Layer
2010
Tisk
Adagtiie paramoter
Pesteres the Finsl P. L'aayee
Fasura of the Finel Layer
Feature
Class tolks
--
Trieae
Attentios modale
Differentiablc variables
Representation
Adaptine Regularization Optimization
The overall framework of the LMSRR. During training, only the last T'' layers of each ViT backbone are trainable, with the rest frozen. Data samples are processed by these ViT backbones to extract feature outputs, which are subsequently stacked. The stacked features are integrated through the proposed MSIDF module before being passed to fully connected classifier for final prediction. In addition, the proposed MLRO approach optimizes the representation networks by penalizing shifts in the parameters, which can ensure the preservation of all previously learned information. Furthermore, we introduce a novel ARO approach adaptively regulate the optimization process of the representation networks, which can relieve wer-regularizztion issues.
Figure 1:
among classes within that task, and the task identifier is provided during both training and evaluation, allowing the model to use tasc-spec output heads or parameters when necessary.
The goal of a model in continual learning is to progressively optimize the parameters as new task data introduced, minimizing the overall training loss across all tasks. Specifically, the model aims to find the optimal set of parameters O from the parameter space Ô, such that the loss function is minimized over all training samples from each task. This problem can be formalized as the following optimization problem:
1 I N Σ N [C fo(x)))
0- argmin o-
(I)
where 0" represents the optimal model parameters, and L(., -) is the loss function, which is commonly implemented as the cross cross-entrope entropy loss to measure the discrepancy between model predictions and true labels. The function fo(:): X' Y represents the classifier with parameter set 6, which maps input samples X₂ E X' to their predicted labels ye E y, where and y denote the data and class label space, respectively. N is the total number of samples in the training subset Dj. Due to the inaccessibility of historical examples in continual learning, many studies have implemented the goal of Eq. (1) by proposing employ memory system to preserve historical examples.
After completing the learning of all tasks (T..... TN J. the model's performance is evaluated using all test datasets {D1.-.. D'x }. This evaluation not only considers the model's performance on the current task but also examines its performance on previous tasks, providing a comprehensive assessment of the model's ability to adapt to a continuously changing data distribution.
3.2 Multi-Source Representation Network
Acquiring robust and semantically enriched representations can markedly enhance model performance across diverse applications
[6]. Numerous studies have leveraged tre-trained neural networks to deliver potent and resilient representations, with the objective of augmenting performance in continual learning [45 (55. Nonetheless, these approaches need to carefully select an appropriate pre-trained backbone, which may not achieve optimal plasticity when confronted with novel data domains. In this study, we propose an innovative framework to manage and optimize several different pre-trained Vision Transformers (ViTs) foundational representation networks, thereby providing robust and

---

semantically enriched representations for continual learning. Let fos: X' + Z denote the i-th pre-trained ViT backbone, which processes the image & input and outputs a feature vector 2 E 2, where = 1.... T and 7 signifies the total number of ViT backbones. Here, E E R and X' E Rd represent the feature and data spaces, respectively, with d, and dz as their respective dimensions.
Integrating the output features from various representation networks, each containing distinct intrinsic properties, can yield a rich diversity of representational information. A straightforward and effective method involves consolidating multi-source features into unified representation for a specific data point x,, as described by:
z' = (xx))) & for (Xx).
where @ signifies the fusion of several feature vectors into an expanded dimensional space. Utilizing the enhanced representation zzz, we can dynamically create new expert to learn a decision boundary for specific task, aiming implement the prediction process. Specifically, the expert is implemented using a linear classifier fa: 2a + V. which receives an augmented representation and returns a prediction, expressed as
Y (o ..... Sor (x.)).
where y' [vi.s F. denotes the predicted probabilities, with C signifying the total number of categories. za E Rd denotes the di=-diinnso feature space associated with the augmented representation 2/, while y E Rd represents the dy dy-dimensional prediction space. Unlike model- averaging or ensemble-based approaches that combine multiple independently trained models, our framework performs feature-space fusion of several pre-trained VIT backbones within unified continual learning setup, maintaining a fixed inference path without parameter growth.
3.3 Multi-Scale Interaction and Dynamic Fusion
The augmented representations formulated in Eq. assume an equal contribution from each representation network towards the learning of new task. However, this approach does not fully exploit the representational capacity. Moreover, simply combining these multi-source features can cause redundancy the representational information, resulting in performance degradation. this research, we tackle these issues by introducing innovative MSIDF mechanism that autonomously filters out redundant information while preserving essential feature components. Specifically, for given input x,, the proposed MSIDF mechanism initially constructs augmented representation by
a fex (x,)
(xx)
where * signifies the operation that stacks multiple vectors {for for(x,,) into a matrix &e xd= Subsequently, the proposed MSIDF framework introduces a set of adaptive attention modules [Ai..... A. J. where each attention module Aj is characterized by learnable matrix W R with window size ky. designed to discern the most pertinent feature components. The process of using a specific attention module (the j-th module) to the representation matrix Z, is articulated as follows:
F2 i) W 2. : kyl.
(5)
where o denotes the Hadamard product and 2g [:]l 8 ky] denotes a matrix starting from the row i and ending at the row i + kiz By using Eq.
5 we can form processed representation by
z F. 0) F FF, kj + 1),
(6)
where Z denotes a representation refined through the j-th attention module. For attention modules with varying window sizes, we utilize symmetric padding techniques ensure that the dimensions of the representations processed by each attention module are consistent with those of other attention modules. Furthermore. facilitate the cooperative optimization of these attention modules. the proposed MSIDF mechanism introduces trainable adaptive parameter P5 to ascertain the significance of each A, during the training phase. To prevent numerical overflow, we normalize each trainable adaptive parameter P5 by:
1' = exp(p))/ Σ exp(pc)
By using the adaptive weights, all processed representations {Z!... ... Zmn are integral by
Z. Σ,- T j=1 {pizz)
(8)
5

---

where 2 denotes the ultimate augmented representation. which is fed into a linear classifier for prediction. contrast to Eq. (2). Eq. (8) can provide more concise and informative representation. maintaining constant feature dimension even the number of representation networks increases.
3.4 Multi-Level Representation Optimization
Refining the parameters of representation networks can facilitate the acquisition of new tasks, thereby enhancing their plasticity. Nevertheless, optimizing the entire parameter set of the model computationally intensive due to the substantial number of hidden layers and nodes within each representation network. Recent research has shown that high-level representations from large- scale pre-trained neural networks provide semantically rich information, which enhances model performance in downstream tasks [38 62). Based on these empirical insights, propose optimizing only the last L' layers to mitigate computational demands. To ensure stability in continual learning, introduce innovative MLRO method, which regulates the representation updating behaviour during the optimization process. Specifically, let f/r denote a representation network trained on the preceding task (1; and kept static during the learning of new task (T,), while for is the active representation network during the new task learning (T,), where j = -1, .T Each representation network Jes consists of I,' trainable feature layers, represented (fo 4² 1 See }, where each
fog: -0-1 2e processes the representation over the feature space go-1 extracted by Seg and outputs the representation over the feature space 2 A representation extracted by specific feature layer of a representation network is articulated as follows:
foz (x))
h =
Filfos, = lay (x)) fog (: Log (Foj (x)))
k: = 2
(9)
3 3 k L.
For given data batch X {x),-... , Xb the i-th task learning, we extract the representations using the j-th active representation network, expressed as
k), 1,... b].
F2 Fx kb) = =
(10)
where denotes the batch size. We can obtain collection feature vectors ZZ (Z))L 20
by leveraging the last active feature layers of the j-th backbone far, where each ((i is computed using Fg (X, fas, k). Similarly, we utilize each frozen representation network fg. to extract set previously acquired feature vectors (İ()L-L') F. (X, fo. k). The proposed MLRO approach incorporates a regularization loss component aimed at minimizing the divergence between the previously acquired and currently active representations, formulated as follows:
20.2} using Eq. (TO). with ğú.A)
(X) Σ (E (Ziis)) 20.0))
(11)
where Fais(t,) :) represents a generic distance metric used quantify the divergence between two sets of feature vectors. We opt for the L2 distance due to its computational efficiency and straightforward implementation. Furthermore, to address the shifts in the representations of historical examples, we incorporate a memory buffer M designed to store and maintain numerous past instances. As the primary focus of this paper is on optimizing representation strategies rather than the memory system, we consider employing simple reservoir sampling method [54) for memory updates, ensuring computational efficiency.
3.5 Adaptive Regularization Optimization
The representation optimization process, delineated in Eq. (T),, presupposes uniform regularization intensity across all representation layers during training, which may not yield optimal plasticity. This paper tackles this limitation by introducing innovative ARO method that selectively constrains parameter alterations in each representation layer throughout the optimization process. Specifically, the proposed ARO method incorporates a trainable switch vector {w] w } for the k-th trainable feature layer within the j-th representation network, where wi and w represent the probabilities

---

of activation and deactivation of the k-th representation layer, respectively. A straightforward method determine the penalization of changes involves converting the switch vector one-hot encoding; however, this approach lacks differentiability. To overcome this challenge, we propose utilizing the Gumbel-Softman distribution
[99] produce differentiable variables, expressed as
exp((lg((w)+) 91)/T)
wit [ (exp((log(w 9t)/T}
(12)
where 9r is drawn from Gumbel(0,1) and wif* is the differentiable approximation of A. represents a temperature parameter and large T encourages samples from the Gumbel Softmax distribution
to become one-hot representations. Using differentiable category variables defined in Eq. 122
can
derive a new regularization loss function:
(x)
T Σ j=ر EE
{wg Fass (Z )
(13)
Compared to Eq. (II. the regularization loss term defined in Eq. (13) changes in the parameters of each representation layer, which can relieve over-regularization issues and enhance plasticity.
) can selectively penalize the
3.6 The Optimization Framework
Algorithm Training procedure of LMSRR
The proposed framework involves T representa- tion networks ... for and a linear clas- sifier fe- In order to update the parameters of these modules, introduce unified objective function at the i-th task learning (T,), defined as
Require: Number
tasks N, dataset (DF IDi..... ..... D training iterations per task Ensure: Trained parameters of Lfar....,for
and classifier fe
for i = 1 to N do
for to n
Ci Yi-Po E (Foc fo (Z.)))
Step 1: Multi-source feature construction
(Bxx [Xi]
Sample a minibatch X {xI...., Xx3
from D
-xx, (x))))
Compute multi-source backbone outputs
{fer (x.)...., (x,)}
where Pp; and PP:; denote the distribution of the dataset Dj and the memory buffer M, re- spectively. PDAa denotes the distribution of the combined dataset D and M. Foc(.:) the cross-entrope function and À is hyperparam- eter that controls the effects of the regulariza- tion term during the optimization process. We provide the detailed learning process of the pro- posed framework in Fig. [ while the detailed pseudocode is provided Algorithm which consists of three steps
Construct stacked representations 2, using
Eq. (4)
Obtain fused representations Z, through the MSIDF module using Eq. (
Step 2: Representation--evel regularization
Obtain active representations (..k) from the last L'' layers using Eq.
TT
Obtain frozen references ZU.N) using
Eq. (IO)
Compute adaptive regularization FA (X)
using Eq. (3
Step 1. Form augmented representations For given data batch X {x1,'.. ,x),. we can obtain fused representations Z, through the MSIDF module using Eq. (8).
Step 3: Parameter update Compute total loss Lloss using Eq. 14 Update the model's parameters
Step 2. Adaptive representation optimization For a given data batch X = {x) x
end for end for
get all active representations (IL)),.... Z(T.L)) well all previously learndd representations (Z..)) (T.LL)
using Eq. (10). The regularization term is calculated using Eq. (13)
Step 3. Optimizing the model: We update all model parameters (Ø) W Wm) using Eq. (1). In addition, we also update the adaptive parameters {Pi, .Pm) well as the parameters
[w].r- W2 -. , w w22
T.I. T.L. of the proposed ARO approach using Eq. (1).
7

---

Table 1: The classification accuracy standard datasets is presented the average three "Average" denotes the average accuracy across all tasks, while "Last" indicates the accuracy of the final task. The "-" in the table signifies that experiments could not be conducted due compatibility issues or intractable training time problems.
CIFAR-10
Tiny ImageNNt
R-MNIST
Buffer
Method
Average
Last
Average
Last
Domain-IL
EWC 51
3.92
97.07.10.77
19.20.10.31
75.15.1..18
77.35.
ST 63
68.05 5.91
94.18-10.88
36.32:10.13
33.80:1..25
71.91:15.83
LwF
23.29-22.35
96.75+0.35
15.85:200.58
77.95=3..00
PNN 30
0..13
96.63+0.110
67.84:1:0229
69.03-=:1.01
96.05::3.39
94.95:1:.22 1.20
ER 49
91.19110.94
07.50:1..33
38.17:1...00
79.40:10.28
$5.01:11..9
GBM
0..44=0.94
06.30:0..35
80.80:t:1 1.15
A-GEM P0
83.88 1.49
07.00
22.77=00.03
38.63
81.91=:0.76
97.07-10.32
28.19:11.48
0.745:100.88
FDR 7
91.01 0.6.8
40.36
81.40.10.70
85,22.1 3..35
200
GSS 3
28.80
0,24
79.50= 0.41
HAL 11
3.220
94.6011..14
$4.02.10.98
DER 8
0.999
97.80:10.28
40.22:1:0.67
79. 0.21
90.04-1:2.61
DER+ 3
01.92200000
97.72-40.38
40,87
78.35::00449
90,43:t:1
(ER+ 56
92.01 3.03
37.63+
47.61=28.87
81.40 1.41
91. 91.64:::2.20
99.35
92.08
1.24
ER 49
93.61 0:0.27
97.15.10.28
48.64-10.46
80.80:1 1.69
88.91 1
GEM
02.16990
96.63-2.117
11.15
A-GEM P
89.48=:1.45
97. 07.40
25.33:4::49
81.00+0:442
80.31:+6.29
88.22:42..62
96.57-10.10
31.55:1..27
FDR 7
93.29
97.32.10.24
49.88
81.10.10.55
89.67 1.63
500
GSS 3
91.02:#1..7
96,97400.24
81.58:+00.50
HAI 1
2.35
94,22.10.00
$5.00.10.99
DER S
93.40:20.39
97.90:10.22
51.78:10.88
30 1.13
92.24
DER++ 3
93.88=0.50
10:0011
51.91=-0.66
5.223
92,77=
(BR++r))
56
03.06:20.38
07.75
54.06:1:0.79
93.28:t:0075
1.58
ER 49
95.34.1.000
97.67-10.67
35.92.10.90
80.30-10.82
90.42
GEM
03.67
97.37-0..17
11.15
A-GEM P
85.61=2..00
97.45=00.22
24.29 1.28
79.63+22.19
81.30:+:5.33 33
91.40:1:1006
0.35
54.0011:.882
FDR 7
94.02.10.64
97.60.10.56
56.05
00.25.1..49
91.68.1 1.01
1000
GSS 3
21.79:22.16
96. 16.10
82.25
HAI. 1
87.3331..46
92.27
89.33.1
DER 8
92.33.10.61
97.72410.07
56.6241..11
78. 50.10.0..4
93.1 03.13::0.228
DER+ 8
04.99=0.226
97.94+0.08
08.05--0.52
79 09.95
03.82-+0.39
DER+Hr)) 56
93.66 1.00
97. 97.40:20001
61.91:1.115
80.45 3.18
93.37:1:0.58
99.43.10.03
96.101...51
97.05.10.04
4 Experiment
4.1 Experimental setting
Datasets.
we conducted extensive experiments on seven different datasets, including CIFAR-
10 (33], TinyImageNet (35), MNIST (36), CIFAR-100 (34), CUB-200 (55), ImageNet-R [22). and Cars 196 (32]. We provide the detailed experiment setting in Appendix A from Supplementary
Material (SM).
4.2 Results on Standard Datasets
In this section, we compare the proposed approach with several basclines on the standard datasets, including CIFAR-10, Tiny ImageNet and R-MNIST, under memory buffer sizes of 200, 500, and 1000. The empirical results are reported Tab. D These results show that LMSRR significantly outperforms the other basclines in terms of classification accuracy. This highlights LMSRR's 's ability to effectively retain previously acquired knowledge as the number of tasks increases, demonstrating its remarkable plasticity and resistance catastrophic forgetting.
Previous CL methods, such as EWC, SI, and LwF, have lower average accuracy. The reason behind this result is that regularization-basee methods typically degrade when the new task contains abundant different information with respect prior tasks. PNN, as a dynamic expansion model, still struggles with scalability when dealing with long sequences of tasks, which significantly reduces its perfor-

---

Table 2: The classification results of various models on complex datasets, with memory buffer size of 500, calculated as the average results of three independent runs.
CIFAR-100
CUB-200
Imagenel-R
Cars196
Method
Average
Last
Average
Last
Average
Last
Average
Last
ER
73.37
1.33
30.57.14.81
35.57.1:14.86
4..85
45.85.10.01
30.52.14.4
54.32.15.07
A-GEM
1P
48.06=0.57
92.80-10.32
13.22440.31
42.18-20.01
16.8742..65
17.56=12231
8.07
16.45:7741
FDR
76.29=1.44
93.00+
23.9440.07
45.58=20.19
15.74=3.09
12.14=
31,41=1.00
58.36=1.17
GSS
57.50=1.93
92.80+22.98
27.04+0.28
42.01=00.08
17.83=0.88
33,44:6.75
34.67=22.27
56.80=44.15
DER 8
74.93=1.06
93.25.1..355
26.19:1.2007
$1.79:11.08
18.26:11.67
25.26:10.47
39.75:10.36
68.02.1...20
DER++
75.64=0.60
92.60:10.14
33.40=:1.48
1..83=1.63
22.87=5883
43.10=10.51
35.39=
60.56=8.45
DER+ +reireshh
56
0..7
93.40:1:.13
35.774:3.20
50.85:00.47
2..74
31.00:0.011
33.94:12.46
60.29:14.73
96.13.10.59
92.54.1.228
03.76.10.53
90.33-10.45
DAP M7
90.11-0.38
92.30:22.12
11.83+1444
22.23-2.85
83.22=
84.61
39.79+
65 65.35:22211
59
95.36=0.12
96.80:20.14
86.30+0.21
90.81=0.244
86.01=0.30
87.50=0.90
79.55=00.86
84.45=0.12
Ours
95.76 0.08
98.70-10.37
88.91100.64
94.31=0.122
84.35=0.52
88,43::0.155
90,14::0.06
95.32=00.99
mance. Experience replay-based methods, such GEM, GSS, DER, DER++ and DeR+erf experience noticeable performance drops when the memory buffer limited. This indicates that these methods struggle to capture critical informative samples when the memory buffer is constrained. Notably, our model maintains excellent performance even with small buffer size, further proving its adaptability and effectiveness across various continual learning scenarios.
4.3 Results on Complex Datasets
We evaluate our method against various baselines on complex datasets, and report the average and last accuracy in Tab. Replay-based methods such as ER. DER, and GSS show clear performance degradation on complex datasets, reflecting their limited ability to capture fine- aen-grained visual seman- ties when constrained by a fixed memory buffer. Although DAP and L2P leverage prompt- based mechanisms to mitigate representation drift and achieve better adaptation, their performance still relies heavily on the alignment between the pre-trained backbone and the target domain. For exam- ple, L.2P performs well on ImageNet-R but struggles on Cars196, where the distribution gap from pre-training data is large.
CoFiMA, which employs multi-model ensemble strategy through fixed-weight logit-level integration and introduces a new adapter for each task, shows strong results on CUB-200, benefiting from its ability to preserve task- specific knowledge. However, its design leads to parameter growth and task- dependent routing during inference, which limits scalability. In contrast, LMSRR attains consistently superior or comparable performance across all datasets within a unified architecture, achieving the highest results on CIFAR-100 and Cars196.
4.4 Ablation Study
In this section, perform full ablation study experiment to investigate the performance of the LMSRR with different configurations. More ablation study results are provided in Appendix from SM.
Backbone, To ensure fair comparison, we adopted the same multiple pre-trained VIT models as our method's backbone for other SOTA methods that do not involve modifications to the backbone network structure. In these methods, each pre-trained ViT model only allowed to update the parameters of the last three feature layers. The feature representations extracted by each pre-trained ViT are concatenated and then fed into a linear classifier to obtain the final output. Fig. 2(a) shows the average accuracy of our method and SOTA models the ImageNet-R dataset under different memory buffer configurations. The results indicate that our method consistently achieved the highest accuracy across various buffer sizes and significantly outperformed other models.
Forgetting rates. Fig. 2(b) presents the forgetting curves of our method and other methods on the ImageNet-R dataset. The results show that some SOTA models exhibit significant forgetting. especially static models like ER and DER, whose performance drops notably as the number of tasks increases. In contrast, our method maintains stable and superior performance, achieving the lowest forgetting rate. This attributed to our ML.O technique, which continuously adjusts the representation optimization process over time, effectively mitigating network forgetting.

---

24
DC877_8v
C19330
1120
D05011 DE8-1Ac
(a) Same backbone comparison.
(b) The forgetting curve.
(c) Configuration comparison.
Figure 2: (a) Comparison of performance of various models with varying buffer sizes on ImageNet-R, where each model uses the same backbone. (b) Comparison of forgetting curves of the proposed approach with other benchmark methods ImageNet-R. (c) Performance variations of the proposed MSIDF method under different configurations.
Different configurations. The MSIDF is driven by multiple attention modules of varying sizes, which can impact model performance based on their dimensions and quantity. To evaluate the MSIDF mechanism, with two attention modules of different sizes-3 & 5; MSIDF with only a size-3 attention module; MSIDF with only size-5 attention module; and baseline model without the MSIDF mechanism. The experimental results, as shown in Fig. [(c) indicate that the MSIDF with two differently sized attention modules achieved the highest classification accuracy, and models using MSIDF outperformed the baseline model without this mechanism. These findings highlight the significance of MSIDF in enhancing overall model performance by effectively capturing more critical feature information through attention modules diverse sizes.
test the following four configurations across multiple datasets: MSIDF
Conclusion and Limitation
This work introduced LMSRR, a framework that leverages multiple pre-trained ViT backbones to obtain diverse and complementary representations, employs MSIDF for multi-scale feature interaction, and incorporates MIRO and ARO to balance plasticity and stability through representation-leve regularization and adaptive layer-wise constraints. Extensive experiments across seven datasets demonstrate that LMSRR consistently improves accuracy and mitigates forgetting under various memory budgets. A limitation of the current study is that LMSRR relies fixed collection of pre-trained backbones and has not yet explored scalability to broader backbone families or multimodal environments. Future work will investigate more flexible backbone integration strategies and extend the framework to more dynamic and open- continual learning scenarios.
Acknowledgements
This work supported by the National Natural Science Foundation of China (Grant Nos. 62506067), the Sichuan Provincial Natural Science Foundation Project (Grant No. 2025ZNS- FSC0510), and the Fundamental Research Funds for the Central Universities (Grant Nos. ZYGX2025XJ024 and ZYGX2025X1025).
10

---

References
[1] A. Achille, T. Eccles, L. Matthey, C. Burgess, N. Watters, A. Lerchner, and I. Higgins. Life-long disentangled representation learning with cross-domain latent homologies. In Advances in Neural Information Processing Systems (NewrIPS), pages 9873-9883, 2018.
[2] Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty-based continual learning with adaptive regularization, In Advances in Neural Information Processing Systems, pages 4394-4404, 2019.
[3] Rahaf Aljundi. Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances neural information processing systems, 32. 2019.
[4] Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung- Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with memory of diverse samples. In Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 0218-8227, 2021.
[5] Jihwan Bang, Hyunseo Koh, Seulki Park, Hwanjun Song, Jung-Was Woo Ha, and Jonghyun Choi. Online continual learning contaminated data stream with blurry task boundaries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9275-92788 June 2022.
[6] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798- 1828, 2013.
[7] Ari S Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. arXiv preprint arXiv: 082889 2018.
[8] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances neural information processing systems, 33:15920-15930 2020.
[9] Hyuntak Cha, Jacho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9516-9525, 2021.
[10] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. Dokania, P. H. S. Torr, and M. A. Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902. 1902..04866 2019.
[11] Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge continual learning. In Proceedings the AAAI conference on artificial intelligence, volume 35, pages 6993-7001, 2021.
[12] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with A-GEM. Int. Conf. on Learning Representations (ICLR), arXiv preprint 2019.
[13] C. Cortes, X. Gonzalvo, V. Kuznetsov, M. Mohri, and S. Yang. Adanet: Adaptive structural learning of artificial neural networks. In Proc. of Int. Conf. on Machine Learning (ICML), vol. PMLR 70, pages 874 883, 2017.
[14] Danruo Deng, Guangyong Chen, Jianye Hao, Qiong Wang, and Pheng-Ann Heng. Flattening sharpness for dynamic gradient projection memory benefits continual learning. Advances in Neural Information Processing Systems, 34:18710 10:18710-18721 2021.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1929, 2020.
[16] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Trans- formers for continual learning with dynamic token expansion. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 9285-9295 2022.
11

---

[17] Jonathan St BT Evans. In two minds: dual-process accounts of reasoning. Trends in cognitive sciences, 7(10):454-45) 2003.
[18] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789-181 1819, 2021.
[19] E. J. Gumbel. Statistical theory of extreme values and some practical applications:a series of lectures. 1954.
[20] Yiduo Guo, Bing Liu, and Dongyan Zhao. Online continual learning through mutual information maximization. In International Conference on Machine Learning, pages 8109-8126. PMLR, 2022.
[21] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[22] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-diitrriution generalization. ICCV, 2021.
[23] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. In Proc. NIPS Deep Learning Workshop, arXiv preprint arXiv:1503.02531, 2014.
[24] Ching-Y Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu- Song Chen. Compacting, picking and growing for unforgetting continual learning. In Advances in Neural Information Processing Systems, pages 13647-13657, 2019.
[25] Rajasegaran Jathushan, Hayat Munawar, H Salman, Khan Fahad Shahbaz, and Shao Ling. Random path selection for incremental learning, arXiv preprint, 2019.
[26] Saurav Jha, Dong Gong, He Zhao, and Lina Yao. Npel: Neural processes for uncertainty-aware continual learning. Advances in Neural Information Processing Systems, 36, 2024.
[27] Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. Generating instance-level prompts for rehearsal-free continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11847-11857, 2023.
[28] Dahuin Jung, Dongjin Lee, Sunwon Hong, Hyemi Jang, Ho Bae, and Sungroh Yoon. New insights for the stability-plasticity dilemma in online continual learning. arXiv preprint arXiv:2302.08741, 2023.
[29] Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, and Chang D Yoo. Forget-free continual learning with winning subnetworks. In International Conference on Machine Learning, pages 10734-10750. PMLR, 2022.
[30] Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[31] Junsu Kim, Hoseong Cho, Jihyeon Kim, Yihalem Yimolal Tirunch, and Seungryul Back. Sddgr: Stable diffusion-based deep generative replay for class incremental object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28772-28781, 2024.
[32] Jonathan Krause, Michael Stark, Jia Deng. and Li Fei-Fei. 3d object representations for fine- grained categorization. In 2013 IEEE international conference on computer vision workshops, pages 554-561, IEEE, 2013.
[33] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Univ. of Toronto, 2009.
[34] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
12

---

[35] Ya Le and Xuan Yang. Tiny imageNet visual recognition challenge. Technical report, Univ. of Stanford, 2015.
[36] Y. LeCun, L. Bottou, Y. Bengio, and Haffner. Gradient-based learning applied to document recognition. Proc. of the IEEE, 86(1 1):2278-22222 1998.
[37] Z. Li and D. Hoiem. Learning without forgetting. IEEE Trans. on Pattern Analysis and Machine Intelligence, 40(11): 29477 2017.
[38] Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang. Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1778-1787, 2018.
[39] David Lopez-Pazz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pages 6467-6476, 2017.
[40] Daniel Marczak, Sebastian Cygert, Tomasz Trzciński, and Bartlomiej Twardowski. Revisiting supervision for continual representation learning. In European Conference on Computer Vision, pages 181-197. Springer, 2024.
[41] Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière. Weighted ensemble models are strong continual learners. In European Conference on Computer Vision, pages 306-324. Springer, 2024.
[42] James Martens and Roger B. Grosse. Optimizing neural networks with soonecckrrfactored approximate curvature. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, JMLR.org,
volume 37 of JMLR Workshop and Conference Proceedings, pages 2015.
[43] Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 36, 2024.
[44] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 11::54-71, 2019.
[45] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsur- prising effectiveness of pre-trained vision models for control. In international conference on machine learning, pages 17359-17371 PMLR, 2022.
[46] Biqing Qi, Xinquan Chen, Junqi Gao, Dong Li, Jianxing Liu, Ligang Wu, and Bowen Zhou. Interactive continual learning: Fast and slow thinking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12882-12892, 2024.
[47] J. Ramapuram, M. Gregorova, and A. Kalousis. Lifelong generative modeling. In Proc. Int. Conf. on Learning Representations (ICLR), arXiv preprint 170..v1700.0990 2017.
[48] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 2001-2010, 2017.
[49] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.1121 2018
[50] Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins. Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu. Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: (66666006677 2016.
[51] Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International conference on machine learning, pages 4528-4537.
PMLR, 2018
13

---

[52] H. Shin, J. K. Lee. J. Kim, and J. Kim. Continual learning with deep generative replay, In Advances Neural Inf. Proc. Systems (NIPS), pages 2990-2999, 2017.
[53] Vinay Kumar Verma, Kevin Liang, Nikhil Mehta, Piyush Rai, and Lawrence Carin. Efficient feature transformations for discriminative and generative continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13865-13875, 2021.
[54] Jeffrey S Vitter. Random sampling with reservoir. ACM Transactions Mathematical Software (TOMS), 11(1):77557 1985.
[55] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- uesd birds-200-2011 dataset. Technical Report CNS-TR-2011-001 California Institute of Technology, 2011.
[56) Zhenyi Wang. Yan Li, Li Shen, and Heng Huang. A unified and general framework for continual learning. arXiv preprint arXiv:2403. 13249, 2024.
[57] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning prompt for continual learning. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
139- 149, 2022.
[58] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965-23998, PMLR, 2022.
[59] Mengqi Xue, Haofei Zhang, Song, and Mingli Song. Meta-attention for vit-backed continual learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 150-159, 2022.
[60] Fei Ye and Adrian G. Bors. Lifelong infinite mixture model based on knowledge-driven Dirichlet process. In Proc. of the IEEE Int. Conf. Computer Vision (ICCV), pages 10695-10704, 2021.
[61] Fei Ye and Adrian G Bors. Dynamic self-supervised teacher-student network learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[62] Matthew D Zeiler, Graham W Taylor, and Rob Fergus. Adaptive deconvolutional networks for mid and high level feature learning. 201 international conference on computer vision, pages 2018 8 2025. IEEE, 2011.
[63] F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In Proc. of Int. Conf. on Machine Learning, vol. PLMR 70, pages 3987-3995, 2017.
[64] M. Zhai, L. Chen, Tung, He, M. Nawhal, and G. Mori. Lifelong GAN: Continual learning for conditional image generation. In Proc. of the IEEE/CVF Int. Conf. Computer Vision (ICCV), pages 2759-2768, 2019.
[65] Renrui Zhang. Liuhui Wang, Yu Qiao, Peng Gao, and Hongsheng Li. Learning 3d representa- tions from 2d pre-trained models via image-to-point masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21769-21780, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of paper in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the algorithm implementation Section 3.6 along with the source code. The detailed experimental setup is documented in Appendix-A.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We provide the source code in Github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
sttpiiisssooooiocyyy. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the algorithm implementation in Section 3.6 along with the source code. The detailed experimental setup is documented in Appendix-A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports average results over multiple experimental runs to ensure statistical reliability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8, Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper explicitly specifies the computational environment and resource consumption for each experiment in Section | and Appendix-A, including hardware type (e.g., GPU model). memory configuration, and execution time, which helps others accurately estimate the resources required for reproduction.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///aaarrrr..ccppppb.caaaaaa innes
Answer: [Yes]
Justification: The research conducted in our paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. ... if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper does not involve any immediate societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve any data models with significant risk of misuse, and therefore no specific safeguards are required.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.paperssit
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or any research involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or any research involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:p///earrrppccc/ooooeereeee/ /ct/////oeeeeeecc//22////.. for what should or should not be described.
21


Reviews:
[{"review_id": "v9uG29UAtu", "reviewer": "Reviewer_ABxL", "summary": "This paper proposes to a method that combined various pre-trained model to solve classical Class Incremental Continual Learning problems. To do so, this survey fusions various representations from pre-trained models by learning various attention matrices. The experimental results show convincing performances on various benchmarks.", "questions": "See above.", "limitations": "See above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Overview\nA lot of engineering which leads to good performances but lacking related work on model soup[3, 1], and containing unbacked overclaims.\n\n# Strengths\n- The motivations of using multiple pre-trained model is straightforward\n- The performances are compelling\n- Many experiments\n\n# Weaknesses\nDespite the performances, I have a lot of problems with the current state of the paper.\n1. The overall position of the paper is disturbing. It should compare with existing multi model strategies [1,3], especially [1] is in a continual learning scenario and is never mentioned. I deeply believe that the related work section should focus more on existing work in model soup usage rather than existing continual learning methods.\n2. A lot of claims are made throughout the paper, but eventually only the improved accuracy is showed. For example l. 56 \"This method captures the most important parts of the representations in response to incoming samples through several learnable attention module, facilitating the interaction among multi-scale features and aiding in uncovering the intricate underlying structure of the data\". There is no evidence of such claim and this is pure interpretation. A gain in accuracy does not justify such interpretation. Since the interaction between pre-trained models is learned, it is crucial to visualize such interaction to backup such claims.\n3. The obvious limitation of the current method is the computation overhead during training and inference. While the authors claim that such information is present in the appendix, I could not find it (in the main paper nor the appendix). Please discuss the spatial and temporal complexity clearly as this is the main limitation.\n4. Similar to my point in 2., the paper claim using a single pre-trained model can be a major issue on certain datasets compared to others. While this can somehow be observe on some tables, this should be more clearly demonstrated, in the main paper or appendix.\n5. A minor weakness might be the novelty as this is mainly a combination of existing idea present in [1,2]. I would advice clarifying the differences more.\n\n# References\n[1] Marouf, Imad Eddine, et al. \"Weighted ensemble models are strong continual learners.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n[2] Douillard, Arthur, et al. \"Dytox: Transformers for continual learning with dynamic token expansion.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n[3] Wortsman, Mitchell, et al. \"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.\" International conference on machine learning. PMLR, 2022.", "was_revised": true, "final_justification": "Overall, I believe the work to be valuable. There are indeed strong points made by the authors during the rebuttal phase, and even though the novelty might not be the strongest, I believe that with careful writing regarding the difference with concurrent methods, updated experiments with latest state-of-the-art approaches, ablation study and interpretation (as mentioned in Q2) this work has its place in such a venue. \n\nI believe the authors have addressed all the concerns I had and therefore raised my score. Given that there were, in my opinion, various missing elements in the first stage of the submission, I cannot give a strong acceptance; however, I believe it to be on the acceptance side of the borderline threshold.", "rebuttal": "**Q1: It should compare with existing multi model strategies.**\n\nWe thank the reviewer for raising this concern. While multi-model fusion methods like [1] and [3] are important in multi-model learning, our work targets Class-Incremental Continual Learning (Class-IL), which involves sequential tasks and catastrophic forgetting. Therefore, our design and comparisons are situated within continual learning literature. \n\nHowever, we agree that the relationship with [1] and [3] needs clarification. Below is a comparison: \n\nComparison with Model Soup [3]: Model Soup fuses models via parameter-space averaging, assuming independent training on static tasks, which contradicts the online adaptation required in continual learning. It also does not address catastrophic forgetting. In contrast, our method uses dynamic feature-space fusion via multi-scale attention, adapting to tasks without independent model training or post-hoc merging. \n\nComparison with Weighted Ensemble [1]: [1] uses a fixed-weight logit-level ensemble, which is static and task-specific. Our approach dynamically learns interactions among multiple pretrained backbones during training, resulting in more adaptive representations. Unlike [1], which trains models from scratch, we use multiple pretrained ViT backbones, offering better resource efficiency and deployment. \n\nAdditional Comparisons: To address the lack of direct comparison with multi-model strategies, we performed experiments on CIFAR-100, CUB-200, Cars196, and ImageNet-R using the protocol from [1], comparing with CoMA and CoFiMA.\n\n| Methods     | CIFAR-100        | Cars196          | CUB-200          | ImageNet-R       |\n| ----------- | ---------------- | ---------------- | ---------------- | ---------------- |\n| CoMA [1]    | 92.00 ± 0.13     | 73.35 ± 0.59     | 85.95 ± 0.29     | 77.47 ± 0.05     |\n| CoFiMA [1]  | 92.77 ± 0.24     | 76.96 ± 0.64     | 87.11 ± 0.56     | 78.25 ± 0.26     |\n| LMSRR (200) | 94.69 ± 0.44     | 88.76 ± 0.10     | 88.26 ± 0.08     | 83.99 ± 0.32     |\n| LMSRR (500) | **95.76 ± 0.08** | **90.14 ± 0.06** | **88.91 ± 0.64** | **84.35 ± 0.52** |\n\nHere, `LMSRR (200)` and `LMSRR (500)` refer to our method evaluated with replay buffer sizes of 200 and 500, respectively. As shown, LMSRR significantly outperforms both CoMA and CoFiMA across all datasets, confirming the effectiveness and adaptability of our learnable feature-level fusion mechanism in continual learning scenarios.\n\n------\n\n**Q2: Visualization results.**\n\nWe thank the reviewer for raising this important point. We understand that improved performance alone does not validate the MSIDF module’s claims. To address this, we designed two additional experiments to provide interpretable evidence: \n\n1. Feature Discriminability Analysis (Structured Class Cluster Separation) We compared Concatenation (baseline) with MSIDF on the CIFAR-10 dataset, using t-SNE for visualization. Due to rebuttal policy, the visual results are included in the supplementary material. The results show that MSIDF provides better inter-class separation, indicating that MSIDF learns discriminative and structured representations. \n2. Visualization of MSIDF’s Dynamic Fusion Behavior We analyzed the activation distribution across MSIDF’s two attention modules, using the L2 norm of the output features. A heatmap was plotted showing the normalized activation strength for each class. Results indicate that different classes activate different attention modules, confirming that MSIDF performs dynamic, task-aware fusion, not a fixed strategy.\n\n------\n\n**Q3: The computational costs.**\n\nWe thank the reviewer for highlighting the concern about computational overhead. We now provide a detailed analysis of LMSRR's computational resource consumption compared to several baseline methods, including replay-based, regularization-based, prompt-based, and multi-model ensemble approaches. We evaluated the following metrics:\n\n- Params: Trainable parameters (millions)\n- GPU Avg/Max: Average and peak GPU memory usage (MiB)\n- CPU Avg/Max: Average and peak CPU memory usage (MiB)\n- Iteration Time: Average time per training iteration (sec/iter)\n\n| Methods | Params    | GPU Avg ↓       | GPU Max ↓       | CPU Avg ↓       | CPU Max ↓       | Iteration ↓   |\n| ------- | --------- | --------------- | --------------- | --------------- | --------------- | ------------- |\n| ER      | 21.42M    | 5876.93 MiB     | 5876.93 MiB     | 2279.25 MiB     | 2342.39 MiB     | 3.07 s/it     |\n| DERPP   | 21.42M    | 5607.20 MiB     | 5607.20 MiB     | 2202.59 MiB     | 2266.29 MiB     | 3.21 s/it     |\n| DERPP(Re) | 21.42M    | 5607.12 MiB     | 5607.12 MiB     | **2196.54 MiB** | **2260.47 MiB** | 3.15 s/it     |\n| DAP     | **0.51M** | **3666.89 MiB** | **3666.89 MiB** | 2694.51 MiB     | 2724.04 MiB     | **1.33 s/it** |\n| CoFiMA  | 85.80M    | 9460.43 MiB     | 9896.50 MiB     | 2311.98 MiB     | 2684.08 MiB     | 2.41 s/it     |\n| LMSRR   | 42.53M    | 4672.31 MiB     | 4672.31 MiB     | 2499.88 MiB     | 2563.55 MiB     | 2.92 s/it     |\n\nPretraining and Fine-Tuning Strategy:\nTo ensure fair comparison, we adopted a consistent protocol across all baseline methods (e.g., ER, DERPP, DERPPRE), using the same ViT backbone and freezing most layers, fine-tuning only the last three layers. This protocol is also applied to LMSRR. Despite fusing multiple ViTs, LMSRR maintains a moderate number of trainable parameters (42.53M), significantly lower than CoFiMA’s 85.80M.\n\nComparison with CoFiMA [1]:\nCoFiMA unfreezes all ViT parameters, leading to 85.80M parameters and higher memory usage. In contrast, LMSRR maintains moderate resource usage while achieving dynamic fusion and task adaptation, providing a better efficiency-performance tradeoff.\n\nComparison with DAP:\n DAP, as a prompt-based method, uses 0.51M parameters, but is sensitive to prompt quality and task shifts. LMSRR increases parameters modestly but shows better performance and stability.\n\nInference and Deployment Efficiency:\n LMSRR uses a single fused representation for inference, avoiding multi-model inference overhead and making inference cost similar to a single model.\n\n**Q4: The demonstraction of the limitation of a single pre-trained model.**\n\nWe thank the reviewer for raising this point. To address the lack of empirical evidence for our claim about the suboptimal performance of single pre-trained models, we conducted a comparative study on CIFAR-100, CUB-200, and Cars196.\n\nExperimental Setup:\n\nWe selected three ViT backbones:\n\n1. ViT1: ViT-B/16, pre-trained on ImageNet-21k and fine-tuned on ImageNet-1k\n2. ViT2: ViT-B/16, pre-trained only on ImageNet-21k\n3. ViT3: ViT-L/14, a larger ViT model\n\nWe evaluated the following configurations:\n\n- Single models: ViT1, ViT2, ViT3\n- Two-model fusion: ViT1 + ViT2\n- Three-model fusion: ViT1 + ViT2 + ViT3\n\nResults (Average Accuracy):\n\nAll experiments were conducted under the Class-IL setting using LMSRR's feature fusion pipeline. The results are reported in terms of Average Accuracy (%):\n\n| Method               | CIFAR100  | CUB200    | Cars196   |\n| -------------------- | --------- | --------- | --------- |\n| ViT1 (In21k-ft-In1k) | 94.17     | 88.03     | 89.02     |\n| ViT2 (In21k)         | 91.81     | 84.73     | 85.78     |\n| ViT3 (ViT-L-14)      | 93.92     | 87.89     | 88.74     |\n| ViT1 + ViT2          | 93.90     | 87.64     | 88.76     |\n| ViT1 + ViT2 + ViT3   | **95.01** | **88.74** | **89.34** |\n\nSingle-model performance varies across datasets: ViT2 underperforms on **Cars196** compared to ViT1 and ViT3, while ViT1 excels on **CIFAR-100**, but not on fine-grained datasets. Multi-model fusion consistently improves performance, with the three-model fusion outperforming all single-model configurations. These results confirm that multi-model fusion enhances robustness and adaptability, supporting our claim.\n\n**Q5: The  novelty.**\n\nCompared to [1], the proposed approach has three differences: (1) [1] only employs a single pre-trained ViT, which has limited learning ability when addressing unknown data domains. In contrast, the proposed MSIDF optimizes several pre-trained representation networks to provide semantically rich features. A novel Multi-Scale Interaction and Dynamic Fusion approach is proposed to integrate features from all representation networks to further enhance the representation capacity; (2) [1] adopts a mixture of experts framework, while the proposed MSIDF adopts a single classifier and does not need to perform the expert selection process. As a result, [1] requires knowing the task information in order to select the suitable expert for each testing sample, while the proposed MSIDF can efficiently predict the data without requiring access to the task information.\n\n\nCompared to [2], the proposed MSIDF has three differences: (1) [2] trains the parameters of the network from scratch while the proposed MSIDF adopts pre-trained ViTs to provide diverse and robust representations; (2) [2] employs the mixture of experts framework, which requires knowing the task information during the inference process. In contrast, the proposed MSIDF does not incrementally create new experts when learning new tasks and can be efficiently performed in the inference process without knowing task information. (3) [2] simply freezes all previously learned parameters to relieve network forgetting, which can decrease plasticity. In contrast, the proposed MSIDF optimizes most parts of the representation networks using the proposed Multi-Level Representation Optimization, which can significantly improve the model's plasticity.\n\n**References:**\n\n[1] Marouf, Imad Eddine, et al. \"Weighted ensemble models are strong continual learners.\".\n\n[2] Douillard, Arthur, et al. \"Dytox: Transformers for continual learning with dynamic token expansion.\".\n\n[3] Wortsman, Mitchell, et al. \"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.\"."}, {"review_id": "fxqWD3sqG0", "reviewer": "Reviewer_cpc8", "summary": "This paper focuses on addressing the fundamental challenge of balancing plasticity and stability in continual learning, i.e., how to effectively learn new tasks without forgetting previously acquired knowledge. To tackle this, the authors propose a new framework called LMSRR (Learning Multi-Source and Robust Representations) that aims to enhance feature robustness and adaptability by integrating multiple pre-trained models as representation backbones.\n\nTo improve plasticity, the method utilizes a series of learnable attention mechanisms to dynamically extract and fuse features from multiple pre-trained ViT models, producing more flexible and semantically rich representations for downstream tasks. To maintain stability, the authors avoid freezing the entire representation network. Instead, they propose two Regularization optimization strategies: MLRO and ARO.\n\nExperimental results on both standard and complex continual learning benchmarks demonstrate the effectiveness of the proposed approach.", "questions": "1. Could the authors expand the related work section to include more regularization-based continual learning approaches (e.g., EWC, MAS, RWalk), and clearly explain how MLRO and ARO differ from those in terms of motivation and implementation?\n2. Given LMSRR's reliance on pre-trained ViTs, how does it handle unseen domains? Has the method been evaluated in cases where the pretrained models were not exposed to task-relevant data distributions?\n3. Please include explicit ablation experiments for each of the three proposed modules. Since LMSRR is composed of several interacting components, understanding their individual effects on performance is crucial for evaluating the contribution of the framework.\n4. It would be valuable to assess LMSRR's performance in low-resource or few-shot learning tasks. As continual learning often deals with limited per-task data, this would significantly strengthen the method's practical relevance.", "limitations": "The authors have discussed limitations both in terms of technicality as well as societal impact of their work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper tackles a highly relevant and challenging problem in continual learning: balancing plasticity and stability. The proposed LMSRR framework provides a promising solution by maintaining strong performance on new tasks while mitigating forgetting on previous tasks.\n2. LMSRR effectively integrates three strategies: MSIDF, MLRO, and ARO. Each component is well-grounded in theory and complements the others: MSIDF improves plasticity through the dynamic fusion of multiple pre-trained backbones, while MLRO and ARO introduce novel regularization techniques that promote stability during continual adaptation.\n3. Extensive experiments on seven datasets consistently demonstrate state-of-the-art results, validating the method's robustness and generalization ability. Particularly under limited memory buffer settings, LMSRR shows impressive performance, highlighting its practical applicability.\n\nWeaknesses\n1. The overall contribution can be viewed as a combination of existing techniques (multi-source feature fusion and regularization-based continual learning strategies). Although the integration is well-executed, the core components are adaptations of established methods, which weakens the paper's originality.\n2. The plasticity of the method may heavily rely on the coverage of the pre-trained ViT backbones. It is unclear how LMSRR would perform in unseen domains, where the representational capacity of the backbone may be limited.\n3. While the paper provides some analysis of MSIDF, it lacks comprehensive ablation studies on all three components (MSIDF, MLRO, ARO). On one hand, ablations on the choice and number of pre-trained models would clarify LMSRR's dependence on backbone diversity. On the other hand, isolating the impact of MLRO and ARO would help identify their individual contributions to stability and plasticity.\n4. Although LMSRR achieves the best absolute performance across metrics, the comparison may be unfair due to differences in architecture or backbone choice. Additionally, LMSRR appears to show greater performance drops (in \"Average\" task accuracy) compared to methods like DAP and PNN, raising questions about its stability.", "was_revised": true, "final_justification": "I thank the authors for their detailed rebuttal and the additional experiments. I decide to maintain my score as 4.", "rebuttal": "**Q1: Lack of Originality, The method mainly adapts existing techniques, reducing originality.**\n\nThis paper presents two significant and previously unexplored contributions to the field. First, we introduce a novel Multi-Scale Interaction and Dynamic Fusion (MSIDF) approach that adaptively synthesizes features from multiple representation networks. In contrast to conventional feature fusion techniques that merely aggregate features into a single representation, the MSIDF framework incorporates a suite of adaptive attention modules, each characterized by distinct attention window sizes to capture diverse local information within the representations. Consequently, each representation is partitioned into several segments, each processed by different attention modules, and subsequently integrated through a trainable adaptive mechanism. This process yields a robust and less redundant representation, marking a novel direction in continual learning research. Second, we propose an innovative Adaptive Regularization Optimization (ARO) method that selectively penalizes parameter updates at each layer of the representation network. Unlike traditional regularization approaches that uniformly penalize changes across all network layers, the ARO method employs a learnable penalization strategy that dynamically assesses the significance of each layer during training. This strategy assigns adaptive weights to modulate the regularization strength per layer, effectively mitigating over-regularization and enhancing network plasticity. Both the MSIDF and ARO approaches represent original contributions that have not been previously addressed in the continual learning literature.\n\n**Q2: Plasticity Dependence, Performance may depend on pre-trained ViT backbones, with unclear performance on unseen domains.**\n\nWe thank the reviewer for raising this important question. To assess LMSRR’s performance on unseen domains, we conducted experiments on two datasets with substantial distributional shifts from ImageNet:\n\n- RESISC45[1]: A remote sensing dataset with 45 scene categories.\n- ISIC[2]: A medical imaging dataset for skin lesion classification.\n\nWe ensured consistent pre-training configurations across methods, using two ViT-B/16 models for LMSRR (In21k and In21k-ft-In1k) and a single ViT-B/16 for baselines.\n\n| Method  | RESISC45  | ISIC      |\n| ------- | --------- | --------- |\n| DERPP   | 48.44     | 64.97     |\n| DERPP(Re) | 36.53     | 62.00     |\n| DAP     | 44.21     | 46.91     |\n| LMSRR   | **50.93** | **79.21** |\n\n\n\nLMSRR outperforms all baselines on both datasets, with +2.49% gain on RESISC45 and +14.24% on ISIC. This is attributed to LMSRR’s multi-backbone fusion and learnable attention-based fusion mechanism (MSIDF), which enables domain-adaptive and robust representations. These results show LMSRR’s effectiveness in generalizing to out-of-distribution tasks, essential for real-world continual learning.\n\n**Q3: Missing Ablation Studies: No comprehensive ablation studies on MSIDF, MLRO, and ARO to clarify their individual impacts.**\n\nWe thank the reviewer for raising the valuable suggestion to analyze the individual contributions of each module within LMSRR. To address this concern, we conducted a comprehensive module-wise ablation study to quantify the impact of the three core components—MSIDF, MLRO, and ARO—across different datasets.\n\nWe performed experiments on CIFAR-100, CUB-200, and Cars196, using Average Accuracy as the evaluation metric. Starting from the complete LMSRR framework, we systematically removed each module while keeping all other settings and architectures fixed:\n\n- `LMSRR`: The full proposed model, including MSIDF, MLRO, and ARO.\n- `w/o MSIDF`: Replaces the multi-source dynamic fusion with simple feature concatenation (static combination of ViT outputs).\n- `w/o MLRO`: Disables multi-level layer selection regularization (no guidance for stable feature selection).\n- `w/o ARO`: Removes the auxiliary representation alignment using the frozen teacher network.\n\n| Method    | CIFAR-100 | CUB-200 | Cars196 |\n| --------- | --------- | ------- | ------- |\n| LMSRR     | 95.76     | 88.91   | 90.14   |\n| w/o MSIDF | 94.47     | 87.49   | 86.53   |\n| w/o MLRO  | 94.89     | 88.02   | 88.76   |\n| w/o ARO   | 95.02     | 87.96   | 88.90   |\n\n------\n\n**Analysis**:\n\n- The MSIDF (Multi-Scale Interaction and Dynamic Fusion) module proves to be the most critical component. Its removal results in the most significant performance drop (up to –3.57%), particularly on more challenging datasets like Cars196. This underscores the importance of dynamic fusion in enhancing the semantic expressiveness of multi-source features.\n- The MLRO (Multi-Level Representation Optimization) module contributes 0.7%–1.3% performance gain, demonstrating its role in guiding the model to prioritize stable and transferable features across tasks, thus enhancing robustness during task transitions.\n- While the ARO (Auxiliary Representation Optimization) module leads to a relatively smaller drop in average accuracy, its primary role lies in enhancing stability during task shifts rather than boosting peak performance. This auxiliary alignment contributes to smoother representation transitions over time.\n\nIn summary, all three modules contribute meaningfully to the final performance and stability of LMSRR, with MSIDF being the most impactful, and MLRO/ARO supporting continual adaptation and resilience to forgetting.\n\n**Q4: Unfair Comparisons & Stability: The comparison may be unfair due to architecture/backbone differences, and LMSRR shows greater performance drops, raising stability concerns.**\n\nWe sincerely appreciate the reviewer’s concern regarding the fairness of our experimental setup. To ensure a fair and consistent comparison across all baselines, we have adopted a unified experimental protocol in both the main paper’s ablation studies (Fig. 2(a)) and Appendix (Fig. 1):\n\n- For baseline methods that do not alter backbone architectures (e.g., ER, DER, DER++), we use the same set of multiple pre-trained ViT models as in LMSRR.\n- All ViT models are trained using a “frozen early layers, fine-tune last layers” strategy to maintain similar learnable capacity across methods.\n- Feature outputs from multiple ViTs are simply concatenated and passed through a shared linear classifier, forming a generic and fair multi-source fusion baseline for comparison.\n\nFor DAP, we clarify that it is fundamentally prompt-based, with a distinct training mechanism. Specifically:\n\n- DAP freezes the entire ViT backbone and introduces an adaptive prompt generator to guide task-specific adaptation.\n- The prompt generator produces instance-level prompts that are injected into every layer of the ViT, enabling the frozen network to generate task-aware representations.\n- In our experiments, DAP is implemented using a single pre-trained ViT (ViT-1K), strictly following the configuration and freezing strategies described in its original paper.\n\nHence, from backbone choices to optimization protocols, all methods are trained under standardized and comparable settings. The performance improvements of LMSRR are therefore attributed to our proposed architectural design and optimization strategies, rather than any unfair experimental bias.\n\n------\n\n2.Regarding the \"Average\" vs. \"Last\" Task Accuracy Gap:\n\nWe acknowledge the reviewer’s observation that LMSRR sometimes exhibits a larger gap between the average and last task accuracy, especially compared to methods like DAP or PNN. We believe this behavior arises naturally from differences in method design philosophies, as detailed below:\n\n- DAP and PNN belong to architecture- or prompt-expansion categories, where task-specific modules or prompts are introduced per task. This design helps preserve prior knowledge and reduces forgetting but comes at the cost of linearly growing parameters and limited scalability.\n- In contrast, LMSRR leverages a unified and shared representation space, enhanced by MLRO and ARO, which dynamically adjust representations across tasks. This allows LMSRR to efficiently balance plasticity and stability, achieving strong last-task performance (reflecting adaptability) while maintaining competitive average accuracy (reflecting retention).\n- Furthermore, as shown in Fig. 2(b) of the main paper and Fig. 2 in the Appendix, we provide forgetting curve analysis on ImageNet-R and other benchmarks. The results demonstrate that LMSRR not only achieves the highest overall accuracy, but also exhibits the smoothest and most stable forgetting curves, indicating strong robustness against catastrophic forgetting.\n\nIn summary, while LMSRR may show a relatively larger gap between last and average accuracy in some cases, this does not indicate weaker stability. Rather, it reflects a trade-off rooted in efficient shared representation learning, which achieves superior scalability, performance, and generalization compared to architecture-expanding methods. We will clarify this point further in the revised manuscript.\n\n**References:**\n\n[1] Cheng, G., et al. *Remote Sensing Image Scene Classification: Benchmark and State of the Art*. IEEE Transactions on Geoscience and Remote Sensing.\n\n[2] Codella, Noel, et al. \"Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic).\"."}, {"review_id": "8YSqyR9OLo", "reviewer": "Reviewer_3Jvb", "summary": "In this paper, the authors propose a new method for class-incremental learning problem. Instead of a single pre-trained model, the authors propose to orchestrate multiple different pre-trained backbones to derive semantically rich multi-source representations. As for optimization, the authors also propose Multi-Level Representation Optimization strategy to adaptively refine the backbones. The experiments show the effectiveness of the proposed method.", "questions": "Please refer to the weaknesses.", "limitations": "Please refer to the weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The integration of leveraging the knowledge of pre-trained models and selectively optimize them shows the great improvement over class-incremental learning.\n2. The writing and mathematical explanation makes the idea easy to understand\n\nWeakness:\n1. The performance comparison is unfair. Most of the methods are proposed before 2021 and the most recent work DAP is proposed in 2023. However, there has been some improvements in class-incremental learning in recent years so that the authors should compare with more SOTA. Second, the compared methods are mostly replay-based or regularization-based but the proposed method is more likely the architecture-based that is naturally more effective for catastrophic forgetting. The authors should compare with some similar methods like [1] and [2] that leverages a combination of small models and optimize part of the representations. Third, the proposed method is based on pre-trained models that already have good representations while the compared methods are not pre-trained. Therefore, the authors should compare with methods with pre-trained knowledge, such as [3], to show their effectiveness. \n2. I wonder how the performance changes when the number of experts changes.\n3. I wonder whether the proposed method is more effective than compared methods when the first task contains more classes.\n\n[1] Random path selection for continual learning\n\n[2] Overcoming catastrophic forgetting with hard attention to the task\n\n[3] Learning to prompt for continual learning", "was_revised": true, "final_justification": "Most of my problems have been solved by the authors' response. I have raised my rating to recommend acceptance.", "rebuttal": "**Q1:Fairness of performance comparison, architectural differences, and pre-training strategy**\n\nWe sincerely thank the reviewer for their professional comments regarding the categorization of our method and the fairness of our experimental comparisons. We provide a unified response to the three sub-issues raised:\n\n**1. Key Distinction from Architecture-Expansion Methods**\n\nAlthough LMSRR utilizes multiple pre-trained ViT models as feature extractors, our method does not expand model structures per task (e.g., by adding branches or task-specific heads). Instead, our goal is to construct a semantically aligned multi-source representation space within a unified architecture.\n\nSpecifically, feature representations from multiple pre-trained models are fused via our proposed MSIDF (Multi-Source Interactive Dynamic Fusion) module. This fused representation is then progressively optimized through MLRO (Multi-Level Representation Optimization) and ARO (Auxiliary Representation Optimization) modules. This design maintains feature continuity and robustness across tasks, and is fundamentally different from task-isolated routing strategies like those used in RPSNet [1] and HAT [2], which suffer from parameter growth as the number of tasks increases.\n\nIn essence, LMSRR achieves parameter sharing + multi-source alignment + localized optimization, offering a unified alternative to architecture expansion-based methods.\n\n------\n\n**2. Pre-training Strategy Alignment and Experimental Fairness**\n\nTo ensure a fair and comparable experimental setting, we adopted a consistent backbone configuration across methods, which is reflected in both our main paper’s ablation study (Fig. 2(a)) and the appendix (Fig. 1):\n\n- For existing methods that do not alter backbone architectures (e.g., ER, DER, DER++), we used the same set of multiple pre-trained ViT models as in LMSRR.\n- All ViT backbones were trained under the same protocol: early layers were frozen, and only the final few layers were fine-tuned to ensure equivalent trainable capacity.\n- These baseline methods fused features from multiple ViTs via concatenation and used a shared linear classifier, serving as a strong and fair multi-source fusion baseline.\n\n------\n\n**3. Comparison with Architecture-Based and Prompt-Based Methods**\n\nTo further address the reviewer’s concern, we additionally compared LMSRR with both architecture-based and prompt-based recent SOTA methods, including:\n\n- RPSNet [1]: A classical architecture-expansion method that employs task-specific path selection masks over a ResNet-18 backbone to dynamically activate sub-networks. While this design reduces forgetting, it introduces structural growth and high memory costs with increasing tasks.\n- DualPrompt [3] and L2P [4]: Prompt-based continual learning methods built on frozen ViT-1K backbones, injecting learned task-specific prompts. These are closely related to DAP and represent the most recent line of pre-trained prompt-injection approaches.\n\nWe faithfully reproduced these methods following their original settings and training protocols. We conducted comparisons on CIFAR-100 and Tiny ImageNet, and, to ensure fairness, we adopted the same buffer size of 2000 used in the RPSNet paper. The evaluation metric was Average Accuracy:\n\n| Method         | CIFAR-100 | Tiny ImageNet |\n| -------------- | --------- | ------------- |\n| RPSNet [1]     | 68.60     | 43.54         |\n| DualPrompt [3] | 91.80     | 93.11         |\n| L2P [4]        | 90.81     | 92.14         |\n| LMSRR          | **95.86** | **93.24**     |\n\n------\n\nThe results demonstrate that LMSRR significantly outperforms RPSNet by more than 20%, and also shows better generalization than recent prompt-based methods. These results confirm the robustness of LMSRR’s unified representation strategy, even when compared to strong architecture-specific or prompt-injection baselines.\n\nWe have thus made every effort to ensure fairness in design, backbone pre-training protocols, and evaluation settings, and we supplemented our main experiments with comparisons to recent competitive approaches.\n\n**Q2:Performance variation with different numbers of experts**\n\nWe appreciate the reviewer’s insightful question. We would like to first clarify a potential misunderstanding regarding the term *“expert”* in our method.\n\nIn our framework, the term *expert* does not refer to a Mixture-of-Experts (MoE) architecture, where independent expert networks are assigned to different tasks or classes. Instead, our method consistently employs a single unified linear classifier $f_\\phi$, which takes as input the fused multi-source representations generated from different pre-trained ViTs and outputs the final prediction.\n\nThus, in our architecture, the number of experts remains one, and increasing the number of ViT backbones only affects the diversity and richness of the input features to be fused, rather than increasing the number of prediction heads or models.\n\nThat said, we fully understand that the reviewer’s intention was likely to ask:\n\n> Does the performance improve as more ViT models are involved in the fusion process?\n\nThis question was already addressed in our response to Reviewer ABxL Q4, where we conducted a systematic analysis. In that experiment, we used combinations of one, two, and three different pre-trained ViTs and evaluated the performance on three datasets. The results clearly showed that adding more ViT models consistently led to improved performance, validating our motivation for designing a multi-model fusion strategy.\n\nIn summary, while our architecture contains only one final classifier (i.e., one expert), the performance gains obtained by increasing the number of fused ViTs demonstrate the effectiveness of leveraging feature diversity, and confirm the value of our fusion mechanism.\n\nWe thank the reviewer again for helping us further clarify this design choice.\n\n**Q3:Effectiveness under a first task with more classes**\n\nWe thank the reviewer for raising this insightful question. To evaluate whether our proposed method maintains its effectiveness under varying task configurations—particularly when the first task contains more classes—we conducted a dedicated experiment on the CIFAR-100 dataset.\n\n**Experimental Setup**\n\nWe compared two task-splitting scenarios, keeping the total number of classes at 100:\n\n- Scenario A (First20): [20,10,10,10,10,10,10,10,10] — First task contains 20 classes\n- Scenario B (First50): [50,10,10,10,10,10] — First task contains 50 classes\n\nFor all methods, the memory buffer size was 500 samples to ensure fairness.\n\n| Methods | First20 | First50 |\n| ------- | ------- | ------- |\n| DERPP   | 75.35   | 70.90   |\n| DERPPRE | 74.06   | 73.50   |\n| DAP     | 90.86   | 85.23   |\n| LMSRR   | 95.32   | 94.62   |\n\nWe also compare with the default continual learning scenario (i.e., uniform splits such as 10 tasks × 10 classes), where LMSRR achieves:\n\n- **LMSRR:** 95.76 ± 0.08\n- **DAP:** 90.11 ± 0.33\n- **DERPP:** 75.64\n- **DERPPRE:** 77.71\n\nFindings and Insights\n\n- LMSRR consistently outperforms all baselines across both settings.\n- The performance of baseline methods drops more significantly when the first task becomes larger (e.g., DERPP drops by 5.0 points).\n- In contrast, LMSRR remains highly stable, with only a minor drop of 0.7 points from 95.32 (First20) to 94.62 (First50).\n\nThis stability is attributed to LMSRR's multi-source representation fusion, which allows the model to dynamically integrate knowledge from multiple pre-trained ViTs. This fusion provides a richer, more flexible feature space, enabling better adaptation even when the first task involves a larger or imbalanced set of classes. Unlike methods like DERPP and DAP, which rely on single-model structures or fixed mechanisms, LMSRR's dynamic fusion ensures that the model remains robust to the variability in task distribution, thus reducing the impact of initial task bias.\n\n**References:**\n\n[1] Rajasegaran, J., et al. *Random Path Selection for Continual Learning*. NeurIPS, 2019.\n\n[2] Serra, J., et al. *Overcoming Catastrophic Forgetting with Hard Attention to the Task*. ICML, 2018.\n\n[3] Wang, Z., et al. *DualPrompt: Complementary Prompting for Rehearsal-Free Continual Learning*. ECCV, 2022.\n\n[4] Wang, Z., et al. *Learning to Prompt for Continual Learning*. CVPR, 2022."}]

Author rebuttal:

Meta-review: This paper addresses the problem of continual learning with a focus on multi-scale aggregation. It received scores from three reviewers and, prior to the rebuttal, was below average for acceptance. After the rebuttal, and particularly after a detailed discussion with reviewer 3Jvb, the authors carefully explained the differences from related work, including architecture-based and architecture-expansion approaches. Furthermore, the authors added a substantial amount of experimental content. The reviewers felt that most of the concerns were addressed.

AC reviewed the issues raised during the discussion and read the original paper and the reviewers' comments. This paper is indeed on the borderline between acceptance and rejection. Although some relevant distinctions were accepted in the rebuttal, it still has several shortcomings, particularly in the original version. However, the paper's motivation is straightforward and clear, and the paper's approach to the problem, especially in comparisons with other relevant work, demonstrates substantial performance advantages. Furthermore, AC noted the authors' further clarifications, including regarding the relationship with multi-model strategies and comparisons with related work, as well as the setup and fairness analysis of the comparative experiments.

The AC considered the article to be of average acceptance level and requested the authors to update these key points in the final accepted version.
Decision: Accept (poster)