Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective
Authors: Aojun Lu
Hangjie Yuan
Tao Feng
Yanan Sun
Venue: ICML 2025 poster (2025)
Keywords: Continual Learning
Incremental Learning
Primary area: deep_learning->algorithms

Abstract: The quest for Continual Learning (CL) seeks to empower neural networks with the ability to learn and adapt incrementally. Central to this pursuit is addressing the stability-plasticity dilemma, which involves striking a balance between two conflicting objectives: preserving previously learned knowledge and acquiring new knowledge. While numerous CL methods aim to achieve this trade-off, they often overlook the impact of network architecture on stability and plasticity, restricting the trade-off to the parameter level. In this paper, we delve into the conflict between stability and plasticity at the architectural level. We reveal that under an equal parameter constraint, deeper networks exhibit better plasticity, while wider networks are characterized by superior stability. To address this architectural-level dilemma, we introduce a novel framework denoted Dual-Arch, which serves as a plug-in component for CL. This framework leverages the complementary strengths of two distinct and independent networks: one dedicated to plasticity and the other to stability. Each network is designed with a specialized and lightweight architecture, tailored to its respective objective. Extensive experiments demonstrate that Dual-Arch enhances the performance of existing CL methods while being up to 87% more compact in terms of parameters.

Rethinking the Stability-Plasticci Trade-off in Continual Learning from an Architectural Perspective
Aojun Lu'
Hangjie Yuan
Tao Feng Yanan Sun'
Abstract
challenge in CL is catastrophic forgetting (McCloskey & Co- hen, 1989; Goodfellow et al., 2013), i.e., directly updating neural networks to learn new data causes rapid forgetting of previously acquired knowledge. To learn continually with- out forgetting, a neural network must balance plasticity, to learn new concepts, and stability, to retain acquired knowl- edge. However, emphasizing stability can limit the neural network's 's ability to acquire new knowledge, while excessive plasticity can lead to severe forgetting, a challenge known as the stability-plastien dilemma (Grossberg, 2013).
The quest for Continual Learning (CI.) seeks to empower neural networks with the ability learn and adapt incrementally. Central this pursuit is addressing the stability-plasticity dilemma, which involves striking balance between two con- flicting objectives: preserving previously learned knowledge and acquiring new knowledge. While numerous CL methods aim to achieve this trade- off, they often overlook the impact of network architecture on stability and plasticity, restrict- ing the trade-off to the parameter level. In this paper, we delve into the conflict between stabil- ity and plasticity at the architectural level. We reveal that under an equal parameter constraint, deeper networks exhibit better plasticity, while wider networks are characterized by superior sta- bility. To address this architectural- dilemma, we introduce novel framework denoted Dual- Arch, which serves as a plug-in component for CL. This framework leverages the complemen- tary strengths of two distinct and independent net- works: one dedicated to plasticity and the other to stability. Each network is designed with a spe- cialized and lightweight architecture, tailored its respective objective. Extensive experiments demonstrate that Dual-Arch enhances the perfor- mance of existing CL methods while being up to 87 % more compact in terms of parameters.
To enhance CL. most of the research efforts (Li & Hoiem, 2017; Henning et al., 2021; Feng et al., 2022) are centered on developing novel learning methods that achieve better trade-off between stability and plasticity. These methods involve adding loss terms that prevent the model from chang- ing, replaying past data, or explicitly using distinct parts of the network for different tasks, etc (Wang et al., 2024a). In particular, architecture-bast
methods have achieved great success across various CL scenarios (Rusu et al., 2016; Rosenfeld & Tsotsos, 2018; Wang et al., 2023). Character- istically, this type of method introduces an extra part of the network that is solely trained on the current data, which is then integrated with other parts that have been continuously trained on the previous data (Yan et al., 2021; Zhou et al., 2023b). Since a new independent parameter space is used to learn the current data, these methods avoid rewriting the original parameters, thus preserving the old knowledge. In this way, the conflict between stability and plasticity the parameter level can be significantly mitigated.
While studies that focus on expanding and allocating archi- tecture have achieved notable success, research on the basic architectures for CL is still in its infancy. This gap is crucial because, despite the ability of advanced learning methods to optimize parameters effectively, the overall CL perfor- mance remains constrained by suboptimal architectures (Lu et al., 2024). In this regard, certain pioneer works have con- cluded that wider and shallower networks exhibit superior overall CL performance, mainly contributing to enhanced stability (Mirzadeh et al., 2022a;b). However, theoretical analyses and practices (Simonyan & Zisserman, 2014; He et al., 2016; Liang Srikant, 2017: Raghu et al., 2017: Zhao et al., 2024) have demonstrated that deeper networks pos-
1. Introduction
Continual Learning (CL) seeks to enable neural networks to continuously acquire and update knowledge. The primary
CColleee of Computer Science, Sichuan University, Chengdu, China *College of Computer Science and Technology. Zhejiang University, Hangzhou, China 'Department of Computer Science and Technology, Tsinghua University, Beijing, China. Correspon- dence to: Tao Feng <fengtao hi @ com'..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Rethinking the Stability-laastiity Trade-off in Continual Learning from an Architectural Perspective
ResNet-18-66
R ResNet-18-64
45
ResNet-10-96
% Accuracy on the New Task
95
ResNet-10-96
% Average Torgetting
1 lower is better
T higher is better
Param. Level
35
90
Weights for plasticity
Weights for stability
25
85
15
80
5-
75
Arch. Level
2
4
5
6
8 9 10
1 2
4
5
6
7
8
10
Tasks Learned
Tasks Learned (b)
Plastic architecture
S Stable architecture
(a)
Figure 1: Left. (a) The average forgetting and (b) the accuracy on the new task of ResNet-18 and its wider and shallower variant. Details presented in Sec. 3. Right. While existing research mainly optimizes weights (represented by node colors) for the stability-plasticie trade-off at the parameter level, this study proposes novel insight for extending this trade-off to the architectural level.
sess enhanced representation learning ability, indicating the important role of depth in facilitating plasticity. These find- ings raise concern regarding whether there an inherent conflict between stability and plasticity at the architectural level under given parameter count constraint.
tectures (Gou et al., 2021). Consequently, our proposed framework, Dual-Architecture (Dual-Arch), leverages the complementary strengths of two distinct architectures, effec- tively balancing stability and plasticity the architectural level. Extensive experiments show that Dual-Areh markedly enhances CL performance with significantly fewer parame- ters when compared to the baselines. Code is available at ht / tub.. com feam/byy byyyyyyyyyy/yyyyyyyyy. 00yyx66665 666 /D6/Duall Arch.
To investigate this, we conducted a comparison between R8 (He et al., 2016) and its wider yet shallower vari- ant, evaluating their average forgetting and accuracy on the new task. As shown in Fig. 1, ResNet-18 achieves higher accuracy on the new task, indicative of better plasticity, whereas the wider yet shallower variant exhibits lower aver- age forgetting, indicative of greater stability. However, both networks underperform in the other aspect, which indicates there may exist a stability-plastictty dilemma at the architec- tural level as well. Given that existing works (Zhou et al., 2023b; Lu et al., 2024) typically employ a uniform architec- ture for both stability and plasticity, this inherent dilemma may limit CL performance, even when the architecture and parameters are finely optimized.
The contributions of this study are outlined as follows:
We empirically demonstrate that existing architectural designs typically exhibit good plasticity but poor stabil- ity, while their wider and shallower variants exhibit the opposite traits. Based on these findings, we propose a novel insight for exploring the stability-plasticii trade- off from an architectural perspective..
We introduce novel CL framework, Dual-Arch, which employs dual architectures dedicated to stability and plasticity and thus combines both advantages. Further- more, Dual-Arch can be naturally incorporated with various methods as a plug-and-play component.
How to balance the stability and plasticity at the architec- tural level? An intuitive and straightforward solution is to combine two independent models with distinct architec- tures: one dedicated to plasticity and the other to stability. Previous studies on CL have demonstrated that incorporat- ing an auxiliary model, specifically trained the eurrent data, can enhance the plasticity of the primary model (Kim et al., 2023; Bonato et al., 2024). Building on these insights, we extend from an architectural perspective, proposing a novel framework that employs a plastic architecture to ac- quire new knowledge, which is then transferred to the main model with a stable architecture. Specifically, knowledge distillation (Hinton et al., 2015; Romero al., 2014) is utilized for this transfer due to its proven efficacy in trans- ferring knowledge between networks with different archi-
Extensive experiments demonstrate that Dual-Arch is parameter-efficient, i.e., attaining better performance with remarkably reduced parameter count than using a single architecture.
2. Related Work
CL involves letting models sequentially learn series of tasks without with limited access previous tasks. Based whether the task identity is provided or must be inferred, CL can be broadly categorized into three typical scenarios: Task/Class/Dmmin Incremental Learning (IL) (Van de Ven et al., 2022). This study mainly focuses on the most general and realistic scenario of these three, i.e., Class II,, where the
2

---

Rethinking the StabilityPlasstiity Trade-off in Continual Learning from an Architectural Perspective
task identity is unknown during inference (Van de Ven et al., 2022; Wang et al., 2024a).
the influence of various network components and scaling on CL performance, demonstrating that certain architectural de- signs are more CL-friendly than existing ones. Furthermore, it is shown that a well-designed architecture can achieve superior CL performance with a smaller parameter count, which is particularly beneficial for memory vironments (Lu et al., 2024). These studies emphasize that the impact of architectural designs on CL performance is at least significant as that of the learning methods. However, it should be noted that existing studies focus exclusively on the impact of architectures on the overall performance of CL. Our work extends this line of inquiry by highlight- ing the inherent conflict between stability and plasticity at the architectural level and subsequently proposing a novel solution to address it.
2.1. Learning Methods for CL.
To address catastrophic forgetting, the CL community has developed a variety of learning methods aimed at striking balance between stability and plasticity (Feng al., 2025). These methods can be broadly categorized into three main approaches: replay-based methods, regularization-basee methods, and architccture-basee methods. Replay-based methods keep a subset of the previous data information in a memory buffer, which is subsequently exploited approx- imate old data distributions during training (Aljundi et al., 2019; Iscen et al., 2020; Qin et al., 2023). A common and effective implementation involves jointly training the model on both the stored data and the current task data (Robins, 1995; Chaudhry et al., 2018). It is important to note that replay-based methods are often combined with other tech- niques to enhance their effectiveness (Rebuffi et al.. 2017: Yan et al., 2021). Regularization-bass -based methods introduce regularization loss term to balance the learning of new tasks with the retention of old tasks.These methods can be further divided into two subcategories based on the target of regularization (Wang et al., 2024a). The first is weight regularization, which selectively constrains the variation of the network parameters based on their importance to pre- viously learned tasks, e.g., EWC (Kirkpatrick et al., 2017) and SI (Zenke al., 2017). The second is function regular- ization (Bian et al., 2024), which employs techniques such as knowledge distillation (Romero et al., 2014) to maintain consistency between the outputs of the original and updated models, c.g., iCaRL (Rebuffi et al., 2017) and WA (Zhao et al., 2020). Architecture-basee allocate task-specific network parameter space for each task to mitigate inter-task interference, thereby balancing sta- bility and plasticity (Yoon et al., 2017). These methods typically involve dynamically expanding the networks, such as DER (Yan et al., 2021) and MEMO (Zhou et al., 2023b).
en-
2.3. Multi Models for CL.
Various existing studies have proposed employing additional models to enhance CL (Li & Hoiem, 2017; Kim et al., 2023; Bonato et al., 2024). In particular, certain works (Pham et al., 2021; Arani et al., 2022) based complementary learning systems (McClelland et al., 1995; Kumaran et al., 2016) utilize two learners (known as slow and fast learn- ers) with different functions to achieve CL. Among them, MKD (Michel et al., 2024) and Hare& & Tortoise (Lee et al., 2024) employ techniques such as exponential moving aver- ages to integrate knowledge across two models, effectively balancing stability and plasticity. Our proposed solution shares a similar conceptual framework, crafting two inde- pendent learners that assume roles of plasticity and stabil- ity respectively during the CL process. However, unlike these prior efforts that employ a uniform architecture for all models, our study emphasizes the importance of spe- cific architectural designs tailored to each learner. By doing so, our study provides novel insights into more effectively leveraging multiple models for CL.
methods dynamically
3. Architectural Dimensions of Stability and Plasticity
Discussion. In principle, the performance of neural net- works is jointly influenced by their parameters and architec- tures. While the learning methods mentioned above mainly enhance CL by optimizing the parameters or extending pa- rameter space, the suboptimal basic architectures might still limit CL performance. Our study aims to address this by proposing a plug-and-play framework that leverages the complementary strengths of two distinct architectures.
This section presents an investigation of the impact of ar- chitectural designs on the stability and plasticity of neural networks. The primary objective of this investigation is to reveal the conflict between stability and plasticity at the architectural level, with a focus on network scaling.
3.1. Empirical Study Settings
2.2. Neural Architectures for CL
A Architectural Variants. ResNet-18 (He et al., 2016) is selected as the foundational architecture, given its exten- sive utilization in existing CL research (Yan et al., 2021; Goswami et al., 2024). Our primary focus is on examining the impact of depth and width on CL. To this end, we vary
Besides learning methods, there is a body of re- search (Mirzadch et al., 2022a;b; Pham et al., 2022) that concentrates on exploring optimal neural architectures for CL. In particular, ArchCraft (Lu et al., 2024) delves into
3

---

Rethinking the Stability-llastiity Trade-off in Continual Learning from an Architectural Perspective
Table I: The performance (%) of the original ResNet-18 (gray background) and its variants. We report the mean and std of 5 runs with different task orders. Note that the "#p' denotes the parameter counts a single architecture here.
Depth
Width
Penultimate Layer
#P (M)
AAN 1
FAF +
FRF +
18
64
GAP
11.23
86.41
35.7611..6
41.09=2.08
10
96
GAP
11.10
83.4440.84 (-2.97)
33.16 (-2.60)
39.15-12.06 (-1.94)
18
64
4 x AvgPool
11.38
84.64+0.43 (-1.77)
34.17=2.03 (-1.59)
39.97+2271 (-1.12)
26
52
GAP
11.56
86.68 (+0.27)
36.02+ (+0.26)
41.20 (+0.11)
the number of layers and initial channel counts in ResNet-18 while maintaining a relatively constant total parameter count. Additionally, we conduct an extended study to investigate the effect of pre-classification width. This involves replac- ing the global average pooling (GAP) layer with a 4 average pooling layer with stride of 3, thereby producing an output feature map of size 2 x 2.
marginally improved plasticity with reduced stability, These results reveal an inherent trade-off between stability and plasticity at the architectural level, governed by architec- tural design choices within specific parameter limits.
4. Dual-Architecttum for CL
In this section, we propose Dual-Arch, a framework that can be easily plugged in existing CL methods, to address the stability-plasticity dilemma at the architectural level. Specifically, we will provide an overview of the Dual-Arch framework and detail its learning algorithm.
Implementation Setup. subset of ImageNet (Deng et al., 2009), known as ImageNet106 (Rebuffi et al., 2017), is utilized as the dataset. It is partitioned into 10 incremental tasks, each comprising 10 classes. All models are trained using iCaRL (Rebuffi et al., 2017), classic learning method in the CL field, with fixed memory size of 2,000 exemplars.
4.1. Preliminaries
Evaluation Metrics.
Before further description, some definitions related the CL are presented. CL aims learn from dynamic data stream. Following convention (Zhou et al., 2023a), we consider a sequence of K tasks (also known as steps) [D1, D,...., Dx} without overlapping classes. Specifi- cally, Dx ~ (.k. V/}} represents the data of the k-th step, containing N₁ classes. Here, X₁ denotes the set of samples, and Vi denotes their respective labels. At the k-th step. the CL model is trained Dirain and then tested which denotes the joint test dataset from task 0 to task k. For replay- methods, parts of data from previous tasks are preserved and incorporated into the Dirain
To evaluate plasticity, we measure the Average Accuracy on New tasks (AAN) across all incremental steps, where higher values indicate greater plasticity. For stability assessment, employ two complementary metrics: Average Forget- ting (AF) and Relative Forgetting (RF). AF measures the absolute stability after learning the k-th task, calculated as ag). where as denotes the current
AF, - - E accuracy on task b and ag represents its peak past accuracy. RF (Wang et al., 2024b) provides a normalized stability measure to avoid penalizing highly plastic models, defined as RF = -- E _t ). We specifically use the final- task values (FAF and FRF) as overall stability indicators, with lower values corresponding to better stability.
Diest
In the traditional CL paradigm using a single learner, the training loss at the k-th step can be formulated as:
3.2. Empirical Study Results
Lsingle = LoF + LOL.
(1)
The performance comparison between ResNet-18 and its variants, under comparable parameter counts (within +3% margin), is summarized in Tab. 1. We observe that making the network shallower but wider decreases both AAN and forgetting metrics (FAF and FRF). indicating enhanced sta- bility but reduced plasticity. This trend similarly emerges when increasing the creccaasifiiattio width through penul- timate layer modification, further confirming that wider ar- chitectures improve stability while compromising plasticity. Conversely, deeper yet narrower variant exhibits slight increase in both AAN and forgetting measures, suggesting
where the loss term LCE is the classification loss calculated using a cross-entropy loss function, and Leg, is specifically defined by the particular used CL methods. Specifically, we consider the CL learner parameterized by weights 0₂ and we use o(x) to indicate the output logits of the learner on input the LoE is defined as:
exp(oy)
LCE(D, y: log
(2)
Em=1 exp(om)
N'

---

Rethinking the Stability-Plastiiity Trade-off in Continual Learning from an Architectural Perspective
Traditional CL Paaddggm
D ata
Input
0t:1
O Ottimize
Learner
Single
Plastic Learner
CL with Dual-Arch
Modify for plasticity and fewer params.
Modify for stability and fewer params.
(1)
Stable Learner
Input
Data
Optimize
Memory Relay
(2)
Input
Regularization
Optimize
Distillation
Dynamic Architecture
0t-1
Figure 2: The formulation of the traditional CL paradigm and CL with Dual-Arch (Ours). Dual-Areh (1) employs two independent learners that are designed by modifying the traditional single learner, and (2) utilizes the stable learner to perform CL with the assistance of the plastic learner. Note that Dual-Arch can be effortlessly combined with existing CL methods (denoted by the dotted lines or box).
4.2. The Formulation of Dual-Arch
roring the training process of the single learner within the traditional CL paradigm. In particular, for replay-based methods, the replay buffer is concatenated with the training data for both the stable and plastic learners.
The overall framework of Dual-Arch is illustrated in Fig. 2. Unlike the existing CL paradigm which relies on sin- gle learner, the Dual-Arch framework distributes the roles of plasticity and stability across two distinct models: the plastic learner and the stable learner. Inspired by existing research (Kim et al., 2023) that employs auxiliary models framework designates the sta-
4.3. Architectures for the Stable and Plastic Learners
This subsection presents the specific architectural designs tailored to the stable and plastic learners, with the objective of achieving superior CL, performance while minimizing parameter counts. Building upon the insights from Sec. employ wide and shallow architecture for the stable learner, denoted as Sta-Net, and a deep and thin architec- ture for the plastic learner, denoted as Pla-Net. Following standard practices (Masana et al., 2022; Goswami et al., 2024), we have chosen ResNet-18 as the foundation for crafting both architectures. Specifically, Sta-Net retains the same width as ResNet-18 but incorporates only half many residual blocks. Furthermore, modify the GAP layer of Sta-Net to produce output feature map of size 2 x 2 in- stead of the original x 1, thereby increasing the width of the classifier. To design Pla-Net, we maintain the depth of ResNet- 18 while reducing its width from 64 42 to align with the parameter count of Sta-Net.
to enhance plasticity, ble learner as the main model, with the plasticity leamne serving an auxiliary model. Throughout the learning process, the plastic learner is dedicated to the extraction of new knowledge, allowing for the potential forgetting of previous knowledge. Conversely, the stable learner is re- sponsible for retaining existing knowledge while integrating new knowledge with the assistance of the plastic learner.
Dual-Arch allows the combination of the strengths of both stable and plastic architectures by employing corresponding architectures for the two learners. Specifically, these archi- tectures are designed through targeted modifications to the original one, with the objective of enhancing plasticity or stability. Additionally, overcome the increased memory consumption associated with incorporating additional model, we concurrently reduce the parameter counts for both learners. It is also worth highlighting that the Dual- Arch framework designed to facilitate integration with a variety CL methods, serving as a plug-and-play compo- nent. This integration can be easily achieved by applying these CL methods when training the stable learner, mir-
4.4. Learning Algorithm Dual-Arch
The learning process of the Dual-Arch framework involves training the plastic and stable learners in sequence. In the
5

---

Rethinking the StabilityPlasstiity Trade-off in Continual Learning from an Architectural Perspective
initial stage of the learning process, our framework trains the plastic learner as new task emerges. At this stage, the primary objective is to facilitate the acquisition of new knowledge, without consideration of the maintenance of previously acquired knowledge. Consequently, the train- ing objective of the plastic learner is simplified to mini- mize the classification loss on the current training data, i.e., plaastic = Lcg. Subsequently, the stable learner is trained to integrate the existing knowledge with that acquired by the plastic learner. This process entails the transfer of recently acquired knowledge from the plastic learner to the stable learner via knowledge distillation. Specifically, a distillation loss term is incorporated into the training objective of the stable learner, to align the logit outputs between the stable and plastic learners. Following convention (Hinton et al., 2015), hard label loss (i.e.. cross-entropy loss) is also em- ployed to minimize the discrepancy between the predictions of the stable learner and the actual labels of the training data. Moreover, established CL methods are implemented during this phase to facilitate the retention of previous knowledge, which can also be expressed as a loss term. In light of the aforementioned considerations, the total learning target of the stable learner can be formulated as:
learner is trained using the composite loss Lstable (Eq. 3), which incorporates knowledge from the teacher model. Af- ter each task, the stable learner is evaluated on all learned tasks (1 to k) to assess its CL performance.
Algorithm 1 Training Procedure of Dual-Arch
1: Input: Stable learner weights On. Plastic learner
weights do
2: Output: Optimized stable learner weights 0x
3: for task & = 1, 2, K do
4: Train Φ 1 for E epochs using normal classification
loss LCF on task to obtain Φx
Freeze @% and save as teacher model
Train 0k-1 for E epochs using Lstable (Eq, (3)) on
task is to obtain 0x
7: Evaluate 02 on tasks (1 to k)
8:
end for
5. Experiment
5.1. Experiment Setup
Benchmark. Following convention (Rebuffi et al., 2017),
Lstable = aLCC + (1=D)LKD + Let.
(3)
We choose CIFAR (Krizhevsky et al., 2009) and Ima- ge (Deng et al., 2009) for evaluation. Both datasets are divided into 10 tasks of 10 classes each and 20 tasks of 5 classes each to construct four benchmarks: CIFAR100/10, CIFAR 100/20, ImageNet100/10, and ImageNet100/200
where LKD denotes the distillation loss and a is a parameter that balances the weight of LKD and Len. We set the default value of to 0.5, following (Hinton et al., 2015).
Within Dual-Arch the distillation loss LKD is employed to enhance the plasticity of the stable learner, which in- volves enabling it to learn from the soft outputs of the plastic learner. Specifically, the LKD is calculated by measuring the Kullback- Leibler divergence between the soft outputs of the teacher model (i.e., the plastic learner) and those of the student model (the stable learner) on the current data. Let T denote the teacher model, and S denote the student model, the L.D is defined as:
Baselines. To assess the efficacy of our proposed method, we integrate it into five distinct CL approaches spanning the three major categories: replay-based, regularizationn based, and architecture-
methods. These methods include iCaRL (Rebuffi et al., 2017), WA (Zhao al., 2020), DER (Yan et al., 2021), Foster (Wang et al., 2022), and MEMO (Zhou et al., 2023b). Specifically, we compare the performance of Dual- with that of the original ResNet- 18 to evaluate the enhancements provided by our method. We also select ArchCraft (Lu et al., 2024) as a baseline, which employs single CL-friendly architecture to improve CL performance, to show the benefits of dual architectures.
N² LKD = - Σ P log Pss
(4)
Implementation Setup. For all experiments, we train all models for 200 epochs in the first task and 100 epochs in the subsequent tasks. The learning rate starts from 0.1 and gradually decays with a cosine annealing scheduler. A fixed memory size of 2,000 exemplars is utilized for all replay-based methods during the learning process. Given the significant impact of hypar-parameters on CL (Cha & Cho, 2024; Mirzadeh et al., 2020), the hyperparameters for all methods adhere to the settings in the open-source library PyCIL (Zhou et al., 2023a) to ensure a fair comparison. Following convention (Mirzadeh et al., 2022b; Zhou et al., 2023a), the first convolution layer and following max pool- ing layer of networks are replaced by 3 convolution
where Pr and Ps represent the soft outputs of the teacher and student models. These soft outputs are derived by applying the SoftMax function to transform the output logits of these models, i.e., Or and Og, into probability distributions. Specifically, Pr = SoltMax(Or/UU and Ps = SoftMax(Os/tt) where is the temperature factor that controls the smoothness of the soft outputs.
The detailed training procedure of the proposed framework is summarized in Alg. 1. For each task k. the plastic learner is first trained to convergence using the standard classifi- cation loss LCg. Its optimized weights are then frozen and preserved as teacher model. Subsequently, the stable
6

---

Rethinking the Stability-Passtiiity Trade-off in Continual Learning from an Architectural Perspective
Table 2: The LA, AIA and FAF (%) using five state-of-the-arr CL methods. '#P" represents the parameter counts of all used models. 'Improvement" represents the boost of Dual-Arch towards original methods. Note that the parameter counts of DER and MEMO vary from incremental settings, resulting in two values for '/20' and 1/10'. Bolded indicates best.
CIFARI00/20
CIFAR100/10
ImageNer100/20
ImageNer100/10
Method
HP (M)
LA
AIA
FAF
LAt
AIAT
FAF
LAt
AIAT
FAFU
LAt
AIAT
FAFL
iCaRL
22,4
49.78
65.63
33.33
54.87
68.30
27.76
46.22
63.89
41,05
51,74
68.47
35.91
w/ ArchCraft
17.4
52.60
68.71
55.52
69.62
45.12
63.98
52.46
68.42
w/ Ours
15.1
52.53
67.80
29.49
57.69
70.40
23.63
47.22
65.06
35.66
54.84
69.37
28.22
Improvement
4 33%
+2.75
42.17
13.84
+2.82
+2.10
J4.13
+1.00
+1.17
45.39
43.10
+0.90
47.69
WA
22.4
46.78
62.75
19.05
56.98
69.16
23.53
46.98
65.76
39.05
57.64
71,20
28.27
w/ ArchCraft
17.4
53.23
69.19
59.79
71.40
49.94
67.20
58.86
71.56
w/ Ours
15.1
55.02
68.84
24.91
59.78
71.57
17.91
52.84
68.79
31.73
60.84
72.57
24.53
Improvement
3.3%
+8.24
+6.09
15.86
+2.80
+2,41
45.62
+5.86
+3.03
47.32
+3.20
+1.37
43.74
DER
224.4/112.2
58.39
70.19
25.63
61.83
72.48
22.13
64.32
74.91
20.51
67.40
75.93
15.22
w/ ArchCraft
173.5/86.8
61.65
73.59
63.94
74.84
63,98
74.50
68.34
77.26
w/ Ours
106.9/55.9
64.08
73.86
20.08
66.22
75.08
17.73
65.40
75.17
16.97
68.52
77.49
12.96
Improvement
4. 525%/50%
+5.69
+3.67
15.55
+4.39
+2.60
|4.40
+1.08
+0.26
43.54
+1,12
+1.56
J2.26
Foster
22.5
49.99
63.39
35.03
58.67
69.95
27.39
54.74
66.77
34.67
62.88
71.09
25.18
w/ ArchCraft
17.5
$7.22
69.99
61.44
72.54
$4.32
66.41
61.94
71.16
w/ Ours
15.4
57.69
71.01
23.75
61.23
73.22
18.23
55.20
67.63
32.42
63.24
72.42
25.04
Improvement
32%
+7.70
+7.62
411.28
+2.56
+3.27
49.16
+0.46
+0.86
42.25
+0.36
+1.33
J0.14
MEMO
171.7/87.2
52.10
67.60
35.71
58.46
70.71
27.99
56,10
69.13
29.64
61.64
73.31
21.87
w/ ArchCraft
126.6/64.6
57.28
72.07
61.93
73.30
57.46
70.54
62.46
74.01
w/ Ours
101.1/53.1
62.39
72.69
24.09
65.35
74.34
20,82
60.26
72.53
24.59
65.40
75.54
16.53
Improvement
4 441%//39%
+10.29
+5.09
411.62
+6.89
+3.63
47.17
+4.16
+3.40
45.05
+3.76
+2.23
45.34
layer with a stride of 1 for CIFARIO0.
Moreover, Dual-Areh outperforms Arch-Craft in most cases, underscoring the advantages of dual architectures over a sin- gle, CL-friendly architecture. In conclusion, Dual-Arch emerges valuable complement to existing CL methods, enhancing both effectiveness and efficiency.
Evaluation Metrics. The overall performance of CL is measured by two metrics: the Last Accuracy (LA) and the Average Incremental Accuracy (AIA). The LA is the total classification accuracy after the last task, which reflects the overall accuracy among all classes. Further, the AIA denotes the average classification accuracy over all tasks, which re- flects the performance across all incremental steps. The higher LA and AIA, the better overall CL performance. Let K be the number of tasks, these two metrics are defined as LA Ax. ALA स Ik_ AAg. where As, represents clas- sification accuracy measured on the test set that covers all tasks learned up to and including the b-th task. Additionally, we report the FAF to quantify catastrophic forgetting.
5.3. Ablation Study
In this subsection, we present the results of our ablation study to show the significance of employing dual networks in conjunction with dedicated architectures. To simplify, we select the CIFAR-100/10 as a representative dataset and utilize AIA as the performance metric for our analysis.
As displayed in Tab. 3, we examine the effects of removing two pivotal components from our method. Specifically, we present the outcomes of employing only Sta-Net to underscore the necessity of the detlonst framework, along with results from using alternative architectures for the two learners, highlighting the importance of tailored designs. We observe from Tab. 3 that the absence of a plastic learner leads a decrease in AIA by an average of 2.63%. Similarly, employing non-specialized architectures for two learners within Dual-Arch results in lower performance, with the AIA declining by an average of 1.74%, 0.65%, and 1.68%. These results clearly demonstrate the benefits of
5.2. Overall Results
Tab. 2 presents the comparative performance of Dual-Aret using five state-of-the-art CL methods. The results demon- strate that across various methods, datasets, and incremental steps, the integration of Dual-Arch consistently outperforms the baseline that employs R as single learner. In particular, adopting Dual-Areh leads to maximum improve- ments of 10.29% in LA and 7.62% in AIA, while simul- taneously reducing the parameter counts by at least 33%.
7

---

Rethinking the Stability-Plasticiy Trade-off in Continual Learning from an Architectural Perspective
Table 3: The ablation study results using five CL methods. We report the mean of runs with different initializations. denotes performing CL with single learner. Bolded indicates the best.
Stable
Plastic
AIA (%) on CIFARIO0/1
Learner
Learner
iCaRL
WA
DER
Foster
MEMO
Average
Sta-Net
Pla-Net
70.21
7.15
75.26
73.18 10.04
74.44
72.92
Sta-Net
None!
66.69
69.
72.47
70.84
72. 1 +
70.29 (-2.63)
Pla-Net
Pla-Net
69.57=t0.10
0.09
74.64+00.28
01.92
69.80=10.15
71.18 (-1.74)
Sta-Net
Sta-Net
69.27=00.6
7..28
7..30
72.65
73.77=t0.05
72.27 (-0.65)
Pla-Net
Sta-Net
69.8
70.31
74.25+
71.86=10.06
69.92-t0.12
71.24 (-1.68)
each component in our proposed solution.
tional efficiency at test time (255M vs. 558M).
However, despite the reduced FLOPs, the training time increases due to the non-parallclizable nature of training the two learners. As illustrated in Table 4, Dual-Arch incurs a 1.39x 1.77x overhead in training time compared to the baselines. This represents limitation of our approach.
76-
15.201
ALA
31.5M
-55.3M
86 BN
0.00%
72-
4.2M
17 2.5M
1.8% higher
14.0N
57% fewer
haterr
70-
22.5M
72-
28.2N
63.2M
11
60
12.7M
Dual-Arch (Ours)
Duai-Arch (Durs)
ArthCrrff
ResNitt11
66
ArchCroft
ResNet-18
Table 4:
Training time (minutes) comparison between Dual- Arch and baselines on CIFAR-100/10
100
125
10
15
20
25
Number of Parameees (M)
Number of Parameters (M)
(a) Results with DER
(b) Results with Foster
Method
iCaRL
WA
DER
Foster
MEMO
Figure 3: Performance of CL vs. Number of Parameters using DER and Foster on CIFAR100/10.
Original (ResNet-18)
40
39
74
93
49
Dual-Arch (Ours)
70
69
106
129
85
5.4. Parameter Efficiency Analysis
5.6. Analysis on the Stability-Plastiiity Trade-off'
To assess the parameter efficiency more comprehensively, we vary the parameter counts of Dual-Areh and ResNet- 18 by reducing the network width by a quarter and half. As il- lustrated in Fig. 3, the Dual-Arch series significantly outper- forms the baseline in terms parameter efficiency. Specif- enhance AIA by 0.90% and 1.94%
To further scrutinize the effectiveness of Dual-Arch in com- bining the strength of both architectures, we compare it to a single learner with one of the architectures (i.e., Pla-Net or Sta-Net). To simplify, we choose the top-performing ap- proach DER the used CL method. We observe from Fig. (a) that Dual-Arch achieves the best overall performance in CL. Moreover, as illustrated in Fig. (b) and (c), the single learner either forgets severely previous tasks (Pla-Net) or underperforms on new ones (Sta-Net), whereas Dual-Arch demonstrates competitive performance in both aspects. This result indicates that Dual-Areh combines the advantages of both types of architecture, leading trade-off between stability and plasticity the architectural level.
ically, Dual-Arch when using DER and Foster as the CL method, respectively, while simultaneously reducing parameter counts by 87% and 81% %. Additionally, Dual-Arch surpasses ArchCraft, a state-of-thhe -oet---rr-hee-tttt solution that enhances parameter efficiency CL by recrafting the network architecture. These empir- ical results highlight the potential of Dual-Arch to signifi- cantly benefit CL in memory-restricted scenarios.
5.5. Computation Efficiency Analysis
5.7. Analysis on Bias-correction
We note that although Dual-Arch involves training two mod- els, the total computational cost, measured FLOPs, re- mains lower than that of the baselines. For instance, CIFAR-100, the FI.OPs for Sta-Net and Pla-Net are 255M
In Class IL. the task-recency bias major cause catas- trophic forgetting, where models tend to misclassify in- stances from earlier tasks belonging to more recently introduced classes during inference (Masana et al., 2022; Zhao et al., 2020). To discover the reasons why Dual-Arch benefits CL, we further evaluate its effectiveness in miti- gating the task recency bias. Specifically, we present the task confusion matrices for the Dual-Arch and the baseline
and 24 respectively, yielding combined total of 496M. In comparison, ResNet-18 requires approximately 558M FLOPs. Furthermore, during inference, only the stable learner is utilized, enabling Dual-Arch to achieve computa-
8

---

Rethinking the Stability-Plasticcg Trade-off in Continual Learning from an Architectural Perspective
90
higher is better
Pla-Net
Sta-Net
30
4 lower is better
Pla-Net Sta-Net
% Accuracy on the New Task
90
T higher better
Sta-Net
Pla-Net
% Average Accuracy
Dual-Arch(ours)
Average Forgeting
Dual-Arch(ours)
Dual-Arch(ours)
85
25
80
80
20
70
75
15
60
70
10
50
65
60
40
2
3
4 5
6
7
8
9
10
2
7 Tasks Learndd
5
6
10
2
3
4
5
6
7
8
9
10
Tasks Learned
Tasks Learned
(a) Average Accuracy
(b) Average Forgetting
(c) Accuracy New Task
Figure 4: The performance of Dual-Arch and two baselines using DER CIFAR100/10.
1 3844
10
12
8
14
6
24
107
431
450
10
19
6
25
18
32
106
323
architectures, such as Vision Transformers (ViTs). To val- idate this, we conduct experiments to transfer our method to Sep (l.i al., 2022), training all models from scratch and using ImageNet100//1 as the benchmark. It important to note that the training settings remain consistent with those described in Sec. 5. with adjustments made to the learning rate and optimizer to align with the official implementation of SiT (Li et al., 2022). The results, summarized in Tab. 5, demonstrate that Dual-Arch consistently improves the CL. performance of SepViT, underscoring potential generalizability to other architectures.
241
11
16
18
30
32
173
457
522
20
12
22
14
24
46
185
329
388
16
11
11
15
20
129
399
15
484
17
14
17
14
31
267
4 13
20
318
19
14
23
44
106433
430
11
17
425
22
18
22
57
125 285
True Task
12
416
13
20
52
111
356
True Task
11
13
870
21
21
72
106-291
10
16
301
44
91
414
6
11
9
27
999
42
96
120
200
13
452
55
130
7
10
7
16
55
115
8
5
20
21
29
480
02
334
2
18
20
32
500
257
9
13
17
886
253
9
11
7
0
12
5
13
772 150
10
0
18
074
10
888
6
8
10
3
4
5
Predicted Task ID
Predicted Task
(a) iCaRL.
(b) iCaRL Dual-Areh
S1S
11
34
21
55
82 158
212
21
23
25
17
29
15
44
2
43
29
32
46
25
55
52
145
148
2
42
33
4
33
67
29
51
38
74
103
3
15
17
19
19
32
107
305
3
30
49
300
33
25
29
14
4B
TIUe Task
4
25
24
545
26
14
52
05
126
33
83
57
43
24
27
42
85
Table 5: The LA and AIA (%) using SepVit Ima- geNet100/10. '#P' represents the parameter counts of all used networks. Bolded indicates the best.
11
26
32
00
27
29
35
67
221
5
30
23
34
34
884
32
17
36
6
21
20
16
574
64
58
126
6
57
25
10
31
75
523
64
13
28
20
10
41
616
48
66
95
7
32
88
17
50
41
47
639
40
10
38
58
71
77
968
102
8
26
20
12
37
54
78
9
25
28
14
23
40
47
9
62
42
42
20
70
21
44
10
14
10
10
22
32
22
782
10
31
10
28
30
33
39
Method
#P (M)
LA
AIA
6
7
8
10
2
3
4
iCaRL.
7.57
43.08
60.62
Predicted Task
ID
Predicted Task
w/ Ours
5.32
46.34 (+3.26)
63.09 (+2.47)
(c) WA
(d) WA Dual-Arch
WA
7.57
38.40
57.67
Figure 5: Task confusion matrices after learning the final task w/ and w/o Dual-Arch plugged in on CIFAR100/10.
w/ Ours
5.32
44.28 (+5.88)
61.15 (+3.48)
which employs a single ResNet-18 Fig. 5. From Fig. 5 (b) and (d), we observe that the integration of Dual-Arch fa- cilitates more precise determination of the correct task ID, thereby reducing inter-task classification errors. Notably, Dual-Arch significantly diminishes the misclassification of data from earlier tasks (c.g., task 1) as belonging to recently learned tasks (e.g., task 10). These observations indicate that Dual-Arch can effectively reduce the task-recency bias, thus mitigating catastrophic forgetting.
6. Conclusion
In this paper, we point out the stability-plastice dilemma at the architectural level and further introduce Dual-Areh, novel CL framework, to address it. Dual-Arch operates at an architectural level, complementary most existing CL methods that focus on parameter optimization, thereby serving plug-in component for enhancing CL. Our extensive experiments demonstrate that Dual-Arch consis- tently outperforms the baselines while significantly reducing the parameter counts. We hope this work inspires further study on exploring a better trade-off between stability and plasticity from an architectural perspective.
5.8. Validation on Vision Transformers
While our study primarily focuses on ResNet. the insights presented in this paper are potentially applicable to other

---

Rethinking the StabilityPlassticty Trade-off in Continual Learning from an Architectural Perspective
Impact Statement
Feng, T., Li, W. Zhu, D., Yuan, H., Zheng, W., Zhang, D., and Tang, J. Zeroflow: Overcoming catastrophic forgetting is easier than you think. arXiv preprint arXiv:2501.01045, 2025.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Goodfellow, J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y. An empirical investigation of catastrophic for- getting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.
Acknowledgement
This work was supported by National Natural Science Foun- dation of China under Grant 62276175 and Innovative Re- search Group Program of Natural Science Foundation of Sichuan Province under Grant 2024NSFTD0035.
Goswami, D., Soutif-Cormerass, A., Liu, Y., Kamath, S., Twardowski, B., and van de Weijer, J. Resurrecting old classes with new data for exemplar continual learn- ing. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
References
Aljundi, R., Lin, M., Goujaud, B., and Bengio, Y. Gradi- ent based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019.
Gou, J., Yu, B., Maybank, S. J., and Tao, D. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):189-11819 2021.
Grossberg, S. Adaptive resonance theory: How a brain learns to consciously attend, learn, and recognize a chang- ing world. Neural networks, 37:1-47, 2013.
Arani, E., Sarfraz, F., and Zonooz, B. Learning fast, learning slow: general continual learning method based on com- plementary learning system. In International Conference Learning Representations, 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Bian, A., Li, W., Yuan, H., Wang, M., Zhao, Z., Lu, A., Ji, P., Feng. T., et al. Make continual learning stronger via c-flat. NeurIPS, 2024.
Henning, C., Cervera, M., D' Angelo, F., Von Oswald, J., Traber, R., Ehret, B., Kobayashi, S., Grewe, B. F., and Sacramento, J. Posterior meta-replay for continual leam- ing. Advances in neural information processing systems, 34:14135-14149 2021.
Bonato, J., Pelosin, F., Sabetta, L.., and Nicolosi, A. Mind: Multi-task incremental network distillation. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11105-11113, 2024.
Buzzega, P., Boschini, M., Porrello, A., Abati, D., and Calderara, S. Dark experience for general continual learn- ing: strong, simple baseline. Advances in neural infor- mation processing systems, 33:15920-15930, 2020.
Hinton, G.. Vinyals. O., and Dean, J. Distilling the knowledge in neural network. arXiv preprint arXiv: 05031500333333 2015.
Iscen, A., Zhang, J., Lazebnik, S., and Schmid, C. Memory- efficient incremental learning through feature adaptation. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow; UK, August 23-28, 2020, Proceedings, Part XVI 16, pp. 699-715. Springer, 2020.
Cha, S. and Cho, K. Hyperparameters in continual learning: a reality check. arXiv preprint arXiv:2403.00000 2024.
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with a-gem. arXiv preprint arXiv: 1042.0040. 2018
Kim, S., Noci, L., Orvieto, A., and Hofmann, T. Achieving a better stability-llasticiy trade-off via auxiliary networks in continual learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11930-11999 11999, 2023.
Deng, J., Dong, W., Socher, R., Li, K., and Fei-Fei, L. Imagenet: A lcrge-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. lece, 2009.
Feng, T., Wang, M., and Yuan, H. Overcoming catastrophic forgetting in incremental object detection via elastic re- sponse distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9427-9436, 2022.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska- Baaoskka-Brwiskka.... A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13)33521-23 -3526, 2017.
10

---

Rethinking the Stability-Plasticiy Trade-off in Continual Learning from an Architectural Perspective
Krizhevsky, A. et al. Learning multiple layers of features from tiny images. 2009.
Mirzadeh, S. I., Chaudhry, A., Yin, D., Hu, H., Pascanu, R., Gorur, D., and Farajtabar, M. Wide neural networks forget less catastrophically. In International Conference on Machine Learning, pp. 15699-15711 PMLR, 2022a.
Kumaran, D., Hassabis, D., and McClelland, J. L. What learning systems do intelligent agents need? complemen- tary learning systems theory updated. Trends in cognitive sciences, 20(7):512-555 534, 2016.
Mirzadeh, S. I., Chaudhry, A., Yin, D., Nguyen, T., Pascanu, R., Gorur, D., and Farajtabar, M. Architecture matters in continual learning. arXiv preprint arXiv:2202.00275. 2022b.
Lee, H., Cho, H., Kim, H., Kim, D., Min, D., Choo, J., and Lyle, C. Slow and steady wins the race: Maintain- ing plasticity with hare and tortoise networks. In Inter- national Conference on Machine Learning, pp. 26416- 26438. PMLR, 2024.
Pham, Q. Liu, C.,, and Hoi, S. Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34:16131-16144, 2021.
Pham, Q., Liu, C., and Steven, H. Continual normaliza- tion: Rethinking batch normalization for online continual learning. In International Conference on Learning Rep- resentations, 2022.
Li, W. Wang, X., Xia, X., Wu, J., J.,J,, Xiao, X., Zheng, M., and Wen, S. Sepvit: Separable vision transformer. arXiv preprint wrXiv:2233.53888 2022.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947 2017.
Qin, H., Ding, Y., Zhang, X., Wang, J., Liu, X., and Lu, J. Diverse sample generation: Pushing the limit of genera- tive data-free quantization. TPAMI, 2023.
Liang. S. and Srikant, R. Why deep neural networks for function approximation? In International Conference on Learning Representations, 2017.
Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., and Sohl- Dickstein, On the expressive power of deep neural networks. In international conference on machine learn- ing, pp. 2847-2854. PMLR, 2017.
Lu, A., Feng, T., Yuan, H., Song, X., and Sun, Y. Revisiting neural networks for continual learning: An architectural perspective. In IJCAI, pp. 4651 2024.
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A. D., and Van De Weijer, J. Class-incremental learning: survey and performance evaluation on image classifica- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-5533, 2022.
Robins, A. Catastrophic forgetting, rehearsal and pseudore-
hearsal. Connection Science, 7(2):123-148 1995.
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6553 2014.
McClelland, J. L., McNaughton, B. L., and O'Reilly, R. C. Why there are complementary learning systems in the hip- pocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Rosenfeld, A. and Tsotsos, J. K. Incremental learning through deep adaptation. IEEE transactions on pattern analysis and machine intelligence, 42(():651-663, 2018.
Rostami, M., Kolouri, S., and Pilly, P. K. Complemen- tary learning for overcoming catastrophic forgetting us- ing experience replay. In Proceedings of the 28th Inter- national Joint Conference on Artificial Intelligence, pp. 3339-3345, 2019.
McCloskey, M. and Cohen, N. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, vol- ume 24, pp. 109-165. Elsevier, 1989.
Michel, N., Wang, M., Xiao, L., and Yamasaki, T. Rethink- ing momentum knowledge distillation in online continual learning. In International Conference on Machine Learn- ing, pp. 35607-35622. PMLR, 2024.
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J.. Kavukcuoglu, K., Pascanu, R., and Had- sell, R. Progressive neural networks. arXiv preprint arXiv: 1oXw:1606.04671,, 2016.
Mirzadeh, S. I., Farajtabar, M., Pascanu, R., and Ghasemzadeh, H. Understanding the role of training regimes in continual learning. Advances in Neural Infor- mation Processing Systems, 53:7308-7320, 2020.
Sarfraz, F., Arani, E., and Zonooz, B. Synergy between synaptic consolidation and experience replay for general continual learning. In Conference on Lifelong Learning Agents, pp. 920-936. PMLR, 2022.

---

Rethinking the Stability Trade-off in Continual Learning from an Architectural Perspective
Serra, J., Suris, D., Miron, M., and Karatzoglou, A. Over- coming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pp. 4548 4557. PMLR, 2018.
Zhou, D.-W,, Wang, E-Y., Ye, H.-J,, and Zhan, D.-C. Pycil: A python toolbox for class-incremental learning, 2023a.
Zhou, D. Wang. Q.-W., Ye, H.-J., and Zhan, D.-C. A model or 603 exemplars: Towards memory- class- incremental learning. In ICLR, 2023b.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for lcage-scale image recognition. arXiv preprint arXiv: 1409, 1556, 2014.
Van de Ven, G. M., Tuyrelaars, T., and Tolias, A. S. Three types of incremental learning. Nature Machine Intelli- gence, 1882)1118511197, 2022.
Wang, F.-Y., Zhou, D.-W., Ye, H.J.,, and Zhan, D.-C. Foster: Feature boosting and compression for class-incremental learning. In European conference on computer vision, pp. 398 414. Springer, 2022.
Wang, F--Y, Zhou, D.-W., Liu, L., Ye, H.-J,, Bian, Y., Zhan, D.-C., and Zhao, P. Beef: Bi-compatible class- incremental learning via energy energy-based expansion and fusion. In The Eleventh International Conference on Learning Representations, 2023.
Wang, L., Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Wang, M., Michel, N., Xiao, L.. and Yamasaki, T. Im- proving plasticity in online continual learning via col- laborative learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13460-23469 2024b.
Yan, S., Xie, J., and He, X. Der: Dynamically expandable representation for class incremental learning. In Proceed- ings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 3014-3023, 2021.
Yoon, J., Yang, E., Lee, J., and Hwang, S. J. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv: / 708.0/547, 2017.
Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995, PMLR, 2017.
Zhao, B., Xiao, X., Gan, G., Zhang, B., and Xia, S.-T. Main- taining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 13208- 13217, 2020.
Zhao, Z., Deng, I.,, Bai, H., Cui, Y., Zhang, Z., Zhang, Y., Qin, H., Chen, D., Zhang, J., Wang, P., and Gool, L. V. Image fusion via vision-language model. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
12

---

Rethinking the StabilityPlasstcity Trade-off in Continual Learning from an Architectural Perspective
A. Appendix
A.1. Implementation Details
We employ the same data augmentation as in PyCIL (Zhou et al., 2023a) for all experiments. For the experiments in Section 3, we report results in different task orders using seeds 2, 4, and 5. For other experiments, we adhere to a fixed seed of 1993, consistent with established conventions (Rebuffi al., 2017; Zhou et al., 2023a). We utilize temperature factor for Dual-Arch of i 4 for CIFAR 100 and =3 3 for ImageNet-100.
Details about Parameter Counts. We compute the sum of the parameter counts of all used models for each incremental step and report their peak values throughout the CL process as the final result. For instance, for iCaRL, we sum the parameter counts of the current and last models. For iCaRL. with Dual-Arch, we sum the parameter counts of the plastic learner, the current stable learner, and the last stable learner. All results are detailed in Tab. 2 of the main paper. Moreover, should be noted that the parameter counts vary slightly between the CIFARI00 and ImageNet100, and we report all results based on CIFARI00.
A.2. Architectural Dimensions of Stability and Plasticity in MLP
We further investigate the impact of network width (i.e., the number of neurons in the hidden layer) and depth (i.e., the number of layers) on the stability and plasticity of the MultiLayer Perceptron (MLP). Following HAT (Serra et al., 2018), employ width of 800 and depth of 4 as the default design for the MLP. Additionally, we design a wider yet shallower variant and deeper yet thinner variant, both with parameter count comparable to the default design. We evaluated all MLPs on the split MNIST dataset, which consists of five tasks, using the LWF (Li & Hoiem, 2017). Note that we train all models with 10 epochs, and report results using five different task orders. The results are reported in Tab. 6. We observe that the wider yet shallower variant exhibits lower values for both AAN and FAF. These results suggest that within fixed parameter budget, the wider and shallower variants offer superior stability at the expense of reduced plasticity, a trend consistent with the findings observed in ResNet architectures.
Table 6: The AAN and FAF (%) of MLP with different depths and widths. Note that the '#p' denotes the parameter counts of single architecture here.
Depth
Width
#P
AAN 1
FAF
4
800
1.92
84.20:15.37
72.10
3
1050
1.94
13.23++5.13 (-4.97)
26.78++5.18 (-15.32)
5
680
1.93
87.35 (+3.15)
59.28:16.71 (+17.18)
A.3. Architectural Dimensions of Stability and Plasticity in Vision Transformers
We further investigate the impact network width (i.e., the dimension of the attention heads) and depth (i.e., the number of blocks) on the stability and plasticity ViTs. Specifically, we use SepViT-Lite (Li et al., 2022) as the default design, which configured with a width of 32 and a depth of 11. Additionally, we design wider yet shallower variant with width of 49 and depth of 5, which has parameter count comparable to the default design. Both ViTs are evaluated on ImageNet-1000/ 100/10 using iCaRL as the learning method (Rebuffi et al., 2017). Note that the training settings are consistent with Sec. 5.1, but the learning rate and optimizer are adjusted to match the official implementation of Sep (Li et al., 2022). The results are reported in Tab. 7. We observe that the wider yet shallower variant exhibits lower values for both AAN and FAF. These results suggest that within a fixed parameter budget, the wider and shallower variants offer superior stability at the expense of reduced plasticity, a trend consistent with the findings observed ResNet architectures.
Table 7: The AAN and FAF (%) of SepVit with different depths and widths. Note that the '#P' denotes the parameter counts of single architecture here.
Depth
Width
#P
AAN t
FAF +
11
32
3.78
79.54
40.51 39.47 (-1.04)
5
49
3.76
78.96 (-0.58)
13

---

Rethinking the Stability-Plastiice Trade-off in Continual Learning from an Architectural Perspective
1
32
18
14
42
65
134
511
28
20
55
85
113
1
554
38
36
43
123
94
2
510
38
32
54
189
326
37
62
119
99
101
2
37
56
195
24
636
39
34
41
19
42
124
25
4
5
534
30
41
53
72
130
87
3
15
602
33
33
20
36
125
83
True Task ID
True Task ID
True Task ID
4
16
27
13
613
34
35
52
94
85
31
7
4
16
524
44
68
116
96
71
4
23
40
33
537
32
33
35
76
99
92
5
19
17
13
594
48
33
122
116
30
4
5
11
15
645
44
37
79
91
69
5
13
23
18
35
580
36
25
67
105
90
6
11
21
13
14
569
39
194
98
30
6
8
0
5
26
594
81
136
75
6
17
27
21
29
50%
41
150
99
80
7
110
133
10
87
7
23
597
135
60
8
11
14
762
3
14
30
72
712
48
8
14
27
37
25
105
80
9
18
11
11
11
871
9
6
54
57
51
35
9
15
14
18
812
50
10
21
12
22
30
40
32
35
67
80
661
10
11
3
12
33
46
31
86
48
721
10
15
14
13
13
15
11
38
57
818
2
5
6
8
9
10
1
2
4
5
7
8
10
i
3
6
7
9
10
Predicted Task ID
Predicted Task ID
Predicted Task
ID
(a)
(b) Foster
(c) MEMO
1
1-643
43
24
36
23
37
33
39
87
35
1-644
644
26
32
39
34
52
33
35
54
51
1
632
39
23
31
21
39
22
44
95
54
2
23
564
33
36
47
34
64
43
122
34
2
22
518
57
57
82
51
65
41
50
57
2
36
538
43
44
51
26
50
43
122
47
3
20
30
695
27
70
3
22
58
47
23
3
26
682
35
38
26
77
39
T ID
True Task ID
True Task ID
True Task
4
14
28
56
75
19
27
659
49
45
39
35
4
23
639
30
37
62
56
5
25
41
69
23
42
753
19
33
33
5
19
20
31
25
76
50
6
28
18
24
614
166
55
35
34
26
31
46
648
60
63
27
43
6
33
38
24
23
23
604
29
138
40
48
7
14
18
10
15
11
17
743
74
83
15
13
19
20
41
44
52
690
46
38
37
7
12
18
9
17
12
22
72
77
72
39
8
13
13
11
28
34
24
20
758
68
31
15
13
11
42
74
101
84
598
25
37
8
12
20
7
27
23
41
28
720
75
47
9
22
21
13
17
11
26
24
837
9
45
41
46
39
95
40
60
58
546
30
9
20
16
17
13
17
18
23
19
823
34
10
16
35
28
10
40
65
602
10
18
27
35
45
2
3
5
6
7
8
9
10
1
2
4
5
7
8
10
1
3
4
6
7
8
9
10
Predicted Task ID
Predicted Task ID
Predicted Task
(d) DER Dual-Arch
(e) Foster Dual-Arch
(f) MEMO Dual-Areh
Figure 6: Task confusion matrices after learning the final task of different CL methods w/ and w/o Dual-Arch plugged in on CIFAR100/10.
A.4. Additional Results Bias-correction
We report the task confusion matrices for the DER, Foster and MEMO with/without Dual-Arch here.
A.5. Validation Long Task Sequences
In this subsection, we report the results on settings with greater number of tasks, specifically CIFAR100/50, which contains 50 tasks, in Tab. 8. These results demonstrate that Dual-Areh consistently outperforms the baselines in this challenging setting, thereby underscoring its generality.
Table 8: The LA and AIA (%) using five state-of-the-art CL methods on CIFAR 100/50. Bolded indicates the best.
iCaRL
WA
DER
Foster
MEMO
Method
LA
AIA
LA
AIA
LA
AIA
LA
AIA
LA
AIA
Original
45.30
63.99
42.12
58.26
55.73
69.53
43,45
59.81
42.44
62.57
w/ ArchCraft
48.70
67.24
39.83
61.02
57.89
71.53
53.02
68.14
54.47
69.96
w/ Ours
48.95
65,93
47.13
64.41
61.88
73.09
53.16
67.83
58.09
71.17
14

---

Rethinking the Stability-Plasticiy Trade-off in Continual Learning from an Architectural Perspective
A.6. Validation CL with Blurry Task Boundaries
Beyond Class-IL, series of works have focused on more challenging and realistic CL scenario where task boundaries not explicitly available, known Generalized Class IL (Buzzega al., 2020; Arani et al., 2022). In this section, we validate the generality of Dual-Arch in this setting. Following convention (Arani et al., 2022; Sarfraz et al., 2022), we report the results on the typical benchmark, GCIL-CIFAR-O0 100, as shown in Tab. 9. Our findings indicate that Dual-Arch consistently enhances CL performance in this scenario, underscoring its broad applicability.
Table 9: The LA (%) GCIL-CIFAR-100 with different buffer sizes. Bolded indicates the best. Note that the benchmark settings follow (Arani et al., 2022).
Method
Buffer Size 500
Buffer Size 1000
ER (Rostami et al., 2019)
20.30
34.13
w/ Dual-Arch (Ours)
27.57 (+7.27)
35.40 (+ 1,27)
DER++ (Buzzega et al., 2020)
25.82
33.64
w/ Dual-Arch (Ours)
30.34 (+4.52)
36.84 (+3.20)
15


Reviews:
[{"review_id": "PbdD0mYeZN", "reviewer": "Reviewer_xKgH", "summary": "This paper investigates the stability-plasticity trade-off in continual learning (CL) from an architectural perspective. The authors empirically demonstrate that deeper networks favor plasticity, while wider networks enhance stability under fixed parameter constraints. To address this, they propose Dual-Arch, a framework combining two specialized architectures: a wide/shallow \"Sta-Net\" for stability and a deep/narrow \"Pla-Net\" for plasticity.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer xKgH for the valuable and constructive feedback.\n\n > Generalizability of architectural insight / Concern about supplementary material.\n\nThere may be a misunderstanding regarding Tab. 5 and 6 results: \n\n- **Shallower yet wider** ViT (5×49, Tab. 5): Lower AAN/FAF value → **reduced plasticity, improved stability** .\n- **Shallower yet wider** MLP (3×1050, Tab. 6): Lower AAN/FAF value → **reduced plasticity, improved stability** .\n- **Deeper yet narrower** MLP (5×680, Tab. 6): Higher AAN/FAF value → **improved  plasticity, reduced stability** .\n\nThese align with ResNet trends (Table 1), supporting our architectural insights (depth -> plasticity, width -> stability). \n\n> Performance gains in Tab. 4.\n\nWe believe that the performance gains on ViT (minimum +2.47%, maximum +5.88%) is not very modest, especially considering the fact that our method reduces 30% parameter counts. \n\n> Concern about dataset limitations.\n\nCIFAR100 and ImageNet100 are common benchmarks in continual learning (CL) research[1-3]. To further address the reviewer's concern, we extend our evaluation to large-scale ImageNet-1K/10 tasks. Following [4], we use the 32×32 downsampled ImageNet-1K for efficiency, with a replay buffer size of 20,000 as in [5]. Results show our method remains effective, achieving gains of +3.37% and 5.11%, indicating its strong scalability.\n\n| Methods | AIA /% ↑          |\n| ------- | ----------------- |\n| iCaRL   | 24.25             |\n| w/ Ours | **27.62 (+3.37)** |\n| WA      | 22.42             |\n| w/ Ours | **27.53 (+5.11)** |\n\n> Concern about primary performance driver.\n\nWe clarify two potential misunderstandings regarding the performance drivers of Dual-Arch.\n\na) The auxiliary (plastic) model’s weights ($\\phi_t$) are **not reinitialized** but are **updated incrementally** from the previous task’s weights ($\\phi_{t-1}$), as shown in Fig. 2. So the performance gains could **not** stem from the plasticity of a freshly initialized model. We will further clarify this in Sec. 4.4 of this revision.\n\nb) Dual-Arch's efficacys stems from two key components as mentioned in Sec. 5.3 (lines 324-325):  \n\n1. Dual networks (stable and plastic learner)\n2. Dedicated architectures (Sta-Net as stable learner, Pla-Net as plastic learner) \n\nWhile dual Sta-Nets perform strongly (still worse than Dual-Arch), this does not weaken our core proposition. Instead, it highlights the importance of:\n\n- Dual networks (dual Sta-Nets still involve stable and plastic learner)\n- Dedicated architectures (Sta-Net as the stable learner)\n\nTo further validate the importance of dedicated architectures, we compare against dual ResNet-18 networks. Results (AIA %) show that our specialized architectures achieve better performance with fewer parameters.\n\n| Architectures          | Parameters (iCaRL) | Avg on Five methods |\n| ---------------------- | ------------------ | ------------------- |\n| Dual ResNet-18         | 33.6M              | 71.46               |\n| Ours (Sta-Net+Pla-Net) | 15.1M (↓55%)       | 72.94 (+1.48)       |\n\n> Why iCaRL was chosen?\n\nSimpler baselines like Experience Replay (ER) may not fully suit CL evaluation—particularly stability. A key limitation is that without additional CL strategies, performance on old tasks might not primarily rely on model's knowledge preservation. \n\nTo investigate this, we introduce an ablation on CIFAR100/10: \"w/ buffer only\", where models:\n\n- Train **only on the last five tasks**,\n- Use an initial replay buffer from the first five tasks,\n- Evaluate performance on the first five tasks after learning the final task.\n\n| Method         | Avg on Task1-5 |\n| -------------- | -------------- |\n| ER             | 38.48          |\n| w/ buffer only | 36.38 (-2.10)  |\n| iCaRL          | 48.64          |\n| w/ buffer only | 41.40 (-7.24)  |\n\nResults show the ablation achieves relatively comparable performance on tasks 1–5 to original ER, despite no direct learning. In contrast, iCaRL shows a larger gap (7.24% vs. 2.10%), indicating the model preserves knowledge beyond buffer memorization. This highlights iCaRL’s suitability for evaluating CL architectures beyond ER.\n\n> Theoretical explanation.\n\nWhile theoretical analysis (e.g., NTK or feature learning) could offer deeper insights, it is beyond this work’s scope. Existing theories often rely on idealized assumptions (e.g., infinite/very large width or two-layer networks), making extensions to our empirical width/depth trade-offs highly non-trivial.\n\n**References**:\n\n[1] Rethinking Momentum Knowledge Distillation in Online Continual Learning. ICML 2024.\n\n[2] Resurrecting old classes with new data for exemplar-free continual learning. CVPR 2024.\n\n[3] Incorporating neuro-inspired adaptability for continual learning in artificial intelligence. Nature Machine Intelligence 2023.\n\n[4] Loss of plasticity in deep continual learning. *Nature* 2024.\n\n[5] Pycil: a python toolbox for class-incremental learning. 2023.", "claims_and_evidence": "**Generalizability of architectural insights:**\nExperiments are limited to ResNet variants and small datasets. ViT/MLP results (Tables 4–6) show inconsistent trends (e.g., deeper MLPs improve AAN but harm FAF), weakening the claim.", "methods_and_evaluation_criteria": "**Dataset limitations**: Experiments on CIFAR100/ImageNet-100 may not reflect real-world scalability. No validation on larger datasets (e.g., ImageNet-1K).", "theoretical_claims": "No, because this paper does not include formal theoretical proofs. The claims are primarily empirical.", "experimental_designs_or_analyses": "**Architectural trade-off as primary performance driver:**\nAblation results (Table 3) show that adding any second model (even non-specialized architectures) improves performance, suggesting gains may stem from dual-model integration rather than architectural specialization. For example, using two Sta-Nets (row 4, Table 3) achieves 72.27% AIA vs. 72.92% for Dual-Arch—only a 0.65% gap. If I understand correctly, in your method design, the auxiliary model is re-initialized as new tasks appear. Then the plasticity of an initialized model is stronger than that of a model that has been trained on a certain task [1]. This can also explain why the algorithm has better performance, because the plasticity of the entire system is improved, but this still has no direct connection with your core proposition.\n\n[1] Dohare S, Hernandez-Garcia J F, Lan Q, et al. Loss of plasticity in deep continual learning[J]. Nature, 2024, 632(8026): 768-774.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "1. Can you clarify why iCaRL was chosen over simpler baselines for architectural analysis?\n\n2. Can you provide a theoretical explanation (e.g., via NTK or feature learning) for why depth/width affect stability/plasticity?", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZCp9648RSg", "reviewer": "Reviewer_MGyK", "summary": "The paper investigates the stability-plasticity trade-off in continual learning from an architectural perspective. Through empirical studies, the authors find that depth enhances plasticity while width favors stability. Building on this insight, they propose an approach that leverages two specialized networks with complementary strengths (stability and plasticity) to address this trade-off. Extensive experiments across multiple datasets and continual learning methods demonstrate that the proposed method not only improves performance but also enhances parameter efficiency compared to baselines.\n\n## update after rebuttal\n\nAfter the rebuttal, all of my concerns have been adequately addressed. I have also read other reviewers' comments. I think the experimental validation is already sufficient, based on my knowledge of the continual learning. Regarding training time, I also agree with Reviewer 2FFZ's updated assessment that it is acceptable. For these reasons, I tend to accept this submission.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer MGyK for the recognition of our insightful research perspective, simple yet effective method, and impressive performance improvements. We are also grateful for the valuable and constructive feedback.\n\n> Validation on replay-free settings.\n\nWhile our current experiments primarily focus on replay-based continual learning settings, our approach is also compatible with state-of-the-art replay-free methods. To demonstrate this, we evaluate our approach in conjunction with three representative replay-free continual learning methods: LWF [1], ADC [2], and LDC [3] as representative replay-free methods. The results (LA /%) consistently show that our Dual-Arch approach enhances the performance of all baseline methods, underscoring its versatility and effectiveness in replay-free continual learning scenarios.\n\n| Method  | CIFAR100/20 | CIFAR100/10 |\n| ------- | ----------- | ----------- |\n| LWF     | 16.98       | 26.63       |\n| w/ Ours | **20.27**   | **33.81**   |\n| ADC     | 30.48       | 40.25       |\n| w/ Ours | **31.90**   | **40.69**   |\n| LDC     | 31.61       | 41.57       |\n| w/ Ours | **34.69**   | **42.91**   |\n\n> Architecture-based methods.\n\nWe appreciate the reviewer’s valuable comments and will revise Sections 2.1 and 2.2 to clarify the distinction between two key concepts. To clarify, architecture-based methods primarily focus on dynamically expanding or allocating networks for each task based on a given basic architecture. In contrast, \"Neural Architecture for CL\" refers to studies that explore and optimize the basic architecture itself to better suit continual learning objectives.\n\n> Parameter counts rules.\n\nWe thank the reviewer for raising this point. To avoid any confusion, we will update the description to explicitly state that the reported parameter counts include all used networks (not just the main network).\n\n> Typo issues and Figure presentation.\n\nWe sincerely appreciate the reviewer’s attention to detail. All mentioned typos have been corrected, and we have increased the font size in Figure 3 for better readability. We will thoroughly proofread the entire paper to ensure clarity and correctness.\n\n**References:**\n\n[1] Li, Zhizhong, and Derek Hoiem. Learning without forgetting. IEEE TPAMI 2017.\n\n[2] Goswami, Dipam, et al. Resurrecting old classes with new data for exemplar-free continual learning. CVPR 2024.\n\n[3] Gomez-Villa, Alex, et al. Exemplar-free continual representation learning via learnable drift compensation. ECCV 2024.", "claims_and_evidence": "The authors conduct thorough empirical studies comparing different architectural configurations (depth vs. width)  in Section 3, providing clear evidence supporting their claims regarding the trade-off between stability and plasticity at the architectural level. The ablation study also effectively support their claims.", "methods_and_evaluation_criteria": "The proposed method presents a simple yet effective solution to the stability-plasticity dilemma by combining the strengths of two specialized architectures. The evaluation criteria, including benchmark datasets and metrics, are consistent with established conventions and well-suited for validating the proposed method.", "theoretical_claims": "This paper is primarily motivated by empirical studies, so there are no formal theoretical proofs to verify.", "experimental_designs_or_analyses": "I have examined the implementation details and reviewed the provided source code. The proposed method do achieve a fair improvement compared with the baselines.", "relation_to_broader_scientific_literature": "While existing research focuses on optimizing weights to address the stability-plasticity trade-off at the parameter level, this study introduces a novel perspective by extending this trade-off to the architectural level. Moreover, by exploring and addressing the inherent conflict between stability and plasticity at the architectural level, the paper distinguishes itself from existing exploration on neural architectures for continual learning.", "essential_references_not_discussed": "The related works section is comprehensive, covering both learning methods and neural architectures for continual learning. To the best of my knowledge, no key related works are missing.", "other_strengths_and_weaknesses": "*Strengths:*  \n1. The paper provides a novel and insightful perspective on continual learning by highlighting the inherent conflict between stability and plasticity at the architectural level. The empirical finding that depth benefits plasticity while width enhances stability has the potential to inspire future research to enhance continual learning by exploring more balanced architectures.   \n2. After reading the line 86-107, I found the proposed method is simple yet intuitive. Moreover, its effectiveness in combining the strengths of two specialized architectures is well validated through experiments and analysis.   \n3. The performance results presented in Table 2 are impressive. The method demonstrates significant and consistent improvements across different continual learning methods and benchmarks, while also achieving substantial gains in parameter efficiency.   \n4. The paper is well-written, clearly structured, and easy to follow.  \n\n*Weaknesses:*  \n1. Recent studies have increasingly focused on replay-free settings. Despite that the effectiveness of the proposed method has been well-demonstrated across various continual learning approaches, further validation in replay-free settings is still beneficial.   \n2. The discussion of \"Architecture-based methods\" in the related work section may be somewhat confusing. I recommend the authors clarify why these methods belong under \"*Learning Methods for CL*\" rather than \"*Neural Architectures for CL*.\"  \n3. The rules for computing parameter counts are reasonable but differ from those used in certain existing studies. To avoid confusion, I recommend the authors detail these rules in the main paper.", "other_comments_or_suggestions": "- In line 113, \"exploit it\" should be corrected to \"exploited.\"\n- There is inconsistent formatting with \"w/ Ours\" and \"w/ ours.\"\n- The font size in Figure 3 is too small.", "questions_for_authors": "Could the authors provide an analysis of the proposed method's effectiveness when combined with state-of-the-art replay-free continual learning methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EuOuyXGTAZ", "reviewer": "Reviewer_Kz16", "summary": "This paper studies the stability-plasticity trade-off in continual learning from an architectural perspective. It finds that increasing depth improves plasticity, while increasing width enhances stability. Motivated by this, it proposes a dual-architecture framework, DualArch, comprising two distinct networks dedicated to plasticity and stability, respectively. The proposed framework can be incorporated into existing continual learning methods to enhance performance while remaining significantly parameter efficient.\n\n## update after rebuttal\nAfter the rebuttal, my primary concerns regarding scalability and computational efficiency remain unaddressed. The experimental validation continues to rely on limited-scale settings, with the ImageNet-1K results offering minimal insight due to significant downsampling. Additionally, the reported increase in training time relative to baselines, even on smaller datasets like CIFAR-10/100, raises further concerns about the method’s practicality in real-world, large-scale scenarios. Therefore, my assessment of the paper remains unchanged.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer Kz16 for the recognition of our intriguing research perspective and interesting idea. We are also grateful for the valuable and constructive feedback.\n\n> Computation overhead.\n\nWe would like to clarify that the total FLOPs of two models (Sta-Net and Pla-Net) in Dual-Arch is lower than the baseline, as detailed in Appendix A.1 (lines 614-617). For instance, on CIFAR100, Sta-Net and Pla-Net require 255M and 241M FLOPs, respectively, totaling 496M—less than the 558M FLOPs of ResNet-18. We thank the reviewer for the valuable suggestion and will emphasize this point in the revised main text for clarity.\n\n*Training time*: While the total FLOPs are reduced as shown in A.1, we acknowledge that the actual training time may increase due to non-parallel training. We note that this trade-off is common in multi-model methods (e.g., [1]) and is justified by the performance gains. As shown in the below CIFAR100/10 experiments, our methods require 1.39× to 1.77× time cost for training.  We will include this analysis in this revision.\n\n| Method              | iCaRL  | WA     | DER     | Foster  | MEMO   |\n| ------------------- | ------ | ------ | ------- | ------- | ------ |\n| Baseline (ResNet18) | 40 min | 39 min | 74 min  | 93 min  | 49 min |\n| w/ Ours             | 70 min | 69 min | 106 min | 129 min | 85 min |\n\n*Inference time*: Since our approach utilizes only the main model (a stable learner) during inference, which is more lightweight compared to the baseline ResNet-18, the computational cost during inference is significantly reduced. Below are the results on ImageNet, demonstrating that our method achieves a 76% increase in inference speed.\n\n| Models              | Frame per second ↑ |\n| ------------------- | ------------------ |\n| Baseline (ResNet18) | 5452               |\n| w/ Ours             | 9604 (+76%)        |\n\n> Long task sequences and large datasets.\n\n*Large datasets:* CIFAR100 and ImageNet100 are common benchmarks in continual learning (CL) research[1-3]. To further address the reviewer's concern, we extend our evaluation to large-scale ImageNet-1K/10 tasks. Following [4], we use the 32×32 downsampled ImageNet-1K for efficiency, with a replay buffer size of 20,000 as in [5]. Results show our method remains effective, achieving gains of +3.37% and 5.11%, indicating its strong scalability.\n\n| Methods | AIA /%            |\n| ------- | ----------------- |\n| iCaRL   | 24.25             |\n| w/ Ours | **27.62 (+3.37)** |\n| WA      | 22.42             |\n| w/ Ours | **27.53 (+5.11)** |\n\n*Long task sequences:* Tab. 7 (Appendix A.5) demonstrates our method’s superior performance on CIFAR100/50, a benchmark with long task sequences (50 tasks). The results below highlight consistent improvements of our method over baselines.\n\n| Method  | LA (%)    | AIA (%)   |\n| ------- | --------- | --------- |\n| iCaRL   | 45.30     | 63.99     |\n| w/ Ours | **48.95** | **65.93** |\n| WA      | 42.12     | 58.26     |\n| w/ Ours | **47.13** | **64.41** |\n\n> Forgetting metrics.\n\nWe would like to clarify that (final) average forgetting (FAF) is a well-established metric in continual learning for measuring stability [4, 5]. Regarding the reviewer's suggestion to compare the model's performance against a universal upper bound, we respectfully note that such an evaluation may represent overall performance rather than forgetting.  Since the upper bound is shared across methods, the resulting metric would primarily reflect final/last accuracy (LA), which inherently combines both plasticity (AAN) and forgetting (FAF), as approximated by $LA \\gtrapprox AAN - FAF$.\n\n> Initialization of networks.\n\nAll models are randomly initialized, consistent with baseline methods. We will add a pseudo-code in this revision to clarify this point. Here is a brief process description:\n\n1. Randomly initialize the stable learner (main model, $\\theta_0$) and plastic learner (auxiliary model, $\\phi_0$).  \n2. For each new task t (1 to N):  \n   - **Train plastic learner**: Update $\\phi\\_{t-1}$ for E epochs using normal classification loss to get $\\phi_{t}$, then freeze and save it.  \n   - **Train stable learner**: Update $\\theta_{t-1}$ for E epochs with the plastic learner's assistance and CL method (loss Eq. 1) to get $\\theta_t$.  \n   - **Test stable learner**: Evaluate $\\theta_t$ on all previous tasks (1 to t).  \n\n**References:**\n\n[1] Rethinking Momentum Knowledge Distillation in Online Continual Learning. ICML 2024.\n\n[2] Resurrecting old classes with new data for exemplar-free continual learning. CVPR 2024.\n\n[3] Incorporating neuro-inspired adaptability for continual learning in artificial intelligence. Nature Machine Intelligence 2023\n\n[4] Loss of plasticity in deep continual learning. *Nature* 2024.\n\n[5] Pycil: a python toolbox for class-incremental learning. 2023.\n\n[6]  A comprehensive survey of continual learning: Theory, method and application.  IEEE TPAMI 2024.\n\n[7] Class-incremental learning: A survey. IEEE TPAMI 2024.", "claims_and_evidence": "The paper provides evidence supporting the efficacy of DualArch in small-scale continual learning settings. It demonstrates that DualArch enhances both plasticity and stability, thereby improving overall performance. However, I have the following concerns:\n\n1. **Compute Overhead**: While DualArch claims to be parameter efficient, it is unclear how it enhances computational efficiency given it involves training two models in two distinct training stages. This setup seemingly incurs approximately 2x compute overhead compared to training a single network. A direct comparison in terms of compute cost (e.g., number of updates or FLOPs) would provide more insight.\n\n2. **Scalability**: The proposed framework is only evaluated on datasets with a maximum of 100 classes. It remains unclear how DualArch would perform on larger datasets like ImageNet-1K or longer task sequences.", "methods_and_evaluation_criteria": "The forgetting metric (as presented in Figure 1a and tables) appears problematic since it is inherently model-dependent. A robust forgetting evaluation typically compares the model's performance against a universal upper bound (a jointly trained model on the entire dataset/sequence). Without this upper bound, the forgetting metric lacks interpretability.\n\nFor example:\n- **Model 1**: Learns less, forgets less → Final accuracy: $x$\n- **Model 2**: Learns more, forgets more → Final accuracy: $y$\n- **Joint model**: Upper bound accuracy: $z$\n\nIn some cases, $z - x > z - y$, meaning that Model 2 actually learns and retains more knowledge despite higher forgetting. Hence, the current forgetting metric can be misleading.\n\nAdditionally, the paper does not report computational efficiency (e.g., FLOPs, training time) compared to a single-model baseline or training-from-scratch models (joint models). This is critical for real-world continual learning applications where a continual learner should be more efficient than the jointly trained model.", "theoretical_claims": "Not applicable", "experimental_designs_or_analyses": "The experiments appear reasonable but do not study *scalability* to larger datasets or tasks. Given the proposed method's reliance on training two models, computational overhead could significantly increase in large-scale tasks.", "relation_to_broader_scientific_literature": "While prior studies separately identify that wider network favors stability or deeper one enhances plasticity, there exists a gap to unify these findings. This paper unifies stability and plasticity from an architectural perspective, which may be informative for the research community.", "essential_references_not_discussed": "The paper seems to discuss relevant prior work adequately.", "other_strengths_and_weaknesses": "Striking a better stability-plasticity trade-off from an architectural perspective seems intriguing. The idea presented in the paper seems interesting, but performing more rigorous experiments will make the paper stronger. In particular, compute efficiency and ability to scale are important criteria to align continual learning with real-world applications.", "other_comments_or_suggestions": "Please see my comments above regarding weaknesses.", "questions_for_authors": "1. How does DualArch perform on datasets with more than 100 classes, such as ImageNet-1K or subsets?\n2. Are the plastic and stable networks initialized with random or pre-trained weights?\n3. Can you provide a comparison of the compute cost (in FLOPs or training time) compared to training a single model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "22ok7F5PWM", "reviewer": "Reviewer_2FFZ", "summary": "This paper tackles the problem of offline Class Incremental Learning but leverage a a dual architecture strategy. The goal is to design one architecture that would be more plastic (focus on new knowledge) and another that would be more stable (focus on older knowledge) and combine both capabilities during training through distillation. Notably, the author show via experiments that deep models are more plastic and wider models are more stable. The authors apply such strategy with various state-of-the-arts methods and demonstrate superior performances in two datasets.\n\n## update after rebuttal\n\nThe identified limitation of computation is indeed important. Hopefully the authors provided interesting additional experiments demonstrating that the method is still valuable, but is more of a tradeoff than previously presented. Interestingly, the achieved tradeoff is more better than previous method leverage multiple models and similarly, leveraging the same computation gives overall better performances. Eventually, in any cases, such approach is valuable for inference. In that sense, given that the authors include the experiments provided in the rebuttal, and refine the presentation of the method including, making more apparent the drawback between training computation, inference optimization, and memory usage, I recommend accepting the paper. In that sense I raised my score to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 2FFZ for the recognition of our insightful empirical findings and interesting idea. We are also grateful for the valuable and constructive feedback.\n\n> Conclusions regarding deeper network\n\nOur main conclusion is that \"existing architectural designs typically exhibit good plasticity but poor stability, while their wider and shallower variants exhibit the opposite traits\" (contribution 1). We acknowledge that further increasing depth further does not significantly boost plasticity, so our plastic network's depth aligns with ResNet-18 (Sec. 4.3). We'll revise Sec. 3.2 to clarify this and avoid ambiguity.\n\n>Relative forgetting\n\nWhile the Average Forgetting is widely used, we agree that *relative forgetting (RF)* [1] offers a fairer stability measure. We've added RF to our analysis. Results on Tab. 1 (partial) support our original claims.\n\n| Depth-Width | Penu. Layer | RF↓ (%)       |\n| ----------- | ----------- | ------------- |\n| 18-64       | GAP         | 41.09         |\n| 10-96       | GAP         | 39.15 (-1.94) |\n| 18-64       | 4x4 AvgPool | 39.97 (-1.12) |\n\n> Essential References\n\nWe thank the reviewer for highlighting these relevant works. We've updated Sec. 2.3 to include a discussion of [2] and [3]. Specifically: (*italicized* indicates new content):\n\n\"..., certain works utilize two learners (known as slow and fast learners) with different functions to achieve CL. *Among them, MKD [2] and Hare & Tortoise[3] employ techniques such as exponential moving averages to integrate knowledge across two models, effectively balancing stability and plasticity.* Our proposed solution ...\"\n\n> Computation time  \n\n*Training*: Despite reduced FLOPs (A.1), training time may increase due to non-parallelism. We note that this common trade-off in multi-model methods (e.g., [1]) is justified by the performance gains. As shown in the below (CIFAR100/10, minutes), our methods require 1.39× to 1.77× time cost for training. We'll include this analysis in this revision.\n\n| Method              | iCaRL | WA   | DER  | Foster | MEMO |\n| --------- | ----- | ---- | ---- | ------ | ---- |\n| Baseline (ResNet18) | 40    | 39   | 74   | 93     | 49   |\n| w/ Ours             | 70    | 69   | 106  | 129    | 85   |\n\n*Inference*: Using only the lightweight main model during inference, our method reduces computational cost, showing a 76% faster speed.\n\n| Models   | Frame per second ↑ |\n| -------- | ------------------ |\n| Baseline | 5452               |\n| w/ Ours  | 9604 (+76%)        |\n\n> Training procedure\n\nAs suggested, we'll include pseudo-code to clarify the training and testing procedure. Brief process is as follows:\n\n1. Initialize the stable main learner $\\theta_0$ and plastic auxiliary learner $\\phi_0$).  \n2. For each new task t (1 to N):  \n   - **Train plastic learner**: Update $\\phi\\_{t-1}$ for E epochs using normal classification loss to get $\\phi_{t}$, then freeze and save it.  \n   - **Train stable learner**: Update $\\theta_{t-1}$ for E epochs with the plastic learner's assistance and CL method (loss Eq. 1) to get $\\theta_t$.  \n   - **Test stable learner**: Evaluate $\\theta_t$ on all previous tasks (1 to t).  \n\nWe clarify that our method is **fully transparent**, though some details may need elaboration: \n\n1. Sequential training of two models is mentioned in Sec. 4.4 (lines 257-258); \n\n2. Fairness is ensured by comparing total FLOPs of both models against single-model baselines in A.1.\n\n> Model ensemble \n\nWe tried ensemble techniques (e.g., exponential moving average) but found them challenging to implement across differing architectures. We believe they hold promise for integrating knowledge and will explore them in our future work.\n\n> Performance of swapping sta and pla networks\n\nThe stable and plastic learners have distinct roles (main and auxiliary) in learning, so swapping their architectures degrades performance. This shows the need for dedicated architectures suited to each learner's function.\n\n> Parameters count in Sec. 5.4\n\n\"87%\" and \"81%\" refer to Fig. 3, where our method remain effective under stricter parameter constraints.\n\n> Dynamic Architecture in Fig. 2\n\n\"Dynamic Architecture\" refers to integrating our method with dynamic architecture techniques by expanding the stable learner’s network only. This has been clarified in the revision.\n\n> Hyper-parameters (HP)\n\nFor fairness and reproducibility, we use the same HP for both baseline and baseline+DualArch without specific tuning. HPs for CL methods follow the PyCIL library [4], with epochs and optimizer scheduler adjusted for consistency across all methods, as noted in Sec. 5.1.\n\n**References:**\n\n[1] Improving plasticity in online continual learning via collaborative learning. CVPR 2024.\n\n[2] Rethinking Momentum Knowledge Distillation in Online Continual Learning. ICML 2024.\n\n[3] Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks. ICML 2024.\n\n[4] Pycil: a python toolbox for class-incremental learning. 2023", "claims_and_evidence": "- Given the standard deviations in Table 1, the conclusions in 3.2 regarding using a deeper network are not very convincing\n- I deeply appreciate that the code is shared\n- Other claims are convincing", "methods_and_evaluation_criteria": "- As shown in a previous study [1], the forgetting metric can advantage more plastic model, so it would also be interesting to consider the relative forgetting metric [1]\n\n[1] Wang, Maorong, et al. \"Improving plasticity in online continual learning via collaborative learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "theoretical_claims": "This paper does not contain any theoretical claim.", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "Key contributions are:\n- helping continual learning practitioners design more efficient models in continual learning\n- giving hindsight on the architectural design of continual learners", "essential_references_not_discussed": "Additionnal methods employing several learners already exist and should be discussed in the current paper. Especially in the case of [2, 3], the authors similarly have a double architecture that focuses on stability and plasticity, so I believe such work should be mentioned. \n\n[2] Michel, Nicolas, et al. \"Rethinking Momentum Knowledge Distillation in Online Continual Learning.\" Forty-first International Conference on Machine Learning.\n\n[3] Lee, Hojoon, et al. \"Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks.\" International Conference on Machine Learning. PMLR, 2024.", "other_strengths_and_weaknesses": "- The proposed approach leverages fewer parameters but it is unclear in terms of computation since it is not computable in parallel. Similarly, the distillation loss term increases computation. I believe sharing computation time is necessary, in addition to the FLOPs given in the appendix.\n- I really like the idea of the paper, however, I have a hard time understanding how the overall training procedure occurs, since both models are optimized. A pseudo-code would really make this procedure clearer in my opinion. In that sense, I checked the code given by the authors, and I am now even more confused. In the code shared, if we check for example the file `models/icarl_t.py`. In the `_init_train()` and `_update_representation()` functions, I can see **two optimization loops of 100 epochs, for each model**. One at line 215 and one at line 248, for `_update_representation()`. This is very important and not mentioned in the paper, from my understanding. Similarly, it impacts the training time. This is for me the main weakness of the paper.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "- How about model ensemble for inference? Have you tried? I understand that it would increase computation but this might also lead to superior performance.\n- Why swapping stab and pla models leads to a strong drop in performances, as observed in the last row of Table 3?\n- In section 5.4 the authors claim a reduction in parameters count of 87\\% and 81\\%, however in the table the reduction is of about 50\\% for the considered methods. Could you elaborate on this?\n- What does this \"Dynamic Arcchitecture\" part mean in Figure 2? Is it because the classification head grows during training? I do not find it very clear at the moment.\n- Could you give more details on how the hyper-parameters are chosen? Did you conduct any search? Do you use the exact same HP for the baseline and baseline+DualArch?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors do a careful empirical study of the importance of width and depth in terms of stability and plasticity. Based on these insights they propose a dual architecture approach, which employs a shallow model for plasticity and a deep one for stability. All reviewers agreed that the work is of good quality and there is sufficient empirical evidence for the observation and proposed method. In the rebuttal the authors provide additional results that solidify their contribution by looking at larger scale models. Overall I recommend acceptance of this work.
Decision: Accept (poster)