Learning without Isolation: Pathway Protection for Continual Learning
Authors: Zhikang Chen
Abudukelimu Wuerkaixi
Sen Cui
Haoxuan Li
Ding Li
Jingfeng Zhang
Bo Han
Gang Niu
Houfang Liu
Yi Yang
Sifan YANG
Changshui Zhang
Tianling Ren
Venue: ICML 2025 poster (2025)
Keywords: Continual learning
Primary area: deep_learning->everything_else
TL;DR: Continual learning

Abstract: Deep networks are prone to catastrophic forgetting during sequential task learning, i.e., losing the knowledge about old tasks upon learning new tasks. To this end, continual learning (CL) has emerged, whose existing methods focus mostly on regulating or protecting the parameters associated with the previous tasks. However, parameter protection is often impractical, since the size of parameters for storing the old-task knowledge increases linearly with the number of tasks, otherwise it is hard to preserve the parameters related to the old-task knowledge. In this work, we bring a dual opinion from neuroscience and physics to CL: in the whole networks, the pathways matter more than the parameters when concerning the knowledge acquired from the old tasks. Following this opinion, we propose a novel CL framework, learning without isolation (LwI), where model fusion is formulated as graph matching and the pathways occupied by the old tasks are protected without being isolated. Thanks to the sparsity of activation channels in a deep network, LwI can adaptively allocate available pathways for a new task, realizing pathway protection and addressing catastrophic forgetting in a parameter-effcient manner. Experiments on popular benchmark datasets demonstrate the superiority of the proposed LwI.

Learning without Isolation: Pathway Protection for Continual Learning
Zhikang Chen 12 Abudukelimu Wuerkaixi 1 Sen Cui Haoxuan Li Ding Li
Jingfeng Zhang 42
Bo Han 52 Gang Niu Houfang Liu" Yi Yang
Sifan Yang'
Changshui Zhang" f1
Tianling Ren
Abstract
Task Task 2 Task Task
Deep networks are prone to catastrophic forget- ling during sequential task learning, i.e., losing the knowledge about old tasks upon learning new tasks. this end, continual learning (CL) has emerged, whose existing methods focus mostly regulating or protecting the parameters associ- ated with the previous tasks. However, parameter protection is often impractical, since the size of parameters for storing the old-task knowledge in- creases linearly with the number of tasks, other- wise is hard to preserve the parameters related to the old-task knowledge. In this work, we bring dual opinion from neuroscience and physics CL: in the whole networks, the pathways matter more than the parameters when concerning the knowledge acquired from the old tasks. Follow- ing this opinion, we propose a novel CL frame- work, learning without isolation (LwI), where model fusion formulated as graph matching and the pathways occupied by the old tasks are protected without being isolated. Thanks to the sparsity of activation channels in deep network, LwI can adaptively allocate available pathways for a new task, realizing pathway protection and addressing catastrophic forgetting in parameter- efficient manner. Experiments on popular bench- mark datasets demonstrate the superiority of the proposed LwI.
Neuron
Pathway Protaction
Protection
Task-aware
Task arak-agnostic
Parameter Protection
Task 1 Task 2 Task 3 Task
Pathway
Ours (Pathway Protection)
Figure I. Left Figures: The illustrative comparison diagram be- tween our method and parameter-pme protecttee approach depicts the key distinctions in methodologies. Bottom Right Fig- ure: The performance comparison between our method and the WSN (Kang et al., 2022a) method. Top Right Figure: We show- case the ability of our method to adapt even task aask-agnsttcc seenar- approach requires knowledge
ios, whereas the parameter-protecee of task identifiers for effective recognition.
mance on all seen tasks, is pivotal for the advancement of artificial general intelligence (Masana et al., 2022; Liang & Li, 2024; Wang et al., 2024). The approach, mirroring the human learning process of acquiring and retaining diverse experiences about the real world, confronts a significant challenge:
catastrophic forgetting (MeCloskey & Cohen, 1989). This phenomenon results in the diminished profi- ciency of model in prior tasks after learning new ones.
1. Introduction
Continual learning, a scenario that requires a model to han- dle continuous stream of tasks while preserving perfor-
Various continual learning approaches have been proposed to mitigate the issue catastrophic forgetting, broadly categorized into three types. Regularization-basse ap- proaches entail adding regularization terms that leverage the weight information of previous tasks during the train- ing of the current task. While this approach catastrophic forgetting to some extent by constraining pa- rameter shifts and ensuring protection of model parame- ters, it tends to result in relatively lower performance when confronted with significant variations in data characteris- tics. Rehearsal- based approaches preserve data segments
"Equal contribution +Corresponding authors 'Tsinghua Uni- versity, Beijing, P.R. China "RIKEN Peking University "The University of Auckland 'Hong Kong Baptist University. Correspondence
Yi Yang <yiyang@ cyiyang@tsinghua. wesiingua.edu.cn.... Changshui Zhang <zes@ mail tsighuu.iingu Tianling Ren <RenTL RRenTL@@singhua.cco edd...ua..d...n...
mitigate
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning without Isolation: Pathway Protection for Continual Learning
from previous tasks or use synthesized pseudo-data to retain previous knowledge while learning new tasks, which can achieve a more unified output range for the classification heads, leading to superior performance in scenarios of task agnostic. However, from the perspective of data privacy protection (Agarwal et al., 2018), this approach does not suffice. Architecture-basee approaches focus on protect- ing parameters through techniques, achieving performance that matches or exceeds that of previous network training. However, they have two drawbacks, one is the requirement to know the task to which the identified object belongs in or- der to achieve accurate recognition, and another is that this method leads to the isolation of tasks, hindering effective communication and information sharing among them.
ing one-to-one correspondence between channels (Singh & Jaggi, 2020). Simple averaging may lead interference and even cancellation of effective components, concern exacerbated during continual learning. Hence, in this pa- per, we align channels before model fusion. Several studies (Zhou et al., 2022; Hu et al., 2022; Yang et al., 2023; Gao et al., 2024) have highlighted the following characteristics of neural networks, in the shallow layers of the deep net- work, where tasks share more common features, we match the channels with high similarity to enhance mutual com- monality. In contrast, in deeper layers, where tasks exhibit more specific characteristics, we match channels with low similarity to facilitate the fusion of distinct task features while preserving their distinctiveness, thus achieving path- way protection.
We argue that existing architecture- continual learning methods do not adequately leverage the overall considera- tion of the sparsity of activation channels in deep networks. As illustrated in Figure 1, We adopt a holistic perspective on the deep network, allocating distinct activation pathways for each task through pathway protection involves assigning unique pathways for information transmission in the deep network. Here, pathway (Kipf & Welling, 2016; Zoph & Le, 2016; Huang et al., 2017: Vaswani et al., 2017) refers to the trajectories the data take through the deep network, travers- ing from the input layer through intermediate layers to the output layer. In this context, the concept of channel is akin to a neuron. The purameter-protcctiv approach primarily in- volves pruning or masking operations on neurons maintains performance when task is known, it lacks consideration for the overall deep network structure. Consequently, in subsequent tasks, the reducible number of learnable param- eters hinders the achievement of optimal performance. As depicted in the Figure 1, our approach is expected to outper- form the latest erramettr-protective methods WSN (Kang et al., 2022a). Meanwhile, considering brain's 's hierarchi- cal, sparsity, and recurrent structure (Friston, 2008), brain activity relies on sparsity connections, where only a few neu- rons respond to any given stimulus (Babadi & Sompolinsky, 2014), brain learns and retains knowledge by re-configuring existing neurons to create more efficient neural pathways. Therefore, pathways protection is all you need.
Figure 2 intuitively demonstrates the effectiveness of our approach. The concept of " "Activation level" refers to the average magnitude of the weights obtained after activation in the last layer of the feature extraction phase. We use activation levels to measure whether pathways associated with different tasks can be distinguished. We present the activation output of data from different tasks in the last convolution layer of the trained model. As depicted in the left subplot of Figure 2, our method consistently exhibits distinctive prominence for each task. In other words, our method adaptively allocates a set of pathways for each task, preventing the knowledge of old tasks stored in deep net- work parameters from being overwritten when learning new tasks, which helps mitigate catastrophic forgetting. In con- trast, the Learning without Forgetting (LwF) method (Li & Hoiem, 2017) probably demonstrates nearly uniform chan- nel activation levels for each task, leading to mixed channel utilization among tasks. As the accuracy plot in the top right corner illustrates, even after training on new tasks, our method maintains consistent or better performance on previous tasks.
It is worthwhile to summarize our key contributions as fol- lows:
1. We explored a new direction, employing pathway pro- tection approach for continual learning.
Inspired by compensatory mechanisms observed in neuro- science and based on the sparsity of activation channels in neural networks, we propose a novel method maintain the
2. We proposed a novel data-free continual learning ap- proach, learning without isolation (IWT), based on graph matching.
overall stability of deep network channels while allocating distinct pathways to different tasks across the network. The proposed approach initiates with training a model on the first task, followed by training a new model for each subsequent task. Then, a matching procedure is employed to fuse the new and old models, yielding merged model. Conventional model fusion methods involve straightforward weight aver- aging (McMahan et al., 2017; Jiang et al., 2017), yet deep network parameterizations are often highly redundant, lack-
3. Our experiments on both CIFAR-100 and Tiny- Imagenet datasets demonstrate that our framework outperforms other methods. The source code of our framework is accessible at com/chhenzk20221 I.....
2

---

Learning without Isolation: Pathway Protection for Continual Learning
tion (CoPE) (De Lange et al., 2021) combines the principles of the nearest-mean classifier with a reservoir- based sam- pling strategy. Dynamic architecture-based methods encom- pass expanding models and employing parameter isolation techniques to retain previous knowledge while accommo- dating new knowledge expansion.
Tack 2
Tick
0.23
LwfF
Nethvatln trvel
69.05
0.00
Pallwway index
Trajatn preatane
0.25
Parameter isolation- continual learning. This ap- proach aims to safeguard parameters to preserve knowledge acquired from previous tasks (Zhang et al., 2024b). The Piggyback method (Mallya et al., 2018) involves learning a series of masks over a post-pretrained model, correspond- ing to various tasks, resulting in a series of task-specific subnetworks. The PackNet method (Mallya & Lazebnik, 2018) uses pruning method to protect neurons, which are important to previous tasks, CLNP method (Golkar et al., 2019) divides neurons in the deep network into active, in- active, and interference parts, utilizing previously learned features and unused weights from the network to train new tasks. Supsup method (Wortsman et al., 2020) employs masking to protect specific parameters important for tasks. Chen et al. (2020) prunes the model to obtaín the optimal subnetwork for the task, thus preserving knowledge and achieves generalization for new tasks through re-growing. GPM (Saha et al., 2021) utilizes gradient mapping to project the knowledge from previous tasks into mutually orthogo- nal gradient subspaces, thereby enabling continual learning. The WSN algorithm (Kang et al., 2022a), based on the lot- tery hypothesis, learns a compact subnetwork for each task while maintaining the weights chosen for previous tasks un- changed. SPU (Zhang al., 2024a) employs causal tracking to select model parameters for updates, thereby facilitating knowledge protection. However, most of these methods involve pruning or masking based on network weights, lead- ing to non-structured modifications that risk compromising the integrity of network. Our approach integrates the chan- nel properties of network, which allows different tasks to utilize distinct pathways for propagation and flow, preserv- ing the overall integrity of the deep network without causing disruption.
Actioatte beve
This Taak Thiek
0.15
0.03
0.00
Rodlaay Iade 76
15
30 Pathway Index
Figure 2. Left Figure: A comparison between our approach and LwF (Li & Hoiem, 2017). The activation values in the last con- volution layer of the models are displayed across channels. The channels of the models have been rearranged along the horizontal axis for clearer demonstration. Bottom Right Figure: An explana- tory legend for the horizontal axis (channel index) in the left figure. Top Right Figure: A comparative analysis under the condition of task awareness between our method and LwF indicates that our accuracy remains largely unchanged, contrasting with substantial decline observed in the case of LwF.
2. Related Work
Continual learning. Deep networks exhibit a static struc- ture, implying that once a task is learned, network parame- ters need to remain fixed to prevent catastrophic forgetting (Wang et al., 2024; Zhao et al., 2024). However, contin- ual learning addresses a more prevalent scenario in which tasks arrive as a continuous data stream for the network to learn. In this context, strategies like regularization- based, rehearsal- based, and dynamic architecture-base based approaches are employed to mitigate catastrophic forget- ting. Regularization-Bee based methods apply constraints to limit changes in weights nodes from past tasks, thereby reducing catastrophic forgetting. For instance, meth- ods like EWC (Kirkpatrick et al., 2017) incorporate the Fisher information matrix of previous task weights, while RWalk (Chaudhry et al., 2018) merges this matrix's 's approx- imation with online path integration to gauge parameter importance. LwF method, on the other hand, employs out- put alignment to prevent the model weights from a large shift. SPG (Konishi et al., 2023) employs the Fisher in- formation matrix to control the updates of each parameter, enabling more granular parameter protection. Rehearsal- based approaches involve preserving portions of data from previous tasks or using some techniques to generate pseudo- data (Shin et al., 2017). This data is then combined with the current dataset during the training for the next task, alleviating catastrophic forgetting. For example, both ap- proaches, LUCIR (Hou et al., 2019) and iCaRL (Rebuffi et al., 2017), leverage the technique of preserving a portion of previously acquired data along with knowledge distilla- tion for incremental learning. Continual Prototype Evolu-
The sparsity of deep network. According to the mech- anisms observed in neuroscience, in the brains of healthy adults, the density of connections remains roughly constant. Despite learning more tasks, the capacity of neurons in the brain remains relatively unchanged. Meanwhile, within deep networks, this phenomenon also manifests. Upon com- pletion of training, deep networks typically exhibit sparse activation, with small proportion of effectively activated neurons (Han et al., 2015; Liu et al., 2015; Fan et al., 2020; Dai et al., 2021). According to the findings of Mao et al. (2017), deep networks exhibit an inverse relationship be- tween overall accuracy and granularity. Specifically, under similar sparsity levels, increased granularity is associated with improved accuracy. Under comparable sparsity con-
3

---

Learning without Isolation: Pathway Protection for Continual Learning
ditions, finer granularity tends to yield optimal accuracy. Concurrently, the MEMO method (Zhou et al., 2022) high- lights similarities in the shallow layers of different models while showcasing differences in the deeper layers. There- fore, we hypothesized that within the coarser granularity (shallow layers) of the deep network, a denser occupation of channels occurs, while in the finer granularity (deeper layers), channel occupation tends be sparser.
the deep network (Singh & Jaggi, 2020). Hence, aligning the channels before fusion becomes erucial step in the integration process.
In this context, we conceptualize the matching process be- tween deep networks as a graph matching problem. In our framework, a deep network is conceptualized as an image. This representation enables the alignment of two deep net- works through the application of graph matching algorithm. At each layer, we interpret the channels within that layer as nodes in an image, and the connections between adjacent layer channels as edges. It is noteworthy that, within deep networks, we assert that matching occurs exclusively within each layer, as cross-layer matching holds no significant rel- evance. This approach facilitates the effective application of graph matching methods in deep networks, given their large-scale neuron configuration. The specific formula for graph matching is presented as follows:
3. Preliminary
In this section, we provide an elucidation of the problems to be addressed and the prerequisite knowledge required for subsequent methods. In Section 3.2, we present an exposi- tion on continual learning. Sections 3.1 and 3.3 introduce the foundational knowledge underpinning our approach, which includes deep network sparsity and graph matching algorithms.
Consider supervised continual learning scenario where learners need to solve T + 1 tasks in sequence without catastrophically forgetting old tasks. At the same time, due to data privacy restrictions, we cannot store data from previous tasks. We use De+1= = (X++1; ) to denote a dataset for task + X = {=i,..., ] and Y1+1 = (y1,-..,1,) represent that the dataset includes n data classes along with their corresponding labels for task t + 1. And we use Mj to denote a trained model for task t. Meanwhile, D1.- = [X1,.., ..., X Yi.... Yτ denotes datasets for all seen tasks from task 1 to task t. We represent the deep network model using the following formula:
N-
N-IN-IN-IN-1
max P
Σ
Pa K (o.cbiid) Padds
a=0
b=0 = d=0
s.t.
Po I; P:=1:
N
(3)
Ploe] =1,
Vm E [1, I. - 1};
N-1
Vm E [1, L 1]. Pm(a.d] =1,
where a and c represent node indices between adjacent layers in modelX as shown in Figure 3.2, b and d represent node indices between adjacent layers in modelY as shown in Figure 3.2, I. represents the index of the last layer in the neural network, N = Em N, represents the sum of the number of nodes across all layers, Nm represents the number of nodes across layer m, K represents the similarity matrix between adjacent layers in modelX and modelY, Po represents the permutation matrix for the first layer, PN represents the permutation matrix for the last layer, Pm, m E [ 1] denotes the permutation matrix for intermediate layers. We need to solve the assignment matrix P, and according to the formula, we can find that the time complexity of using the graph matching method is O However, based on the above analysis, we use layer-by- layer calculation of the assignment matrix in this paper to align the channels at each layer of the deep network, so that N is not the number of all channels, but the number of channels in each layer. More analysis could be found in the appendix A.3.
Mxx) = f()),
and standard continual learning scenario designed to learn series of tasks by minimizing optimization problems at each step:
min L)f(Xx+1;;))
where T. denotes the loss function used when training task t +++ 1. It is well known that simply optimizing the loss function can easily lead to catastrophic forgetting.
3.1. Graph matching for deep network fusion.
Recently, some studies have employed graph matching ap- proaches for model fusion (Su et al., 2021). Graph match- ing bears resemblance to a quadratic assignment prob- lem (QAP) (Loiola al., 2007). with the objective of estab- lishing correspondences between the nodes in an image and the edges connecting these nodes. The activation distribu- tion of deep network channels is not fixed across training iterations, resulting in some neurons exhibiting high acti- vation for one task, but low activation for another. a straightforward averaging fusion is performed, it may lead to interference and blending of effective components within
3.2. Problem Statement
In practical applications, highly precise matching results are not necessary, and the majority of current work focuses on the approximate matching of nodes or edges. The previous
4

---

Learning without Isolation: Pathway Protection for Continual Learning
Model Matching matrix
Model structure
Graph structure
Model X
Model Y
Mouel y
Task
Task t+1
L3
L4
Append operation
Model X
Model Y
Graph X
Graph Y
Deep Layers Minimum Similarity
Model x
OutputLayee
L4
Shallow Layers
Input Layer
M Maximum Similarity Madel Y
L3
Mddel Y
L2
Edge Comeepandence
L1
L2
L1
Model
Node Comespondence
Figure 3. The overall structure of our proposed algorithm. In the right diagram, represent the deep network in four parts: L.1 corresponds to the input layer, 1.2 to the shallow layers, 1.3 to the deeper layers, and L4 to the output layer. The channels in the deep network can be analogous to nodes side, LI requires no matching operation. L4 only needs to append operations for the output heads of different tasks. L2 matches the channels with maximum similarity. Conversely, L3 undergoes minimization similarity matching.
graph, the connections between channels correspond the edges the graph. the left
work can be divided into classical methods and deep graph matching methods. In this paper, a more commonly used method, Sinkhorn algorithm (Cuturi, 2013), used. The Sinkhorn algorithm, rooted in entropy regularization, trans- forms binary 0-1 matrix into soft matching matrix with sum of through process of bi-directional relaxation.
with the entire network in different tasks while employing fewer parameters and requiring shorter training times. Re- garding the sparsity aspect of deep networks, the Lottery Ticket Hypothesis asserts that only small subset of con- nections (weights) within the network is crucial for learning and performance, while the remaining connections can be pruned (set to zero) without significantly affecting the per- formance of network.
3.3. The sparsity of deep network.
The primary rationale behind this approach stems from the sparsity of deep networks. To accommodate future learning tasks, continual learners often utilize peratearameerrieed deep networks. The reason is that continual learning fre- quently relies on over-parameterized deep deep networks to allow flexibility for future tasks.
Previous studies show that deep networks are sparse- not all parameters need computation, as many are inactive. By assigning different channels different tasks, we lever- age this sparsity to match mitigating catastrophic forgetting in continual learning.
surpass performance while
4. Methodology
We believe that deep network is composed of multiple layers, and we of the deep network. Meanwhile, it has been observed in previous studies (Zhou et al., 2022) that shallow layers across models of different tasks exhibit notable similarities, whereas deeper layers demonstrate distinct characteristics. The deep network model can be decoupled into a classifier, denoted G(-), and feature extractor, represented as F())- The feature extractor further bifurcates into shallow-layer deep network S(-) and deep-layer deep network D(-) in (4):
to represent the index of one layer
In this section, we delve into a comprehensive discussion of proposed continual learning structure based on graph matching, along with its specific implementation. Section 4.1 outlines the framework of proposed method for contin- ual learning based on graph matching. We fuse the model trained on a new task with the one trained on previous tasks. Rather than merely averaging the parameters of two models, we conduct pathway alignment based on graph matching before fusion. To achieve the protection and shar- ing of knowledge,
employ different similarity matrix across different layers of the deep network. Section 4.2 provides detailed account of the optimization process.
( = G. (F((z)) G. (De (S))
(4)
The Lottery Ticket Hypothesis (Frankle & Carbin, 2018) posits that within deep networks, there exist specific "win- ning tickets" generated during the training process. These winning tickets, it suggests, enable comparable performance
5

---

Learning without Isolation: Pathway Protection for Continual Learning
4.1. Graph-not-Matching for Continual Learning
Algorithm 1 Model Fusion Process
Input: the weight matrix between the layer / - 1 and / is
Unlike previous architecture--ased methods, which usually mask weights crucial for the previous task, strategy stems from the holistic nature and sparsity of neural net- works. We believe that safeguarding previous tasks through misaligned addition represents a matching approach. This method not only protects the overall integrity of the deep network, but also allocates different directional channels for various tasks.
denoted as W(( p(l- represents the corresponding
permutation matrix, fusion coefficient is k.
Output: the fusion model W fusion.
for layer I.... N do
Calculate the permutation matrix p(l-1.1) according to the Algorithm 2 in appendix:
if layer = then
Calculate w + -((0))) W!..)) end else
The overall structure of our proposed IwI algo- rithm is shown in Figure 3. When new task arrives, we train a new model for it. We then employ graph matching for channel alignment before model fusion, by analogizing neural network nodes to nodes in graph matching and con- nections between deep network channels to edges in graph matching. The specific aligmment operations, as illustrated in the left diagram, involve matching the channels with high similarity in shallow layers and low similarity in deep lay- ers. Meanwhile, matching operations are not required in LI, and in L4, only append operation is necessary. The overall process of our proposed method is illustrated in Algorithm 3 in appendix.
Overvíew.
Calculate W ( 1.) +- Wo (lll-1.1) 1.1) p(l-
Calculate W. (1-1 f p((-1,1) (1-11.0)
end
W = W + (1 k) * *W! (!1.
fusion
end W.
W
Fusioni
the most similar or dissimilar channels accordingly. This process achieves channel alignment within the current layer. Graph-Matching and Graph-not-Matchingg We adopted the combined approach of maximizing and minimizing sim- ilarities for the following reasons: 1). To facilitate col- laboration between tasks. 2). Considering the sparsity of deep network channels, we allocate different channels for different tasks in the sparse layer, thereby preserving the characteristics of each task. The key to implementing soft matching in our method lies in calculating the optimal trans- port matrix, which is the matching matrix P. Here, we provide a more detailed explanation of Algorithm 1. Our goal to use the similarity matrix K to obtain the matrix P. where Pas represents the optimal amount of mass to transport the a-th neuron in the l-th layer of modelX to the b-th neuron in the I-th layer of modelY. The implementa- tion process is that, in the shallow layers, we observe that different tasks occupy denser channels with shared features. Consequently, for these distinct tasks, we consolidate their most similar channels, facilitating mutual reinforcement of common features, for the collaboration of knowledge among different tasks. Meanwhile, in the deeper layers, we observe that different tasks occupy sparser channels, em- phasizing distinct characteristics. Thus, for these tasks, we consider the misaligned fusion of channels that represent unique traits of each task, aiming to safeguard the individual characteristics.
Model Fusion Process. The fusion process of the new and old models is illustrated in Algorithm 1. We use Eu- clidean Distance in (5) to compute distances between weights and subsequently employ specific graph matching algorithms:
K K(a,obb.d =Ileac- Oball2. 2,
(5)
where K denoted the similarity matrix, lee and Cbd denote the similarity relationship between the channels of adjacent layers in two models, specifically focusing on the edges (a, c) and d), respectively. We also utilize cosine simi- larity for measurement, and specific details can be found in the ablation study and the appendix C.6. In the shallow layers, we need to maximize similarity matching for pro- tecting old knowledge and promoting the collaboration of different tasks. Hence, the similarity matrix between two edges is itself. Conversely, in the deeper layers, we repeat similar process but utilize minimizing similarity approach for protecting the individuality of different tasks, facilitating misaligned fusion of channels. Therefore, the similarity matrix between two edges is represented by its negation. The specific matching process can be found in Algorithm 2 in the appendix. This process yields a permutation matrix, enabling us to perform matrix multiplication between the old model and the permutation matrix. Subsequently, the permutation matrix of similarity from the previous layer is multiplied with the parameter matrix of the current layer, ensuring the coherence of the connections between the chan- nels. The matrix multiplication of the permutation matrix for the current layer is performed with itself, positioning
4.2. Optimization
Knowledge distillation aims to mitigate semantic discrep- ancies between the new and old models, otherwise, model fusion loses its significance. Additionally, in training a
6

---

Learning without Isolation: Pathway Protection for Continual Learning
Table 1. Task-agnostic and Task- accuracy (%) of different methods. Our approach is based data-free, but the results of exemplar- based methods are also provided.
Task agnk-agnottic
Task-aware
Datasel
Architecture
Method
Exemplar
5 splits
10 splits
20 splits
5 splits
10 splits
20 splits
EWC RWalk
31.81 1.45
21.14 0.98
12.32
2.32.100 0.56
64.22
0.83
65.86 +: 1.55
63.43
1.59
21.40 +: 1.22
20.07
1.91
12.49
1.36
64.98
0.97
69.16 + 1.29
67.98 = 1.38
LwF
37.54 + 0.43
25.78
+ 0.43
15.86
1.15
74.63 + 0.72
75.98 + 1.03
76.37
1.44
SPG
30.74 0.27
22,54 : 1.23
11.28
0.22
62.22 = 1.54
70.34 - 0.52
72.39 = 0.05
SPC
DO
34.56 + 0.93
23.44 = 0.36
17.33
66.02 = 0.47
73.31 + 0.21
78.34 0.47
CIFAR-100
ResNet32
GPM
71.72 = 0.35
78.74 1.17
80,47 = 0.33
WSN
75.47 = 0.48
80.12 - 0.60
82.51 = 0.50
Ours
43.42 + 0.58
30.62
1.08
20.31
0.77
76.10 0.33
81.12 + 0.90
83.19 0.35
iCaRL
37.23 +: 0.74
36.88
2.33
33.88 + 3.03
62,98 = 0.79
73.40 +: 1,46
81.74
1.65
2000
LUCIR
48.48 + 1.16
41.10 + 1.98
36.46 + 1.83
75,40 = 0.57
80.05 # 1.00
84.95 = 0.99
EWC
30.84 +.247
18,66 0.62
9.21 = 0.25
61,25 = 0,46
56.53 - 1.84
51,34 0.72
RWalk
38.81 + 2.08
21.78 + 0.53
7.82 + 1.07
69.41 + 1.70
61.91 + 0.62
57.57 = 1.16
LwF
44.66 + 0.97
30.41 + 0.82
16.66 + 1.36
79.96 + 0.52
81.35 + 0.51
81,45 - 0.67
SPG
26.32 2 0.57
20.16 1,51
10.54 + 0.14
64.98 = 0.97
69.16 = 1.29
67.98 = 1.38
SPC
43.79 + 0.40
25.12 # 0.48
16.08 + 0.71
74.63 = 0.72
75.98 : 1.03
76.37 = 1.44
CIFAR-100
ResNet18
GPM
78.23 = 1.13
81.42 : 1.43
86.21 = 0.46
WSN
78.65 = 1.33
83.08 + 1.57
86.10 = 0.25
Ours
51.95 0.56
36.36 1.06
22.99
0.39
81.10 0.80
84.90 +: 0.36
86.49 0.55
iCaRL
49.44 1: 0.78
39.27
1 0.37
28.48 1: 1.57
73.84
0.36
76.63 1: 0.62
78.49 0.74
2000
LUCIR
55.67 1.04
0.97 42.56
33.84 1.95
81.22
0.25
84.41 +: 0.22
86.19 0.25
EWC
19.21 0.31
10.32 2: 0.29
4.69 = 0.39
42.84 = 0,54
36.21 : 1.07
30,82 = 2.06
RWalk
21.69 + 0.64
12,94 0.38
7.84 + 0.21
55.67 + 1.27
56.14 + 0.29
59.58 0.40
LwF
26.76 0.50
20.14 0.28
13.09 +: 0.24
59.66 + 0.47
63.52 +: 0.57
70.59 0.47
SPG
22.80 + 0.26
12.03 0.73
7,86 = 0.24
54.50 = 0.47
57.81 +: 0.23
59.67 = 0.44
SPC
25.50 + 0.40
19.98 0.06
13.44 0.18
$7.15 0.31
59.93 +: 0.08
63.64 = 0,38
mmy-lmmgenet
ResNet18
GPM
58.45 = 0.38
13.17:0.20
70.16 = 0.42
WSN
57.38 = 0.51
64.12 0.43
71,54 = 0.43
Ours
34.33 0.51
26.15
0.22
15.59 0.84
62,97 0.14
68.67 1 0.36
72.74 0.27
iCaRL
28.81 0.14
23,37 0.24
14.68 0.35
56.17 = 0.34
59,49 1 0.91
61.00 = 0.67
2000
LUCIR
30.17 0.37
20.15 0.63
13.48
60.25
65.52 +
66.56 0.66
model for a new task, leveraging the universally applicable knowledge from the old task model, such as shallow-level enhances the efficiency of learning through distillation. To leverage prior task knowledge, we employed previous mod- els as pre-trained models, integrating their parameters into the current model for subsequent task training. Simulta- neously, throughout the entire training process, the feature extractor of the classifier undergoes continuous modifica- tions. If there is a noticeable drift in the feature space of the classifier, the knowledge memorized by the model may become outdated. Consequently, it is imperative to maintain relative consistency in the feature space of the classifier during the training process. Further details can be found in the appendix C.7 and C.8.
5.1. Settings
Datasets. Following the work (Masana et al., 2022), we evaluate different methods on benchmark datasets with set- tings, including CIFAR-I00 and Tiny-Imagenet datasets. Under the condition of continual learning, we use three task-splitting settings: 5 splits, 10 splits, and 20 splits.
Architecture. In order to verify our proposed method can achieve knowledge protection for different tasks, we con- ducted a large number of experiments to study the effect of model size on performance. In this article, we use ResNet32 and ResNet18 architectures (He et al., 2016) for compari- sthe sizes and parameter counts of the two models are detailed in the appendix B.3).
Baselines. In order to demonstrate the effectiveness of our approach, we conduct comparative tests against dif- ferent continual learning methods. Specifically, baseline methods include regularization-
5. Results and Discussion
In the main text, we present the results of three experiments, including the application of the ResNet32 architecture to the CIFAR-100 100 dataset, and ResNet18 8 to both the CIFAR-100 and
frameworks, like EWC (Kirkpatrick et al., 2017), LwF (Li & Hoiem, 2017), RWalk (Chaudhry et al., 2018) and SPG (Konishi al., 2023), architecture-basee framework, like GPM (Saha et al., 2021), WSN (Kang et al., 2022a) and SPU (Zhang et al., 2024a), which is inapplicable in scenarios where task is un- known, and some classical rehearsal- methods, such as LUCIR (Hou et al., 2019) and iCaRL (Rebuffi et al., 2017).
datasets. The remaining experimental results are included in the appendix C.
7

---

Learning without Isolation: Pathway Protection for Continual Learning
Implementation Details. We trained the model for 200 epoches and optimized it in conjunction with SGD, setting the batch size of the dataset as 64, For tnnearsal-based methods, we set 2000 exemplars using the herding method to select (Masana et al., 2022). In addition, we evaluate the methods on task-aware and task-agnostic settings, More experimental details could be found the appendix B.
5.4. Experimental testing of forgetting rates for different methods.
The experimental results for testing forgetting rates show that we used the ResNet18 architecture to evaluate forgetting rates on the CIFAR-100 dataset. The Figure 4 indicates that we achieve lower forgetting rates, and our method also demonstrates improved learning capabilities.
5.2. Different deep network architectures on CIFAR-100 dataset.
0.5
LwFF ICaRRI
LUCIR
The performance of all methods on the same dataset, that is CIFAR-100 100 dataset, is shown in the Table 1. Our approach surpasses the baseline performance of all without exemplar in the comparative experiments. Furthermore, when com- pared to methods employing exemplar such as iCaRL and LUCIR, our approach exhibits superior performance across the majority of test results.
kWalk
Ours
0.4
0.376
Forgetting Rate
0.315
0.3
0.257
0.305
0.2
0.118
ILORS
0.1
0.055
0,012
0
8.10
6.045.0.05
6021
0,834
0.0
5 splits
10 splits
20 splits
Table 2. Task-agnostic accuracy (%) of methods on using mini- similarity matching on different layers.
Figure 4. Task- aware forgetting rates of different methods.
Task agnostic
Method
5 splits
10 splits
20 splits
5.5. Ablation Studies
Ours
43.42 + 0.58
30.62
1.08
20.31 0.77
Ours Zlayers
26.84 + 0.86
23.50 = 0.30
16.05 0.28
To validate the effectiveness of different modules in our proposed method, LwI, we conducted ablation experiments on the model. In this context, "w/o task diversion" signifies match the channels with high similarity for every layer of the deep network, while "Ours n layers" indicates apply- ing minimization of similarity matching for the different layers. The term "with cosine" indicates employing co- sine similarity for channel similarity measurement. More experimental details and results are available in appendix C.
Ours 3layers
20.22 + 1.08
19.71 0.82
13.53 1: 0.50
Ours 4layers
15.91 1 0.95
13.69 0.58
10.42 0.39
Table Task-aware accuracy (%) of methods on using minimum similarity matching on different layers.
Task-aware
Method
5 splits
10 splits
splits
Ours
76.10 1 0.33
81.12 = 0.90
83.19
0.35
Minimum similarity matching on different layers. The re- sults show the effectiveness of minimizing similarity match- ing in the final layer.
Ours 2layers
61.92 + 0.66
71.30 0.83
75.42 + 0.44
Ours 3layers
51.01 0.92
66.49 = 0.37
71.14 0.85
Ours 4layers
42.09 + 1.45
55.27 0.41
67.48 + 0.81
Table 4. Task-agnosit accuracy (%) of methods on the validation of using task diversion module and similarity measurements.
As the network capacity increases, our performance in task- agnostic scenarios improves significantly. This is primarily attributed to the fact that, under the conditions of smaller network models, channels are more densely occupied by various tasks. As the size of the network model increases, the sparsity of the occupied channels increases.
Task agnostti
Method
5 splits
10 splits
20 splits
Ours
34,33 0.51
26.15 0.22
15,59 + 0.84
Ours w/o diversion
30.75 0.43
21.29 0.34
14.30 + 0.26
Ours with cosine
34.17 0.54
25.98 + 0.25
15,48 + 0.65
5.3. Different datasets based on ResNet18 architecture.
Effectiveness with task diversion module. Using mini-
mization of similarity matching in the final layer, facilitating channel diversion for task segregation and consequently en- suring protection across distinct tasks.
The performance of all methods in the same deep network architecture is shown in the latter two blocks in Table 1. With the escalation of dataset complexity, channels within the same structured deep network are more extensively lever- aged. Consequently, judiciously preserving channels occu- pied by different tasks becomes essential to achieve better performance under task-agnostic conditions.
Different similarity measurement methods. When mea- suring model similarity for the purpose of model fusion, the use of Euclidean distance yields slightly higher performance compared to cosine similarity.
8

---

Learning without Isolation: Pathway Protection for Continual Learning
Table 5. Task-aware accuracy (%) of methods on the validation of using task diversion module and similarity measurements.
Tuytelaars, T. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference computer vision (ECCV), pp. 139-154, 2018.
Task-aware
Method
splits
10 splits
20 splits
Ours
62.97 0.14
68.67 + 0.36
72.74 0.27
Ours w/o lusk diversion
62.12 0.26
66.94 + 0.50
71.72 0.60
Bahadi, B. and Sompolinsky, H. Sparseness and expansion in sensory representations. Newron, 83(5):1213-15 1226, 2014.
Ours with cosine
62.69 = 0.32
68.86 = 0.24
72.30 + 0.47
6. Conclusion
Chaudhry, A., Dokania, P. K., Ajanthan, T., and Tom, P. H. Riemannian walk for incremental learning: Understand- ing forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV), pp. 532-547, 2018.
This paper proposes framework for continual learning. LwI, achieving pathway protection between different tasks using model fusion approach. We validated our approach using two network structures of different sizes, and further validation can be performed on larger models. Our method acknowledge some limitations, notably the lack of validation of the proposed method using large models. Additionally, the graph matching algorithm be accelerated in future work by employing sparse matrix techniques, we will in- vestigate more effective and efficient matching processes in future work. We hope this work opens the new direction for future research, pathway protection for continual learning.
Chen, T., Zhang, Z., Liu, S., Chang, S., and Wang, Z. Long live the lottery: The existence of winning tickets in life- long learning. In International Conference on Learning Representations, 2020.
Cho, M., Lee, J., and Lee, K. M. Reweighted random walks for graph matching. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Herak- lion, Crete, Greece, September 5-11, 2010, Proceedings. Part V 11, pp. 492-505. Springer, 2010.
Impact Statement
We propose novel pathway protection-based continual learning approach. Our method is under the condition of data-free, which has significant implications for data pri- vacy protection. The introduction of novel method in our research represents& a significant technological advancement. In future work, this innovation can potentially improve the performance of Large Language Model (LLM) under the circumstance of streaming tasks.
Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Dai, W., Cahyawijaya, S., Liu, Z., and Fung, P. Multimodal end-to-end sparse model for emotion recognition. arXiv preprint MXXXv:2103.09666 2021.
De Lange, M.. Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. A continual learning survey: Defying forgetting in classifi- cation tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):33663335,, 2021.
Acknowledgement
This work was supported by the National Key Research and Development Program of China (2022YFB3204100) Scientific and Technological Innovation Project of China Academy of Chinese Medical Sciences (ZN2023A01, C12023C002YG), China Institute for Rural Studies (CIRS2025-9), and Institute for Intelligent Healthcare, Ts- inghua University. BH was supported by the NSFC General Program No. 62376235, GDST Basic Research Fund Nos. 2022A151501 1652 and 2024A1515012399 Sen Cui would like to acknowledge the financial support received from Shuimu Tsinghua scholar program.
Fan, Y., Yu, J., Mei, Y., Zhang, Y., Fu, Y., Liu, D., and Huang, T. S. Neural sparse representation for image restoration. Advances in Neural Information Processing Systems, 33:15394 15404, 2020.
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803 1803.8863335 2018
Friston, K. Hierarchical models in the brain. PLoS compu- tational biology, 4(11)xe1000211 2008
References
Gao, C., Chen, K., Rao, J., Sun, B., Liu, R., Peng. D., Zhang, Y., Guo, X., Yang, J., and Subrahmanian, V. Higher layers need more lora experts. arXiv preprint arXiv:2402.08562. 2024.
Agarwal, N., Suresh, A. T., Yu, F. X., Kumar, S., and McMahan, B. cpsgd: Communication-effiieeet and differentialll--rriatee distributed sgd. Advances in Neural Information Processing Systems, 31, 2018.
Gold, S. and Rangarajan, A. A graduated assignment algo- rithm for graph matching. IEEE Transactions on pattern analysis and machine intelligence, 18(4):377-388 1996.
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and

---

Learning without Isolation: Pathway Protection for Continual Learning
Golkar, S., Kagan, M., and Cho, K. Continual learning via neural pruning. arXiv preprint arKiv: (19990304476 2019.
Kuhn, H. W. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83- 97, 1955.
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Leordeanu, M. and Hebert, M. spectral technique for correspondence problems using pairwise constraints. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, volume 2. pp. 1482-1489. IEEE, 2005.
He, K., Zhang, X., Ren, S., and Sun, Deep residual leamn ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Leordeanu, M., Sukthankar, R., and Hebert, M. Unsuper- vised learning for graph matching. International journal of computer vision, 96:28 00.28-45,220 2012.
Hou, S., Pan, X., Loy, C. C., Wang, Z., and Lin, D. Learn- ing a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 831-839, 2019.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40<12):2935-2441 2017.
Liang. Y.S.. and Li, W.-J. Loss decoupling for task-agnostic continual learning. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1,2):, 2022.
Liu, B., Wang, M., Foroosh, H., Tappen, M., and Pensky, M. Sparse convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 806-814, 2015.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 4700-4708, 2017.
Loiola, E. M., De Abreu, N. M. M., Boaventura-Nette P. O., Hahn, P., and Querido, T. A survey for the quadratic assignment problem. European journal of operational rexearch, 176(2):657-690, 2007.
Jiang, Z., Balu, A., Hegde, C., and Sarkar, S. Collaborative deep learning in fixed topology networks. Advances in Neural Information Processing Systems, 30, 2017.
Kang, H., Mina, R. L., Madjid, S. R. H., Yoon, J., Hasegawa-Johnson M., Hwang, S. J., and Yoo, C. D. Forget-free continual learning with winning subnetworks. In International Conference Machine Learning, pp- 10734-10750. PMLR, 2022a.
Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Mallya, A., Davis, D., and Lazebnik, S. Piggyhack: Adapt- ing single network to multiple tasks by learning to mask weights. In Proceedings of the European conference on computer vision (ECCV), pp. 67-82, 2018.
Kang, M., Park, J., and Han, B. Class-increment learning by knowledge distillation with adaptive feature consol- idation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16071 16080, 2022b.
Mao, H., Han, S., Pool, J., Li, W., Liu, X., Wang. Y., and Dally, W. J. Exploring the granularity of sparsity in con- volutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 13-20, 2017.
Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. arXiv preprint arXiv: 1609.02007 2016.
Kirkpatrick, J.. Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Trabska-Barwinskk A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)3551-3522 2017.
Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A. D., and Van De Weijer, J. Class-incremental learning: survey and performance evaluation on image classifica- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-5533 2022.
Konishi, T., Kurokawa, M., Ono, C., Ke, Z., Kim, G., and Liu, B. Parameter-level soft- masking for continual learn ing. In International Conference on Machine Learning, pp. 77492-17505.. PMLR, 2023.
McCloskey, M. and Cohen, N. J. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, vol- ume 24, pp. 109-165. Elsevier, 1989.
10

---

Learning without Isolation: Pathway Protection for Continual Learning
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communicaaion--fffceeett learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273- 1282. PMLR, 2017.
Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In International conference machine learning, pp. 3987-3995. PMLR, 2017.
Zhang, W., Janson, P., Aljundi, R., and Elhoseiny, M. Over- coming generic knowledge loss with selective parameter update. In Proceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition, pp. 24046- 24056, 2024a.
Rebuffi, S. A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pp. 2001-2010, 2017.
Rubner, Y. Tomasi, C., and Guibas, L. J. The earth mover's distance metric for image retrieval. International journal of computer vision, 40:99-121, 2000.
Zhang, X., Song, D., and Tao, D. Continual learning on graphs: Challenges, solutions, and opportunities. arXiv preprint arXiv:2402. 2024b.
Saha, G., Garg, I., and Roy, K. Gradient projection memory for continual learning. arXiv preprint arXiv:2103.09762, 2021.
Zhao, L.,, Zhang, X., Yan, K., Ding, S., and Huang, W. Safe: Slow and fast parameter-efficiet tuning for con- tinual learning with pre-trained models. arXiv preprint arXhv:2411.02175 2024.
Shin, H., Lee, K., Kim, J., and Kim, J. Continual learning with deep generative replay. Advances neural informa- tion processing systems, 30, 2017.
Zhou, D.-W., Wang, Q.-.,, Ye, H.-J., and Zhan, D.C. model or 603 exemplars: Towards memory class- incremental learning. arXiv preprint arXiv:2205.13218. 2022.
Singh, S. P. and Jaggi, M. Model fusion via optimal trans- port. Advances in Neural Information Processing Systems, 33:22045-22055 2020.
Zoph, B. and Le, Q. V. Neural architecture search with reinforcement learning, arXiv preprint 1.01577 2016.
Su, Y., Zhang, R., M. Erfani, S., and Gan, J. Neural graph matching based collaborative filtering. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp. 849-858, 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N. Kaiser, L., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Wang, Z., Li, Y., Shen, L., and Huang. H. A unified and general framework for continual learning, arXiv preprint arXiv:2403.1 / 13249, 2024.
Wortsman, M., Ramanujan, V., Liu, R., Kembhavi, A., Rastegari, M., Yosinski, J., and Farhadi, A. Supermasks in superposition. Advances in Neural Information Pro- cessing Systems, 33:15173 111111184411,1111 2020.
Yang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:2310.02575, 2023.
Yu, T., Wang, R., Yan, J., and Li, B. Learning deep graph matching with channel -ndeeeeenddeennddnnt embedding and hun- garian attention. In International conference on learning representations, 2019.
Zaslavskiy, M., Bach, F., and Vert, J.-P. A path following algorithm for the graph matching problem. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, (12):2277-2242, 2008

---

Learning without Isolation: Pathway Protection for Continual Learning
A. Theoretical Supports
A.I. Analysis
We analyze one layer of deep network channel, and first-order Taylor expansion is used for analysis (Kang et al., 2022b);
L [G2 (Z(), C [GG, (Ze)
C₁ + Ze, (Ge (Zz) v), Zi, - Zt.o)
(6)
Based on the above, We find that the first-order term is deviation due to the deviation of the channel c, so we need to use some ways to reduce this deviation. Naturally, we think about whether we can make full use of the information different channels brought by different tasks, so that different tasks can occupy different channels to minimize inter-task interference.
A.2. Some methods related to graph matching
The classical methods mainly include the path-following strategy (Zaslavskiy et al., 2008), graduated assignment algo- rithm (Gold & Rangarajan, 1996), spectral matching algorithm (Leordeanu Hebert, 2005), rundom-walk algorithm (Cho al., 2010) and sequential Monte Carlo sampling (Leordeanu et al., 2012). The method of deep graph matching (Yu et al., 2019) has also received more and more attention in recent years.
The specific implementation of graph matching is illustrated in the following diagramS. Assuming that the nodes in graph X are labeled from 1 to 6, and the nodes in graph Y are labeled from A to F, the similarity matrix for pairwise nodes is shown in the upper right corner. Meanwhile, nodes are interconnected, forming various edges, such as 1-2, 3-5 in graph X, and A-B, C-E in graph Y, as indicated. The number of formed edges far exceeds the number nodes, making node matching a linear assignment problem, while graph matching poses a quadratic assignment problem. Aligning the matched graphs allows the identification of the most similar parts.
Graph Matching matrix
Graph structure
Graph X
Graph Y
X
Node Similarty
Edge Similartt
Figure The illustration graph matching. The graphs to be matched, Graph Graph are depicted on the left figure, each annotated with corresponding nodes and partial connections. The diagrams the right represent the similarity matrices between nodes between edges.
A.3. ddattive algorithm
The specific calculation formula for Sinkhorn is as follows:
P = exp(M/T),
Pg t Psg
(the sum of each row is 1),
E,
(7)
Poj
Pg + Σ, Poj
(the sum of each column is 1).
12

---

Learning without Isolation: Pathway Protection for Continual Learning
The second and third lines of the formula represent the process of enforcing bilateral constraints. The second line scales each row to while the third line scales each column 1.
The delineation of the specific computational process is articulated in Algorithm 2. notable reduction in computational requirements. Exploiting this attribute makes it particularly apt for mitigating the heightened computational complexity that arises from sparsity or the expansive nature of the assignment matrix.
The diminished complexity results in
For the similarity matrices corresponding to two channels, denoted by R. in the shallow layers of the neural network, we use the original R for the computation of the permutation matrix. Conversely, in the deep layers the deep network, we employ the inverse of R, which is R, for the computation of the permutation matrix.
Through these two phases, we enable the extraction of richer information in the shallow layers upon task arrival, while facilitating the divergence of different tasks in the deeper layers. This mechanism guarantees the preservation of task distinctiveness by permitting them to traverse separate pathways.
In this paper, we employ the Sinkhorn algorithm; however, when r S Tmins the Sinkhorn algorithm and the Hungarian algorithm (Kuhn, 1955) exhibit consistent trends.
The Earth Mover's ' Distance (EMD) algorithm (Rubner et al., 2000) involves solving optimization problem known as the transportation problem. It is the manifestation of the Sinkhorn algorithm in low-dimensional space, and the specific algorithmic formula is as follows: Given two probability distributions P and Q, represented by histograms p; and 9j for i == I,..., m and I,..., n respectively, the EMD can be calculated as follows:
EMD(P. Q)
min (FT
Tiy d(c;, dj),
where ((2, 4) is the set of all possible transportation plans (joint distributions) between P and Q. Viy represents the amount of mass to be transported from p; to qj- d(c;, dj) is the ground distance between the bin in the source histogram and the bin j in the target histogram. The EMD can be calculated using linear programming techniques:
EMD(P, = min
Ry . d(c; dz),
72
m
s.t.
No = Ps
Vi [1,m,,
Mo = 4j
20
Vi e [1, j [1,
j=1
A.4. The overall framework of LwI
The overarching framework of our algorithm operates in Algorithm 3: tasks stream into the deep network, the new model undergoes training with the input data. Upon completion of training, fusion of models occurs through maximizing similarity matching in the shallow layers and minimizing similarity matching in the deeper layers. We have observed that our method excels in merging old and new models under data-free conditions, achieving superior task preservation across different tasks. Additionally, our approach, employing misaligned fusion, provides distinct channels for different tasks, better preserving the overall integrity of the deep network.
A.5. Analysis of time complexity
In the context of our hierarchical matching, the analysis of its time complexity is presented below. Assuming a deep network with Nr layers, each layer containing C channels, the conventional graph matching ineurs time complexity of O(N*), where N represents the total number of nodes in the graph. However, by adopting a hierarchical matching strategy for deep networks, we compute the time complexity for each layer individually and subsequently sum them up. As a result, our final time complexity is O NT N4) determined by this summation:
N
O
C4) = >
(4)
0
N4). N3
(8)
Nr.
13

---

Learning without Isolation: Pathway Protection for Continual Learning
Algorithm 2 Adaptive algorithm
Input: Similarity Matrix R. Total number of iterations E, Parameter T for control the difference between Hungarian algorithm and Sinkhorn algorithm:
for each round 1,.... F do
it P. not converged then
if r <= Tmin then
adaptive algorithm Hungarian algorithm;
end
else
adaptive algorithm <- Sinkhorn algorithm: end
if layer is deep then
R
end
else
R = R end
P adaptive algorithm( R, r);
end
end
Output: the learned permutation metrics P.-
B. Implementation Details
B.1. Evaluation
In this paper, we employ two measures, task-agnostic and task-aware, to simultaneously evaluate the performance of these methods in scenarios known tasks (such as task incremental learning) and unknown tasks (such as class incremental learning). Task-agnostic refers to appending all of the classifier's 's head to given data and then taking the maximum value, where the label corresponding the maximum value is assigned the category. Task-aware, on the other hand, involves already knowing the task associated with given data and directly obtaining the maximum value from the corresponding classification head, where the label corresponding to the maximum value is assigned the data's 's category. Due to the lack of uniformity in the output of the classification head in our framework, the final performance of Task-agnostic is generally lower than that of Task-aware. Based on the findings in Table and the results below, it is evident that our approach has outperformed even the exemplar-basee methods iCaRL and LUCIR in the majority
aggnosicc scenarios.
Assuming that learning has been conducted for T tasks, the model possesses T classification heads corresponding to the tasks indicated as 1 to T, with each classification head containing the respective classes denoted as ni, -.. fry Consequently, for the two measurement methodologies mentioned above, evaluate performance using the following formulas:
Accuracy- = En Uk N
{ 1 Predicts == labela- else.
Uk =
99)
The formula for the task-agnostic method can be expressed as follows: the classification involves selecting the prediction with the highest value from total of n1 +... + ny classes to serve as the final output:
Predict, argmar((oo- .... tny
(10)
The formula for the task-aware method is follows: given that it is the f-th task, the classification involves selecting the prediction with the highest value from a total of n, classes to serve as the final output:
Predicts = argmar([ovv ... ())-
(11)
where represents i-th output of the deep network.
In the coarser granularity layers of the neural network, we match the channels with high similarity to enhance mutual common features. Conversely, in the finer granularity layers, we employ minimization of similarity matching to enable misalignment fusion of distinct task features, thereby achieving protective effect.
14

---

Learning without Isolation: Pathway Protection for Continual Learning
Algorithm 3 LwI
Input: Sequential tasks T.,..., TN. Sequential data {X1, V1), (X2, Y2) (XN, Ysh. New Model for training model_neet new, Old model for fusion model Lold. Randomly initialize model old and model new for task & Ti, ".,....Txxxx do
1a The training process of model_nen new
for epoch = ...., n do Initialize Total loss 0;
if == 1 then
Update model new whew +- neew
model training: minimize loss function defined as Ltotel = Looi
end else
initialize model new W new W fusion:
Update model_new w weew +- " new!
model training: minimize loss function defined as Crotal Loe + x Lidi
end end
1s The training process of model old
#1
if' == then
initialize model old Wold +- Wnews end
else
get model_old according Algorithm
end
end
B.2. Experiments details
We validate method several benchmark datasets against relevant continual learning baselines. We followed similar experimental setups and framework described in (Masana al., 2022). We utilized the SGD optimizer for training, and batch sizes for the training. validation and testing sets were consistently set to 64 in all experiments. During network training, the learning rate was initialized at 0.1. Furthermore, the learning rate was decreased by a factor of 0. in the 80th and 120th epochs, and the total number of training epochs was set to 200. The model architecture and training hyperparameters are the same for different methods. When employing ResNet32, the momentum for the SGD optimizer was set to 0.9, while, for ResNet18, the momentum for SGD optimizer was set to 0.0.
To gauge the distributional disparity between the and old models, introduce divergence measurement, and derive the objective of knowledge distillation through the following theoretical deductions:
log p(x) g(x))
DKL(pl)g) Expp(t)
= Σ p (xt) [log (x+) - log (a:)]
(12)
Σ 1-p log) (=p(x:) log (x:)))
In order to measure the distribution difference between the new and old models, we introduce Kullback-Leiiler(KL)) Divergence to measure, and get the optimal object of knowledge distillation through the theoretical equation. The last term in (12)ss final line represents cross-entropy, while the subsequent term signifies entropy. Consequently, when dealing with the same dataset, entropy remains constant, and the divergence between two distributions is determined by cross-entropy.
15

---

Learning without Isolation: Pathway Protection for Continual Learning
B.3. Architecture details
The model utilized three convolution blocks, each block containing five convolution layers. The number of output channels ranged from 16 to 32 and culminated in 64. In addition, a fully connected (FC) layer consisting of 64 units was employed, and the output was divided into multiple heads based on task requirements.
ResNet32:
The model utilized four convolution blocks, with each block containing two convolution layers. The number of output channels ranged from 64 to 128, 256 and increased to 512. A single fully connected (FC) layer with 512 units was employed, and the output was divided into multiple heads based on the task requirements.
ResNet18:
Table 6. Comparison between different architecture models.
Architecture
Total parameters
Model size
ResNet33
466,896
1.84MB
ResNet18
11,220,132
42.87MB
B.4. Datasets splits details
CIFAR-100 dataset contains 100 classes, each of which contains 600 32*32 color pictures, 500 are for training, and 100 are for testing. The Tiny-Imagene' dataset contains 200 classes, each of which contains 500 64*64 color images, 400 images among which were used for training, 50 used for validation, and 50 for testing.
CIFAR-100: If set to splits, it corresponds to 20 classes per head. If set to 10 splits, it corresponds to 10 classes per category. If set to 20 splits, it corresponds to 5 classes per category.
Imay-Im: If set to 5 splits, it corresponds to 40 classes per head. If set to 10 splits, it corresponds to 20 classes per category. If set to 20 splits, it corresponds to 10 classes per category.
B.5. Baselines
We compared our method with three regularization-han based methods, one architecture-baser method, and two exemplar-based methods. Regularization-baee methods involve adding regularization terms to the loss function during training to protect knowledge from previous tasks. The architecture-base method, specifically the WSN method used in this paper, identifies the optimal subnetwork using masking to achieve continual learning, making effective under task-aware conditions. Exemplar-ban based methods involve saving some data from previous tasks, mixing it with the current task's dataset for training, which contributes uniformity across different task heads and is beneficial for continuous learning in task-agnostic scenarios.
EWC (Kirkpatrick et al., 2017): This is regularization method aimed at protecting previously learned knowledge to prevent forgetting of prior tasks during new task training. It uses the Bayesian formula to constrain the distribution of model parameters, making the crucial parameters from prior tasks less susceptible to modification during new task learning. The formula shown below:
L(() = Lnew + A Σ
2 (,, - -
where Q respresents the fisher information matrix about parameters.
SI: The Path Integral method (SI) (Zenke et al., 2017) accumulates changes in each parameter along the entire learning trajectory in an online manner. The authors of this paper posit that batch updates to weights during parameter updates may lead to an overestimation of importance, while commencing from a pre-trained model may result in its underestimation.
MAS: Memory aware syASs)es(MAS) (Aljundi et al., 2018) computes the regularization term online by accumulating the sensitivity (gradient magnitude) of the learning function.
RWalk (Chaudhry et al., 2018): This method integrates the approximation of the Fisher information matrix and online path integral into a single algorithm to compute the importance of each parameter. As the outcomes of this method typically surpass those of SI and MAS methods, the comparative experiments in the main body of this paper employ this approach for evaluation.
LwF (Li & Hoiem, 2017): The core concept is to retain the knowledge from previous tasks when learning a new task,
16

---

Learning without Isolation: Pathway Protection for Continual Learning
ensuring that the model does not entirely forget the content it has already learned. By employing knowledge distillation, the outputs aligned to achieve the effect of knowledge preservation.
WSN (Kang et al., 2022a): The Lottery Ticket Hypothesis theory is employed, which posits that there exists optimal path within a neural network for given task, and this is utilized to apply channel masking. Therefore, this method is typically utilized for tasks with known training and testing processes.
iCaRL (Rebuffi et al., 2017): The model incorporates exemplars and employs knowledge distillation to preserve knowledge. The formula is shown below:
(()) = Σ
Σ
6y=n loggy (Ix) + dysu. log(1 - Iy (x:))
Lu=e
8-1
d logg, (xi) + (1 - (00-")))gg log(1 - 9y (xi)))
LUCIR (Hou et al., 2019): The use of exemplars is accompanied by the application of several strategies to mitigate the issue of the new class weight vector being larger than the old class, leading to catastrophic forgetting and the model's tendency to classify old class data as new class. In this study, we employed Cosine Normalization, Less- Gess-Forgee Constraint, and Inter-Class Separation as several methods to alleviate this issue.
Some work has explored the application of pruning methods in continual learning. However, such methods tend disrupt the overall deep network architecture. Non-structured pruning, in particular, sometimes lead to severe consequences. Based on the analysis and experiments mentioned above, we opted to employ method called "maximizing similarity matching" in the coarser granularity section. This method facilitates the fusion of different deep networks as different tasks occupy denser channels that contain more common features. In the finer granularity section, we employ a method called "minimizing similarity matching" to perform misalignment fusion different deep network channels, thereby safeguarding the distinct characteristics of different tasks.
C. More experiments
C.1. The performance of various methods when the number of tasks increases by an order of magnitude.
We increased the number of tasks by an order of magnitude for testing, dividing the Tiny-ImageNer dataset into 100 tasks, each with two categories. The comparison methods primarily focus on the latest approaches to parameter protection and the experimental results are shown in Figure 6. From the Figure, we can observe that our method achieves the best performance when compared to other approaches, The analysis suggests that the use isolation based methods reduces the number of learnable parameters in the network, leading decreased learning ability for subsequent tasks and demonstrates that our pathway protection approach can preserve knowledge of old tasks while generalizing to task knowledge.
SPG
WSN
LwF
0.8
SPU
EWC
Ours
0.646
0.606
0.6
0.568
Accuracy
0.529
0.506
0.434
0.4
0.2
0.0
Methods
Figure 6. Twsk-aware accuracy of methods when the number of tasks is 100.
17

---

Learning without Isolation: Pathway Protection for Continual Learning
C.2. Discussion about Table 1
Through the comparison of results and analysis of the four experimental sets, we summarize the findings and elucidate the underlying reasons.
Firstly, regarding the network model capacity, we posit that, under identical scenarios, the gradual increase in model capacity leads to a sparser channel occupancy. This sparsity constitutes a key aspect of our proposed methodology. Thus, the conclusions drawn from the experiments, particularly the higher performance gains achieved by ResNet18 over ResNet32 under task-agnostic conditions, validate the correctness of our proposed task diversification concept, as depicted in Figure 2.
Secondly, with respect to the dataset, our observations indicate that under equivalent deep network architectures, the superiority of our method becomes more pronounced with increasing dataset complexity. This emphasizes the efficacy of our approach in handling intricate datasets. EWC and RWalk methods are designed to address issues arising from significant data variations, making it challenging for these regularizations to effectively constrain parameter shifts, LwF, primarily employed for training different tasks, experiences the blending of task knowledge, as illustrated in Figure 2. This blending is likely to result in outcomes inferior to our method. WSN requires a mask when dealing with various tasks, limiting its applicability to task-aware testing. Additionally, as the number tasks increases, the reduction in learnable parameters diminishes its effectiveness. iCaRL and LUCIR methods benefit from partial datasets of all previous tasks during the training of subsequent tasks, offering advantages for task-agnostic testing.
Thirdly, in the situation of task agnostic, our deep network exhibits lower performance compared to exemplar-based continual learning (CL) methods on the CIFAR-100 dataset. We posit that, while our deep network learns each task individually, the persistent setup of learning classification heads results in inconsistent output sizes for these task-specific heads, thereby posing challenges in scenarios of task uncertainty. The utilization of exemplars involves incorporating partial data from previous tasks into the current dataset during training, mitigating the inconsistency classífication heads. However, with the complexity of datasets such as Tiny-Imagenet, the performance improvement derived from exemplar usage is surpassed by the benefits brought about by our approach of task-specific streams.
C.3. Experiments analysis in Table
According to the experiment on dataset CIFAR- 100, architecture ResNet32, our approach surpasses the baseline performance of all non-replay pools in the comparative experiments. Compared to the best-performing regularization method LwF, our approach demonstrates maximum improvement of 5.88% under task-agnostic conditions. In scenarios of task awareness, the performance is further enhanced, showing improvement of 6.28%. In comparison to the WSN method, which primarily designed for task incremental learning, hence not applicable to scenarios of task agnosticism. Under the task-aware setting, our method achieves an approximately 1% improvement. When contrasted with exemplar- based approaches, our method achieves its peak performance under task-aware 5/10 splits conditions.
With the increase in model size of deep network, the improvement of our method becomes more pronounced under task- agnostic conditions. As shown in the second block, that is the experiment dataset CIFAR- 100, architecture ResNet18 88, our method outperforms other comparative approaches under task-aware conditions. In all other conditions, method surpasses the performance of the methods employed in the comparative experiments. When compared to the best-performing regularization method, LwF, our approach exhibits a maximum improvement of 5.04% under task-aware conditions and an even more substantial improvement of 7.29% under task-agnostic conditions. In contrast to the WSN method, our approach demonstrates a performance improvement of around 2.45% in task-aware scenarios. In comparison with exemplar- approaches, our method attains its peak performance under task-aware 10/20 splits conditions.
C.4. Experiments on Timy-Imagenet dataset using ResNet32.
According to Table 7 and 8, our method surpasses almost all comparative results, except for iCaRL under task-agnostic 5-splits conditions. In comparison the LwF method, our approach exhibits a maximum improvement of up to 3. % under task-agnostic conditions and 7.18% under task-aware conditions. When contrasted with the LUCIR method, our performance surpasses by maximum of 3.15% under task-agnostic conditions and 10.25% under task-aware conditions.
Under conditions of task-agnostic, analysis of the results in Table 7 reveals that our method, with the exception of a slight underperformance compared to the iCaRL method in the 5-splits scenario, consistently outperforms the comparative experiments in all other cases. In comparison to the LwF method, which exhibits the best performance among regularization
18

---

Learning without Isolation: Pathway Protection for Continual Learning
methods, our approach demonstrates improvement of up 3.1%. Furthermore, when contrasted with the EWC method, method achieves maximum improvement of 9.61%. Notably, when compared exemplar-based methods on the Tiny-Imagenet dataset, our approach even surpasses them, highlighting the advantages of our task-shifting methodology. This is evident in the ability of our method to achieve higher activation levels for each channel corresponding to a specific task, even without the unification of classification heads. Thus, task specialization is achieved, with the activation intensity for each channel surpassing that of all other tasks, emphasizing the effectiveness of our ssisftii approach.
Table 7. Task-agnostic accuracy (%) of methods on the Tiny-Imagenet dataset based on the architecture of ResNet32.
Task-agnostic
Method
Exemplar
5 splits
10 splits
20 splits
EWC
7.76 = 0.75
3,80 = 0.32
2.60 = 0.19
RWalk
11.10
4.71 = 0.21
4.54 + 0.63
no
LwF
20.12 0.63
13.72 0.52
9.11 = 0.40
Ours
22.21 + 0.39
16.75 + 0.21
12.21 0.29
iCaRL
22.45 + 0.14
16.48 = 0.84
9.94 0.24
2000
LUCIR
20.05 0.16
13.60 = 0.42
10.38 = 0.40
Under conditions of task-aware, examination the results in Table reveals that our method consistently outperforms the comparative experiments.
In comparison to the LwF method, which demonstrates the best performance among regularization methods, approach exhibits maximum improvement of up to 7.18%. Furthermore, when contrasted with the RWalk method, our approach surpasses it even more significantly. reaching up to 25.71%. In comparison to the architecture- method WSN, the advantages of our method under conditions of task knowledge are not particularly pronounced, with the highest improvement being 2.36%. However, when compared to exemplar-base based methods, the superiority of our approach becomes notably evident.
Table Task-aware accuracy (%) of methods on the Tiny-Imagent Imagenet dataset based on the architecture ResNet32.
-aare
Method
Exemplar
5 splits
10 splits
20 splits
EWC
31.97 + 1.18
36.71 = 1.23
42.91 = 0.31
RWalk
43.75 + 2.08
43.22 + 2.11
37.66 + 1.74
LwF
no
47.78 + 0.98
52.39 0.60
56.19 0.54
WSN
50.06 + 0.37
56.29 0.75
63.24
Ours
52.42 + 0.59
57.84 + 1.15
63.37 = 0.44
iCaRL
44.87 + 0.61
49.42 + 1.29
54.43 1 0.73
2000
LUCIR
44.53 + 0.68
46.86 + 1.29
53.12 = 0.73
Based the above experimental results, we observe that method not only accomplishes task channel specialization under conditions of task agnostic without the need for deliberate unification of classification heads but also, under conditions of task awareness, exhibits comparative advantage. In contrast to other methods, our approach shows minimal catastrophic forgetting of previously acquired knowledge and, in certain instances, even demonstrates facilitating effect. This observed promotion of cooperation among tasks is a notable outcome of our method.
C.5. Experiments on CIFAR-100 dataset using AlexNet.
According to Table 9 and 10, our method surpasses all comparative results.
Under conditions of task- agnostic,, analysis of the results in Table 9 reveals that our method consistently outperforms the comparative experiments in all cases. In comparison to the LwF method, which exhibits the best performance among regularization methods, our approach demonstrates an improvement of up to 2.0%. Furthermore, when contrasted with the EWC method, method achieves maximum improvement of 16.1%. Thus, this is evident that task specialization is achieved, with the activation intensity for each channel surpassing that of all other tasks, emphasizing the effectiveness of our task-shifting approach.
Under conditions of task-aware, examination of the results in Table 10 reveals that our method consistently outperforms the comparative experiments. In comparison to the LwF method, which demonstrates the best performance among regularization methods, our approach exhibits a maximum improvement of up to 1.8%. Furthermore, when contrasted with the SI method,
19

---

Learning without Isolation: Pathway Protection for Continual Learning
Table 9. Task accuracy (%) of methods on the CIFAR 100 dataset based on the architecture of AlexNet.
Task Task-agnostic
Method
Exemplar
splits
10 splits
20 splits
EWC SI
13.8 + 1.4
6.9 = 2.0
4.5 = 0.8
14.2 + 1.4
6.8 + 2.0
3.84 0.3
RWalk
14.0 = 1.7
8.4 = 1.4
3,7 1.4
MAS
no
14.3 + 1.1
5.3
LwF
27.9 - 1.7
19.5 1.6
10.7 111
Ours
29.9 + 0.6
20.4 + 0.9
11.2
approach surpasses it even significantly, reaching up 27.8%. In comparison to EWC method, the advantages of our method, with the highest improvement being 26.6%.
Table 10. Task aware accuracy (%) of methods on the CIFAR-100 dataset based on the architecture AlexNet.
Task-aware
Method
Exemplar
splits
10 splits
20 splits
EWC SI
34.6 + 2.0
38.9 = 2.7
45.5 1 3.2
35.9 + 1.2
37.7 + 1.2
43.7 + 3.2
RWalk
37.2 + 1.7
38.5 + 1.1
43.9 $3.9 2.9
no
MAS
37.1 + 1.3
42.1 + 1.9
50.5 + 4.0
LwF
58.8 + 1.1
64.8 - 1.8
68.6 0.8
Ours
60.6 + 0.5
65.5 + 0.8
68.8 1.0
Based on the above experimental results, we observe that our method not only accomplishes task channel specialization under conditions of task agnostic without the need for deliberate unification of classification heads but also, under conditions of task awareness, exhibits comparative advantage. In contrast other methods, our approach shows minimal catastrophic forgetting of previously acquired knowledge and, in certain instances, even demonstrates facilitating effect. This observed promotion of cooperation among tasks is notable outcome of our method.
C.6. Experiments different similarity measurement formulas.
In this study, the Euclidean distance and cosine similarity were employed measure the distance between two model channels.
The Euclidean distance primarily quantifies the distance between two vectors in space, with smaller absolute values indicating closer proximity. It is a commonly used distance measurement formula. On the other hand,
Cosine similarity gauges the angle between two vectors within the same sphere, mainly reflecting directional differences. Larger numerical values denote smaller angle discrepancies, indicating closer proximity in space. It widely used formula for measuring similarity.
This section compared and validated the use of Euclidean distance and cosine similarity to measure channel proximity and revealed that, in most cases, using distance measurement is preferable to using cosine similarity.
Euclidean Distance
||a bll2
(13)
a b
Cosine Similarity
(14)
|a|| ||b||
where a and represent two vectors.
According to Table 11 and 12, it is observed that under two different testing conditions, when measuring model similarity for the purpose of model fusion, the use of Euclidean distance consistently yields slightly higher performance compared to cosine similarity. This trend holds true across various scenarios, with the notable exception of the task-agnostic 10-splits condition, where results obtained using Euclidean distance are recorded at 30.62%, while those using cosine similarity are slightly higher at 31.16%. Consequently, based on the comparative experimental outcomes presented in this paper, the choice made employ Euclidean distance for model fusion, facilitating comprehensive evaluation of testing effectiveness.
20

---

Learning without Isolation: Pathway Protection for Continual Learning
Table II. Task-agnostic accuracy (%) of methods between different similarity mcasurement formulas.
Task -ansssanostic
Dataset
Architecture
Method
5 splits
10 splits
20 splits
Ours
43.42 + 0.58
30.62 1.08
20.31 + 0.77
CIFAR-100
ResNet32
Ours with cosine
43.09 = 0.95
31.16 + 0.78
20.03 + 0.93
Ours
51.95 + 0.56
36.36 + 1.06
22.99 + 0.39
CIFAR-100
ResNet18
Ours with cosine
51.78 + 0.92
37.06 1.12
22.77 + 0.57
Ours
22.21 + 0.39
16.75 + 0.21
12.21 + 0.29
Imay-em
ResNet32
Ours with cosine
22.26 + 0.60
16.96 0.16
11.94 + 0.41
Table 12. Task-aware accuracy (%) of methods between different similarity measurement formulas.
Task-aware
Dataset
Architecture
Method
splits
10 splits
20 splits
Ours
76.10 + 0.33
81.12 + 0.90
83.19 + 0.35
CIFAR-100
ResNet32
Ours with cosine
75.63 + 0.46
79.35 0.86
83.05 + 0.36
Ours
81.10 + 0.80
84.90 + 0.36
86.49 + 0.55
CIFAR-100
ResNet18
Ours with cosine
80.72 + 1.09
84.01 + 1.02
85.91 + 0.85
Ours
52.42 + 0.59
57.84 1.15
63.37 + 0.44
Tiny-Imagenet
ResNet32
Ours with cosine
52.68 + 0.53
58.37 + 0.90
62.59 + 0.81
C.7. Experiments on without using knowledge distillation module.
According to Table 13, in order to verify the effectiveness of our method, we also carried out ablation experiments on the knowledge distillation module, and the results showed that this case, the knowledge generation would be shifted to a large extent, thus reducing the effect.
Table 13.. Task-aware accuracy (%) our methods without using knowledge distillation module.
Dataset
Architecture
Method
5 splits
10 splits
20 splits
Ours
81.10 - 0.80
84.90 + 0.36
86.49 + 0.55
CIFAR-100
ResNet18
Ours w/o KD
75.91 = 0.85
73.91 + 0.51
71.98 + 1.24
C.8. An example of whether to use knowledge distillation module.
According to Table 14, Table 15 and Table 16, applying knowledge distillation to each layer method results in minimal changes in the model's parameter space. Conversely, without using knowledge distillation method leads to significant differences. The following three tables depict the accuracy(%) obtained from utilizing the ResNet32 model under the same conditions for five tasks on the CIFAR-100 dataset. can be observed that our method sometimes achieves better performance after training
tasks than after the initial training.
Table 14. Task-aware accuracy (%) of our method using knowledge distillation module for every layer.
Task-ID
Task
Task2
Task3
Task4
Task5
Overall
Task
78.2
0
0
0
0
78.2
Task2
78.0
68.1
0
0
0
73.1
Task3
77.6
63.1
68.7
0
69.8
Task4
74.4
59.8
62.0
64.7
0
65.2
Task5
74.2
60.9
53.5
65.0
63.0
63.3
21

---

Learning without Isolation: Pathway Protection for Continual Learning
Table 15.
Task-aware accuracy (% of methad.
Task-ID
Taskl
Task2
Task3
Task4
Task5
Overall
Taskl
78.2
0
0
0
0
78.2
Task2
75.3
74.2
0
0
0
74.8
Task3
74.8
76.6
75.2
0
0
75.5
Task4
75.3
76.3
75.7
76.0
0
75.8
Task5
74.2
75.3
75.5
77.2
76.1
75.7
Table 16. Task-aware accuracy (%) of our method without using knowledge distillation module.
Task-ID
Task1
Task2
Task3
Task4
Task5
Overall
Task1
78.2
0
0
78.2
Task2
70.1
77.7
0
0
73.9
Task3
61.0
73.0
72.7
0
68.9
Task4
54.8
69.0
65.8
84.6
0
68.5
Task5
52.3
59.5
58.9
78.4
81.7
66.2
C.9. Experiments on the validation of effectiveness with task diversion module.
In order to validate the effectiveness of our proposed task diversion method, we compared it with an approach that does not perform deep-level minimization of similarity. This approach involves using maximization of similarity at all layers for model fusion. Our findings indicate that our method yields better results as the complexity of the task increases.
The entry labeled "Ours w/o task diversion" in the Table 17 and 18 signifies that each layer of the deep network model employs maximization of similarity matching, signifying the absence of task-specific parameter diversion during model fusion. As evidenced by the results, our approach incorporates the minimization of similarity matching in the final layer, facilitating channel diversion for task segregation and consequently ensuring protection across distinct tasks. Consequently, under equivalent conditions, our method consistently outperforms approaches solely relying on matching the channels with high similarity.
Table 17. Task-agnostic accuracy (%) of methods on the validation of effectiveness with task diversion module.
Task aank-agnostic
Dataset
Architecture
Method
5 splits
10 splits
20 splits
Ours
43.42 = 0.58
30.62 + 1.08
20.31 - 0.77
CIFAR-100
ResNet32
Ours w/o task diversion
41.73 = 0.41
29.94 = 0.89
18.02 + 0.97
Ours
51.95 - 0.56
36.36 + 1.06
22.99 + 0.39
CIFAR-100
ResNet18
Ours w/o task diversion
51.13 0.43
35.70 + 0.88
21,49 + 0.54
Ours
22.21 0.39
16.75 + 0.21
12.21 + 0.29
Imageema
ResNet32
Ours w/o task diversion
20.80 - 0.46
15.31 + 0.75
10.68 + 0.53
Table 18, Task-aware accuracy (%) of methods on validation effectiveness with diversion module.
Task-aware
Dataset
Architecture
Method
splits
10 splits
20 splits
Ours
76.10 + 0.33
81.12 + 0.90
83.19 + 0.35
CIFAR-100
ResNet32
Ours w/o task diversion
75.54 + 1.37
79.38 + 1.02
81.43 + 1.51
Ours
81.10 + 0.80
84.90 + 0.36
86.49 + 0.55
CIFAR-100
ResNet18
Ours w/o task diversion
80.17 - 0.35
83.60 + 0.86
84.15 - 0.96
Ours
52.42 - 0.59
57.84 + 1.15
63.37 - 0.44
Tiny-Imagent
ResNet32
Ours w/o task diversion
50.26 = 0.66
55.07 = 1.18
60.98 + 1.15
22

---

Learning without Isolation: Pathway Protection for Continual Learning
C.10. Experiments using minimum similarity matching on different layers.
In order to determine the most effective layers for performance improvement through minimizing similarity matching, we conducted extensive comparative experiments with the ResNet32 model. Specifically, we tested the impact of the final layer, last two layers, last three layers, and last four layers. Remarkably, we observe very similar results across these configurations.
According to Table 19 and 20, we observe that when minimizing similarity matching for the final two, three, and four layers, the ultimate results are comparable to those obtained by minimizing similarity matching for a single layer. However, the majority of cases, the performance is lower than when minimizing similarity matching for just one layer. Therefore, based on the results of our previous comparative experiments, opted minimize similarity matching for the final layer.
Table 19. Task Task-agnostic accuracy (%) of methods using minimum similarity matching different layers.
Task-agnostic
Dataset
Architecture
Method
5 splits
10 splits
20 splits
Ours
22.21 + 0.39
16.75 + 0.21
12.21 + 0.29
Ours 2layers
22.05 + 0.60
16.43 + 0.38
11,40 + 0.21
Imay-nt
ResNet32
Ours 3layers
21.65 + 0.87
16.65 + 0.37
11.44 + 0.31
Ours 4layers
21.61 + 0.79
16.66 + 0.17
11.54 + 0.27
Table 20, Task-aware accuracy (%) of methods on using minimum similarity matching on different layers.
Task-aware
Dataset
Architecture
Method
5 splits
10 splits
20 splits
Ours
52.42 + 0.59
57.84 + 1.15
63.37 = 0.44
Ours 2layers
52.41 + 0.58
56.10 + 0.59
61.01 = 0.58
Imay-lmagenet
ResNet32
Ours 3layers
51.80 + 0.81
56.60 + 0.31
60.85 + 1.07
Ours 4layers
52.20 + 0.53
56.84 + 0.22
61.33 + 0.67
According to Table 19 and 20, observe that when minimizing similarity matching for the final two, three, and four layers, the ultimate results are comparable to those obtained by minimizing similarity matching for single layer. However, in the majority of cases, the performance is lower than when minimizing similarity matching for just one layer. Therefore, based on the results of our previous comparative experiments, we opted to minimize similarity matching for the final layer.
C.11. Devices
In the experiments, we conduct all methods on a local Linux server that has two physical CPU chips (Intel(R) Xeon(R) CPU E5- E5-2640 v4 2.40GHz) and 32 logical kernels. All methods are implemented using Pytorch framework and all models are trained on GeForce RTX 2080 GPUs.
23


Reviews:
[{"review_id": "WCwVWERcNu", "reviewer": "Reviewer_mxBD", "summary": "The paper introduces a novel approach to continual learning, termed \"Learning without Isolation\" (LwI), which aims to mitigate catastrophic forgetting by protecting distinct activation pathways for different tasks within a deep network. The key idea is to allocate unique pathways for each task, ensuring that knowledge from previous tasks is preserved while learning new tasks. The authors propose a data-free continual learning method based on graph matching, which aligns channels in the deep network before model fusion. This approach is inspired by the sparsity of activation channels in neural networks and the hierarchical structure of the brain. The method is evaluated on CIFAR-100 and Tiny-Imagenet datasets, demonstrating superior performance compared to existing continual learning methods, particularly in task-agnostic scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to W1:** Thank you for the reviewer's reminder. We have supplemented our theoretical derivations.\n## 1. Core Theoretical Framework\n### 1.1 Shallow Layers (Shared knowledge)\n**Core:** Minimize weight differences through optimal transport alignment of similar channels\n1. Parameter Update\nMerged shallow weights:\n$$W_S^{\\text{merged}} = (1-\\eta)W_S^{\\text{old}} + \\eta P_S W_S^{\\text{new}}$$\nwhere $P_S$ is the permutation matrix maximizing similarity.\n2. Difference Bound\n- Define channel similarity matrix $K_S$ where:\n$$\nP_S = \\arg\\max_P \\text{tr}(P^\\top K_S) \n\\quad \\text{where} \\quad \nK_S[i,j] = \\frac{w_i^\\top w_j}{\\|w_i\\|\\|w_j\\|}, \nw_i = W_S^{\\text{old}}[:,i], \nw_j = W_S^{\\text{new}}[:,j]\n$$\n- Optimal transport guarantees existence of $\\kappa_S = \\max K_S[i,j]$:\n$$\\|P_S W_S^{\\text{new}} - W_S^{\\text{old}}\\|_F^2 = 2(1 - \\kappa_S)$$\n- Therefore:\n$$\\|W_S^{\\text{merged}} - W_S^{\\text{old}}\\| \\leq \\eta \\sqrt{2(1 - \\kappa_S)}$$\n### 1.2 Deep Layers (Task-specific isolation):\n**Core:** Limit inter-task interference through sparse channel separation\n1. Channel Overlap\nLet tasks $t$ and $t'$ share $\\epsilon C$ channels where:\n$$|\\mathcal{A}_t \\cap \\mathcal{A}_{t'}| \\leq \\epsilon C$$\n2. Probability Bound\nUsing Chernoff bound:  \n$$\\mathbb{P}(\\text{Overlap} \\geq \\epsilon C) \\leq e^{-\\epsilon C/2}$$\n3. **Worst-case Gradient Conflict**\nWith Lipschitz constant $\\lambda$ and task separation $\\Delta$:\n$$\\| \\nabla L_t - \\nabla L_{t'} \\| \\leq \\lambda / (\\epsilon \\Delta^2)$$\n**Simplified Form**:\nWhen $\\Delta \\propto 1/\\epsilon$ (implied by sparsity):  \n$$\n\\text{Deep Term} = \\lambda \\epsilon\n$$\n### 1.3 Dynamic Error Term Derivation\n**Core:** Azuma-Hoeffding Inequality for martingales\n1. Regret Definition\nFor task sequence with losses $L_t(\\theta_t)$:\n$$R(T) = \\sum_{t=1}^T L_t(\\theta_t) - \\min_\\theta \\sum_{t=1}^T L_t(\\theta)$$\n2. Concentration Bound\nWith probability $1-\\delta$:\n$$R(T) \\leq \\sqrt{2T \\log(1/\\delta)}$$\n3. Per-Task Error\nSetting $\\delta = 1/T$:  \n$$\\frac{R(T)}{T} = \\mathcal{O}\\left(\\sqrt{\\frac{\\log T}{T}}\\right)$$\n### 1.4 Forgetting Bound Theorem\n$$\\mathcal{F}(T) \\leq \\eta\\sqrt{2(1-\\kappa_S)} \\, \\text{(Shallow)} + \\lambda \\epsilon \\, \\text{(Deep)} + O\\left(\\sqrt{\\frac{\\log T}{T}}\\right)$$\nWhere:\n- $\\kappa_S = \\max \\mathbf{K}_S[i,j]$ (Peak shallow similarity)\n- $\\epsilon$ = Channel overlap ratio\n- $\\lambda$ = Task conflict intensity (Lipschitz constant)\n### 1.5 Hyperparameter Settings\n|Parameter|Effect on Forgetting|\n|-|-|\n|$\\eta$|$\\eta \\downarrow \\Rightarrow \\mathcal{F}_S \\downarrow$|\n|$\\epsilon$|$\\epsilon \\downarrow \\Rightarrow \\mathcal{F}_D \\downarrow$|\n|$\\kappa_S$|$\\kappa_S \\uparrow \\Rightarrow \\mathcal{F}_S \\uparrow$\n\n## 2. Task Capacity Bound\n1. Define regret $R(T) = \\sum_{t=1}^T L_t(\\theta_t) - L_t(\\theta^*)$\n2. Apply Azuma-Hoeffding:\n$$ \\mathbb{P}(R(T) \\geq \\sqrt{2T \\log T}) \\leq 1/T $$\n3. Per-task average: $\\frac{R(T)}{T} = \\mathcal{O}(\\sqrt{\\log T/T})$\n4. Expected overlap: $\\mathbb{E}[X] = \\epsilon^2 C$\nConcentration:\n$$ \\mathbb{P}(X \\geq \\epsilon C) \\leq e^{-\\epsilon^2 C/2}$$\n5. Conflict bound: $\\lambda \\epsilon C$\n6. Channel combinations: $\\binom{C}{\\epsilon C} \\geq T$\n7. Stirling approximation:\n$$ \\log T \\leq C H(\\epsilon) $$ where $H(\\epsilon)$ is binary entropy\n8. Final bound:\n$$ T_{\\max} \\leq \\frac{C}{2\\epsilon} \\log(1 + \\frac{C}{\\epsilon \\Delta^2}) $$\n## 3. Experiments\nWe divided Tiny-ImageNet into 100 tasks and compared different approaches.\n\n|Method| SPG |SPU|WSN|EWC|LwF|Supsup|Ours\n|-|-|-|-|-|-|-|-\n|100 splits|43.40|52.90|60.60|50.61|56.82|49.47|**64.63**\n\n**Response to W2:** Thank you for the reviewer's suggestions.\n- At first, we considered that CL is designed for resource-constrained edge devices (e.g., mobile phones) where models must adapt quickly to new tasks with limited computation/memory. \n- To address the reviewer's concern. We conducted corresponding experimental tests on the ImageNet-R dataset and also performed tests using ResNet50.\n\n**(1) ResNet32-based model on the Imagenet-R**\n|Method|5 splits|10 splits|20 splits|\n|-|-|-|-|\n|EWC|24.6|27.1|29.3\n|LwF|25.0|28.6|30.3\n|RWalk|26.1|28.4|26.6\n|WSN|28.2|30.4|32.1\n|SPG|23.2|24.1|22.7\n|SPU|26.2|27.6|27.1\n|GPM|26.7|27.1|29.3\n|Ours|**32.8**|**34.3**|**37.3**\n\n**(2) ResNet50-based model on the Cifar100**\n|Method|5 splits|10 splits|20 splits|\n|-|-|-|-|\n|EWC|70.9|62.7|57.6\n|LwF|79.2|80.3|82.7\n|RWalk|70.3|66.3|62.1\n|WSN|80.2|81.4|84.3\n|SPG|55.3|57.2|55.1\n|SPU|60.5|62.1|61.7\n|Ours|**84.4**|**87.5**|**89.7**\n\n**Response to Q1:** Thanks for the comments. We would like to explain them as follows:\n- When the number of tasks grows or data complexity rises, the model requires more channels to mitigate inter-task interference.\n- ResNet18's deep channels (512) far exceed those of ResNet32 (64), enabling ResNet18 to better meet the demands of sparse allocation in complex task scenarios. We verified that, under the same scenario, the sparser the deeper layers of the network, the greater the performance improvement.", "claims_and_evidence": "The claims made in the submission are generally supported by clear and convincing evidence, particularly through extensive experimental results on CIFAR-100 and Tiny-Imagenet datasets, which demonstrate the method's superiority over state-of-the-art continual learning approaches. The ablation studies further validate the effectiveness of pathway protection and graph matching. However, some claims, such as the scalability to larger models and the theoretical foundations of pathway protection, lack sufficient evidence. The paper primarily validates the method on smaller models (ResNet32, ResNet18), and a deeper theoretical analysis is absent. Addressing these gaps by including experiments on larger models and providing more theoretical insights would strengthen the overall credibility of the claims.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for addressing the problem of continual learning, particularly in mitigating catastrophic forgetting. The use of benchmark datasets like CIFAR-100 and Tiny-Imagenet, along with task-agnostic and task-aware evaluations, provides a robust framework for assessing the method's performance.", "theoretical_claims": "The authors analyze one layer of a deep network channel, and first-order Taylor expansion is used for analysis. It's been correctly demonstrated.", "experimental_designs_or_analyses": "Please refer to Claims And Evidence part", "relation_to_broader_scientific_literature": "The proposed method of pathway protection via graph matching builds on prior work in regularization-based, rehearsal-based, and architecture-based continual learning approaches.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "## Paper Strengths\nThe paper presents a novel and theoretically grounded approach to continual learning by leveraging pathway protection and graph matching. This is a significant departure from traditional methods that rely on regularization, rehearsal, or dynamic architectures.\n\nThe proposed method does not require storing data from previous tasks, which is a significant advantage in terms of data privacy and storage efficiency.\n\nThe authors provide extensive experimental results showing that their method outperforms several state-of-the-art continual learning methods, particularly in task-agnostic settings. The results are convincing and well-supported by ablation studies.\n\nThe paper effectively leverages the sparsity of activation channels in deep networks, drawing parallels with neuroscience to justify the approach. This adds a layer of biological plausibility to the method.\n\n\n## Major Weaknesses\nWhile the empirical results are strong, the paper lacks a thorough theoretical analysis of why the proposed method works. For instance, the authors could provide more insights into the conditions under which pathway protection is most effective or how the method scales with the number of tasks.\n\nThe paper primarily validates the method on relatively small models (ResNet32, ResNet18). It would be beneficial to see how the method performs on larger models, such as those used in modern large-scale applications.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. Why is the model limited to the ResNet18 and ResNet32?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ACCtcDcwxy", "reviewer": "Reviewer_EQUe", "summary": "This paper proposes a new framework for continual learning (CL) called Learning without Isolation (LwI), which introduces pathway protection as a mechanism to mitigate catastrophic forgetting. Unlike traditional CL methods that focus on parameter protection, LwI prioritizes preserving activation pathways in deep neural networks, inspired by neuroscientific principles of sparsity in neural activations. The model fusion process is framed as a graph matching problem, where activation pathways in neural networks are aligned using a similarity-based approach. The proposed method is rehearsal-free and the authors evaluate it on CIFAR-100 and Tiny-ImageNet, using ResNet-32 and ResNet-18, demonstrating that the proposed method outperforms the baselines.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to C1:** Thank you for the reviewer's reminder.\n- We conducted a corresponding analysis, including the optimization of our proposed method in terms of time complexity. In the context of our hierarchical matching approach, we analyze its time complexity as follows. Given a deep network with $N_L$ layers, each containing $C$ channels, traditional graph matching incurs a time complexity of O(N^4), where $N$ denotes the total number of nodes in the graph (All neurons in the neural network). However, by employing a hierarchical matching strategy for deep networks, we can compute the time complexity separately for each layer and then aggregate the results. Consequently, the overall time complexity of our approach is:\n$$ \\begin{split} & O(\\sum_{1}^{N_L} C^4) = O(\\sum_{1}^{N_L} (\\frac{N}{N_L})^4) = O(\\frac{1}{N_L^3} N^4). \\end{split} $$\n- To address the reviewers' concerns, we tested the time cost and found that due to the addition of the matching fusion module, our method's runtime is second only to LwF.\n\n**Runtime of ResNet32 on CIFAR-100 (min)**\n|Method|5 splits|10 splits|20 splits|\n|-|-|-|-|\n|EWC|157|159|201\n|LwF|105|145|173\n|RWalk|225|236|260\n|WSN|276|298|321\n|Ours|110|157|186\n\n**Response to C2 and Experimental2,3:** Thank you for the reviewer's reminder. \n- During inference, the space and time complexity of [1] are both **O(N)**, where N is the number of tasks. In contrast, our method has both time and space complexity of **O(1)**. To ensure a fair comparison under **O(1)** complexity, we tested using the mask of the last task as well as the intersection of masks from different tasks, ensuring that the time complexity during testing remains **O(1)**. The results of methods such as **WSN** and **GPM** were obtained through comparisons in task-incremental learning.\n**ResNet18-based model on the Cifar100.**\n|Method|10 splits|20 splits|\n|-|-|-| \n|Ours|**50.9**|**47.7**\n|HAT + [1]\\(Final)|10.2|12.9\n|HAT + [1]\\(Intersection)|13.5|15.4\n|SupSup + [1]\\(Final)|15.2|18.8\n|SupSup+ [1]\\(Intersection)|18.7|23.4\n\n- Due to differences in the **training environment, dataset splitting methods, and training details** (e.g., we use **200 epochs**, whereas [1] uses **700 and 1000 epochs**), the final training results vary. Additionally, the baseline methods in our paper are implemented based on the code from **[2]**, which aligns well with the corresponding results in **[2]**. We integrated the **LwF** method into the training process of **[1]**, leading to the following results:\n**ResNet18-based model on the Cifar100.**\n|Method|10 splits|20 splits|\n|-|-|-| \n|Ours|**50.9**|**47.7**\n|LwF|42.2|40.8\n\n[2] Class-incremental learning: survey and performance evaluation on image classification. TPAMI,2022.\n\n**Response to C3 and Experimental2,3:** Thank you for the reviewer's reminder.\nThank you for the reviewer's reminder. \nThe reviewer is indeed correct—it is possible to infer the task ID, just not using the approach in [1]. The SupSup paper presents an algorithm for task ID inference based on optimization rather than an O(N) complexity method. However, the inferred task ID may not always be accurate (especially when the number of tasks is large, such as 100 tasks). When we compared this approach with ours, we found that our method outperforms it in task-agnostic scenarios. Lastly, we will revise our wording and include additional experiments in task-agnostic settings.\n|Method|Supsup|Ours\n|-|-|-\n|100 splits|7.47 |**13.63**\n\n**Response to Theory:** We greatly appreciate the reviewer for pointing out this issue. We have performed the corresponding theoretical derivation and will include it in the paper.\n\nThe detailed derivation process for 1 and 2 can be found in the response to Reviewer mxBD.\n### 1. Forgetting Bound\n$$\\mathcal{F}(T) \\leq \\eta\\sqrt{2(1-\\kappa_S)} \\, \\text{(Shallow)}+\\lambda \\epsilon \\, \\text{(Deep)}+O\\left(\\sqrt{\\frac{\\log T}{T}}\\right)$$\n### 2. Task Number\n$$T_{\\max} \\leq \\frac{C}{2\\epsilon} \\log(1 + \\frac{C}{\\epsilon \\Delta^2})$$\nWe divided Tiny-ImageNet into 100 tasks and compared different approaches.\n|Method| SPG |SPU|WSN|EWC|LwF|Supsup|Ours\n|-|-|-|-|-|-|-|-\n|100 splits|43.40|52.90|60.60|50.61|56.82|49.47|**64.63**\n\n### 3. Naive Avg Bound:\n$$\\mathcal{F}_{\\text{avg}} \\geq \\frac{1}{2}\\sqrt{\\sum \\delta_c^2} + \\lambda C + \\mathcal{O}(1)$$\n- $\\mathcal{F}_{\\text{avg}}$: Forgetting measure (naive averaging)\n- $\\delta_c$: Channel misalignment distance for filter $c$\nWhen channel alignment quality $\\kappa_S > 0.5$ and overlap ratio $\\epsilon < 1/C$, the forgetting ratio satisfies:\n$$\n\\frac{F_{\\text{LwI}}}{F_{\\text{avg}}} \\approx \\sqrt{2(1-\\kappa_S)} \\, \\text{(alignment gain)} \\cdot \\epsilon \\, \\text{(sparsity gain)} + O(T^{-1/2})\n$$\n1. Our method provides an **upper bound** on forgetting\n2. Naive averaging only provides a **lower bound**\n3. Experiments\n|Method|5 splits|10 splits|20 splits\n|-|-|-|-\n|Naive Avg|72.13 |74.80 |75.30 \n|Ours| **81.10** | **84.90** |**86.49**", "claims_and_evidence": "Claims with support:\n\n- The authors argue that parameter protection leads to task isolation and inefficient parameter usage. Empirical results show that LwI outperforms parameter isolation methods (e.g., WSN) in both task-aware and task-agnostic settings.\n\n- The paper introduces a graph-matching approach to align activation pathways before merging models. The method achieves better performance, as shown in ablation studies, where removing pathway matching leads to lower accuracy.\n\nClaims that need further support:\n\n- The paper states that graph matching has a complexity of O(N^4), but uses a layer-wise approach to reduce computational cost. No detailed runtime analysis or comparison with other CL methods is provided.\n\n- The paper argues that the proposed method is effective. The proposed method does outperform the baselines, but the accuracies are too weak as there existing methods such as [1] that perform significantly better on both CIFAR-100 and Tiny-ImageNet. For example, [1] is a rehearsal-free method and it achieves more than 65% accuracy on task-agnostic CIFAR-100 10-splits while the proposed method achieves only 30% accuracy.\n\n[1] A theoretical study on solving continual learning. NeurIPS 2022\n\n- The authors claim that parameter-isolation methods need to know the task identity for inference. However, this is not correct. Task incremental learning (TIL) methods such as parameter isolation can be task-agnostic as theoretically demonstrated in [1].", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria largely align with the problem of continual learning.", "theoretical_claims": "No theoretical analysis was provided. Some arguments made by the authors need theoretical justification. For example, \"Simple averaging may lead to interference and even cancellation of effective components, a concern exacerbated during continual learning\" in lines 104-107.", "experimental_designs_or_analyses": "The experimental design in the paper is generally well-structured, but there are some areas that could be improved or clarified.\n\n- CIFAR-100 and Tiny-ImageNet are widely accepted benchmarks for continual learning. The datasets are split into 5, 10, and 20 tasks, allowing the evaluation of performance under different levels of granularity.\n\n- Missing comparisons with some recent CIL methods such as [1] and parameter-isolation methods such as SupSup [2].\n\n[1] A theoretical study on solving continual learning. NeurIPS 2022  \n[2] Supermasks in superposition. NeurIPS 2022\n\n- Task-incremental learning (TIL) methods such as WSN can also be used for task-agnostic evaluation [1].", "relation_to_broader_scientific_literature": "- Relation to CL: LwI introduces a new paradigm where pathway protection is emphasized over parameter protection, leveraging graph matching for model fusion.\n\n- Relation to model fusion and graph matching: Moves beyond naive weight averaging by using structured pathway alignment, ensuring better knowledge transfer between tasks.", "essential_references_not_discussed": "Refer to the previous comments", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "There are several minor mistakes. \n\n- ”Activation Level” in lines 087. In latex, use `` for opening a quotation. This incorrect quotation appears in other places throughout the paper.\n\n- \"... over-parameterized deep deep networks to allow flexibility for future tasks.\" The word 'deep' appeared twice", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OAQR4yDDDG", "reviewer": "Reviewer_Wbqt", "summary": "The paper proposes a novel continual learning framework that assigns distinct neural pathways to different tasks, enabling knowledge retention while replacing traditional masking & pruning methods. The authors use graph matching for model fusion, leveraging neural network properties by maximizing similarity alignment in shallow layers and minimizing it in deeper layers. Knowledge distillation is applied to constrain parameter deviations. The framework is validated on networks of different sizes and datasets of different sizes.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "****Response to C1:**** Thank you for the reviewer's reminder. We sincerely apologize for the misunderstanding caused by our negligence. A precise description will be provided in future revisions. The concept of \"Activation Level\" refers to the average magnitude of the weights obtained after activation in the last layer of the feature extraction phase. We utilize activation levels to measure whether pathways associated with different tasks can be distinguished. Through the left side of Figure 2, we observed that among these channels, in our method, there consistently exists a channel specific to a task. In contrast, for the LwF method, the activation levels of its channels exhibit a phenomenon of intermixing.\n\n****Response to C2:**** Thanks for pointing out the issue. We would like to explain it as follows.\n- We conducted a corresponding analysis, including the optimization of our proposed method in terms of time complexity. In the context of our hierarchical matching approach, we analyze its time complexity as follows. Given a deep network with $N_L$ layers, each containing $C$ channels, traditional graph matching incurs a time complexity of O(N^4), where $N$ denotes the total number of nodes in the graph. However, by employing a hierarchical matching strategy for deep networks, we can compute the time complexity separately for each layer and then aggregate the results. Consequently, the overall time complexity of our approach is:\n\n$$ \\begin{split} & O(\\sum_{1}^{N_L} C^4) = O(\\sum_{1}^{N_L} (\\frac{N}{N_L})^4) = O(\\frac{1}{N_L^3} N^4). \\end{split} $$\n\n- Although graph matching is generally an NP-hard problem, using bilateral relaxation transforms it into a solvable problem by converting the general graph matching problem into a bipartite graph matching problem. The original graph matching problem may involve graphs with arbitrary topological structures, but bipartite graph matching is relatively easier to solve computationally. In this paper, we use the Sinkhorn algorithm, which is a polynomial-time algorithm.\n\n****Response to Essential References:**** Thank you for providing these papers. We indeed omitted the comparisons in our manuscript. We will incorporate references to these papers in our study and conduct comparative analyses accordingly.\n\n**Comparison with [1]**:\n\n1. This method, like ours, focuses on **protecting connection pathways**.\n2. **The specific connection points protected are different**. [1] focuses on protecting network connections between adjacent layers.\n3. **The methods for preserving knowledge from previous tasks differ**. [1] protects important connections for previous tasks by freezing them, whereas we use a soft protection method.\n4. **The number of trainable parameters for the next task is different**. [1] does not train frozen parameters.\n5. **The protection of data privacy protection is different**. [1] requires storing previous data, whereas our method does not require storage.\n\n**Comparison with [2]**:\n\n1. **The different approaches for protecting task-related knowledge**. [2] protects task-specific knowledge by compressing and safeguarding neurons that are crucial for particular tasks. In contrast, our method achieves task knowledge protection through matching-based approach.\n2. **[2] aims to reduce the interference between different tasks, whereas our method achieves knowledge sharing through interference.**\n\n**Comparison with [3]**:\n\n1. **The different approaches for protecting task-related knowledge**. [3] employs dynamic masking (including for parameters and gradients) to protect task knowledge, whereas our approach utilizes a matching-based method.\n2. **Our method facilitates the propagation of knowledge, while [3] prohibits it**. [3] employed in this paper utilizes hard masking, which is detrimental to the sharing of common knowledge across different tasks.\n3. **The protection of data privacy protection is different**. [3] requires storing previous data.\n\nWe replicated two of the above methods and integrated them into our framework for comparative analysis.\n\nResNet32-based model on the CIFAR-100 dataset.\n|Method|5 splits|10 splits|20 splits\n|-|-|-|-\n|NISPA [1]|69.34±0.81|73.41±0.32|76.42±0.27\n|Sparcl [3]|72.04±0.48|75.08±0.81|77.40±0.21\n|**Ours**|**76.10±0.33**|**81.12±0.90**|**83.19±0.35**\n\n****Response to Weaknesses:**** We conducted corresponding experimental tests on the ImageNet-R dataset and also performed tests using ResNet50.\n\n**(1) ResNet32-based model on the Imagenet-R.**\n|Method|5 splits|10 splits|20 splits|\n|-|-|-|-| \n|EWC|24.6|27.1|29.3\n|LwF|25.0|28.6|30.3\n|RWalk|26.1|28.4|26.6\n|WSN|28.2|30.4|32.1\n|SPG|23.2|24.1|22.7\n|SPU|26.2|27.6|27.1\n|GPM|26.7|27.1|29.3\n|Ours|**32.8**|**34.3**|**37.3**\n\n**(2) ResNet50-based model on the Cifar100**\n|Method|5 splits|10 splits|20 splits|\n|-|-|-|-|\n|EWC|70.9|62.7|57.6\n|LwF|79.2|80.3|82.7\n|RWalk|70.3|66.3|62.1\n|WSN|80.2|81.4|84.3\n|SPG|55.3|57.2|55.1\n|SPU|60.5|62.1|61.7\n|Ours|**84.4**|**87.5**|**89.7**", "claims_and_evidence": "Most claims made in the submission are clear. However, there are some areas that need further clarification:\n1. Although the 'Activation level' in Figure 2 is explained in the text, some aspects remain unclear. Could the authors provide a more detailed explanation of the results shown in Figure 2?\n2. Claim about the trade-off between performance and cost. The submission asserts that pathway protection maintains high performance while reducing computational overhead. While the experiments show promising results in terms of performance, a more detailed analysis comparing the computational cost (e.g., memory usage, FLOPs)  would provide stronger evidence.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, no issues found", "experimental_designs_or_analyses": "Yes. 2. Although multiple datasets were used, no corresponding comparative experiments were conducted on larger datasets, such as ImageNet-R.", "relation_to_broader_scientific_literature": "The method contributes to the broader scientific literature by addressing the continual learning problem with a novel method.", "essential_references_not_discussed": "This manuscript lacks comparisons with some relevant approaches, such as [1, 2, 3].\n\n[1] \"NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks.\" International Conference on Machine Learning. \n[2] \"Spacenet: Make free space for continual learning.\" Neurocomputing.\n[3] \"Sparcl: Sparse continual learning on the edge.\" Advances in Neural Information Processing Systems.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper explores the lottery ticket hypothesis under neural network sparsity. By analyzing the complete pathway from input to output, it ensures task-specific pathway preservation.\n2. To validate the effectiveness of the proposed framework, experiments are conducted on networks of different sizes across datasets of varying complexity.\n3. The paper is well-structured and written, enhancing readability and facilitating a deeper understanding of the content.\n\nWeaknesses:\nThe authors need to conduct experiments on larger datasets (Imagenet-1K?) and backbones (maybe resnet50? vit?).", "other_comments_or_suggestions": "Please check the above concerns.", "questions_for_authors": "Please check the above concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tZnVpPmPhr", "reviewer": "Reviewer_bzga", "summary": "The paper introduces a novel continual learning method specifically designed to address the problem of catastrophic forgetting, which is a significant challenge in the field of machine learning when models are required to learn from a sequence of tasks. The proposed approach leverages pathway protection techniques, which aim to preserve the essential parts of the neural network that are crucial for previously learned tasks while allowing the network to adapt to new tasks. Unlike traditional methods that focus on isolating parameters for each task or using memory replay to store past experiences, pathway protection emphasizes maintaining the integrity of critical pathways within the network. This mechanism not only helps to safeguard previously learned representations but also facilitates the efficient integration of new knowledge, enabling the model to continue learning without losing valuable information from prior tasks. By doing so, the proposed method aims to strike a balance between retaining old knowledge and acquiring new skills, ultimately leading to more robust and scalable continual learning systems.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Claims:** Thanks for pointing out the issue. We would like to explain it as follows.\n\n1. In continual learning, considering the lack of correspondence between neurons in Model 1 and Model 2, it is possible that the function of the p-th neuron in Model 1 is very similar to that of the (p+1)-th neuron in Model 2, despite their different positions. Therefore, when using the OT algorithm in neural networks, **we aim to optimally (minimizing cost) transfer neurons from a specific layer in Model 1 to the corresponding layer in Model 2**, achieving model alignment.\n2. **The advantages of using a soft matching algorithm.**\n   - Using entropy regularization and bidirectional relaxation ensures that the optimization function is smoother.\n   - **One-to-one matching hinders knowledge sharing.** One weight in Model 1 might be similar to multiple weights in Model 2. In such cases, using one-to-one matching may set the corresponding weights with similar knowledge to zero, hindering knowledge sharing.\n3. We compared direct fusion and fusion using hard matching, and the results show that soft matching is more advantageous. ResNet18-based model on the CIFAR-100 dataset, measured under task-aware scenarios.\n\n|Method|5 splits|10 splits|20 splits\n|-|-|-|-\n|Ours w/o alignment|66.13±0.98|68.80±1.65|70.30±1.27\n|Ours with OT|75.73±0.58|77.73±0.76|80.60±0.71\n|**Ours**|**81.10±0.80**|**84.90±0.36**|**86.49±0.55**\n\n**Response to Essential References:** Thanks for the meaningful suggestions. We indeed omitted the comparisons in our manuscript.\n\n**Comparison with [1]**\n1. **The different approaches for protecting task-related knowledge.** [1] employs a genetic algorithm. It selects the parameter subset for the next task while fixing the important parameters of the previous tasks, whereas our method achieves knowledge protection through a matching-based approach.\n2. [1] does not train frozen parameters, whereas we train all parameters.\n\n**Comparison with [2]**\n1. [2] selectively retrains network parameters and adapts to different tasks by dynamically changing neurons. However, our method protects knowledge of different tasks by finding the optimal pathways for each task and using matching techniques.\n2. The method proposed in [2] dynamically increases network capacity. In contrast, our method uses a fixed network capacity approach.\n\n**Comparison with [3]**\n1. [3] protects knowledge through parameter decomposition and masking techniques. However, our method protects important pathways through matching techniques.\n2. [3] focuses on protecting network connections between adjacent layers, whereas we consider a complete pathway from input to output.\n\n**Response to Weakness:** Thanks for the valuable question.\n\n- Regarding the number of learnable tasks, our method employs soft protection, allowing for a greater number of learnable tasks. Compared to hard mask methods like SupSup, we use all network parameters and protect important task paths through misaligned fusion. Based on the reviewer's suggestions, we conducted experiments on Tiny-ImageNet with 100 tasks, each comprising two classes. The results show that our method performs very well.\n- To verify the performance of our method when increasing the task number by an order of magnitude, we split Tiny-ImageNet into 100 tasks and conducted tests using different methods. The results show that our method achieves better performance.\n\n| Method | SPG |SPU\t|WSN\t|EWC\t|LwF\t|Supsup\t| Ours |\n| --- |  ---  | --- |  --- |---| --- |  ---  | --- |\n| 100 splits | 43.40±0.45|\t52.90±1.02|\t60.60±0.43|\t50.61±0.61|\t56.82±0.55|\t49.47±0.52 | **64.63±0.14**|", "claims_and_evidence": "The claims presented in the paper are largely substantiated by comprehensive experimental results, demonstrating the effectiveness of the proposed method for mitigating catastrophic forgetting in continual learning scenarios.\n\nNevertheless, there appear to be some details missing regarding the use of OT in Section 3.2. OT is typically used to find the minimum effort required to transport one distribution to another. However, in the paper, you use OT, specifically the Sinkhorn approximation, to 'transform a binary 0-1 matrix into a soft matching matrix with a sum of 1 through a process of bi-directional relaxation.' Could you kindly clarify how this approach relates to the traditional use of OT? It would be helpful to elaborate on this process further in the main text.", "methods_and_evaluation_criteria": "While the authors have demonstrated the feasibility of the proposed method across various techniques, datasets, and architectures, I have a few suggestions for further improvement.\n\n1.Although the authors evaluated forgetting rates using the ResNet18 architecture on the CIFAR-100 dataset, it would still be beneficial to test the forgetting rates of ResNet32 on CIFAR-100 and ResNet18 on the Tiny-ImageNet dataset.\n\n2.Related works that rely on different paths in a network and sparsity are not cited and compared to qualitatively or empirically [1,2,3].", "theoretical_claims": "The authors have provided a detailed explanation of the theories used.\n\n1.\tPathway Protection Enhances Knowledge Retention (Guaranteed by sparsity)\n\nThe theoretical foundation of the proposed method is rooted in the concept of pathway protection, which is designed to safeguard key pathways or network parameters that represent learned knowledge from previous tasks.\n\n2.\tGraph Matching Method\n\nThe authors propose a graph-based method for integrating weights from previous models into the current model, which theoretically capitalizes on the structural similarity between shallow and deep layers. Shallow layers are assumed to share common representations across tasks, while deeper layers specialize in more task-specific features.", "experimental_designs_or_analyses": "1.The authors have selected a wide range of well-established benchmark datasets for their experiments, including CIFAR-10, CIFAR-100, and Tiny-ImageNet, to assess the effectiveness of their proposed method in various continual learning settings.\n\n2.The authors clearly define their experimental setup, including the use of ResNet18 and ResNet32 architectures for evaluating the accurate rates on the given datasets.\n\n3.One of the strengths of the experimental design is the inclusion of a series of ablation studies to evaluate the individual components of the proposed method. These studies help isolate the effects of key factors, such as pathway protection and the graph-based integration of model weights, on the overall performance. By systematically varying the design choices (e.g., whether pathway protection is applied or not), the authors are able to demonstrate the contributions of each component to the method’s success in mitigating catastrophic forgetting.\n\n4.The paper includes experiments in both task-incremental and class-incremental settings, two common setups in continual learning.", "relation_to_broader_scientific_literature": "The proposed method in this paper draws inspiration from concepts found in neuroscience, particularly regarding how the human brain handles learning and memory retention over time. In continual learning, one of the primary challenges is catastrophic forgetting—where the model \"forgets\" previously learned information as new tasks are learned. This mirrors how human brains can forget previously learned information when new knowledge is acquired, a phenomenon known as interference in cognitive psychology.\n\nRecent research in neuroscience has shown that the brain employs mechanisms such as synaptic consolidation and neuroplasticity to minimize forgetting. In particular, the brain strengthens and modifies synaptic connections between neurons as new information is learned while maintaining previously established pathways. This process ensures that long-term memories are not easily overwritten. The method proposed in this paper mimics this idea by implementing pathway protection strategies, which involve protecting previously learned features or representations during the learning of new tasks, in a manner similar to how the brain \"protects\" neural connections when learning new information.", "essential_references_not_discussed": "Related works that rely on different paths in a network and sparsity are not cited and compared to qualitatively or empirically, such as [1, 2, 3].\n\n[1] PathNet: https://arxiv.org/abs/1701.08734\n\n[2] DEN: https://arxiv.org/abs/1708.01547\n\n[3] APD: https://arxiv.org/abs/1902.09432", "other_strengths_and_weaknesses": "Stengths:\n\n1.\tThe authors propose a method called Learning without Isolation (LwI), where, at each step in continual learning, a new model is trained on a task while distilling knowledge from an existing model. This idea aligns closely with the concept of information flow in the human brain, while also integrating the characteristics of neural networks into this research.\n\n2.\tThe authors conceptualize the model’s parameters as a graph, they construct paths as convex combinations of new and old model weights, applying a permutation matrix to the new model's weights. For shallow layers, the permutation matrix is determined by the similarity between adjacent layer weights, while for deeper layers, it is based on the negative similarity between graph nodes. This encourages shared paths across different tasks, promoting feature reuse in the shallow layers, and fosters the use of distinct paths in the deeper layers.\n\n3.\tExtensive experiments are conducted on image classification continual learning tasks, utilizing benchmarks such as CIFAR-10, CIFAR-100, and Tiny-ImageNet, as well as using different architectures of neural networks to validate the effectiveness of proposed method. The authors examine both task-agnostic and task-aware settings.\n\nWeaknesses:\n\nIn the experiments presented in the paper, the authors evaluate the method on a maximum of 20 tasks. This raises an important question: how does the model perform when the number of tasks is increased by an order of magnitude? It seems reasonable to expect that as the number of tasks grows, there will be a tradeoff between the network's ability to accommodate more tasks and the overall model size. This tradeoff is likely to impact both the efficiency of the network and its capacity to preserve knowledge across tasks.", "other_comments_or_suggestions": "Please address the concerns I raised in the weaknesses section. Based on your responses, I would be happy to reconsider my scores.", "questions_for_authors": "Please check the above concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach for continual learning by performing pathways protection, and not just parameter protection during training, with some good performance overall. The rebuttal period raised intense discussions, and some reviewers increased their scores. At the end, three reviewers lean towards acceptance, while one reviewer did not oppose it clearly during the AC – reviewers discussion phase. Therefore, the AC urges the authors to include all rebuttal discussions and new experiments in the final version of the paper.
Decision: Accept (poster)