Detect, Decide, Unlearn: A Transfer-Aware Framework for Continual Learning
Authors: Yiwen Wang
Diana Benavides-Prado
Yun Sing Koh
Venue: ICLR 2026 Poster (2026)
Keywords: Continual Learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Continual learning (CL) aims to continuously learn new tasks from data streams. While most CL research focuses on mitigating catastrophic forgetting, memorizing outdated knowledge can cause negative transfer, where irrelevant prior knowledge interferes with new task learning and impairs adaptability. Inspired by how the human brain selectively unlearns unimportant information to prioritize learning and to recall relevant knowledge, we explore the intuition that effective CL should not only preserve but also selectively unlearn prior knowledge that hinders adaptation. We introduce DEtect, Decide, Unlearn in Continual lEarning (DEDUCE), a novel CL framework that dynamically detects negative transfer and mitigates it by a hybrid unlearning mechanism. Specifically, we investigate two complementary negative transfer detection strategies: transferability bound and gradient conflict analysis. Based on this detection, the model decides whether to activate a Local Unlearning Module (LUM) to filter outdated knowledge before learning new task. Additionally, a Global Unlearning Module (GUM) periodically reclaims model capacity to enhance plasticity. Our experiments demonstrate that DEDUCE effectively mitigates task interference and improves overall accuracy with an average gain of up to 4.55\% over state-of-the-art baselines.

Published as a conference paper at ICLR 2026
DETECT, DECIDE, UNLEARN: A TRANSFER-AWARE FRAMEWORK FOR CONTINUAL LEARNING
Yiwen Wang', Diana Saaavide--Pradoo & Yun Sing Koh
University of Auckland, Auckland, New Zealand "Qeenn Mary University London, London, England {yiwen. wang, y. y.koh)@auckland ac.nz, w.w.aaa................eee...... d.benavvidesprrdd@@@l...cc
ABSTRACT
Continual learning (CL) aims to continuously learn new tasks from data streams. While most CL research focuses mitigating catastrophic forgetting, memo- rizing outdated knowledge can cause negative transfer, where irrelevant prior knowledge interferes with new task learning and impairs adaptability. Inspired by how the human brain selectively unlearns unimportant information to priori- tize learning and recall relevant knowledge, we explore the intuition that effective CL should preserve and selectively unlearn prior knowledge that hinders adapta- tion. We introduce DEtect, Decide, Unlearn in Continual ]Earning (DEDUCE), a novel CL framework that dynamically detects negative transfer and mitigates it by a hybrid unlearning mechanism. Specifically, we investigate two comple- mentary negatíve transfer detection strategies: transferability bound and gradient conffict analysis. Based on this detection, the model decides whether to activate Local Unlearming Module (LUM) to filter outdated knowledge before Iearning a new task. Additionally, Global Unlearning Module (GUM) periodically re- claims model capacity to enhance plasticity. Our experiments demonstrate that DEDUCE effectively mitigates task interference and improves overall accuracy with an average gain of up 4.55% over state-of-the-art baselines.
INTRODUCTION
Continual Learning (CL) enables models to continuously learn from data streams, while mitigating
catastrophic forgetting (CF) (McCloskey & Cohen 1989) and enhancing knowledge transfer (Chen
et al., 2019). To address CF, various methods have been proposed that focus retaining previ- ously learned knowledge (Schwarz al., 20188) Buzzega al. 20200) Ramesh & Chaudhari [02TT) Despite their effectiveness, these methods often overlook another critical challenge: memorizing outdated information can hinder adaptation and lead to task interference (Wang et al. (025). Imag- ine driver assistance system that must continuously adapt to dynamic environments. Retaining outdated information, such as past lighting conditions, can hinder learning new traffic patterns and interfere with recalling relevant prior knowledge. In CL., this phenomenon manifests negative transfer, where conflicts between old and new tasks degrade both forward and backward transfer, leading to suboptimal performance on both new and old tasks. We argue that addressing this should not only focus on preserving knowledge, but also on selectively unlearning interfering knowledge.
At first glance, introducing unlearning into CL seems counterintuitive: if one of the goals is re- duce CF, why deliberately unlearn? The key lies in what is unlearned. By selectively unlearning interfering knowledge before it causes conflict, task interference is reduced, thereby enabling better adaptation to new tasks and mitigating interference-driver forgetting of old tasks, as illustrated in Fig. 1 This intuition draws from dual perspectives. Neuroscience shows that when old and new experiences conflict, the human brain actively forgets irrelevant information to mitigate interference and support knowledge transfer (Gravitz) 2019; Feldman & Zhang 2020). From machine learning, unlearning has been demonstrated to remove the influence of specific data without degrading overall performance (Shibata et al.) 2021), and even to discard domain -seeiif knowledge to improve gen- eralization (Basak& & Yin 2024). Motivated by these, argue that unlearning should not be viewed as contradictory to knowledge retention. Instead, we propose a new perspective for CL: how can continual learners selectively unlearn interfering knowledge when negative transfer occurs, thereby

---

Published as a conference paper at ICLR 2026
enhancing adaptability and reducing task interference? This leads us two guiding questions: (i) How to detect negative transfer? (ii) How to selectively unlearn interfering prior knowledge?
To answer these questions, introduce DE-
DUCE (DEtect, Decide, Unlearn in Contin- ual |Earning), transfer-aware CL framework that dynamically identifies negative transfer and mitigates it by selective unlearning. DE- DUCE uses two complementary strategies for detecting negative transfer: transferability bound. which estimates the task transferabil- ity gap based on their data distributions (Ben- David et al. 2010);; and gradient conflict anal- ysis, which identifies interference during op- timization (Yu al. 2020). To selectively unlearn interfering knowledge, DEDUCE em- ploys hybrid unlearning mechanism that op- erates unlearning at the batch-level, named Local Unlearning Module (LUM), and at the network-level, named Global Unlearning Mod- ule (GUM). Specifically, upon detecting nega- tive transfer, LUM is triggered to unlearn out- dated prior knowledge before new learning. In parallel, GUM enhances plasticity by monitoring neuron activity and resetting low- www-contrbbuting and unimportant neurons, freeing capacity for future learning. Together, these modules enable DEDUCE to dynamically respond to negative transfer, promoting more efficient knowledge integration across tasks. Our main contributions are as follows:
Gradients of Task
+
Task somples
Task proxies
Gradients of Task
Task B somples
Task proxies
Learning Tosk
Learving
Learning Task (DEDUCE)
Figure 1: Illustration of negative transfer and the
effect of unlearning. Left: Task A is learned, es- tablishing its gradients and feature space. Mid- dle: Directly learning Task B introduces gradi- ent conflict with Task A, causing negative forward and backward transfer. Right: DEDUCE selec- tively unlearns interfering knowledge upon nega- tive transfer detection, reducing the influence of old gradients on new learning and vice versa.
We introduce a awanseer-aware CL framework that adaptively detects and responds to nega- tive transfer. This framework provides a novel perspective for CL, where integrating selec- tive unlearning into the learning process can improve both forward and backward transfer.
We explore two complementary negative transfer detection strategies: transferability bound and gradient conflict analysis. These strategies provide CL with a systematic way to decide when to activate unlearning.
We present a hybrid unlearning mechanism, which enables continual learners to dynami- cally regulate knowledge retention and unlearning, laying the foundation for more adap- tive continual learning system. This mechanism integrates LUM for batch-level interfer- ence removal and GUM for reclaiming underutilized capacity at the network level.
2 RELATED WORK
Continual Learning, Existing CL methods can be broadly divided into three categories: Memory-
based (Rebuffi al. 2017 Buzzega al. 2020 Architecture-based (Mallya & Lazebnik 2018
Kang et 2022). and Regularization-based methods (Zenke et al.) 2017 SSchaar et [0188] De-
spite their effectiveness in mitigating CF, they are often limited in their ability to enhance knowledge transfer (Lopez-Paz & Ranzato (017)) and adaptively handle interference from outdated information
Wang et al. 2025). To avoid negative transfer. Batch Spectral Shrinkage (Chen al.) 2019) shrinks dominant feature components; AFEC (Wang et al. 2021) prunes negatively contributing neurons based on gradient signals. For enhancing positive transfer. GPM (Saha al., 202I) leverages sub- space projections to mitigate interference; TRGP (Lin et al. (2022) introduces Trust Region Gradient Projection. However, these methods still prioritize maximizing knowledge reuse, without offering a flexible strategy that dynamically balances knowledge retention and adaptability.
Machine Unlearning (MU). MU was initially introduced to remove specific data traces for privacy compliance without full retraining (Guo et al. 2020; Wu et al. 2020). Beyond privacy (Bourtoule) jt al. 2021; Shibata et al. 2021). recent works have extended MU to broader scenarios, such as filtering domain-specific representations (Basak & Yin 2024) and eliminating outdated knowledge (Wang et al. (224c)). These works demonstrate that unlearning prior knowledge can improve model performance. However, a critical limitation remains: most existing MU methods do not explicitly
2

---

Published as a conference paper at ICLR 2026
DETECT
DECIDE
UNLEARN
Transferability Updrse
(a) Activate Local Unlcarning Module
New Tiisk Dr
Dg
Lacal
Negutive transfer detected
Informaticn
Ualearning
Munis
New Task
2 Unlcsm
(LUM)
Xy
Backsone fo
Yes
Burter
Eg00) 2ddeen
Reinitllssé
Global Unlearning Modale (GUM)
Gradicnt Conflict Analysis
(b) Freeze Lecal Unlearning Module
Unlenrning (LUM)
Buffer
+
New task
Directly
(b)
learning with GUM
BufTer
Leam
Reinitiatis
Tolerance
clIVElf, M)Il: I9II 19L(fa.D))
Lalearning (GUM)
Figure 2: DEDUCE first detects potential negative transfer when a new task arrives, using either the transferability bound or gradient conflict analysis. If negative transfer is detected, the Local Unlearning Module (LUM) is activated, as shown in (a). The model first performs local unlearning in steps (-2). Then, the new task is learned while randomly replaying previous tasks in step 3 with global unlearning (GUM) occurring throughout this process in step ④. If negative transfer detected, LUM remains frozen, as shown in (b), and the new task is learned directly with GUM.
consider when unlearning necessary, which potentially limits positive transfer by discarding use- ful information excessively (Wang et al. [2021; 2024b). This highlights two key challenges that remain underexplored in CL.: when to activate unlearning during the learning process, and how to leverage unlearning to enhance both forward and backward transfer. Motivated by these gaps, our work introduces transfer-aware framework that detects potential negative transfer and dynamically regulates knowledge retention and unlearning to mitigate task interference and promote adaptability.
METHODOL.OGY
To address negative transfer in CL, we design DEDUCE, transfer-aware framework that integrates selective unlearning into the learning process. The core intuition is that, instead of indiscriminately preserving all prior knowledge, continual learners should dynamically unlearn interfering knowl- edge while retaining information that supports future learning. Implementing this intuition requires addressing two key challenges: detecting negative transfer to identify conflicts between old and new tasks before they cause catastrophic interference, and (ii) deciding how to selectively unlearn outdated knowledge while preserving critical knowledge.
DEDUCE couples negative transfer detection with a hybrid unlearning mechanism. Detection is achieved via two complementary strategies. First, we introduce a transferability bound, inspired by domain adaptation theory (Ben-David et al., (2100). To make it practical, we employ LEEP (Nguyen al. 2020) as proxy, enabling taskl compatibility estimation in CL. which differs fundamen- tally from its original use in offline transfer analysis. Second, unlike GEM (Lopez-Paz & Ranzato) 2017). , which constrains updates to rigidly preserve old tasks, we repurpose gradient conflict analysis to detect interference and trigger unlearning. This shifts the focus from strict preservation to adaptive regulation of stability and plasticity. Upon detecting negative transfer, the Local Unlearning Module (LUM) is activated selectively unlearn batch-level interference and facilitate immediate adapta- tion. In parallel, the Global Unlearning Module (GUM) continuously monitors neuron contribution, reclaiming underutilized neurons to sustain long-term plasticity. Together, these modules allow DE- DUCE to dynamically balance stability and adaptability across tasks, mitigating negative transfer while preserving essential knowledge. Fig. provides an overview of the DEDUCE framework.
3.1 PRELIMINARIES
CL aims to learn from dynamic data stream D (Di...., D....., Dr). where each task dataset D. = (#f, vi. Ny,N, is sampled from a task-specific distribution p.(x, y). Here, xi E X' is the i-th in- put of task I, vi E D. C y D__,. 3%. is its corresponding lahel, and t T (1,2,..., T) represents
3

---

Published as a conference paper at ICLR 2026
the task identity. The objective is to learn model fe. parameterized by 8, while the learner only has access to the current task data and an optional memory buffer M. In task-incremental learning (TIL), the task identity is available during both training and testing. A TIL system learns function f: x T + V, such that for test instance, the model predicts f (a, f) E V%- In dass-incremmntal learning (CIL), the task identity is unavailable at test time. Thus, CIL system must learn a unified predictor f : X + y' that distinguishes all previously seen classes
3.2 DETECTING NEGATIVE TRANSFER TO DECIDE WHEN TO SELECTIVELY UNLEARN
Intuitively, when the new task introduces classes that are disjoint from those of previous tasks, updating the model can easily lead to negative transfer. This arises due to the task interference, which hinders the new knowledge learning and can even exacerbate CF (Chaudhry et al. 2019). Therefore, detecting negative transfer between new and prior tasks is clear signal for selectively unlearning interfering prior knowledge. Based on the above analysis, CL can be naturally framed within the transfer learning paradigm, where previous tasks correspond to the source domain Xs and new task correspond to the target domain Xy. Following the theoretical error bound (Ben-) David al., 2010), can compute the transferability bound of target domain based on source domain. By comparing this bound with the target test error, we can assess whether the knowledge transfer from the prior tasks to the new task negative. If the comparison suggests a high risk of negative transfer, selective unlearning is activated to mitigate interference before new learning. Specifically, for any hypothesis H. the theoretical target error (h)) can be upper bounded by:
(h) = (h) + 2 ddnsn(Xs.x7) +
(1)
where (h) is the expected source error under hypothesis h, duAH denotes the H dddverggenee, which measures the discrepancy between the source and target distributions, and À is the error of the ideal joint hypothesis h², defined as À Es(h")+ ET & In practice, directly computing is intractable since it requires access to the optimal hypothesis over both source and target domains. To address this, we approximate the theoretical transferability bound by estimating three practical quantities (Ben- David et al. 2010): (1) the error of source domain {s);; (2) the divergence between the source and target domains dRaR (Xs, X=) (3) the transferability of the source model to the target X. By estimating these terms, we can approximate the bound and use it predict the risk of negative transfer. We now explain how each component is computed in our framework. We estimate &s (h) by evaluating the model's s error previous tasks. To calculate dxax classifier hu to discriminate between samples from the source domain Xs and samples from the target domain Xy. The test error ē(ha) of this classifier can be used to approximate the divergence as (Theoretical proof in Appendix I.I:
train a domain
dMAm (Xs. 2/1 - 22(ha))
where ε(ha) denotes the classification error of ha on a validation set composed of source and target instances. Intuitivell, dHAH measures how well a classifier can distinguish whether input comes from the source or target domain. A high divergence indicates a large distribution shift and a high risk of negative transfer. To estimate S, we employ LEEP (gguyen et al. 2020).
Definition 1 (LEEP Score) For a given source model feg. the LEEP score measures the transfer- ability between the source model and the target data X7, which is given by:
n Σ log Σ Ply'lux)PPv.lat""
E(Jos, XT)
n
B.EV's
where P(y (yx)) is the estimated probability of target label y² E 3' given the source label y. E Ys. P(y (xe) is the model's predicted probability distribution over source classes for input x' E XT. and n is the number of target instances. From its definition, the LEEP score is always negative and a larger value (i.c., smaller absolute value) indicates better transferability. We can approximate Ā as function of LEEP using empirical relationship:
À A c) E/E X7)
where [E(fsg. indicates how well the label distribution predicted by the source model aligns with the target domain, and is scaling factor. Combining the source error (h),,, the estimated
4

---

Published as a conference paper at ICLR 2026
divergence duan (Xs, Xr), and the estimated X, obtain practical proxy for the theoretical transferability bound of the target error as:
(h)) S Eb(h) + 2 HHA (s, (7) + s Es(h) +11 2ē(ha)) :(e(fes
(5)
3.2.1 DETECTING NEGATIVE TRANSFER ONLINE
According to the transferability bound. if the target test emror exceeds this bound, it signals the presence of potential negative transfer (Zhang al. 2023). In such cases, DEDUCE activates the LUM to proactively unlearn interfering prior knowledge. However, accurately estimating the target error requires access the full target data, typically achieved by one full pass (epoch) of the task data, which limits its use in strict online CL settings, where each sample must be processed only once and learning decisions must made instantly (Wang et al. 2024a).
To complement this and to enable real-time detection of potential negative transfer, we introduce complementary strategy based on gradient conflict analysis. By comparing gradients from the cur- rent mini-batch with those from previous tasks, DEDUCE can dynamically detect negative transfer, allowing timely adaptation without access to the complete target domain. Intuitively, when the new task involves classes that are disjoint from previously learned ones, the gradient induced by the new task will often conflict with the gradient previous tasks. Updating the model with such conflicting gradients can cause negative forward and backward transfer. Therefore, detecting the presence of gradient conflict before fully training on the new task offers practical signal: when negative gradi- ent conflicts are observed, suggests that direct learning of the new task may harm prior knowledge and lead to poor adaptation; thus, selective unlearning should be triggered. Based on the memory buffer M, we define the loss computed over stored exemplars from previous tasks as follows:
1 C(fo. M) = M
Σ ((fo(x).v).
(6)
(a.9))M
While minimizing the loss on the new task, we need to avoid negative gradient conflict between the new task and previous tasks. To formally characterize this conflict between tasks, we introduce the following task gradients condition inspired by the gradient constrains of the update model (Lopez- Paz & Ranzato 2017).
Definition (Negative Transferability) For any new task D. {ao, vf M / [1, we say has negative conflict with previous tasks M if for some e E [-1,0]
(VC(fo M), VC(fo, D.)) M V D.)
Specifically, Definition introduces negative transfcrability between tasks, in the sense that learning the task t is said to be at risk of negative transfer if the model gradient of the new task t conflicts with the model gradient of previous tasks. Formally, when the tolerance = 0, the correlation between the gradient of the new task and the gradients of previous tasks is negative, it indicates strong dissimilarity or opposition in the update directions. In this case, continuing to optimize the model on the new task will likely lead negative forward and backward transfer.
is worth noting that the transferability bound calculated by Eq. and the negative transferability evaluated by Eq. can provide two different strategies for detecting negative transfer before learn- ing the new task 6, which can be chosen according to the specific task settings and demands. Further analysis of these two strategies for detecting negative transfer presented Appendix A.4
3.3 HOw TO SELECTIVELY UNLEARN?
To address the challenge of selectively unlearning outdated and interfering knowledge while pre- serving critical information, we propose a hybrid unlearning mechanism that enhances knowledge retention and adaptation in CL. This mechanism consists of two complementary modules: local unlearning and global unlearning. Upon detecting negative transfer, the Local Unlearning Module (LUM) is activated to unlearn outdated knowledge before learning new batch, thereby mitigating batch-level interference and enabling rapid adaptation. In parallel, the Global Unlearning Mod- ule (GUM) continuously monitors neuron contribution and periodically reinitializes inactive and
5

---

Published as a conference paper at ICLR 2026
unimportant neurons, reclaiming underutilized capacity and restoring plasticity. Together, LUM and GUM enable the model to balance knowledge preservation with plasticity, thereby facilitating more effective integration of new tasks while reducing interference from outdated information.
3.3.1 LOCAL UNLEARNING MODULE (LUM)
When negative transfer occurs, directly fine-tuning a new task can be hindered by interference from prior knowledge, leading to poor adaptation and CF. To address this, LUM is activated to proac-
tively unlearn interfering knowledge before new task learning. From a theoretical perspective, the
unlearning process minimizes the KL divergence between the current CL model parameter posterior po))) and the target unlearned model parameter posterior Pu. (0). Following Wibisono) (2018)
represent the target unlearned posterior as an energy-based form Pa (0) = where w LCL-
This KL divergence can be further decomposed as:
kT(pellew) = / (c( (og Pr d0 Pu(O)
/ (c( log Pr. (u))
(8)
= Ep. log Pa. + Ep, log
E.p, LCL + E.p, log
which corresponds to optimizing an energy functional that increases the loss on the current mini- batch, thereby encouraging the model to step toward the target unlearned parameter distribution. Inspired by this KL-based formulation, the unlearning loss is defined as:
Cunteorn (fe (xx). v1) + aD (0,, 0+)
where De (0,, 0 ||o, 0: > 0 represents form of regularization prevent the model from unlearning the early mini-batches knowledge of the current task, and 0 refers to the model parameters after learning the first & mini bati-batches. In CL. interfering knowledge refers to previously acquired information whose parameter gradients conflict with those of the new task, thereby hinder- ing its learning and inducing negative transfer (Riemer et al. 2019). In contrast, useful knowledge denotes information encoded in parameters that are highly sensitive and important to previous tasks, ie.. parameters with high Fisher Information Matrix (FIM) values (Kirkpatrick
al.) 2017). To ensure selective unlearning, our LUM is explicitly regularized using the FIM. Then, we take the gradient with respect 0₂ for the RHS of the Eq. 99, previously learned tasks:
we can obtain the following unlearning for the
0 = 0, + 8F Ve 0:) Vo, (x:), ye)l
(10)
where F is the diagonal approximation of the FIM. In (T). we leverage the FIM and unlearning rate å to regulate the unlearning process so as to progressively unlearn interfering prior knowledge before learning the new task. Therefore, the unlearning update focuses precisely on low-importance, interfering prior knowledge that would cause negative transfer, while high-FIM parameters that
encode useful information protected. Following local unlearning the current mini-batch the model is trained on this batch by minimizing the following objective function:
Licarn LoE(Se; (2x)), v1) 3( 0.-1) F( 0.-1)
(11)
where Lreg = (0) 0.-1) - 0,-1) is a regularization term used to encourage the use of parameters that not important previous tasks to learn new knowledge, 3 2 O, and 0,-1 refers to the optimal model parameters that were learned for the previous tasks.
3.3.2 GLOBAL UNLEARNING MODULE (GUM)
As models learn sequentially, their plasticity naturally decreases, making adaptation to new infor- mation increasingly difficult. This limitation is further compounded by task interference and finite network capacity. To address this, we propose monitoring neuron activity to identify neurons that remain largely inactive across recent tasks, based the intuition that such neurons contribute little to learning the current task (Dohare et al. 2024). However, reinitializing neurons solely based on low-activation risks eliminates those that sparsely active yet critical for specific tasks.

---

Published as a conference paper at ICLR 2026
To mitigate this, we introduce importance scores that more accurately assess each neuron's 's contribu- tion before reinitialization. Activity scores measure the actual influence of neurons on model outputs during the current task, while importance scores quantify their historical significance. Neurons iden- tified as both low-activation and unimportant are selectively reinitialized, reclaiming model capacity for future learning without interfering with essential prior knowledge. As it continuously monitors neuronal contributions throughout training and globally reinitializes low-contributing neurons, we refer this module Global Unlearning Module (GUM). To measure the contribution of each neu- ron during training, we maintain running average updated with decay rate 7) Specifically, in feed-for forward network, the contribution Ci of the i-th features in layer I time T updated as:
77, (1 7)) 1T
Σ /To. + C
о (PT..)
(12)
where hi, denotes the output of the i-th feature in layer / at time and wis is the weight con- necting the i-th neuron to the k-th neuron in layer 1 + 1. The term 7141 is the number of neurons in layer 1 + 1. The gating factor o (f%)) is sigmoid function of the normalized importance score Fi,,. which ensures that important neurons are less affected. The importance of the i-th neuron Fi,, is calculated by aggregating the importance scores of all outgoing connections F7,, Ersr F To reclaim model capacity, we periodically reinitialize low-contributing neurons. When a neuron is reinitialized,, its outgoing weights are reset to zero, effectively making it inactive. However, immedi- ately after reinitialization, a neuron would naturally exhibit zero contribution, risking rapid repeated resets. To prevent this, we introduce a maturity threshold m: neurons are protected from further reinitialization until they have survived for at least m update steps and are considered mature. At each update step, a fraction of mature neurons with less contribution é, called the global unlearning rate, is reinitialized in every layer. The Pseudocode of DEDUCE is shown in Appendix A.2
EXPERIMENTS
Baselines. We compare DEDUCE with oEWC (Schwarz et al., 2018), , A-GEM (Chaudhry al.,
2019) ER (Rolnick et al. 2019). DER+ (Buzzega et al. 2020). HAL (Chaudhry et al.] 2021) } PCR Lin al.] (023), OnPro Wei al. 2023). MOSE Yan al. 2024), and STAR (Eskandar ct al. 2025). In addition, we report Joint Training (upper bound) and Fine-tuning (lower bound) as standard reference baselines.
Evaluation Metrics. To evaluate the learning performance, we use Averaged Accuracy (ACC) to measure the final accuracy across all tasks. Backward Transfer (BWT) measures the influence of learning the new task on all previous tasks (Lopez-Paz & Ranzato 2017). To assess plasticity, we use the accuracy of new task learning as a proxy, since it reflects the model's ability to effectively acquire novel knowledge (Dohare et al. (024))
Datasets. We evaluate the performance of DEDUCE on four datasets under TIL and CIL settings. Specifically, CIF (Krizhevsky) 2009) is split into 10 tasks with 10 disjoint classes each: CIFAR-10 10 (Krizhevsky) 2009) into 5 tasks with 2 disjoint classes each; Tiny- ImageNet (Deng et al.) 2009) into 10 tasks with 20 disjoint classes each; and CORE-50 (Lomonaco & Maltoni 2017) into 24 tasks with overlapping classes across tasks. addition, we validate DEDUCE on DN4IL (Gowda ct al., 2023) under the domain incremental setting, as shown in Table 3 For the above datasets, we adopt ResNet-18, 18, similar to Kang et al. (2022), and further evaluate DEDUCE with ViT-Base (Dosovisskiy et al. (202T) shown in Table More details are provided in Appendix A.3.1 A.3.4
4.1 MAIN RESULTS
Overall Performance. Table reports the final averaged accuracy (ACC) of all baseline methods across four datasets in the CIL and TIL settings. By integrating our proposed DEDUCE, we consis- tently enhance the overall ACC of memory-based (HAL, ER, DER++), www.larizationn-baieeeee based (oEWC, A-GEM, STAR). and online CL methods (PCR, MOSE, DER++) in the CIL and TIL settings. In particular, DEDUCE delivers substantial improvements to HAL, boosting accuracy by over 13.2% in the challenging CIL settings and over 8.3% in the TIL settings on both CIFAR- 100 and CIFAR-10. Similar benefits are observed for the regularization-baee method oEWC in the online CL setting.
7

---

Published as a conference paper at ICLR 2026
Table 1: ACC on different datasets. All methods use a replay buffer of size 500. OUR(B) indicates DEDUCE using transferability bound for detecting negative transfer, OUR(G) indicates DEDUCE using gradient conflict analysis for detecting negative transfer, "-' indicates not applicable, and indicates another setting in Appendix A.3.3 (A.3.3 The first 11 rows correspond to evaluations under the online CL setting, while the remaining rows report results under the offline CL setting.
CIFAR-100
CIFAR-10
CORE-50
Method
CIL
TIL.
CIL
TIL
CIL.
TIL
CIL
TIL
Fine-tuning
4.2.10.2
26.411..7
15.0.10.5
60.8_1.0
3.7:0.2
188810.8
6,910.9
34.7:0..9
PCR
20.4:10.9
52.512.6
2..41004
31,8.10.8
w/OUR(G)
21.0-10.4
53,6-11.0
1.2
32.3.10.6
MOSE
27.7
63.8=0.3
13.7=0.9
41.3+:0.6
w/OUR(G)
29.3110.6
64.7.101
14.4
43.2.10.4
OnPro
19,5+0.8
6..9=1.2
6.9=0.2
32.1+1.4
w/URRGG)
21.4+0.1
68.4=088
7.6=0.2
34.7+00.5
oEWC
4.8.10.2
21.411.11
15.8
1.3
4.0.1..4
13.6.1..7
7.8.1..4
34.3.1..7
w/OUR(G)
5.640.6
1.39
16.4=1.0
62.9=111
4.8
1..3+1.4
7.9+0.3
35.0
DER++
12.4.0.4
56.610.3
1..
82.0=0.2
6.9.0.6
37.710.4
28.2411.0
66.5
w/OUR(G)
15.1
57.3.10.3
5.3
86.8
8.8
41.9
29.8
70.7:10.9
Fine-tuning
9.3+0.1
37.
19.6+0.00
61.2=2.8
8.1
18.9+0.9
8.7+0.11
33.4+0.3
Joint training
70.6+0.6
91.1+0.1
92.1=0.1
98.3=0.1
59.5=0.3
81.8+0.11
70.9+0.3
91.2+00.1
A-GEM
9.3.1..1
56.2.10.2
19.9.0.4
87.
8.1.0.0
22.610.55
8,910.3
30.1
w/UR
9.5+0.11
59.3+1.44
21.4=0.2
88.0=0.7
8.4=0.2
25.1+005
9.1+0.1
55,4=0.4
STAR(DER)
38.8.10.7
76.4.0.33
73.1
94.2-0.9
16.7.0.9
53.3.1.8
36.810.77
73.3....3
w/OUR(G)
40.9110.8
77.710.3
74.7-0.6
9.5
1..5...6
55.1
42.2.10.2
78.5
STAR(XDER)
41.440.6
84,8+0.3
63.8=1.6
95.4=0.3
19.7=0.4
55.7+:0.
38.4
75.9=0.4
w/OUR(G)
44.2.10.4
85,5.0.4
6..6.1177
95.3-006
2..1.1066
56.0.10.8
39.811.77
76.6.10.5
HAL
11.6+0.8
45.1=2.0
44.9=1.0
84.9=3.2
3.7=0.1
23.6#1.9
15.7
51.7=3.6
w/URR(B
15.8+0.1
61.3+0.2
53.2=1.3
90.4=0.1
6.7=0.1
34,8+0099
16.0#2.22
51.7+3.4"
W/URR(G
24.8+0.7
72.8+0.4
67.0=0.8
93.2=0.9
9.6-0.1
48.9+0.4
16,9+233
53.1+3.3
ER
21.4.10.3
73.4.0.5
57.7-0.3
93.6-0.33
9.8.1.44
50.0.10.4
29.910.77
71.3.0.6
w/OUR(B)
24.2.1.11
74.710.9
67.6.12.0
9..4-0.4
10.6.0.2
51.7.10.6
34.1110.3
74.911..8
w/URR(G)
26.1
76.0
70.4=0.4
94.1=0.6
10.6=0.0
51.8+0.6
33,4+0.7
73.9+0.9
DER+
36.8.10.9
75.6.10.5
71.111.
93.4-0.2
15.6.1.9
51.111.77
40.111..5
76.0.12.0
w/OUR(B)
37.8100.7
75.8.1.11
74.310.0
94.4-0.2
19.710.8
53.511.66
43.911..0
78.0
w/OUR(G)
39.8
78.4.0.44
74.6.10.7
9..
2.5
55.6110.1
41.21114
76.8.1111
though the magnitude varies across datasets. Even when compared with the strong rehearsal-based baseline DER++, DEDUCE achieves margins 3.0% and 3.5% in CIL on CIFAR-100 and CIFAR-10 10. respectively. On the more challenging Imageen,,,, which contains more classes per task and demands higher discriminative power, DEDÚCE achieves a gain of 6.5% in the CIL and a gain of 4.5% in the TII., showcasing its robustness under increased task complexity. In the case of CORE-50, where task boundaries are more ambiguous due to overlapping classes across tasks, DE- DUCE still delivers consistent improvements. The bound-based variant (OUR(B)) utilizes the trans- ferability bound to detect potential negative transfer at the task level. In contrast, the gradient-basse basedd variant (OUR(G)) performs detection at the batch level, enabling much finer-grained identification of task interference. This adaptive, batch-wise control allows the model to respond immediately to emerging conflicts, which explains why OUR(G) achieves higher performance. Nevertheless, is important to note that both variants consistently outperform the baselines, demonstrating that DE- DUCE serves as a versatile and effective enhancement framework across diverse CL paradigms and dataset characteristics.
Comparison on Learning Process. For a more comprehensive comparison, we evaluate the learn- ing process on CIFAR- 100 and Timy-Ima as shown in Fig. 3 First, we report the ACC of each new task, where DEDUCE consistently outperforms all baselines, with particularly substantial gains observed for HAL. This suggests that the hybrid unlearning mechanism effectively enhances forward transfer and improves the model's plasticity. Next, we track the accuracy on Task I over time, Task is the most vulnerable to forgetting, making strong indicator of method's
8

---

Published as a conference paper at ICLR 2026
GEWI OEWCC HOEDUCE
A GEM DEGUM
PCR POR-DELUCE
HAL
DER+ DUR+++UEDUUCE
MAL-DEDUCE
EX+DEEUUCE
CIFAR-100
CIFAR-100
CIFAR-100
New Task Accuracy (86
Averaged Accuracy (%)
Accuracy
60-
80
60
40
40-
40-
20-
20-
6
10
2
6
8
10
10
Thsi
Taiks
Thiks
(a) Performance of new task
(b) Performance of Task I
(c) Performance of all learned tasks
Tiny-ImagoNat
Tiny-ImageNet
Tiny-ImageNet
New Task Accuracy (%)
80
80
Averaged Accuracy (%)
80
60-
Tosk Accuracy
60-
60
40
40-
200
50-
20-
0
10
10
Tasks
Tasks
(d) Performance of new task
(c) Performance of Task 1
(f) Performance of all learned tasks
Figure 3: Performance on observed learning stages based on CIFAR-100 and Imag--m
HAL
DER
BECC+DEDJGE
A-GREM
DEDUCE DEDUCE
PCR-DRDUCE
MAL- DEDUCE
BR+DEDUCK
DER++ +DEDULE
CIFAR-100
Ima-eeageNett
CORE-50
-0
BWT (%)
BWT (%)
-40
-40
-60-
-60
GBWC
4
POR
HAL
BR
ORR
GEWC
A-GEM
PCR
HAL
BR
DER+
GEWC
AGEM
PCR
HAL
En
DER
Figure 4: Backward transfer (BWT) on CIFAR-100, Imay-Im and CORE-50.
ability to mitigate CF. As illustrated in Fig. and Fig. e although all methods suffer from forgetting, models equipped with DEDUCE exhibit slower degradation. This indicates that DE- DUCE mitigates CF by reducing task interference through the timely activation of the LUM. Finally, as depicted in Fig. 3 and Fig. 3 i. ACC across all learned tasks shows that DEDUCE helps the model consistently maintain more favorable balance between stability and plasticity over time.
Comparison on Knowledge Transfer: As shown in Fig. 3 DEDUCE enhances forward knowledge transfer when learning new tasks. To further evaluate the effect of DEDUCE, report backward transfer (BWT) Fig. Across all datasets, baselines with DEDUCE consistently achieve higher BWT. For instance, DEDUCE improves DER++ by 6.4%, 9.2%. and 2.8% on CIFAR- 100, Tiny- ImageNet, and CORE-50, respectively. These results demonstrate that DEDUCE not only avoids introducing additional CF but also helps mitigate negative backward transfer. Although HAL alone achieves slightly better BWT on CIFAR- 100 and Tiny- ImageNe,, its performance on forward trans- fer and overall accuracy remains inferior to that of HAL combined with DEDUCE (Detailed analysis is provided in Appendix A.3.5). suggesting that the overall benefits of integrating DEDUCE are still substantial.
4.2 ABLATION STUDIES
Validation of DEDUCE. We conduct a comprehensive ablation study to assess the contribution of each component DEDUCE, Several key observations be drawn from Table (i) Each compo-

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study of the OUR proposed framework (DEDUCE).
CIFAR-100
CIFAR-10
Timy-ImageNet
CORF-50
Method
CII.
TII.
CII.
T1I.
CII.
TII.
CIL
TII.
+ER++
36,8:10.9
75.6:10.5
71.1=1.3
93,410.2
15.61119
51.1=0.7
40.111..5
76.0:2..0
w/LUM
39.4.10.6
76.6.10.3
72.610.6
93.910.55
16.610.5
52.6.0.9
40.5.114
76.3:112
w/GUM
3..7+0.3
76,4+0.5
72.4=0.1
93.6+0.0
16,3+0.8
52.2=0.9
42.1+1.2
77.4+14
wol
39.6+0.3
77.2+002
73.6=0.3
94.0
19.6+0.7
53.8=0.6
41.0+1.1
76.5
w/OUR(G)
39.8+0.9
78.4:0.4
74.6=0.7
94.8+0.2
22.1
55.6=0.1
41,2#11
76.8
nent of DEDUCE contributes positively to overall performance, with the combination of DER++ and DEDUCE yielding the most significant improvement. (ii) The LUM, which targets the unlearning of interfering prior knowledge, is especially beneficial in CIL, where the risk of task interference and negative transfer more prominent. (iii) The GUM achieves consistent performance improvements across datasets, demonstrating the need to enhance the plasticity of the model while overcoming CF. More specific results are detailed in Tables 7 of Appendix J5..nd [A.3.5]
Sensitivity Analysis. We study the impact of three key hyperparameters: the local unlearning rate 6, the global unlearning rate @, and the tolerance c for triggering unlearning (Detailed Tables 10 of Appendix). For both and Φ. performance initially improves and then declines as their val- ues increase, suggesting that moderate unlearning effectively removes interference while preserving useful knowledge. For we observe that smaller values lead to performance degradation, as stricter activation of LUM reduces ACC. Notably, when = O, the model achieves the best ACC and BWT. These observations highlight the importance of investigating when to selectively unlearn, as excessive unlcarning can harm performance, while timely unlearning can mitigate negative transfer.
Impact of Buffer Size. We report ACC and BWT of DEDUCE under varying memory capacities 100, 200, 500, 1000 (Detailed in Tables 11 12 of Appendix). Across all settings, DEDÚCE consis- tently improves over baselines. Even with large buffer size of 1000, where replay- based methods already perform strongly, DEDUCE delivers further gains, highlighting its complementary value. More importantly, under constrained memory of 100, DEDUCE achieves substantial improvements, effectively mitigating the degradation caused by limited replay. Overall, these results demonstrate that DEDUCE effectively mitigates task interference and facilitates knowledge integration, yielding robust continual learning across varying memory budgets.
5 CONCLUSION
In this work, we addressed the challenge of negative transfer in CL by introducing DEDUCE, transfer-aware framework that proactively manages knowledge retention and update. DEDUCE de- tects negative transfer through transferability bound or gradient conflict analysis, and mitigates it with a hybrid unlearning mechanism that integrates dynamic local unlearning of interfering knowl- edge with periodic global unlearning of low-contributing knowledge. Unlike prior approaches that focus mainly on CF. DEDUCE emphasizes selective unlearning as a means to enhance both forward and backward knowledge transfer. Extensive experiments demonstrate that DEDUCE consistently improves state-of-the-a CL methods by reducing task interference. In future work, we plan to investigate mechanisms that explicitly promote beneficial transfer while suppressing interference, further advancing the adaptability of CL systems.
ACKNOWLEDGEMENTS
This research was partially supported by the Program of China Scholarship Council (Grant No. 20.202400000033). We also gratefully acknowledge the Advanced Machine Learning and Data Analytics Research (MARS) Lab at the University of Auckland.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Hritam Basak and Zhaozheng Yin. Forget more to learn more: Domain-specifie feature unlearning for semi-supervised and unsupervised domain adaptation. European Conference on Computer Vision, 2024.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
Lucas Bourtoule, Varun Chandrasckaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy, 2021.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in Neural Information Processing Systems, 2020.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations, 2019.
Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip Torr, and David Lopez-Paz. Using hind- sight to anchor past knowledge in continual learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In Advances in Neural Information Processing Systems, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Shibhansh Dohare, J. Fernando Hemmandez-Garci.. Qingfeng Lan, Parash Rahman, A. Ruapm Mah- mood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 632:768-774, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Masih Eskandar, Tooba Imtiaz, Davin Hill, Zifeng Wang, and Jennifer Dy. STAR: Stability-inducing weight perturbation for continual learning. In International Conference on Learning Representa- tions, 2025.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: discovering the long tail via influence estimation. In Advances in Neural Information Processing Systems, 2020.
Shruthi Gowda, Bahram Zonooz, and Elahe Arani. Dual cognitive architecture: Incorporating biases and multi- memtry systems for lifelong learning. Transactions Machine Learning Research, 2023.
Lauren Gravitz. The importance of forgetting. Nature, 571:812-S14,2011 2019.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning. 2020.
Haeyong Kang. Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hosngawa-Johnson, Sung Ju Hwang, and Chang D. Yoo. Forget-free continual learning with winning subnetworks. In Proceedings of the 39th International Conference on Machine Learning, 2022.
11

---

Published as a conference paper at ICLR 2026
Daniel Kifer, Shai Ben- Sen-David, and Johannes Gehrke. Detecting change in data streams. In Proceed- ings of the Thirtieth International Conference on Very Large Data Bases, 2004.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Taao Ramalho, Agnieszka Grabska- Barwinska, Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy Sciences, 114(13)3351-33526 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 2009.
Huiwei Lin, Baoquan Zhang, Shanshan Feng, Xutao Li, and Yunming Ye. Per: Proxy-based con- trastive replay for online class dass-incrrmmenal continual learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), 2023.
Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang, TRGP: Trust region gradient projection for continual learning. In International Conference on Learning Representations, 2022.
Vincenzo Lomonaco and Davide Maltoni. Core50: new dataset and benchmark for continuous object recognition. In Conference on Robot Learning, 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology Learning and Motivation, 24:109- (150-9111111110 1989.
Cuong V. Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: new measure to evaluate transferability of learned representations. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Rahul Ramesh and Pratik Chaudhari. Model zoo: A growing brain that learns continually. In International Conference on Learning Representations, 2021.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interfer- ence. In International Conference on Learning Representationss 2019.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Greg Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, 2019.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In Internutional Conference Learning Representations, 2021.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Crabska-Barwinska. Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Takashi Shibata, Go Irie, Daiki Ikami, and Yu Mitsuzumi. Learning with selective forgetting. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, 2021.
Liyuan Wang, Mingtian Zhang, Zhongfan Jia, Qian Li, Kaisheng Ma, Chenglong Bao, Jun Zhu, and Yi Zhong. Afec: active forgetting negative transfer in continual learning. Advances in Neural Information Processing Systems, 2021.
12

---

Published as a conference paper at ICLR 2026
Liyuan Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5362-533) 5383,, 2024a.
Weiqi Wang, Zhiyi Tian, Chenhan Zhang, and Shui Yu. Machine unlearning: A comprehensive survey. arXiv preprini arXiv:2405.07006 2024b.
Zhenyi Wang, Yan Li, Li Shen, and Heng Huang. A unified and general framework for continual learning. International Conference on Learning Representations, 2024c.
Zhenyi Wang, Enneng Yang. Li Shen, and Heng Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. IEEE Transactions Pattern Analysis and Machine Intelligence, (4(3)::464-1488 2025.
Yujie Wei, Jiaxin Ye, Zhizhong Huang, Junping Zhang, and Hongming Shan. Online prototype learning for online continual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Andre Wibisono. Sampling optimization in the space of measures: The langevin dynamics as composite optimization problem. In Conference on Learning Theory, 2018.
Yinjun Wu, Edgar Dobriban, and Susan Davidson. DeltaGrad: Rapid retraining of machine learning models. In Proceedings the 37th International Conference Machine Learning, 2020.
Hongwei Yan, Liyuan Wang, Kaisheng Ma, and Yi Zhong. Orchestrate latent expertise: Advancing online continual learning with multi-level supervision and reverse self-distillation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference Machine Learning, 2017.
Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu. A survey on negative transfer. JEEE/CAA Journal of Automatica Sinica, 10(2):305-32, 2023.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THEORETICAL PROOF
Following the theoretical error bound proposed by Ben-David al. (2010), we aim estimate the learning difficulty of the new task under the influence of prior knowledge. Specifically, for any hypothesis h E H. the theoretical target error (h) can be upper bounded by:
1 (h) S Es(h) + 2 ddann(X X7) +
(13)
where (h) is the expected source error under hypothesis h, dHAH denotes the H which measures the discrepancy between the source and target distributions, and À is the error of the ideal joint hypothesis h", defined
Es(h") + (h")).
To estimate dHAH (T), we train a domain classifier ha to discriminate between samples from the source domain Xs (labeled 0) and samples from the target domain Xy (labeled as 1). The error ē(ha) of this classifier can be used to measure the divergence as:
dHAH(Xs, X7) 2 2h(h)))-
(14)
where é(ha) denotes the classification error of ha on set composed of source and target examples. Intuitively, dHHz measures how well a classifier can distinguish between source and target domains. A high divergence indicates a large distribution shift and a high risk of negative transfer.
To show the proof of Eq.(14), we give the specific definitions of the symmetric difference hypothesis space H.H and HAH--divegeen (Kifer al. 2004). which are useful in reasoning about error.
Definition 3 -HAAH-divergeece) Given a hypothesis space H. the symmetric difference hypothesis space is defined as HAH {h(x)) N'(x) h, h' H}. where is the XOR operation. The HAAH-divergence between Xss and Xy is defined as
dHaH = 2 sup
Pr (h(x) + (l(x)) xx
Pr (h(x) + f((x)) x
(15)
As shown in Eq. the definition of the H.AH--ivvrgeenc measures the maximum difference in disagreement rates between the two distributions under the hypothesis class H. By introducing the symmetric difference hypothesis class H. where h(x)) = 1[h(x) + h'(x)), h E H, the divergence can be rewritten
= 2 sup Pr (h(x:) = =1) Pr (h(x) = 1)1 x~XT
hEA x~X8
sup
Pr (A) Pr (A)l
(16)
AEAR wrXs x~XT
where (h- E H. represents the of the inputs that satisfy h(() induced by H. This transformation expresses the divergence in terms of the probability of the symmetric difference classifier h assigning a sample to the positive class (the set of hxx) = 1) under the source and target distributions. The next lemma shows that we can compute the divergence by training a domain classifier which attempts to separate one domain from the other.
Lemma 1 For a symmetric hypothesis class H and samples of size n from Xs, Xy, the H- divergence between X and X is
(Xs, =
1 min NER
Σ =~=(@)==
1 1 1 Σ 1 E XT|
(17)
=~=(===0
where 7|x E Xs) is the binary indicator variable which O when x E X and when x E Xy- Our basic plan of attack will be follows: Label each source instance with 0 and the target instance
(0, Xss under H. to discriminate between
with 1. Then train a domain classifier h((x)
11 x E XT
14

---

Published as a conference paper at ICLR 2026
source and target instances. The HAA--divergence is immediately computable from the enror
duau(Xs. Xy) =dx(Xs,X7) Xy)
(18)
2 - (errors + errort)l-
where errors and errorT are the error rates of the domain classifier on the source and target do- mains, respectively. Since ē(ha) denotes the classification error of ha on a validation set of size 2n which is composed of source and target examples, we can obtain ε(ha) erroraterrort Therefore, the Hiv--iiveeeence can be calculated by 14.(14)
A.2 PSEUDOCODE
Algorithm Local Unlearning Module (LUM)
Require: Sequence of tasks D {Di, D.....; Dr).
Dr), Fisher Information Matrix F, local unlearn- ing rate 6, model learning rate p, model parameters
for to 7 do
2:
X₁ < LoaddDaaa (D;)
3:
for each (It. V1 X, do
4:
0 - 0. + 5F Vo, Lumiearn (6z)
5:
- 0 Linarn (0;)
6: end for
7: end for
8: return for
Algorithm Global Unlearning Module (GUM)
Require: Sequence of tasks = [D1, D2...., Dr}. memory buffer M, global unlearning rate X, decay rate 7), Fisher Information Matrix F, maturity threshold m, network layers L, the number of neurons per layer {n.,.... n,), model parameters 0, model learning rate p Initialize contributions {Ci, Cr.) number of neurons reinitialize {ci, Cr ages
{a),...,( } to
2: Initialize the network weights {w .... wu in which wi sampled from distribution di
3: for = 1 to 7 do
4: X, < -LoaDData
5: for each (It, E X, do
6:
02 < 0 Liearn (0,)
7:
update the weights {wi,... Wr) using stochastic gradient descent
8:
for / to do
9:
for i 1 n4 do
10:
Calculate importance score for neurons: Ft,, = -E
1M41 Fi,,k
Fin-minPi, Normalized importance score for neurons: Fl,i maaFi,, -min
11:
Update age: dj,; f- dr,i +
13:
Update contribution by Equation 12 end for
15:
Find eligible neurons: neltigibee number of neurons with age more than
16:
Update number neurons to reinitialize a + C; + Aneligible if c then
17:
18:
Find the neuron with smallest contribution and record its index as r
19:
Reinitialize input weights: resample ui-1 [r] from distribution di Reinitialize output weights: reset wy [r] to 0
20:
Update number of neurons reinitialize qj Ct
22:
end if end for
24:
end for
end for
26:
return Ser
15

---

Published as a conference paper at ICLR 2026
Algorithm 3 DEDUCE
Require: Sequence of tasks D {D1, De...., Dr). memory buffer M, local unlearning rate 6,
global unlearning rate X, maturity threshold m, Fisher Information Matrix F, model parameters
0, model J
1: for 11=1 to T do
2:
X; <- (ooaDDat((DD)
3:
for each (æx; 3/) E Xx do if UseTransferabiiityyouul Bound then
> Detect Phase
4:
5:
Calculate transferabiliity bound (y by Equation 5
6:
S + ComparcBoundl (h), (h))
7:
else UneeGradiinnCCnnficctaallal then
8:
S 5 CompareGradientt D,, M) by Equation
9:
end
Decide Phase
if there exists potential negative transfer then
11:
ActivateLUM < True
else
13:
ActivateLUM <- False
end if if ActivateL_UM then
Unlearn Phase
15:
0, <- LU,(0,, F,
17:
end if
18:
02 f- GU, X, m)
19:
M UpdateBuffer(MM æt, ye)
20: end for
21:
Update Fisher Information Matrix
22:
end for return for
23:
A.3
SETUPS AND ADDITIONAL EXPERIMENTS
A.3.1 BASELINES
We compare DEDUCE with the following baselines on the datasets mentioned above. (1) oEWC Schwwrz et al. 2018): online regularization-bant
method that estimates parameter importance by FIM. (2) A-GEM (Chaudhry et al. 2019): constrains the new task learning with the gradient calculated using the stored data of previous tasks. (3) ER (Rolnick et al.) 2019): interleaves the data stored in the buffer with the current data during training. (4) DER++ (Buzzega et al.) 2020):
extends ER also storing and distilling past model logits. (5) HAL (Chaudhry al. 202 aligns
current representations with class-specific prototypes (anchors) from replayed data. (6) PCR (Lin et al. )023): online CL method that maintains class-level proxies and applies contrastive loss to mitigate CF in an online CII. setting.
(7) OnPro (Wei et al.) 2023): an online CL method that uses online prototypes learn stable representations and maintain class discriminability. (8) MOSE (Yan et al. 2024): an online CL method that orchestrates latent expertise with multi-level supervision and reverse self-distillation. (9) STAR (Eskandar et al. 2025): stabilizes training by introducing
mability-induchio weight perturbations.
A.3.2 EVALUATION METRICS
To evaluate the learning performance, we use Averaged Accuracy (ACC) measure the final ac- curacy averaged over all tasks. In addition, we use the Backward Transfer (BWT) to measure the average influence of learning the new task on all previous tasks. To assess plasticity, we use the accuracy of new task learning novel knowledge. To evaluate forgetting, we track the accuracy on Task over time, as Task 1 the most vulnerable to forgetting, making it strong indicator of method's ability mitigate CF.
proxy, since it reflects the model's ability to effectively acquire
T ACC
Σ
Ar.
(19)
T
16

---

Published as a conference paper at ICLR 2026
BWT-
(AT, As,a)
(20)
where A represents the testing accuracy of task j after learning task i.
A.3.3 DATASETS
We evaluate the performance of DEDUCE on CIFAR-100 (Krizhevsky 2009). CIFAR-10 (Krizhevsky 2009). Imag-eea (Deng al.} 2009), and CORE-50 (Lomonaco & Maltoni 2017) in both TIL and CIL settings. Furthermore, we validate DEDUCE on DN4IL dataset (Gowda al. 2023) under the domain incremental setting. The results demonstrate that DEDUCE still maintain its advantage in mitigating negative transfer under such challenging settings, as shown in
Table 3
Notably, CORE-50 is comprised of 50 classes, with around 2400 examples per class. For training, we split the dataset into 24 sequential tasks, where different tasks have joint classes. For testing, we split this dataset into 9 sequential tasks. Since different tasks have overlapping classes, when we use transferability to detect negative transfer, we cannot know which samples in the buffer come from the new task; thus, dividing the source and target domains becomes a great challenge. To evaluate the effectiveness of the transferability bound, we conduct experiments on CORE-50 in another set- ting. Assuming that classes sharing semantic similarity are more likely to induce positive knowledge transfer, we compute the transferability bound by classifying the new classes as the target domain and the previous classes that have been learned as the source domain when we detect new classes in a new task. Therefore, the results of DEDUCE(B) on CORE-50 in Table are derived based on the above setup. We can see from Table that case of CORE-50, where the task boundaries are more ambiguous. Furthermore, DEDUCE(B) also shows notable improvements under the new setting, demonstrating that it effectively mitigate inter- ference and promote forward transfer. This suggests that DEDUCE is robust in complex scenarios involving potential negative transfer.
that DEDUCE(G) can deliver consistent improvement in the
Table 3: Validation of DEDUCE on domain incremental setting.
Methods
ACC
BWT
A-GEM
26.98 0.82
-37.96-10.67
w/OUR(G)
28.25 0.58
-3..21+
ER
30.11=0.78
-31.19++0.66
w/URR(G)
32.12-0.31 0.33
-26.38
DER++
40.03=0.53
-20.02+:.44
w/OUR(G)
4.13
-16.68-10.37
A.3.4 IMPLEMENTATION DETAILS
For all CL methods, we use the ResNet18 as backbone on all datasets, similar to Kang et al. (0022) Furthermore, we investigate the ViT-Base (Dosovitskiy et al. 202T) for CIFAR- and report the results in Table 4 We train the model with an SGD optimizer for all the datasets and compare methods on an Nvidia Tesla A100 GPU 40GB. For all the methods compared, we set the same batch size (32) and replay batch size (32) for fair comparisons. We reproduce all baselines in the same environment with their source code and adopt the hyperparameters from DER++ codebase (Buzzega et al. 2020) as the baseline settings for the methods we compared in the experiments. For CIFAR-100 and CIFAR-10, we train the network for 50 epochs. For Tiny- ImageNn, we train the network for 100 epochs. For CORE-50, we train the network for 20 epochs. For DN4IL, we train the network for 50 epochs. In the experiments, we set buffer size M 500, local unlearning rate å = 0.001, global unlearning rate & = 10 10-5, and e = 0.0. We run each setting five times and report the average well the standard deviation.
17

---

Published as a conference paper at ICLR 2026
Table 4: Performance of ViT-Base DEDUCE on CIFAR-100.
ACC
BWT
Method
CIL
TIL
CIL
TIL
DER+
43.03=#023 9.87=-0887
-5..56-1:.33 44.8
-16.77+00277
w/)UR(G)
11.56=00.47
-50.1410.18
-14.69400.33
A.3.5 MORE DETAILED EXPERIMENTAL RESULTS
Comparison on Knowledge Transfer As shown in Fig. DEDUCE enhances forward knowledge transfer when learning new tasks. To further evaluate the effect of DEDUCE, we report backward transfer (BWT) in Fig. Across all datasets, most baselines with DEDUCE consistently achieve higher BWT. However, when integrated with the proposed method, the BWT of HAL is decreased.
its final accuracy after all tasks. Because our method substantially enhances HAL's learning of new tasks, this initial aceuracy becomes much higher, increasing the BWT gap even when both initial and final accuracies improve. For example. for Task 2 in CIFAR-100 (Table 5). HAL's accuracy drops from 52.2-+2.9 (A=49.3), while HAL+OUR(G) drops from 33.6-133.2 13.2 (A=70.4). Although both 83.6 and 13.2 exceed the HAL baseline, the larger initial gain (83.6 vs. 52.2) yields numerically larger drop, and thus lower BWT.
BWT measures the difference between task's accuracy immediately after its training
Table 5: Performance of Task 2.
Method
T1
T2
T3
T4
T5
T6
T7
T8
T9
TI0
HAL w/ OUR(G)
52.2
15.4
10.5
3.7
7.2
6.6
4.0
2.1
2.9
83.6
45.1
30.9
20.4
18.5
16.8
14.3
8.3
13.2
This effect also relates to HAL's inherent design limitations. HAL anchors past activations to pre- serve old knowledge. providing strong stability but placing much tighter constraints the learning of new tasks than on the retention of old ones. In other words, HAL's static anchoring restricts the model's ability to flexibly acquire representations far more than it prevents forgetting. When our transfer-aware unlearning mechanism integrated, it alleviates these constraints, leading to significant improvement in learning tasks and moderate improvement in retaining old ones.
Therefore, although HAL achieves slightly better BWT on CIFAR- 100 and Tiny- ImageNe,,,,, its per- formance on forward transfer and overall accuracy remains inferior that of HAL+DEDUCE, as shown in Fig. 5 suggesting that the overall benefits of integrating DEDUCE are still substantial.
HAL HAL+DEDUCE
90
HAL HAL+DEDUCE
Averaged Accuracy (%6
80
Task Accuracy
70
70
60
50
50
40
Task
Task
(a) Performance of all learned tasks
(b) Performance of new task
Figure Performance of HAL on CIFAR-100.
Ablation Study We conduct comprehensive ablation study to assess the contribution of each com-
ponent DEDUCE. Table 2 reports the overall accuracy (ACC), while Tables 6 7 present the backward transfer (BWT) results in the CIL and TIL settings. As shown in Table 2 each mod-
18

---

Published as a conference paper at ICLR 2026
ule of DEDUCE contributes positively overall accuracy. The combination of DEDUCE with the strong baseline DER++ yields the most significant accuracy gains on most datasets. An exception is observed on CORE-50, where the combination of DER++ and the GUM achieves the best per- formance. We hypothesize that this is due to the unique characteristics of CORE-50, where task boundaries ambiguous: many tasks contain overlapping or even identical classes. This overlap may undermine the effectiveness of the LUM because it will inadvertently unlearn the knowledge of the same classes the current task. In contrast, the GUM agnostic to task semanties and focuses periodically reclaiming model capacity, which remains effective regardless of class over- lap. Importantly, even this challenging scenario, DEDUCE still outperforms the DER++ baseline, demonstrating its robustness in complex and overlapping task settings. Furthermore, BWT results Tables 6 7 confirm that DEDUCE consistently alleviates negative backward transfer across all datasets, suggesting that DEDUCE is effective in mitigating task interference.
Table 6: Ablation study of the OUR proposed framework (DEDUCE).
CIFAR-100
CIFAR-10
Method
CIL
TIL
CIL
TIL
DER+
-5..71=1.60
-14.72+00.6
123.55++.377
.4.89+0043
w/LUM
49.76=0087
-1..21+0.24
-25.29
...0++0.33
w/GUM
-11.45+1.118
-14.54+0.54
-24.79+-.110
4.322+0.21
wolLreg
47.61+0093
01..11+0.62
+2..31+0.53
-3.21+0.25
w/)
4..3
013228+0.46
-20.84+-0.9
0.224
Table 7: Ablation study of the OUR proposed framework (DEDUCE).
Tiny-ImageNe
CORE-50
Method
CII.
TIL.
CIL.
THL.
DER+
-4.442+0.2
-28.52.00.33
14.998
-1.9942.35
w/LUM
-63.89=0.64
11.001
-13.82+1143
-0.70+1.27
w/GUM
-63.67=0.60
-27.75=10.69
111.97+:1.22
-1.14+1..20
wol Lreg
-58.28=0.88
-26.57+:0788
-12.35+1.10
-0.47+1.38
w/OUR(G)
-55.21
-24.57:10.41
-12.1911.52
1.62:1..28
Hyperparameter Analysis We analyze the impact of three key hyperparameters: the local unlearn- ing rate 8 in Table the global unlearning rate À in Table[9 and the tolerance for triggering local unlearning in Table 10 For both å and X, we can observe that the performance first increases and then declines the values grow, indicating that moderate unlearning effectively removes interfering knowledge while preserving useful information. Regarding c, we observe that ACC and BWT de- grade decreases. Specifically, smaller imposes stricter criterion for activating LUM, which can lead to performance degradation. In particular, when e O, the model achieves the highest ACC and BWT. This suggests that it is beneficial to trigger local unlearning whenever there exists potential negative transfer, even if the gradient conflict is not strong. It is worth noting that different baselines based on different datasets will have different optimal hyperparameters.
Table 8: Analysis of local unlearning rate ó on CIL.
Dataset
0.0001
0.001
0.01
0.03
0.05
CIFAR-100
ACC
37.77
39.84
44.08
40.87
31.04
BWT
-53.76
-49.36
-22.98
-20.24
-16.80
CIFAR-10
ACC
71.07
74.60
74.87
70.45
66.01
BWT
-28.55
-20.84
-19.17
-26.36
-31.08
Tiny- ImageNett
ACC
15.40
22.13
17.68
14.72
13.57
BWT
-58.11
-55.21
-56.95
-59.93
-60.27
19

---

Published as a conference paper at ICLR 2026
Table 9: Analysis of global unlearning rate ø on CIL.
Dataset
le-6
1e-5
le-4
5e-4
CIFAR-100
ACC
39.12
39.84
37.93
31.26
BWT
-50.70
-49.36
-52.26
-59.48
CIFAR-10
ACC
71.15
74.60
71.95
70.58
BWT
-27.51
-20.84
-27.59
-28.38
Tiageem
ACC
18.60
22.10
18.47
17.30
BWT
-57.70
-55.21
-58.51
-59.78
Table 10: Analysis of tolerance CIL.
Dataset
0.2
0.0
-0.2
-0.5
-0.7
CIFAR-100
ACC
37.10
39.84
35.97
35.64
35.54
BWT
-53.54
-49.36
-55.33
-55.89
-57.33
CIFAR-10
ACC
70.03
74.60
71.67
71.33
69.47
BWT
-29.63
-20.84
-27.38
-28.68
-31.16
ImageNt
ACC 19.11
22.10
17.49
16.51
12.67
BWT
-58.05
-55.21
-60.57
-61.96
-63.90
Impact of Buffer Size We report the performance of DEDUCE under varying memory capacities 100, 200, 500, 1000 in Tables [1] 12 Across all settings, DEDUCE consistently enhances basclines performance. Notably, even with large buffer size 1000, where replay-based baselines already achieve strong performance due to abundant memory, DEDUCE provides further gains, demon- strating complementary value. More importantly, under constrained memory as 100, DEDUCE yields even greater relative improvements, effectively mitigating the performance drop caused by limited exemplars. These results highlight DEDUCE's ability to reduce task interference and en- hance knowledge integration, enabling robust continual learning across memory budgets.
Table 11: Overall accuracy on CIFAR-100 with different memory sizes.
Method
M 100
M = 200
M 500
M 1000
ER
58.23++1.49
66.53+
03.43++0.54
78.71+00.9
W/URR(G
62.65:11.47
66.72-11.33
76.03:10.23
78.8311072
DER+
56.38:00.35
67.04
79.53 00.54
w/ORR(G)
60.26:10.65
70.85:10.21
78.43:10.36
80.19:10.11
Table 12: Backward Transfer on CIFAR-100 with different memory sizes.
Method
M 100
M 200
M = 500
M 1000
ER
-3..44-0.66
-15.82:1.62 10.5
-11.80-00.88
w/URR(G)
-27.89 1.01
-22.75+1.14
-13.92-1..99
-10.39
DER+
-3..04-0.53
10..10
-14.72-10.9
-10.67-0.44
w/OUR(G) -31.70 0.54
20.444.0.30
-13.24
-10.47+-0.61
Impact Training Epochs To assess the robustness DEDUCE, we investigate the effect of dif- ferent training epochs performance, increasing the number of epochs helps reduce optimization instability. We therefore further evaluate whether DEDUCE can continue to provide benefits under extended training. Table 13 reports results for DER++ and DER++ with DEDUCE under 50, 70, 90, 110, and 130 training epochs on CIFAR-100.
Two observations can be made: (i) Stable improvements across training lengths. DEDUCE consis- tently outperforms DER++ in both ACC and BWT, demonstrating that its benefits persist as training epochs increase. (ii) Efficiency advantage. DEDUCE at 50 epochs achieves higher accuracy than
20

---

Published as a conference paper at ICLR 2026
Table 13: Analysis of training epochs.
Method
50
70
90
110
130
ACC
DER++
36.85
35.92
36.55
37.86
38.86
w/)
39.84
39.35
37.89
38.83
39.57
BWT
+ER+
-53.71
-55.32
-54.65
-54.96
-52.50
w/ORR(G)
-47.36
-49.08
-50.84
-48.97
-49.30
DER++ trained for 130 epochs, showing that it reduce training cost while improving perfor- mance. These observations confirm that longer training alone cannot resolve task interference, and DEDUCE remains necessary
when training epochs increased.
Computation Efficiency of DEDUCE To quantify the computational overhead introduced by our method, report training time per epoch the CIFAR- 100 dataset for various baselines, with and without our proposed modules in Table 14
Table 14: Computation efficiency of the proposed method (CIFAR-100, one epoch in seconds)
Method
Baseline
w/ OUR(B)
w/ W/OUR(
PCR
11.19s
57.38s
MOSE
12.32s
235.23s
oEWC
6.94s
61.30s
STAR
23.12s
37.46s
85.06s
AGEM
10.99s
29.17s
74.77s
HAL
13.60s
17.66s
86.50s
ER
11.23s
16.49s
68.69s
DER++
11.90s
19.73s
74.88s
As shown in Table 14 while detect-decide-unleen framework introduces additional computa- tion, the overhead remains moderate. Importantly, the bound-based variant OUR(B) adds minimal extra cost, its runtime is very close the original baselines (e.g., HAL: 13.60s 17. 17.66s; DER++: 11.90s + 19.73s). This provides a highly efficient option for settings with limited computational budgets. The gradient- based variant (OUR(G)) incurs higher overhead because it performs batch- wise detection. However, its runtime is still within a reasonable range, and several baselines com- bined with OUR(G) (e.g.. DER+ OUR(G), STAR + OUR(G)) are still faster than MOSE alone, indicating that the cost is far from prohibitive.
Specifically, our method requires one extra forward pass for detection and one forward/hackward pass for unlearning when unlearning is triggered. In many cases, unlearning is not activated, re- sulting in only a single additional forward pass. To further balance efficiency and performance, provide multiple detection strategies with different granularities (task-level, epoch-level, and batch-level). These options act as built-in optimization strategies: users can choose the appropri- ate granularity depending computational constraints, while all variants still provide consistent performance improvements (Appendix A.4.2).
A.4 ANALYSIS OF NEGATIVE TRANSFER DETECTION STRATEGIES
To motivate our design, we provide two perspectives that illustrate how selective unlearning allevi- ates negative forward and backward transfer. Each perspective naturally leads corresponding detection strategy in DEDUCE.
A.4.1 TRANSFERABILITY BOUND
From the feature space perspective, negative transfer arises when tasks induce overlapping repre- sentations, leading to blurred decision boundaries. As shown in Fig. 6 a
after learning Task A, introducing Task B can cause Task A's features to drift and overlap with Task B's, thereby degrad- ing both tasks' performance. Selective unlearning helps reduce such overlap. stabilizing Task A's representation while improving Task B's adaptation, shown in Fig. 6 bb. Coneretely, upon de-
21

---

Published as a conference paper at ICLR 2026
tecting negative transfer, DEDUCE selectively unlearss outdated prior knowledge that drives task interference. This prevents features of carlier tasks from drifting into the representation space of new tasks, thereby reducing overlap and sharpening task boundaries. In contrast to indiscriminate un- learning, which discards all prior knowledge, selective unlearning preserves useful representations and removes only the interfering knowledge, striking balance between stability and adaptability. Motivated by this, we adopt a transfcrability bound to quantify task compatibility.
60
40
40
20
20
-20
-20
-40
Task after Task A
Task A after Task A
Task after
Tesk after
lask efter lask
lesk efter lask B
-80
40 -20
-60
-40 -20
20 40
(a) Task A drift without unlearning
(b) Task A drift with unlearning
Figure 6: Toy example on feature-space effects of selective unlearning.
We further validate this strategy on CIFAR-100. The transferability bound predicts negative transfer when learning Task 2, but not Task 7. Consistent with this, Task 1 undergoes distribution drift and overlap with Task 2 without DEDUCE (Fig. 7 while DEDUCE reduces the drift and overlap to some extent (Fig. 7 D.. In contrast, when learning Task 7, the bound does not trigger unlearning, and Task 6 remains stable with relatively clear separation from Task 7 (Fig, c). These results highlight two points: (i) The transferability bound provides reliable signal for detecting negative transfer. (ii) DEDUCE can effectively reduce task interference and mitigate negative backward transfer.
distriution distibuutionn
The at
60
The Reriiitten Garss EntiLtto
of
The sorbuuion Strioutionn
40-
20
20
-20
20
20-
-40
-00
-40
-60
-60
-40
-20
20
40
60
-30
-40
-20
0
20
40
-50
-25
25
50
(a) Task drift without DEDUCE
(b) Task drift with DEDUCE
(c) Task 6 drift without DEDUCE
Figure 7: Analysis of transferability bound on CIFAR-100.
A.4.2 GRADIENT CONFLICT ANALYSIS
From the optimization perspective, negative transfer manifests as confticting gradients between old and new tasks. As illustrated in Fig. [ when learning Task B after Task A, gradient directions may conflict, hinder adaptation, and induce forgetting. Selective unlearning mitigates this by unlearn- ing interfering knowledge, thereby reducing gradient conflict and enabling smoother knowledge integration. This motivates second detection strategy, gradient conflict analysis, which directly measures the alignment between task gradients. When strong conflicts are detected, the Local Un- learning Module (LUM) is activated to selectively unlearn outdated knowledge before learning the new batch, adaptively balancing stability and plasticity.
We evaluate variants: epoch-level, which compares gradients once per epoch, and betch-level, which monitors conflicts at mini-batei batch granularity. Table[[5]eport the frequency of LUM activation under each scenario, while Table 16 presents the corresponding accuracy achieved by DEDUCE. As
22

---

Published as a conference paper at ICLR 2026
shown in Table I6 DEDUCE with both epoch-level and batch-level detection strategies effec- tively improve accuracy across settings, validating the utility of gradient conflict as a proxy for task interference. Notably, the batch-level variant achieves higher accuracy improvements while activat- ing the LUM less frequently, as shown in Table 13] This demonstrates its efficiency and precision in identifying negative transfer moments for unlearning. These results support the following conclu- sions: (i) Gradient conflict analysis provides reliable online signal for detecting negative transfer. (ii) Timely activation of LUM is crucial, as immediate responses detected interference enhance overall learning accuracy. (iii) DEDUCE can selectively unlearn interfering prior knowledge at the right time, thereby enabling better task adaptability and reducing interference.
Table 15: Frequency of LUM activation under different scenarios.
Scenario
CIFAR-100
CIFAR-10
Tiny-ImageNet
Epoch-level 92.22% 35.99%
84.66% 26.55%
47.97%
Batch-level
36.68%
Table 16: ACC of DEDUCE with gradient conflict analysis at epoch-level (G-E) and batch-level (G-B)
CIFAR-100
CIFAR-10
Imagel
Method
CIL
TIL
CIL
TIL
CIL
TIL
DER++
36.85=1098 75.64
1..14
93.4610.29
1..61 51.10
w/UR(GGBB)
39.01=0.58
94.02:00.15 73.12 78.31
17.83#1.17 22.09:10.46 55.60
05.49::0.42
w/OUR(G-B)
39.84-0.93
74.56-0.73 78.4410.63
94.7710.24
A.4.3 COMPUTATION EFFICIENCY OF DETECTION
We investigate two complementary strategies for detecting negative transfer. Table compares the training time of the detection strategies. The transferability bound is highly efficient because operates at the task-level, providing quick estimation of task transferability. In contrast, gradient conflict analysis, especially at batch-level, offers finer-grained and more stable detection, making particularly suitable for online continual learning where task boundaries are unavailable. However, this comes at a higher computational cost, as reflected the longer runtime. Overall, transferability bound is preferable when efficiency is critical, while gradient conflict analysis provides stronger precision and robustness in settings that demand accurate, online detection of interference.
Table 17: Computation efficiency of different detection strategies (CIFAR-100, one epoch).
DER+
DEDUCE(B) DEDUCE(G)
Time(seconds)
11.90
14.32
52.95
A.5 ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. It does not involve human subjects or sensitive data, and all datasets are publicly available. The proposed methods are intended for scientific purposes only, and we have ensured fairness, transparency, and compliance with ethical and legal standards.
A.6 REPRODUCIBILITY STATEMENT
We present details of experimental settings in Appendix A.3.3
well as all the used pyreme-arameters in Appendix (A..35) Moreover, we present detailed pseudocode in Appendix A.2 Additionally, source code is publicly available in the supplementary document.
A.3.4
23

---

Published as a conference paper at ICLR 2026
A.7 USE OF LARGE LANGUAGE MODELS
Large language models (LLMs) were used solely for language refinement and polishing of the manuscript. They did not contribute the conception of the research, design of methods, execution of experiments, or analysis of results.
24


Reviews:
[{"review_id": "RpcW56zzaT", "reviewer": "Reviewer_GPEL", "summary": "The paper proposes DEDUCE, a transfer-aware framework for continual learning that dynamically detects and mitigates negative transfer, where outdated or irrelevant prior knowledge interferes with learning new tasks. The framework combines two detection strategies: a transferability bound derived from domain adaptation theory to estimate when prior knowledge harms generalization, and gradient conflict analysis to identify real-time interference between old and new task gradients. Experiments across CIFAR100, TinyImageNet, and CORE50 show consistent improvements over strong CL baselines.", "questions": "What is the computational overhead of the proposed method compared to other methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Introduces a novel approach to detect and decide when to unlearn, bridging machine unlearning and continual learning communities.\n2. Dual-strategy detection plus hybrid unlearning are well-motivated and mathematically backed.", "weaknesses": "1. Relies on stored exemplars and known task identities, and the applicability to online task-free CL remains unclear.\n2. All experiments use vision classification benchmarks, which may limit the paper's generality.\n3. The framework adds two auxiliary modules, which may cause additional compute complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gU0UiAjrhV", "reviewer": "Reviewer_aygx", "summary": "This paper proposes a novel approach to continual learning that aims to effectively balance stability and plasticity. The key idea is to detect interference from new tasks on past task knowledge. When such interference is detected, a Local Unlearning Module selectively removes conflicting knowledge while preserving critical parameters. A Global Unlearning Module identifies less important network components to allocate more learning capacity for new tasks, thereby reducing forgetting. The method is supported by theoretical justifications and evaluated empirically on standard CL benchmarks.", "questions": "- What is the explanation for the results where the bound-based variant (OUR(B)) underperforms the gradient-based variant (OUR(G))?\n\n- Why does backward transfer (Figure 4) decrease for HAL when combined with the proposed method?\n\n- Why does EWC perform worse than the simple fine-tuning baseline in your experiments?\n\nAlso refer to the weakness section above.\n\n\n**Additional Remarks:**\n\nI am open to increasing my score if the concerns about evaluation fairness and clarity are addressed in the rebuttal.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A novel and well-motivated approach that integrates neuron importance estimation with targeted unlearning based on interference signals.\n\n- Strong theoretical foundation supporting the role of local and global unlearning.\n\n- The proposed method indicates a meaningful contribution to continual learning research.", "weaknesses": "The evaluation setup requires more clarity. See details below:\n- It is unclear whether all methods (including the proposed one) are trained in a fully online continual learning scenario.\n\n- The buffer size used in Table 1 should be explicitly stated.\n\n- The DER++ baseline results differ noticeably from previously published online CL work (e.g., OnPro). I suggest re-running DER++ and comparing under the OnPro (ICCV 2023) experimental protocol for fairness:\n\n  - OnPro: Online Prototype Learning for Online Continual Learning, ICCV 2023\n\n- Results for the STAR baseline also differ from what has been reported in prior work. How is STAR implemented here?\n  - According to the STAR paper, STAR > DER++ in most settings, so STAR being significantly weaker raises concerns of evaluation fairness.\n  - When comparing with STAR, shouldn’t the comparison be against X-DER + STAR?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NTfxJLAZZ9", "reviewer": "Reviewer_6M6c", "summary": "This paper addresses the problem of negative transfer in continual learning (CL) by proposing a 3-stage framework, Detect, Decide, and Unlearn, which can be plugged into existing CL methods to improve the performance. It presented two strategies, task-level transferability bound and batch-level gradient conflict analysis, to detect negative transfer. If the interference between old and new tasks is detected, it performs unlearn-learn update to future batches. This approach was tested on several existing CL approaches with standard CL benchmarks, showing its effectiveness of increasing the CL performance.", "questions": "1. Clarify Eq8, i.e., how does maximizing the loss on the current batch can reliably unlearn prior interfering knowledge, rather than unlearning new knowledge\n2. Provide a more detailed analysis on the extra computational cost and propose/discuss optimiziation to make the framework more lightweight \n3. How to address the limitations of using diagonal FIM? \n4. How will the proposed framework perform with more recent CL including those using pre-trained models?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has a novel contribution to CL, i.e., focusing on negative transfer, while many others are concerned with catastrophic forgetting.\n\nThe two different mechanisms of detecting negative transfers are designed at different levels and can complement each other. \n\nThe LUM is designed to balance between removing the interference (negative transfer) and important parameters (prior knowledge). The GUM can help keep those low activated but important neurons.", "weaknesses": "This detect-decide-unlearn process introduces significant computational costs, and the partial of the costs is mentioned in the appendix. The work needs to be further analyze the extra costs and be optimized to make this the process lightweight. \n\nIn Eq8, it is unclear why L_{unlearn} is defined in that way. Maximizing the error on the new task's current batch does not necessary mean to unlearn the old task. It could be unlearning the new task as well. \n\nThe LUM relies on a diagonal FIM to compute the importance of parameters, which makes it inherits all the known limitations of diagonal FIM, such as inaccurate approximation,  scalability issue of storing FIM, and saturation of FIM with more tasks are coming. \n\nThe experiment of the paper should include several more recent CL baselines, including those leveraging pre-trained models, to give a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fwuKdCisQb", "reviewer": "Reviewer_wJFi", "summary": "This paper introduces DEDUCE, a continual learning framework that detects negative transfer between tasks and mitigates it through selective unlearning. The core idea is that not all prior knowledge should be preserved - some interfering knowledge should be unlearned to improve adaptation to new tasks. The framework uses two detection strategies (transferability bounds and gradient conflict analysis) and two unlearning modules (local and global) to dynamically balance knowledge retention and plasticity. Experiments show consistent improvements across multiple datasets and baselines.", "questions": "1. Can you provide theoretical or empirical evidence that gradient conflict during optimization actually correlates with negative transfer at test time?\n\n2. How exactly does maximizing cross-entropy loss lead to selective unlearning of only interfering knowledge? What prevents this from removing useful knowledge as well?\n\n3. In Tab. 13, the LUM activation frequency varies dramatically across datasets (92% for CIFAR-100 epoch-level vs 36% for batch-level). What does this suggest about the reliability of the detection mechanism?\n\n4. Could the performance improvements be explained by regularization effects rather than actually addressing negative transfer or improving plasticity?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses an underexplored aspect of continual learning - that blindly preserving all knowledge can hurt performance. This perspective is refreshing and the neuroscience motivation about selective forgetting is compelling.\n\n2. The experimental evaluation is comprehensive, testing across multiple datasets (CIFAR-100, CIFAR-10, Tiny-ImageNet, CORE-50) and showing that DEDUCE works as a plug-in enhancement for various existing methods. The consistency of improvements across different baselines suggests the approach has broad applicability.\n\n3. The paper provides two complementary detection strategies, which is useful given different computational and accuracy trade-offs. The ablation studies are thorough and help understand the contribution of different components.", "weaknesses": "1. The author may ignore the optimization-generalization gap. There's a fundamental conceptual issue with the gradient conflict detection mechanism. Gradient conflict is an optimization phenomenon, while negative transfer concerns generalization. The paper doesn't justify why opposing gradients during training would necessarily indicate poor test-time transfer between tasks. This conflation undermines the theoretical foundation of one of the main detection strategies.\n\n2. The implementation of the transferability bound also has gaps. The use of LEEP scores as a proxy for λ isn't well justified, especially since LEEP was designed for offline transfer learning with fixed source models, not continual learning where models evolve. The connection between Eq.4 and the actual bound is hand-wavy at best.\n\n3. The paper doesn't clearly explain what constitutes **interfering knowledge** versus **useful knowledge**. How does maximizing cross-entropy loss (Eq. 8) specifically target interference rather than just degrading performance broadly? The mechanism seems too coarse-grained for selective unlearning.\n\n4. The global unlearning module appears disconnected from the negative transfer detection - it periodically resets neurons regardless of whether negative transfer is detected. This seems more like a generic plasticity mechanism than targeted unlearning.\n\n5. The paper claims that unlearning improves plasticity but provides no direct evidence for this. While they show accuracy improvements on new tasks, they don't demonstrate that this is due to increased plasticity rather than other factors. Including analysis of network capacity in [1] could help.\n\n[1] Dohare S, Hernandez-Garcia J F, Lan Q, et al. Loss of plasticity in deep continual learning[J]. Nature, 2024, 632(8026): 768-774.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)