A Unified and General Framework for Continual Learning
Authors: Zhenyi Wang
Yan Li
Li Shen
Heng Huang
Venue: ICLR 2024 poster (2024)
Keywords: Continual Learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Continual Learning (CL) focuses on learning from dynamic and changing data distributions while retaining previously acquired knowledge. Various methods have been developed to address the challenge of catastrophic forgetting, including regularization-based, Bayesian-based, and memory-replay-based techniques. However, these methods lack a unified framework and common terminology for describing their approaches. This research aims to bridge this gap by introducing a comprehensive and overarching framework that encompasses and reconciles these existing methodologies. Notably, this new framework is capable of encompassing established CL approaches as special instances within a unified and general optimization objective.
An intriguing finding is that despite their diverse origins, these methods share common mathematical structures. This observation highlights the compatibility of these seemingly distinct techniques, revealing their interconnectedness through a shared underlying optimization objective. Moreover, the proposed general framework introduces an innovative concept called *refresh learning*, specifically designed to enhance the CL performance. This novel approach draws inspiration from neuroscience, where the human brain often sheds outdated information to improve the retention of crucial knowledge and facilitate the acquisition of new information. In essence, *refresh learning* operates by initially unlearning current data and subsequently relearning it. It serves as a versatile plug-in that seamlessly integrates with existing CL methods, offering an adaptable and effective enhancement to the learning process. Extensive experiments on CL benchmarks and theoretical analysis demonstrate the effectiveness of the proposed *refresh learning*.

Published as a conference paper at ICLR 2024
A UNIFIED AND GENERAL FRAMEWORK FOR CONTIN- UAL LEARNING
Zhenyi Wang', Yan Li', Li Shen", Heng Huang' University of Maryland, College Park 2JD Explore Academy {zwang169, yanli18, heng @mdd. edu, mathshenli@gmail.c
ABSTRACT
Continual Learning (CL) focuses on learning from dynamic and changing data distributions while retaining previously acquired knowledge. Various methods have been developed to address the challenge of catastrophic forgetting, includ- ing eggularization-baseee
techniques. However, these methods lack a unified framework and common terminology for describing their approaches. This research aims bridge this gap by introducing comprehensíve and overarching framework that encompasses and reconciles these existing methodologies. Notably, this new framework is capable of encompass- ing established CL approaches as special instances within a unified and general optimization objective. An intriguing finding that despite their diverse origins, these methods share common mathematical structures. This observation highlights the compatibility of these seemingly distinct techniques, revealing their intercon- nectedness through shared underlying optimization objective. Moreover, the proposed general framework introduces an innovative concept called refresh learn- ing, specifically designed to enhance the CL performance. This novel approach draws inspiration from neuroscience, where the human brain often sheds outdated information to improve the retention of crucial knowledge and facilitate the quisition of new information. In essence, refresh learning operates by initially unlearning current data and subsequently relearning it. It serves versatile plug-in that seamlessly integrates with existing CL methods, offering an adaptable and effective enhancement to the learning process. Extensive experiments on CL benchmarks and theoretical analysis demonstrate the effectiveness of the proposed refresh learning.
Bayesian-based and memory-replay-ben
INTRODUCTION
Continual learning (CL) is dynamic learning paradigm that focuses acquiring knowledge from data distributions that undergo continuous changes, thereby simulating real-world scenarios where new information emerges over time. The fundamental objective of CL is adapt and improve model's performance as it encounters new data while retaining the knowledge it has accumulated from past experiences. This pursuit, however, introduces a substantial challenge: the propensity forget or overwrite previously acquired knowledge when learning new information. This phenomenon, known as catastrophic forgetting (McCloskey & Cohen, 1989), poses a significant hurdle in achieving effective CL. As result, the development of strategies to mitigate the adverse effects of forgetting and enable harmonious integration of new and old knowledge stands as a critical and intricate challenge within the realm of CL research.
plethora of approaches have been introduced to address the challenge of forgetting in CL. These methods span range of strategies, encompassing Bayesian- based techniques (Nguyen al., 2018; Kao et al., 2021), regularization-diie -itiiattioondrieen solutions (Kirkpatrick et al., 2017: Cha et al., 202 I), and memory-replay -orieted methodologies (Riemer et al., 2019; Buzzega et al., 2020). These methods have been developed from distinct perspectives, but lacking a cohesive framework and a standardized terminology for their formulation.
In the present study, we endeavor to harmonize this diversity by casting these disparate categories of CL. methods within unified and general framework with the tool of Bregman divergence. As

---

Published as a conference paper at ICLR 2024
Table I: A unified framework for CL. We define a generalized CL optimization objective as LCL LeE( (zz, y) + +aDD (()hh/(()))))))))())) ;)) 3D (0, O.d)). Where a: 0, 2 0, CC y) is the loss function on new task, D& (ho (z), =) is output space regularization represented as a Bregman divergence associated with function $, D& (0, ata) is weight space regularization represented as Bregman divergence associated with function V. Several existing representative CL methods can be recovered from this general optimization objective by setting different Φ; v and Bregman divergence.
Category
Method
Ref
Recover Setting
VCL
Nguyen et al. (2018)
a 0. (p) = p(a) log d(
Bayesian-bused
NCL
Kao et al. (2021)
+ (p)
Σi-1 p. log pi. V 1100
EWC
Kirkpatrick et al. (2017)
0. (0) 1 P0
regularization-basee
CPR
Cha et al. (2021)
+ (p))
> log
Memory-replay-basee
ER
Chaudhry et (2019b)
0,
(p)
Di=1 P: log
DER
Buzzega al. (2020)
(x(
||/]|
Novel CL method
Refresh Learning
Ours
Unleam-releam plug-in
outlined in Table 1, we introduce a generalized CL optimization objective. Our framework is designed to flexibly accommodate this general objective, allowing for the recovery of a wide array of representative CL methods across different categories. This is achieved by configuring the framework according to specific settings corresponding to the desired CL approach. Through this unification. we uncover an intriguing revelation: while these methods ostensibly belong to different categories, they exhibit underlying mathematical structures that are remarkably similar. This revelation lays the foundation for a broader and more inclusive CL. approach. Our findings have the potential to open avenues for the creation of a more generalized and effective framework for addressing the challenge of knowledge retention in CL scenarios.
Our unified CL framework offers insights into the limitations of existing CL methods, It becomes evident that current CL techniques predominantly address the forgetting issue by constraining model updates either in the output space or the model weight space. However, they tend to prioritize the preservation of existing knowledge while potentially neglecting the risk of over-memorization. Over- emphasizing the retention of existing knowledge doesn't 't necessarily lead to improved generalization, as the network's capacity may become occupied by outdated and less relevant information. This can impede the acquisition of new knowledge and the effective recall of pertinent old knowledge.
To address this issue, we propose a refresh learning mechanism with a first unlearning, then relearn the current loss function. This is inspired by two aspects. On one hand, forgetting can be beneficial for the human brain in various situations, as helps in efficient information processing and Docision-making (Davis & Zhong, 2017; Richards & Frankland, 2017; Gravitz, 2019; Wang et al., 2023b). One example the phenomenon known as "cognitive load" (Sweller, 2011). Imagine person navigating through a new big city for the first time. They encounter multitude of new and potentially overwhelming information, such as street names, landmarks, and various details about the environment. If the brain were to retain all this information indefinitely, it could lead to cognitive overload, making challenging to focus on important aspects and make decisions effectively. However, the ability to forget less relevant details allows the brain to prioritize and retain essential information. Over time, the person might remember key routes, important landmarks, and necessary information for future navigation, while discarding less critical details. This selective forgetting enables the brain to streamline the information it holds, making cognitive processes more efficient and effective. In this way, forgetting serves natural filter, helping individuals focus on the most pertinent information and adapt to new situations without being overwhelmed by an excess of irrelevant details. On the other hand, CL involves adapting to new tasks and acquiring new knowledge over time. If model were to remember every detail from all previous tasks, it could quickly become impractical and iesuurce-itensiie.. Forgetting less relevant information helps in managing memory resources efficiently, allowing the model to focus on the most pertinent knowledge (Feldman & Zhang, 2020). Furthermore, catastrophic interference occurs when learning new information disrupts previously learned knowledge. Forgetting less relevant details helps mitigate this interference, enabling the model to adapt to new tasks without severely impacting its performance on previously Iearned tasks. Our proposed refresh learning is designed as straightforward plug-in, making it easily compatible with existing CL methods. Its seamless integration capability allows it to augment the performance of CL techniques, resulting in enhanced CL performance overall.
To illustrate the enhanced generalization capabilities of the proposed method, we conduct a compre- hensive theoretical analysis. Our analysis demonstrates that refresh learning approximately minimizes
2

---

Published as a conference paper at ICLR 2024
the Fisher Information Matrix (FIM) weighted gradient norm of the loss function. This optimization encourages the flattening of the loss landscape, ultimately resulting in improved generalization. Extensive experiments conducted various representative datasets demonstrate the effectiveness of the proposed method. Our contributions are summarized as three-fold:
We propose a generalized CL optimization framework that encompasses various CL. ap- proaches special instances, including Bayesian- based, patllarizatioo-bseed. and memory- replay-based CL methods, which provides a new understanding of existing CL methods.
Building upon our unified framework, we derive a new refresh learning mechanism with an unlearn-relearn scheme to more effectively combat the forgetting issue. The proposed method a simple plug-in and can be seamlessly integrated with existing CL methods.
We provide in-depth theoretical analysis prove the generalization ability of the proposed refresh learning mechanism. Extensive experiments on several representative datasets demonstrate the effectiveness and efficiency of refresh learning.
RELATED WORK
Continual Learning (CL) (van de Ven et al., 2022) aims to learn non-stationary data distribution. Existing methods on CL can be classified into four classes. (1) Regularization-basee methods regularize the model weights or model outputs to mitigate forgetting. Representative works include (Kirkpatrick et al., 2017; Zenke et al., 2017b; Chaudhry et al., 2018: 8: Aljundi et al., 2018; Cha et al., 2021; Wang et al., 2021; Yang et al., 2023a). (2) Bayesian- based methods enforce model parameter posterior distributions not change much when learning new tasks. Representative works include (Nguyen et al., 2018; Kurle et al., 2019; Kao et al., 2021; Henning et al., 2021; Pan et al., 2020; Titsias al., 2020; Rudner et al., 2022). (3) Memory-replay-baseee bbase methods maintain a small memory buffer which stores a small number of examples from previous tasks and then replay later to mitigate forgetting. Representative works include (Lopez-Paz & Ranzato, 2017: Riemer al., 2019; Chaudhry et al., 2019c; Buzzega et al., 2020; Pham et al., 2021: Arani et al., 2022; Caccia et al., 2022: Wang al., 2022b;a; 2023cta; Yang et al., 2023b). (4) Architecture- based methods dynamically update the networks or utilize subnetworks to mitigate forgetting. Representative works include (Mallya & Lazebnik, 2018; Serra et al., 2018; Li et al., 2019; Hung et al., 2019). Our work proposes a unified framework to encompass various CL methods special cases and offers new understanding of these CL methods.
Machine Unlearning (Guo et al., 2020; Wu et al., 2020; Bourtoule et al., 2021; Ullah et al., 2021) refers to the process of removing or crasing previously Iearned information or knowledge from a pre-trained model to comply with privacy regulations (Ginart et al., 2019). In contrast to existing approaches focused on machine unlearning, which seek to entirely eliminate data traces from pre- trained models, our refresh learning is designed to selectively and dynamically eliminate outdated or less relevant information from CL model. This selective unlearning approach enhances the ability of the CL model to better retain older knowledge while efficiently acquiring new task information.
PROPOSED FRAMEWORK AND METHOD
We present preliminary and problem setup in Section 3.1, our unified and general framework for CL in Section 3.2, and our proposed refresh learning which is built upon and derived from the proposed CL optimization framework in Section 3.3.
3.1 PRELIMINARY AND PROBLEM SETUP
Continual Learning Setup The standard CL problem involves learning sequence of N tasks, repre- sented as D'r = (Dfr, D ... ). The training dataset D for the tthh task contains a collection of triplets: (xf // Th) where x. denotes the 7th data example specific to task k. yf represents the associated data label for x and Ty is the task identifier. The primary objective is to train a neural network function, parameterized by 0, denoted go(x). The goal is to achieve good performance on the test datasets from all the learned tasks, represented as e = (Dfs, Drr. ,,D)) while ensuring that knowledge acquired from previous tasks is not forgotten.
3

---

Published as a conference paper at ICLR 2024
Bregman Divergence Consider c: Q - R as strietly convex differentiable function and defined on a convex set 12. The Bregman divergence (Banerjee et al., 2005) related to & for two points p and within the set & can be understood as the discrepancy between the o value at point p and the value obtained by approximating & using first-order Taylor expansion at q. It is defined as:
D+ (p. q) = ppp) (
(V q). q)
where (q) is the gradient of o at q. (.) denotes the dot product between two vectors. In the upcoming section, we will employ Bregman divergence to construct a unified framework for CL.
3.2 A UNIFIED AND GENERAL FRAMEWORK FOR CL
In this section, we reformulate several established CL algorithms in terms of general optimization objective. Specifically, a more general CL optimization objective can be expressed as the following:
LCL (o v) to D& (ho( 2) +B D (0, Pota)
(2)
new Lask
outpul space
weight spuce
where denotes the CL model parameters. LCE( y) is the cross-entropy loss on the labeled data (xx 3) for the current new task. 2 0, 2 0. The term D& (?)) represents form of regularization in the output space of the CL model. It is expressed as the Bregman divergence associated with the function T The constant vector serves the model from forgetting previously learned tasks. Essentially, it is responsible for reducing changes in predictions for tasks the model has learned before. On the other hand, D& (0, Osta) represents form of regularization applied to the weight space. It is also expressed as a Bregman divergence, this time associated with the function L. The term Oold refers to the optimal model parameters that were learned for older tasks. is used to ensure that the model doesn't It adapt too rapidly to new tasks and prevent the model from forgetting the knowledge of earlier tasks. Importantly, these second and third terms in Eq. 2 work together to prevent forgetting of previously learned tasks. Additionally, it's worth noting that various existing CL. methods can be seen as specific instances of this general framework we've described above. Specifically, we cast VCL (Nguyen et al., 2018) 8), NCL (Kao et al., 2021). EWC (Kirkpatrick et al., 2017), CPR (Cha et al.. 2021), ER (Chaudhry et al., 2019c) and DER (Buzzega et al., 2020) as special instances of the optimization objective, Eq. (2). Due to space constraints, we will only outline the essential steps for deriving different CL methods in the following. Detailed derivations can be found in Appendix A.
reference value and helps us prevent
ER as A Special Case Experience replay (FR) (Riemer et al., 2019; Chaudhry et al., 2019c) is a mepory-replay based method for mitigating forgetting in CL. We denote the network softmax output as ge (z) = coftmarr(ug (ue())) (z)) and as the one-hot vector for the ground truth label. We use KL to denote the KL-divergence between two probability distributions. We denote M as the memory buffer which stores a small amount of data from previously learned tasks. ER optimizes the objective:
CLL = LCr((x v) + a Lcee:))MMCn((...
In this case, in Eq. (2), we set 3 = O. We take + to be the negative entropy function, i.c., (p)) = Ei=r p; log p.. We set p go (2), i.e., the softmax probability output of the neural network on the memory buffer data and q to be the one-hot vector of the ground truth class distribution. Then, D& (p.q) q) = KL(ge (x), y). We recovered the ER method.
DER as Special Case DER (Buzzega al., 2020) is nemory-replat based method. DER not only stores the raw memory samples, but also stores the network logits for memory buffer data examples. Specifically, it optimizes the following objective function:
CCL LCF(w, v) + alE( [u(x)) =112
where up ua(x)) is the network output logit before the softmax and the network output logit when storing the memory samples. In this case, in Eq. (2), we set / 0. We take o(x) = ||m||² Then, we set p = (x) and q = z. Then, Do (p, q) = [[/g(=) - =112. We recover the DER method.
CPR as A Special Case CPR (Cha et al., 2021) is a regularizatiion-asse method and adds an entropy regularization term to the CL model loss function. Specifically, solves:
cL LCr(2, a (go (z)) + BD (0, Oata)
(5)
4

---

Published as a conference paper at ICLR 2024
Where H (go(2)) is the entropy function on the classifier class probabilities output. In Eq. (2), we take & to be the negative entropy function, i.c., { Σi= P; log P;. We set p ge a+) i... the probability output of CL model on the current task data and q = v, i.c., the uniform distribution on the class probability distribution. For the third term, we can freely set any proper regularization on the weight space regularization. D& q) = (x)),, v) We then recover the CPR method.
EWC as A Special Case Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017), is a regularization-basee technique. It achieves this by imposing penalty on weight updates using the Fisher Information Matrix (FIM). The EWC can be expressed as the following objective:
LLL Lor(E y) + 3 - -Ott)) F(0 - Oata)
(6)
where dod is mean vector of the Gaussian Laplace approximation for previous tasks, F is the diagonal of the FIM. In Eq. (2),
= 0, we take (0) _ F9. We set p 0 and q Ooid- De (p. 9) (0- Ootd) F(O otad). Then, we recover the EWC method.
VCL as A Special Case Variational continual learning (VCL) (Nguyen et al., 2018) is Bayesian- based method for mitigating forgetting in CL. The basic idea of VCL is to constrain the current model parameter distribution to be close to that of previous tasks. It optimizes the following objective.
C LCE (z, v) + (P PP.c), P(O (P.-11)
where Di.r denotes the dataset from task to 1. Pi)) is the posterior distribution of the model parameters on the entire task sequence Dt. P(0 (Di:z-) P( is the posterior distribution of the model parameters the tasks 11.t-1- In this case, P(0 P1:a) and P(ootal (Da--1) are both continuous distributions. In this case, in Eq. (2), we sct a = 0. we take & to be (p) = p(0) log p(0) We then set p = P(0) and q- P(Oatd) D1=-1). We then recover the VCL method.
Natural Gradient CL as A Special Case Natural Gradient CL (Osawa et al., 2019; Kao et al., 2021) (NCL) is soyyesia--based CL method. Specifically, NCL updates the CL model by the following damped (generalized to be more stable) natural gradient:
0 0x+1 = 02 - nfaFF + 31)
(8)
where F is the FIM for previous tasks, / is the identity matrix and vj is the learning rate. For the second
loss term in Eq. (2), we take & to be the negative entropy function, i.c., L(p) = >iEr pi log p.. For the third loss term in Eq. (2), we adopt the (0) = 0 In Eq. (2), we employ the first-order Taylor expansion to approximate the second loss term and employ the second-order Taylor expansion
to approximate the third loss term. We then recover the natural gradient CL method. Due to the space limitations, we put the detailed theoretical derivations in Appendix A.6.
3.3 REFRESH LEARNING AS GENERAL PLUG-IN FOR CL
The above unified CL framework sheds light on the limitations inherent in current CL methodologies. It highlights that current CL methods primarily focus on addressing the problem of forgetting by limiting model updates in either the output space or the model weight space. However, these methods tend to prioritize preserving existing knowledge at the potential expense of neglecting the risk of over-memmrizzaiinn Overemphasizing the retention of old knowledge may not necessarily improve generalization because it can lead to the network storing outdated and less relevant information, which can hinder acquiring new knowledge and recalling important older knowledge.
In this section, we propose a general and novel plug-in, called refresh learning, for existing CL methods to address the above-mentioned weer-mmmorizatin. This approach involves two-step process: first, unlearning on the current mini-batch to erase outdated and unimportant information contained in neural network weights, and then relearning the current loss function. The inspiration for this approach comes from two sources. Firstly, in human learning, the process of forgetting plays a significant role in acquiring new skills and recalling older knowledge, as highlighted in studies like (Gravitz, 2019; Wang et al., 2023b). This perspective aligns with findings in neuroscience (Richards & Frankland, 2017), where forgetting is seen as essential for cognitive processes, enhancing thinking abilities, facilitating decision-makee making, and improving learning effectiveness. Secondly, neural networks often tend to overly memorize outdated information, which limits their adaptability to learn and relevant data while retaining older information. This is because their model capacity
5

---

Published as a conference paper at ICLR 2024
becomes filled with irrelevant and unimportant data, impeding their flexibility in learning and recall, as discussed in (Feldman & Zhang, 2020).
Our refresh learning builds upon the unified framework developed in Section 3.2. Consequently, obtain a class novel CL methods to address the forgetting issue more effectively. serves as a straightforward plug-in and can be seamlessly integrated with existing CL methods, enhancing the overall performance of CL techniques. We employ a probabilistic approach to account for uncertainty during the unlearning step. To do this, we denote the posterior distribution of the CL model parameter as p(0)= == PD) where == denotes a definition. This distribution is used to model the uncertainty that arises during the process of unlearning, specifically on the current mini-batoh data D.
The main objective is to minimize the KL divergence between the current CL model parameters posterior and the target unlearned model parameter posterior. We denote the CL model parameter posterior at time t as Pt. the target unlearned posterior as ja. The goal is to minimize [ Following Wibisono (2018), we define the target unlearned posterior as a energy function μ and w _c This KL divergence can be further decomposed as:
(T(palli)) = / Pr(O) log (())
Pc(0) d0 = - (:(0) loggl/o)))))) ((0)d + / P:(()) log ((0)do
(9)
(p), (p.))
where Hpe, u) =- -Bp, log is the cross-entropy between Pt and (pt) := - -Ep log Pe is the entropy of Pt. Then, we plug in the above terms into Eq. (9), and obtain the following:
(L(pl)la) = p. log μ + Exp log Ps = E, CCL + Ep, log Pt
(10)
The entire refresh learning includes both unleaming-releamnin can be formulated the following:
min p.gs LCL (releam)
(11)
s.t. Popt mi) = -E + +E, log pl (unlcarn)
(12)
where Eq. (12) is unlearn the current mini- batch by optimizing energy functional in function space over the CL parameter posterior distributions. Given that the energy functional { as defined in Eq. (12), represents the negative loss of _ it effectively promotes an inerease in loss. Consequently, this encourages the unlearning of the current mini-batch data, steering it towards the desired target unlearned parameter distribution. After obtaining the optimal unlearned CL model parameter posterior distribution, Pope, the CL model then relearns on the current mini-batch data by Eq. (1I). However, Eq. (12) involves optimization within the probability distribution space, and is typically challenging to find a solution directly. To address this challenge efficiently, we convert Eq. (12) into a Partial Differential Equation (PDE) as detailed below.
By Fokker- Planck equation (Kadanoff, 2000), gradient flow of KL divergence is as following:
0pt Ot
(KT((elll)) (p)
= =dv p:
(13)
8p
div. (q) = Ei= (z)) is the divergence operator operated vector-valued function q Rd Idd where x' and q' are the th element of z and q. Then, since the first-variation of KL-divergence, SKL(p)llm) ie., 6p (p.) = log Pt + 1 (Liu et al., 2022). We plug it into Eq. 13, and obtain the following:
(p.
(0) 1)) = (P)(()))))))))))))))))))))))(()) (w)
(14)
Ôt
μ
Then, (Ma et al., 2015) proposes a more general Fokker-Planck equation as following:
dp. (i)(( + Q(0)) + (W))
(15)
Øt
where D(0) is positive semidefinite matrix and Q()) is a skew-symmetric matrix. We plug in the defined w = _gOL into the above equation, we can get the following PDE:
a
(p) = + "0 (p(())
(16)
Øt

---

Published as a conference paper at ICLR 2024
Intuitively, parameters that are less critical for previously learned tasks should undergo rapid unlearn- ing to free up more model capacity, while parameters of higher importance should unlearn at a slower rate. This adaptive unlearning of vital parameters ensures that essential information is retained. To model this intuition, we set the matrix D(0) = F-1, where F is the FIM on previous tasks and set Q(0)= (Patterson & Teh, 2013). Eq. (16) illustrates that the energy functional decreases along the steepest trajectory in probability distribution space to gradually unlearn the knowledge in current data. By discretizing Eq. (16), we can obtain the following parameter update equation:
e = - 1 V ( + +N(0022F7)
(17)
where in Eq. (17), the precondition matrix F-1 aims to regulate the unlearning process. Its pur- pose is to facilitate slower update of important parameters related to previous tasks while al- lowing less critical parameters update more rapidly. It's important to note that the Hessian matrix of KL divergence coincides with the FIM, which characterizes the local curvature of param- eter changes. practice, this relationship is ex- pressed as KTL(((=) (z) 0) |p(z(0 + d)) x dd Fd. This equation identifies the steepest direction for achieving the fastest unlearning of the output probability distribution. To streamline computation and reduce complexity, we employ a diagonal approximation of the FIM. is important to note that the FIM is only computed once after training one task, the overall computation cost of FIM is thus negligible. The parameter y represents the unlearning rate, influencing the pace of unlearning. Additionally, we introduce random noise (0, 2-F-1) to inject an element of randomness into the unlearning process, compelling it to thoroughly explore the entire posterior distribution rather than converging solely to single point estimation.
Algorithm Refresh Learning General CI.,
1: REQUIRE: model parameters 0, CL model learn-
ing rate 7).
2:
for = 1 to K do (number CL steps)
3:
for j 1 to J do (unlcarn steps)
0
+ C (0)-1))
+
[00 2-F
5:
end for
0x+1 O2 1/ (O.) (relearn step)
7: end for
Refresh Learning As a Special Case Now, we derive our refresh learning as special case of Eq. 2:
Luntearn (ar( v) + 20 (h) x) 3D (0, O Oota) -a (ho((z)),
(18)
Ceu
In Eq. (18): we adopt the second-order Taylor expansion on (ho (x), z) the following:
D& (ho( 2) = D (he. (x), x) V (he (z). 0x)
0x) F 0x) (19)
Since Da (x)), =) is close to zero at the stationary point, i.e., Ox. we thus only need to optimize the leading quadratic term in Eq. 19. we adopt the first-order Taylor expansion on LeL as:
(C xe (0x) + VeLcz(Ox)(e 0x)
(20)
In summary, the approximate loss function for Eq. (18) can be expressed as the following:
Luntearn VoLci(0x) 0x) @ 0x) F 0k)
(21)
We then take the gradient with respect to 0 for the RHS of the Eq. (21), we can obtain the following:
VeLcl(O,) aF Ox) = 0
(22)
Solving the above equation leads to the following unlearning for the previously learned tasks:
0 0 + - -
(33)
Equation (23) is nearly identical to Equation (17), with the only distinction being that Equation (17) incorporates an additional random noise perturbation, which helps the CL model escape local minima Raginsky et al. (2017) and saddle point Ge et al. (2015). The constant I now takes on a new interpretation, serving as the unlearning rate.
In summary, we name our proposed method as refresh, which reflects our new learning mechanism that avoids learning outdated information. Algorithm 1 presents the general refresh learning method with unlearn-releee framework for general CL. Line 3-5 describes the unlearn step for current loss at each CL. step. Line 6 describes the relearn step for current loss.
7

---

Published as a conference paper at ICLR 2024
4 THEORETICAL ANALYSIS
Our method can be interpreted theoretically and improves the generalization CL by improving the flatness of the loss landscape. Specifically, refresh learning can be characterized the FIM weighted gradient norm penalized optimization by the following theorem.
Theorem 4.1. With one step of unlearning by Eq. (17), refresh learning approximately minimize the following FIM weighted gradient norm of the loss function. That is, solving Eq. (11) and Eq. (12) approximately solves the following optimization:
min (CL + VolV (0)F
(24)
where ø > O is constant.
The above theorem shows that refresh learning seeks to minimize the FIM weighted gradient norm of the loss function. This optimization objective promotes the flatness of the loss landscape since smaller FIM weighted gradient norm indicates flatter loss landscape. In practice, flatter loss landscape has been demonstrated with significantly improved generalization (Izmailov et al., 2018). It is important to note that our method is more flexible and efficient than minimizing the FIM weighted flexibly control the degree of unlearning with different
gradient norm of the loss function since number of steps, which may involve higher order flatness of loss landscape. Furthermore, optimizing Eq. (24) necessitates the calculation of the Hessian matrix, computationally intensive task. In contrast, our method offers a significant efficiency advantage as it does not require the computation of the Hessian matrix. Due the space limitations, we put detailed theorem proof in Appendix B.
EXPERIMENTS
5.1 SETUP
Datasets We perform experiments on various datasets, including CIFARIO (10 classes), CIFARI00 (100 classes), Imageee (200 classes) and evaluate the effectiveness of our proposed methods in task incremental learning (Task-IL) and class incremental learning (Class-IL), Following Buzzega et al. (2020), we divided the CIFAR- 10 dataset into five separate tasks, each containing two distinct classes. Similarly. we partitioned the EIFAR-100 dataset into ten tasks, each has ten classes. Additionally, for Tiny-ImageNet, we organized into ten tasks, each has twenty classes
Baselines We compare to the following baseline methods for comparisons. (1) Regularization-basee methods, including oEWC (Schwarz et al., 2018), synaptic intelligence (SI) (Zenke et al., 2017a), Learning without Forgetting (LwF) (Li & Hoiem, 2018), Classiferr-roojection Regularization (CPR)
(Cha et al., 2021). Gradient Projection Memory (GPM) (Saha et al., 2021). (2) Bayesian- based methods, NCL (Kao et al., 2021). (3) Architecture- baseed methods, including HAT (Serra et al., 2018 8). (4) Memory- based methods, including ER (Chaudhry et al., 2019b), A-GEM (Chaudhry et al., 2019a), GSS (Aljundi et al., 2019), DER++ (Buzzega et al., 2020), HAL(Chaudhry et al., 2021).
Implementation Details We use ResNet18 (He et al., 2016) on the above datasets. We adopt the hyperparameters from the DER+ codebase (Buzzega et al., 2020) the baseline settings for all the methods we compared in the experiments. Additionally, to enhance runtime efficiency in our approach, we implemented the refresh mechanism, which runs every two iterations.
We evaluate the performance of proposed refresh method by integrating with several existing methods with (1) overall accuracy (ACC), which is the average accuracy across the entire task sequence and (2) backward transfer (BWT), which measures the amount of forgetting on previously learned tasks. If BWT > which means learning on current new task helpful for improving the performance of previously learned tasks. If BWT < 0, which means learning on current new task can lead to forgetting previously learned tasks. Each experiment result is averaged for 10 runs with mean and standard deviation.
Evaluation Metrics
5.2 RESULTS
We present the overall accuracy for task-IL and class-IL in Table 2. Due to space limitations, we put BWT results in Table 9 in Appendix C.5. We observe that with the refresh plug-in, the
8

---

Published as a conference paper at ICLR 2024
Table 2: Task-IL and class-IL overall accuracy on CIFARIO, CIFAR-I00 and Tmny-lmageNer, respectively with memory size 500. indicates not applicable.
Algorithm Method
CIFAR-10
CIFAR-100
Tiny-ImageNee Task-IL.
Class-IL
Task-IL.
Class-IL.
Task-IL
Class-IL.
fine-tuning Joint train
19.62 +: 0.05
61.02
3.33
9.29 +: 0.33
33.78 + 0.42
7.92 0.26
18.31 +: 0.68
92.20 = 0.15
98.31
0.12
+1.32
91.31 = 0.17
59.99 = 0.19
82.04 + 0.10
ST
19.48 + 0.17
68.05
5.91
9.41 + 0.24
31.08 + 1.65
6.58
0.31
36.32 + 0.13
LWF
19.61 1 0.05
63.29
1. 2.35
9.70 0.23
28.07 1: 1.96
8.46 1 0.22
15.85 1. 0.58
NCL
19.53 + 0.32
64.49 4.06
8.12 + 0.28
20.92 + 2.32
7.50 + 0.36
16.29 + 0.87
GPM
90.68 + 3.29
72.48 + 0.40
UCB
79.28 1.87
57.15 = 1.67
HAT
92.56
72.06 0.50
A-GEM
+ 0.57
89.48 1.45
9.30 0.32
48.06 + 0.57
8.06 0.04
25.33 0.49
GSS
49.73 = 4.78
91.02 1.57
13.60
57.50 + 1.93
HAL
41.79 + 4.46
84.54 + 2.36
9.05 + 2.76
42.94 + 1.80
OEWC
= 0.12
6131
4.31
8.24 0.21
21.2 2.08
7.42 0.31
15.19 0,82
oFWC+refrsh
20.37 + 0.65
66.89 + 2.57
8.78 + 0.42
23.31 + 1.87
7,83 + 0.15
17.32 + 0.85
CPR(EWC)
19.61 + 3.67
65.23
3.87
8.42 + 0.37
21.43 + 2.57
7.67
0.23
15.58 0.91
CPR(EWC)rrfresh
20.53 + 2.42
67.36 3.68
9.06 + 0.58
22.90 + 1.71
8.06 0.43
17.90 + 0.77
ER
57.74 + 0..77
93.61
01.77
20.98 0.35
73:37 + 0.43
9.99
0.29
48.64 + 0.46
ERerffrsh
61.86 + 1.35
94.15 0.46
22.23 + 0.73
75.45 + 0.67
11.09 = 0.46
50,85 0.53
DER+
72.70 + 1.36
93.88
0.50
36.37 + 0.85
75.64 + 0.60
19.38
1.41
51.91 0.68
DeR+s-eefresh
74.42 + 0.82
94,64 + 0.38
38.49 + 0.76
77.71 + 0.85
20.81
1.28
54.06 0.79
performance of all compared methods can be further significantly improved. Notably, compared to the strong baseline DER++ our method improves by more than 2% in many cases CIFARI 10, CIFAR100 and Tiny--mageNet. The performance improvement demonstrates the effectiveness and general applicability of refresh mechanism, which more effectively retain important information from previously learned tasks since can more effectively utilize model capacity to perform CL.
5.3 ABLATION STUDY AND HYPERPARAMETER ANALYSIS
Hyperparameter Analysis We evaluate the sensitivity analysis of the hyperparameters, the unlearning rate and the number of unlearning steps J in Table in Appendix. We can observe that with increasing number of unlearning steps the CL performance only slightly improves and then decreases but with higher computation cost. For computation efficiency, we only choose one step of unlearning. We also evaluate the effect of the unlearning rate the CL. model performance.
Effect of Memory Size To evaluate the effect of different memory buffer size, we provide results in Table 4 in Appendix. The results show that with larger memory size of 2000, our refresh plug-in also substantially improves the compared methods.
Computation Efficiency To evaluate the efficiency of the proposed method, we evaluate and compare DER+ reefresh learning with DER++ on CIFAR100 100 in Table 8 in Appendix. This running time indicates that refresh learning increases 0.81 x cost compared the baseline without refresh learning. This shows our method is efficient and only introduces marginal computation cost.
6 CONCLUSION
This paper introduces an unified framework for CL. and unifies various existing CL approaches as special cases. Additionally, the paper introduces a novel approach called refresh learning, which draws inspiration from neuroscience principles and seamlessly integrates with existing CL methods, resulting in enhanced generalization performance. The effectiveness of the proposed framework and the novel refresh learning method is substantiated through series of extensive experiments various CL datasets. This research represents a significant advancement in CL, offering a unified and adaptable solution.
Acknowledgments
This work was partially supported by NSF IIS 2347592, 2347604, 2348159,
2348169, DBI 2405416, CCF 2348306. CNS 2347617.

---

Published as a conference paper at ICLR 2024
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pp. 139-154, 2018.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems 30, 2019.
Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general con- tinual learning method based on complementary learning system. In International Confer- ence Learning Representations, 2022. URL //tttpee// net.eeewweeeeeeeeeeeeeeeeeeeee wereereewww.nee/foruniii.... met/forum?id UXXFrDwrE7Y.
Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, Joydeep Ghosh, and John Lafferty. Clustering with bregman divergences. Journal of machine learning research, 6(10), 2005.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Pictro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 23:15920-15930, 2020.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky.
New insights on reducing abrupt representation change in online continual learning. In International Conference on Learning Representations, 2022. URL ht attps: : //pp/ppeeeeeeeeeeeeeeeeeeeeeeeee net forum? 1d-NeMaByOZUfb.
Sungmin Cha, Hsiang Hsu, Taeback Hwang, Flavio Calmon, and Taesup Moon. Cpr: Classifier- projection regularization for continual learning. In International Conference on Learning Repre- sentations, 2021.
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings the European conference on computer vision (ECCV), pp. 532-547, 2018.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. Proceedings of the International Conference on Learning Represen- tations, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc' Aurelio Ranzato. Continual learning with tiny episodic memories. htnp::/aanxiv.org//b////000222 10486, 2019b.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 10486, 2019c.
Arslan Chaudhry, Albert Gordo, Puneet K Dokania, Philip HS Torr and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. Association for the Advancement of Artificial Intelligence (AAAI), 2021.
Ronald Davis and Yi Zhong. The biology of forgetting-aa perspective. Neuron, 95(3):490-503 2017.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891, 2020.
Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points online stochastic gradient for tensor decomposition. In Conference learning theory, pp. 797-842. PMLR, 2015.
10

---

Published as a conference paper at ICLR 2024
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Lauren Gravitz. The importance of forgetting. Nature, 571(ully:S12-SA4 S14- 2019.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In International Conference on Machine Learning, pp. 3832-3842. PMLR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Christian Henning, Maria Cervera, Francesco D' Angelo. Johannes Von Oswald, Regina Traber, Benjamin Ehret, Seijin Kobayashi, Benjamin F Grewe, and João Sacramento. Posterior meta-replay for continual learning. Advances in Neural Information Processing Systems, 34:14135-14 14149,
2021.
Ching Yi Hung, Cheng-Har Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. Advances Neural Information Processing Systems, 32, 2019.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Av- eraging weights leads to wider optima and better generalization. Annual Conference on Uncertainty in Artificial Intelligence, 2018.
Leo P Kadanoff. Statistical physics: statics, dynamics and renormalization. World Scientific, 2000.
Ta-Chu Kao, Kristopher Jensen, Gido van de Ven, Alberto Bernacchia, and Guillaume Hennequin. Natural continual learning: success is a journey, not (just) destination. Advances in neural information processing systems, 34:28067-28079, 2021.
James Kirkpatriek, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Baand Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Richard Kurle, Botond Cseke, Alexej Klushyn, Patrick Van Der Smagt, and Stephan Günnemann. Continual learning with bayesian neural networks for non-stationary data. International Confer- on Learning Representations, 2019.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting, In International Conference on Machine Learning, pp. 3925-3934. PMLR, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
Shu Liu, Wuchen Li, Hongyuan Zha, and Haomin Zhou. Neural parametric fokker-planee equation. SIAM Journal on Numerical Analysis, 1385::135-1149, 2022.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Yi-An Ma, Tianqi Chen, and Emily Fox. A complete recipe for stochastic gradient meme, Advances in neural information processing systems, 28, 2015.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEFE conference on Computer Vision and Pattern Recognition, pp- 7765-7773, 2018.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
11

---

Published as a conference paper at ICLR 2024
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In International Conference on Learning Representations, 2018.
Kazuki Osawa, Siddharth Swaroop, Mohammad Emtiyaz E Khan, Anirudh Jain, Runa Eschenhagen, Richard E Turner, and Rio Yokota. Practical deep learning with bayesian principles. Advances in neural information processing systems, 32, 2019.
Pingbo Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard Turner, and Moham- mad Emtiyaz E Khan. Continual deep learning by functional regularisation of memorable past. Advances in Neural Information Processing Systems, 33:4453-4464, 2020.
Sam Patterson and Yee Whye Teh. Stochastic gradient riemannian langevin dynamies the probability simplex. Advances in neural information processing systems, 26, 2013.
Quang Pham, Chenghao Liu, and Steven HOL. Dualnet: Continual learning, fast and slow. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: ftctp:://openrvvieee httpsiiwwweeeeeeeee..eeeee.ee formm? ? eQ7Kh- QeWnO.
Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky, Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis. In Conference on Learning Theory, pp. 1674-1703, PMLR, 2017.
Blake A Richards and Paul W Frankland. The persistence and transience of memory. Neuron, 94(6):
1071-1084, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2019.
Tim GJ Rudner, Freddie Bickford Smith, Qixuan Feng, Yee Whye Teh, and Yarin Gal. Continual learning via sequential function-space variational inference. In International Conference on Machine Learning, pp. 18887. PMLR, 2022.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. ICLR, 2021.
Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki, Agnieszka Graaska-BBrrinskee Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress and compress: A scalable framework for continual learning. In Proceedings of the International Conference Machine Learning, 2018.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International conference machine learning, pp. 4548-4557, PMLR, 2018.
John Sweller. Cognitive load theory. In Psychology of learning and motivation, volume 55, pp. 37-76. Elsevier, 2011.
Michalis K. Titsias, Jonathan Schwarz, Alexander G. de G. Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with gaussian processes. In International Conference Learning Representations, 2020. URL https: http:::://ppeeevvieeee net / net/forum? forum? id=HkxCZeHFDER
Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pp. 4126-4142. PMLR, 2021.
Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4( 4(12): 185-1197, 2022.
Liyuan Wang, Mingtian Zhang, Zhongfan Jia, Qian Li, Chenglong Bao, Kaisheng Ma. Jun Zhu, and Yi Zhong. Afec: Active forgetting of negative transfer continual learning. Advances in Neural Information Processing Systems, 242299 2021.
12

---

Published as a conference paper at ICLR 2024
Zhenyi Wang, Li Shen, Le Fang, Qiuling Suo, Tichang Duan, and Mingehen Gao. Improving task-free continual learning by distributionally robust memory evolution. In International Conference on Machine Learning, pp. 22985-229999 PMLR, 2022a.
Zhenyi Wang, Li Shen, Le Fang, Qiuling Suo, Donglin Zhan, Tiehang Duan, and Mingchen Gao. Meta-learning with less forgetting on large-scale non-stationary task distributions. In European Conference on Computer Vision, pp. 221-238. Springer, 2022b.
Zhenyi Wang. Li Shen, Tiehang Duan, Qiuling Suo, Le Fang, Wei Liu, and Mingchen Gao. Dis- tributionally robust memory evolution with generalized divergence for continual learning. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2023a.
Zhenyi Wang, Enneng Yang, Li Shen, and Heng Huang. comprehensive survey of forgetting in deep learning beyond continual learning. arXiv preprint arXiv:2307.09211, 2023b.
Zifeng Wang. Zheng Zhan, Yifan Gong, Yucai Shao, Stratis loannidis, Yanzhi Wang, and Jennifer Dy. Dualhsie: Hsic-bottleneck and alignment for continual learning. In International Conference on Machine Learning, 2023c.
Andre Wibisono. Sampling as optimization the space of measures: The langevin dynamics as composite optimization problem. In Conference Learning Theory, pp. 2093-3027, PMLR. 2018.
Yinjun WU, Edgar Dobriban, and Susan Davidson. Deltagrad: Rapid retraining of machine learning models. In International Conference Machine Learning, pp. 10355-10366. PMLR, 2020.
Enneng Yang. Shen, Zhenyi Wang. Shiwei Liu, Guibing Guo, and Xingwei Wang. Data aug- mented alatness-aware gradient projection for continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5630-5639, 2023a.
Enneng Yang, Li Shen, Zhenyi Wang, Tongliang Liu, and Guibing Guo. An efficient dataset condensation plugin and its application continual learning. Advances in Neural Information Processing Systems, 36, 2023b.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 3987- 3995, 2017a.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995, PMLR, 2017b.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference Machine Learning, pp. 26982- 26992. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2024
Appendix
A RECAST EXISTING CL METHODS INTO OUR UNIFIED AND GENERAL
FRAMEWORK
e - v) + (hD + 3D (0, Oatd)
(25)
The following is the definition of Bregman divergence:
(p q) = p x (V@())) q)
(26)
A.1 CAST CPR INTO THE GENERAL FRAMEWORK
In Eq. (25), we take (p)) E log P; Here, p and q probability simplex, i... E P.
and Σ qi- 1. Then, we plug ((p) into Eq. (26). We can obtain the following equation:
imn
imn
Dz ②)
Σ
Pe log p:
qi log qi
(log(g) + 1,p - q)
(27)
i=1
i=n
im
i=n
Σ 1=1
Pe log P
ps log qi
Σ
P: Σ
qi
(28)
1=1
1=1
Σ 1=1
Pe log
P:
(29)
qi
(H(p) + (p.
(30)
<L(pplg)
(31)
where (p) is the entropy for the prohability distribution p. and H (p. q) is the cross entropy between probability distributions p and q.
When take the probability distribution p go(x)) i.e., the CL model output probability distribution over the classes, and v. i.e., the uniform distribution over the underlying classes, D& (p, q) (L))o u). This precisely recovers the CPR method.
A.2 CAST EWC INTO THE GENERAL FRAMEWORK
In Eq. (25), we a we take (z( O0000 FO. We set p O and q Ootd- where F the diagonal Fisher information matrix.
D& (p.q) {pp) ((g) (VØ(q).P q)
(32)
0 FO
Or
F'Old (0ttl 0 - Oota)
(33)
2
TT FO +
OT
FOald (O F,0 0)
(34)
2
(0 FF( Oaia)
(35)
Then, we recover the EWC method.
A.3 CAST DER INTO THE GENERAL FRAMEWORK
In Eq. (25), set 3 0 and take ppp) ||p
14

---

Published as a conference paper at ICLR 2024
Do = (p)) ( (V q)
(36)
= -|p| [ 2 (2q. p ạ)
(37)
pp + lg 2 2(p.9)
(38)
= (p q)²
(39)
Next, take p uo(x) and q Dz |110(()) =11?. method.
Then, recover the DER
A.4 CAST ER INTO THE GENERAL FRAMEWORK
In Eq. (25), we set 3 = 0, we take p = i.e., the one-hot representation of the ground-truth label and q = (o))), i.e. the CL model output probability distribution over the classes. Then, (p. q) is equivalent to the cross-entropy loss H (p. Q) according to Eq. (30). As result, we recover the ER method.
A.5 CAST VCL INTO THE GENERAL FRAMEWORK
In Eq. (25), we set a = 0, we take (p) = p(0) logr(O)d0. p(00)d = Then, the following Bregman divergence can expressed as:
(p, (p) (1))) - (V
(40)
/ p(0) loggo(d)) (()) log (0)) 00
-o (1gg))((0)(0(0 - {(0))d0
(41)
/ p(0) p(0) log (00) (0)d0 p(0) log d0)
(42)
/
p(0) log ((0)
d0
(43)
(
(L(((())l((())
(44)
Variational continual learning (VCL) Nguyen et al. (2018) is Bayesian-based method for mitigating forgetting in CL. The basic idea of VCL is to constrain the current model parameter distribution to be close to that of previous tasks. optimizes following objective.
COL CoE( v) (KL P(Oate(Paa)))))
(45)
where DI.r denotes the dataset from task task L. P(0)D1:) the posterior distribution of the model parameters on the entire task sequence D1.-- P(Botd D is the posterior distribution of the model parameters on the tasks D1a-1. In this case. P(:)P and P(Oold Pi---1) are both continuous distributions. In this case, in Eq. (2), we set = O. we take V to be (p)p) p(0) log (0))d0 We then set ((.)).:) and g = P(f (P.te1) We then recover the VCL method.
A.6 CAST NATURAL GRADIENT CL INTO THE GENERAL FRAMEWORK
In Eq. (25), we adopt the first-order Taylor expansion on the first loss the following:
(CE ~ (Cx + VoLCE 0x)
(46)
For the second loss term in Eq. (25), we take (p)) = Σi= p; log one-hot vector for the labeled data. Then, the second loss term is the cross entropy loss the
p:. z to be the ground truth
15

---

Published as a conference paper at ICLR 2024
previously learned tasks. We adopt the second-order Taylor expansion on the second loss term the following:
- 10 -
())h(());z) se D& (ho, (z), 2) + VoDa (z), 0z)
0x) (47)
where F is the Fisher information matrix (FIM) of the loss D& (he())) z) previously learned tasks. Since Vo (ho (x), x) is close to zero at the stationary point, i.c., Ox. we thus only need to optimize the quadratic term in Eq. 47.
For the third loss term in Eq. (25), we adopt the 11 Thus, the third loss term becomes De (0, 0x) = 1 0 1x112.
In summary, the approximate loss function for Eq. (25) can be expressed as the following:
VeLCE(0.) 0x) 2 F 02)
- 2 0x11
8
(48)
We then apply first-order gradient method on the Eq. (48), we can obtain the following:
VoLce(Ok) + a(O -x) 3(0 0x) =0
(49)
We can obtain the following.
(aF + BI)0 (aF + (1) ((aF Voocc
(50)
We can get the following natural gradient CL method.
0x+1 0₂ - +
(51)
when / 0. the above equation recover the standard natural gradient CL method without damping as the following:
0k+1 02 Va-
(52)
B THEOREM PROOF
Proof. Proof sketch: We outline our proof in the following. We denote the weighted gradient norm regularized CL loss function as (e) and our refresh learning loss function as Crefresh Then, calculate their gradient VeLaN (0) and VoCre (0), respectively. Finally, show their gradient is approximately the same, i.c. VoLGN x VoCrefresh (0), then the conclusion follows.
(1) Calculate the gradient VoLaN
We define the gradient norm regularized CL loss function
LaN (
LCL (0) + V (0)F-11
(53)
Then, we take the derivative with respect to 0 in Eq. (53), we got the following:
Vo
(54)
10 (0)
(55)
[0 (0FCL(())-11
VOLCL
(56)
VoLCH())
(0)F
(57)
VoLCL())
16

---

Published as a conference paper at ICLR 2024
Table 4: Task-IL and class-IL overall accuracy on CIFAR-100 and Tiny-ImageNet, respectively with memory size 2000. '_' indicates not applicable.
Algorithm
CIFAR-100
Tiny-ImageNet
Method
Class
Task
Class-II
Task II.
ER
36.00
09.09
0.71
58.19 0.69 59.42 0.51
ERerefresh
37.29 0.85
83.21 1.23
16.93 0.86
DER++
1.09
62
tre+st
52.81
0.80
84.05
0.77
27.37
1,53
64.31 1. 0.98
(2) Calculate the gradient VoLrefresh(O) the following:
Then, define the refresh learning loss function as
Prefresh = (0 s6)
(58)
where, we set F-1 Voceh (0)
+ (0,0 Then, we take the first-order Taylor expansion
on Crefresh Zhao al. (2022) as the following:
(0+ (0)
(59)
VoLOL(O)
70)00(0)6 V3LCL [ VoL + + 2, (
(60)
(3) Show that these two loss gradients are approximately the same, ie., VeLaN(0) Vo VoCrefresh
2
VoL (0 sō) VoL ( CLL (0)so
(61)
VOLCu(
VoL + CCL F-1
13
(62)
VOL + 10 N10
(63)
where the additional random Gaussian noise Eq. (63) helps the CL model escape local minima and saddle points to achieve global minima solution. Furthermore, Eq. (63) indicates that
VoLrefresh ~ Vol + ( = VoLoN(0)
(64)
In other words, the gradient of the refresh learning approximately the same as that of weighted gradient regularized CL loss function. The conclusion then follows.
C MORE EXPERIMENTAL RESULTS
C.1 RESULTS ON MNIST
Table Domain-IL overall accuracy on P-MNIST and R-MNIST, respectively with memory size
500.
Algorithm Method
P-MNIST
R-MNIST
Domain-IL.
Domain-IL.
oEWC
59.57 +: 2.37
77.35 1: 5.77
oEWC+refresh
61.23 +: 2.18
79.21 +: 4.98
ER
78.45 + 0.72
88.91 1.44
ER+refresh
80.28 +:
90.53
1.67
DER++
88.21 +: 0.39
92.77 + 1.05
88,93 +: 0.58
93.28 + 0.75
17

---

Published as a conference paper at ICLR 2024
C.2
RESULTS WITH MEMORY SIZE OF 2000
C.3
HYPERPARAMETER ANALYSIS
Table 5: Analysis of unlearning rate yy and number of unlearning steps on CIFARIO0 with task-IL.
0.02
0.03
0.04
Accuracy 77.23 1: 0.97
77.71 1 0.85
77.08 1 0.90
2
3
Accuracy 77.71 1 0.85
77.76 1 0.82
75.93 1 1.06
Table 6: Analysis of unlearning rate y and number of unlearning steps J on CIFAR1O with task-IL.
0.02
0.03
0.04
Accuracy 94.27 = 0.42
94.64 + 0.38
94.82 + 0.51
2
3
Accuracy 94.64 = 0.38
94.73 + 0.43
93.50 + 0.57
Table 7: Analysis of unlearning rate y and number of unlearning steps on task-IL.
with
0.02
0.04
Accuracy 53.27 0.72
54.06
0.79
54.21 0.83
Accuracy $4.06 0.79 54.17 0.91 52.29 0.86
C.4 COMPUTATION EFFICIENCY
Table 8: Computational efficiency of refresh learning on CIFARI00 with one epoch training
CIFARI00 running time (seconds)
DER++ 8.4
DER++refrrsh
15.2
C.5 BACKWARD TRANSFER
We evaluate Backward Transfer (BWT) in Table 9.
Table 9: Backward Transfer of various methods with memory size 500.
CIFAR10
CIFAR100
Tiny- ImageNet
Method
Class-IL
Task-IL
Class-IL
Task-IL
Class-IL
Task-IL
finetuning
96.39 0.12
46.24 +: 2.12
-89.68 + 0.96
62.46
0.78
78.94 + 0.81
-67.34 +: 0.79
AGEM
94.01 1.16
14.26 +: 1.18
88.5 1: 1.56
45.43
2.32
78.03
+ 0.78
59.28 1 1.08
GSS
62.88 2.67
-7.73
+ 3.99
-82.17 + 4.16
-33.98
1.54 13.60
HAL
62.21
-5.41 1.10
49.29 1:
45.35 0.07
-3.54 + 0.35
74.84 + 1.38
16.81
0.97
-75.24 + 0.76
-31.98 + 1.35
ER+refresh
40.89
0.86
+:
-73.78 1.59
0.87 -15.65
-74.49
0.80
-30.66
1.51
DER+
22.38 4.41
4.66 1 1.15
53.89 1.
14.72
64.6 1 0.56
27.21 1: 1.23
DER++ refresh
-22.03 3.89
-4.37 1.25
-53.51 0.70
-14.23 0.75
-63.90
0.61
-25.05
1.05
18


Reviews:
[{"review_id": "F0lwHgwJR8", "reviewer": "Reviewer_ngze", "summary": "In this paper, the authors provide a unified framework for various types of continual learning (CL) algorithms, Specifically, by using Bregman divergence, they show that well-known CL approaches can be viewed as instances of the general objective coming from different parametrization of the Bregman divergence function. Then, they propose a new CL algorithm, which consists of a two-step process: unlearn and relearn. The authors provide empirical results that show the merits of their approach as compared to other relevant CL algorithms.", "questions": "- Could the proposed refresh learning algorithm be derived by the unified CL framework?\n- Why energy functional is defined as in (13)? Could you please provide more intuition? Are there any other ways to promote unlearning?\n- What is the loss function in eq. (6) of Algorithm 1?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The unified framework provided by the authors is interesting and sheds light on particular characteristics of existing algorithms.\n- The main idea behind the proposed \"refresh learning\" algorithm seems to be reasonable.\n- The authors integrated their \"refresh learning\" approach to existing algorithms showing empirical results on three different datasets.", "weaknesses": "- The first part of the paper (unified framework) seems to be unrelated to the second one (Refresh learning algorithm). \n- The refresh learning objective eqs (12), (13) is insufficiently motivated.\n- It's not clear why unlearning should be enforced on the current batch. The authors do not provide any motivation. \n- The conversion of the constraint to a PDE is not obvious to me and is not sufficiently explained in the paper.\n- The refresh learning algorithm ends up in a preconditioned ascent followed by a decent step. But the descent step seems to be applied in a different loss (??) (eq. 6 in Algorithm 1). This is a bit confusing.\n- The authors integrate the proposed algorithm into various existing schemes. However, this is arbitrarily introduced in the experimental section only, and not clear how this integration can naturally arise from the problem introduced in section 3.3 or the unified CL framework of section 3.2.\n\n** Post-rebuttal comment: I appreciate the authors' efforts to address my concerns. The revised version of the paper has now been improved. The proposed refresh learning framework is better motivated and the derivations provided by the authors make the paper easy to follow. Therefore, I have decided to increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4OmNbmtjFY", "reviewer": "Reviewer_Dw6m", "summary": "This paper proposes a unified optimization objective which is capable of encompassing existing CL approaches, including regularization based/Bayesian-based and memory-replay based methods. From the objective, the authors identified a novel method of refresh-learning, which act as a plug-in to augment the performance of CL methods.", "questions": "1.Why can't the over-memorization issue be solved by properly tuning the regularization strengths alpha/beta?\n2.Can you provide an intuitive explanation why unlearning the posterior first and relearning a single set of parameters from the unlearned posterior is a good method for encouraging forgetting of outdated information?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a unified framework for different continual learning methods, and derived a new CL approach from the unified objective. The paper presents detailed theoretical derivations of the algorithm and comprehensive experimental results to demonstrate the advantage of the new method of refresh learning.", "weaknesses": "For me the more interesting and novel part of the paper is the refresh learning method, its derivation, intuition behind it, and its performance, while the part where the unified approach corresponds to different CL methods in different setup is more expected and easier to follow. I would recommend the authors shorten the part of how the unified objective corresponds to different special cases and further elaborate on refresh learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "epxZkFtL4F", "reviewer": "Reviewer_tVq3", "summary": "The paper considers a unified objective that includes prior methods as special cases.\n\nIt further proposes an unlearn-relearn method to minimize the proposed objective function.\n\nNext, there is a short section on theoretical analysis.\n\nFinally, experiments are conducted in comparison to prior methods.", "questions": "I have no specific questions.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "Omitted.", "weaknesses": "In my opinion, the paper has the following weaknesses:\n- The proposed framework is not very interesting. Basically, it says that prior work has loss functions $L_1,L_2,L_3$, therefore let us propose a unified objective $\\alpha_1L_1 + \\alpha_2 L_2 + \\alpha_3L_3$. In my eyes this proposal is incremental and has limited novelty.\n\n\n- The work is not solid and there are issues with writing. For example:\n   - The first part of the paper (pages 1-5, until section 3.3), appears to be a review of prior works. This is more than half of the main paper. \n   - There are many repetitive sentences. This is just one example (and there are more): There is a long paragraph on page 2 discussing refresh learning, and then a highly similar paragraph appeared on page 6.\n\nI was not very convinced by what the paper argues about over-memorization. But after witnessing the repetitive style of the paper I realized that over-memorization is indeed harmful.\n\nThe theoretical analysis is very informal. I don't see how the theory statement connects to the proofs. The proofs seem to be written in a rush. Please justify that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0eDg2D90uk", "reviewer": "Reviewer_KmRA", "summary": "The paper proposes a new objective function for task-based continual learning (TBCL). There are two key contributions: The first one consists in highlighting that all regularisers proposed in regularised-based, memory-replay and Bayesian approaches can be seen as specific instantiations of a Bregman divergence term, thus casting the whole formulation of TBCL into a single overarching objective criterion. The second contribution consists of proposing an additional regulariser to promote generalisation, which is based on the minimisation of the L2 norm of the weighted gradient loss (here, the weight corresponds to the inverse of the Fisher Information matrix computed using the parameters from the previous tasks). A corresponding learning algorithm is proposed based on an alternating minimisation scheme. Indeed, the training algorithm updates the parameters firstly by approximately minimising the gradient loss term (the approximation avoids (i) the computation of the Hessian matrix, as assuming to be an identity, and (ii) it introduces Gaussian noise to promote exploration) and secondly by minimising the cross-entropy on the current task. Experiments on CIFAR-10, CIFAR-100 and Tiny-Imagenet demonstrate the effectiveness of the new regulariser in enhancing the generalisation performance of existing approaches. Consequently, the proposed regulariser effectively complements the ones proposed in the literature of TBCL.", "questions": "All questions are related to the main weaknesses:\n1. Why not directly minimising the (weighted) gradient penalty? Also, what is the advantage of introducing the Gaussian noise (it would be good to see its necessity in practice)?\n2. Can you provide the complete ablation analysis on all datasets and also include some experiments about MNIST?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea behind the two contributions is original and novel. Specifically, it is nice to see a unified view of the three classes of TBCL leveraging the general definition of Bregman divergence. Additionally, while the proposed new regulariser is known to increase generalisation in supervised deep learning, as promoting solutions characterised by flat minima [1], its adaptation to the TBCL is novel and non-trivial (**Originality**)\n2. Overall, the paper is quite clear and easy to follow, except for one part (see weaknesses) (**Clarity**)\n3. The solution and the experiments are convincing, demonstrating its benefits (**Significance**)\n\n[1] Penalising Gradient Norm for Efficiently Improving Generalization in Deep Learning. ICML 2022", "weaknesses": "1. The main concern I have is with the overstated claim about the introduction of an innovative concept, “refresh learning”. The idea boils down to approximately minimising the weighted gradient norm, known to promote flatness in the achieved minima and therefore to increase generalisation performance. Section 3.3 is a bit handwavy and unclear. Perhaps, it is better to rephrase it directly in terms of gradient penalty as originally proposed in [1] and focus more on its extension to the TBCL setting. This would help to have a clearer understanding and intuition on why the proposed regulariser works (**Quality**).\n2. It would be good to provide the ablation study for all datasets. Indeed how are the hyperparameters chosen in practice and how should practitioners choose them in general? This should help to give a better sense on how sensitive the hyperparameters are on different datasets (**Quality**)\n3. All experiments are conducted on a similar family of natural images. It would be good to see how the proposed approach works on a traditional and more different MNIST-like benchmark (**Quality**)\n4. Code is not available (**Reproducibility**)\n\n**MINOR**\n\nSome typos:\n1. Section 3, remove “In this section”\n2. Last paragraph page 4 (and also later in page 5) -> the NEGATIVE entropy function\n3. Eq. (5) misses the expectation term in its second addend\n4. Eq. (8) and all occurrences of F -> missing L", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)