Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning
Authors: Mohamed Elsayed
A. Rupam Mahmood
Venue: ICLR 2024 poster (2024)
Keywords: catastrophic forgetting
loss of plasticity
plasticity
stability
continual learning
streaming learning
online learning
incremental learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Deep representation learning methods struggle with continual learning, suffering from both catastrophic forgetting of useful units and loss of plasticity, often due to rigid and unuseful units. While many methods address these two issues separately, only a few currently deal with both simultaneously. In this paper, we introduce Utility-based Perturbed Gradient Descent (UPGD) as a novel approach for the continual learning of representations. UPGD combines gradient updates with perturbations, where it applies smaller modifications to more useful units, protecting them from forgetting, and larger modifications to less useful units, rejuvenating their plasticity. We use a challenging streaming learning setup where continual learning problems have hundreds of non-stationarities and unknown task boundaries. We show that many existing methods suffer from at least one of the issues, predominantly manifested by their decreasing accuracy over tasks. On the other hand, UPGD continues to improve performance and surpasses or is competitive with all methods in all problems. Finally, in extended reinforcement learning experiments with PPO, we show that while Adam exhibits a performance drop after initial learning, UPGD avoids it by addressing both continual learning issues.

Published as a conference paper at ICLR 2024
ADDRESSING Loss OF PLASTICITY AND CATAS- TROPHIC FORGETTING IN CONTINUAL LEARNING
Mohamed Elsayed
A. Rupam Mahmood Department of Computing Science University of Alberta CIFAR Canada Al Chair, Amii ca
Department of Computing Science
University of Alberta
Alberta Machine Intelligence Institute
mohamedel sayedualberta...aa.....
ABSTRACT
Deep representation learning methods struggle with continual learning, suffering from both catastrophic forgetting of useful units and loss of plasticity, often due rigid and unuseful units. While many methods address these two issues separately, only few currently deal with both simultaneously. In this paper, we introduce Utility-based Perturbed Gradient Descent (UPGD) as a novel approach for the continual learning representations. UPGD combines gradient updates with per- turbations, where it applies smaller modifications to more useful units, protecting them from forgetting, and larger modifications less useful units, rejuvenating their plasticity. We use challenging streaming learning setup where continual learning problems have hundreds of non-stationarities and unknown task bound- aries. We show that many existing methods suffer from at least one of the issues, predominantly manifested by their decreasing accuracy over tasks. On the other hand, UPGD continues to improve performance and surpasses is competitive with all methods in all problems. Finally, in extended reinforcement learning ex- periments with PPO, show that while Adam exhibits performance drop after initial learning, UPGD avoids it by addressing both continual learning issues."
CHALLENGES OF CONTINUAL LEARNING
Continual learning remains significant hurdle for artificial intelligence, despite advancements in natural language processing, games, and computer vision. Catastrophic forgetting (McCloskey & Cohen 1989, Hetherington & Seidenberg 1989) in neural networks widely recognized as major challenge of continual learning (De Lange et al. 2021). The phenomenon manifests as the failure of gradient-based methods like SGD or Adam to retain or leverage past knowledge due to forgetting or overwriting previously learned units (Kirkpatrick et al. 2017). In continual learning, these learners often relearn recurring tasks, offering little gain over learning from scratch (Kemker et al. 2018). This issue also raises concern for reusing large practical models, where finetuning them for new tasks causes significant forgetting of pretrained models (Chen et al. 2020, He et al. 2021).
Methods for mitigating catastrophic forgetting are primarily designed for specific settings. These include settings with independently and identically distributed (i.id.) samples, tasks fully contained within batch or dataset, growing memory requirements, known task boundaries, storing past sam- ples, and offline evaluation. Such setups are often impractical in situations where continual learning paramount, such as on-device learning. For example, retaining samples may not be possible due to the limitation of computational resources (Hayes et al. 2019, Hayes et al. 2020, Hayes & Kannan 2022, Wang et al. 2023) concerns over data privacy (Van de Ven al. 2020).
In the challenging and practical setting of streaming learning, catastrophic forgetting is more severe and remains largely unaddressed (Hayes al. 2019). In streaming learning, samples are presented to the learner as they arise. which is non-i.i.d. in most practical problems. The learner cannot retain the sample and is thus expected to learn from immediately. Moreover, evaluation happens online on the most recently presented sample. This setup mirrors animal learning (Hayes et al. 2021, also c.f., list-learning, Ratcliff 1990) and practical for many applications, such as robotics or autonomous on-device learning. In this work, we consider streaming learning with unknown task boundaries.
"Code available https: / thub. wtt.ccomm/oo.aacccc sayed/upea

---

Published as a conference paper at ICLR 2024
Label-pemuted EMNIST
Average Online Accuracy
input-permuted MNIST
0.95
permtte MNIST
Online Office
Average Accuracy
0.6
Accuracy
0.8
0.85
0.4
0.3
9.70
Task
Task
Task 240
Task
Task Number
Task Number
10
11
241
(a) Catastrophic Forgetting
(b) Loss of Plasticity
(c) Closer look at Loss of Plasticity
UPGD
Adam
Adam-Restarts
Figure I: (a) Adam suffers from catastrophic forgetting and hence hardly improves performance. (b & c) Adam loses plasticity as newer and newer tasks are presented and performs much worse than Adam with restarts later. In contrast, our proposed method, UPGD, quickly learns and maintains plasticity throughout learning. See Appendix I.1 for experimental details.
Streaming learning provides the learner with a stream of samples (#t,, yr) generated using a non- stationary target function ft such that y/: = f(x)). The learner observes the input æ, E outputs the prediction Yt: E 1RT and then observes the true output 3/: E R strictly in this order. The learner is then evaluated immediately based on the online metric E(y,, ).) for example, accuracy in classification squared error in regression. The learner uses neural network for prediction and E or related loss to learn the network parameters immediately without storing the sample. The target function is locally stationary, where changes to Jr occur occasionally, creating a nonstationary continual learning problem composed of a sequence of stationary tasks.
Fig. I(a) illustrates catastrophic forgetting with Adam in the streaming learning setting. Here, sequence of tasks based on signed to be highly coherent, where the features learned in one task are fully reusable in the other. Full details of the problem are described in Section 4.4. If the learner can remember and leverage prior learning, it should continue to improve performance as more tasks are presented. However, Fig. I(a) reveals that Adam can hardly improve its performance, which remains at a low level of ac- curacy, indicating forgetting and relearning. Although catastrophic forgetting is commonly studied under offline evaluation (solid lines), the issue also manifests in online evaluation (dashed lines). This result indicates that current representation learning methods are unable to leverage previously learned useful features but instead forget and relearn them in subsequent tasks.
EMNIST is presented to the learner. The tasks are de-
Yet another pernicious challenge of continual learning is loss of plasticity, where the learner's ability to learn new things diminishes. Recent studies reveal that SGD or Adam continues to lose plastic- ity with more tasks, primarily due to features becoming difficult to modify (Dohare et al. 2021, Lyle al. 2023). Several methods exist to maintain plasticity, but they generally do not address catastrophic forgetting. Figures 1(b) and I(c) illustrate loss of plasticity, where Adam is presented with a sequence of new tasks based on Siput-Permutce performance degrades with more tasks and becomes worse than Meaamm-esttrts, which learns from scratch on each task. The stable performance of AdstaRestarts indicates that the tasks are of similar difficulty. Yet, Adam becomes slower to learn over time, demonstrating loss of plasticity.
MNIST (Goodfellow al. 2013). Adam's
method that preserves useful units, such as features or weights, while leaving the other units adaptable would potentially address both catastrophic forgetting and loss of plasticity. Although a few methods address both issues simultaneously, such methods expect known task boundaries, maintain replay buffer, or require pretraining, which does not fit streaming learning (Hayes et al. 2022). In this paper, we intend to fill this gap and present a continual learning method that addresses both catastrophic forgetting and loss of plasticity in streaming learning without such limitations.
2 RELATED WORKS
Addressing Catastrophic Forgetting. Different approaches have been proposed to mitigate catas- trophic forgetting. For example, replay-based methods (e.g., Chaudhry et al. 2019, Isele & Cosgun 2018, Rolnick et al. 2019) address forgetting by using replay buffer to store incoming non-i.i.d. data and then sample from the buffer i..d. samples. Catastrophic forgetting is also addressed by parameter isolation methods (e.g., Rusu et al. 2016, Schwarz et al. 2018, Wortsman al. 2020, Ge et al. 2023) that can expand to accommodate new information without significantly affecting previ- ously learned knowledge. There also sparsity methods (e.g., Liu et al. 2019) that work
2

---

Published as a conference paper at ICLR 2024
by maintaining sparse connections that the weight updates can be localized and not affect many prior useful weights. Finally, regularization-base methods use a penalty to discourage the learner from moving too far from previously learned weights (Kirkpatrick et al. 2017, Aljundi et al. 2018, Aljundi et al. 2019) or approximations (Ian et al. 2023). The penalty amount is usually a function of the weight importance based on its contribution to previous tasks.
Addressing Loss of Plasticity. Dohare et al. (2023a) introduced a generate-and-tes method (Ap- pendix M, Mahmood & Sutton 2013) that maintains plasticity by continually replacing less useful features and showed that methods with continual injection of noise (e.g., Ash & Adams 2020) also maintain plasticity. Later, several methods were presented to retain plasticity. For example, Nik- ishin et al. (2023) proposed dynamically expanding the network, Abbas et al. (2023) recommended adapted activation functions, and Kumar et al. (2023) proposed regularization toward initial weights.
Addressing Both Issues. The trade-off between plasticity and forgetting has been outlined early by Carpenter & Grossberg (1987) as the stability- plasstitty dilemma, trade-off between maintaining performance on previous experiences and adapting newer ones. The continual learning commu- nity focused more on improving the stability aspect by overcoming forgetting. Recently, however, there has been new trend of methods that address both issues simultaneouslly Chaudhry et al. (RWalk, 2018) utilized a regularization based approach with a fast-moving average that quickly adapts to the changes in the weight importance, emphasizing the present and the past equally. Jung et al. (2022) introduced different techniques, including grucctre-wise distillation loss and pretrain- ing to balance between plasticity and forgetting. Gurbuz al. (2022) proposed using connection rewiring to induce plasticity in sparse neural networks. Finally, Kim et al. (2023) proposed using separate network for learning the new task, which then is consolidated into a second network for previous tasks. Despite the recent advancement in addressing the two issues of continual learning, most existing methods do not fit the streaming learning setting since they require knowledge of task boundaries, replay buffers, or pretraining.
Importance Measure in Neural Network Pruning. Pruning neural networks requires an impor- tance metric to determine which weights to remove. Typically, the network is pruned using different measures such as the weight magnitude (e.g., Han et al. 2015, Park al. 2020), first-order informa- tion (e.g., Mozer & Smolensky 1988, Hassibi & Stork 1992. Molchanov et al. 2016), second-order LeCun information LeCun et al. 1989, Dong et al. 2017), or both (e.g.. Tresp et al. 1996, Molchanov
et al. 2019). Similar to pruning, regularization-ber based methods that address catastrophic forgetting use weight-importanee measures such as the Fisher information diagonals to weigh their penalties.
3 METHOD
Our approach is to retain useful units while modifying the rest, which requires a metric to assess the utility usefulness of the units. In this section, we introduce measure for weight utility and outline an efficient method for computing it. Weight utility can be defined as the change in loss when setting the weight to zero, essentially removing its connection (Mozer & Smolensky 1988, Karnin 1990). Removing an important weight should result in increased loss. Ideally, both immediate and future losses matter, but we can only assess immediate loss at the current step. Finally, we devise a gradient-based update rule that protects or modifies weights based on this utility measure.
To define utility precisely, let us consider that the learner produces the predicted output ÿ using a neural network with L layers, parametrized by the set of weights W = (WI..., W.). Here Wi the weight matrix at the 1-th layer, and its element at the i-th row and the j-th column is denoted by Wi.... At each layer 1, we get the activation output h, of the features by applying the activation function o to the activation input at: by = (a,). We simplify notations by defining hg = æ. The activation output h; is then multiplied by the weight matrix W1+1 of layer I + 1 to produce the next activation input: a:-1, = Σi= hi. Vi, where hy e RK4, Here, a applies activation element-wise for all layers except for the final layer, which becomes the softmax function.
The utility (Z) of the weight i at layer I and sample Z is defined as
(Z) = L(W .1.5): 2) - L(W,
where L(W, Z) is the sample loss given W, and C(W Z) is a counterfactual loss where W- is the same as W except the weight Way is set to O. We refer to it as the true utility
3

---

Published as a conference paper at ICLR 2024
to distinguish it from its approximations, which referred to either approximated utilities or simply utilities. Note that this utility is a global measure, and it provides a total ordering for weights according to their importance. However, computing it is prohibitive since it requires additional N... forward passes, where N₂ is the total number of weights.
3.1 SCALABLE APPROXIMATION OF THE TRUE UTILITY
Since the computation of the true utility is prohibitive, we aim approximate it such that no addi- tional forward passes are needed. To that end, we estimate the true utility by a second-order Taylor approximation. We expand the counterfactual loss L(W- Z) around the current weight W L,, and evaluate weight zero. Hence, the quadratic approximation of (Z) can be written
(Z) = L(W ([ii)) 2) L(W.
Z) + ac(W, 2) (0- W1...)
1 0CC
(0 300-WWas)-CCO L(W,2)
+ 2 Baw
OWi,i.
2 Wi. + 1WW, Z)
W
OW,i.
OW?,5
We refer to the utility measure containing the first term the first-order utility and the measure con- taining both terms as the second-order utility. The computation required for the second-order term has quadratic complexity. Therefore, use the approximation by Elsayed and Mahmood (2022)
that provides Hessian diagonal approximation in linear complexity. This makes the computation of both of our utility approximations linear in complexity and therefore scalable. Moreover, we present a way for propagating our approximated utilities by the utility propagation theorem in Appendix B. We also define the utility of feature and provide its scalable approximation in Appendix C and D.
3.2 UTILITY- BASED PERTURBED GRADIENT DESCENT (UPGD)
Now, we devise a new method called Utility-based Perturbed Gradient Descen! (UPGD) that per- forms gradient-basee learning guided by utility-based information. The utility information used as a gate, referred to as utility gating, for the gradients to prevent large updates to already useful weights, addressing forgetting. On the other hand, the utility information helps maintain plasticity by perturbing unuseful weights which become difficult to change through gradients (see Dohare et al. 2023a). The update rule of UPGD is given by
Oc Wii. a' Owi,,.
+5) -=)(1- U(,s), (1-U,,))
WL..
(3)
where & ~ (O0 1) is noise, ar is the step-size parameter, and Üt, E [0, is a scaled utility. For important weights with utility U1,. = 1. the weight remains unaltered even by gradient descent, whereas unimportant weights with Üt,a,j
get updated by both perturbation and gradient descent.
Another variation of UPGD, which we call non-protecting UPGD, is to add the utility-based pertur- bation to the gradient Wh,i <- Wh,i. af88/own..s + E(1 U1,,))- However, such update rule can only help against loss of plasticity, not catastrophic forgetting, as useful weights are not protected from change by gradients. We include non-protecting UPGD in our experiments vali- date that using the utility information as a gate for both the perturbation and the gradient update is necessary to mitigate catastrophic forgetting. We provide convergence analysis for both UPGD and Non-protecting UPGD on non-convex stationary problems in Appendix A.
Utility scaling is important for the UPGD update rule. We present here global scaling and present local scaling variation in Appendix E. The global scaled utility requires the maximum utility of all weights (e-g., instantaneous or trace) at every time step, which given by Ür.,, = ((Ura//1) Here 77 is the maximum utility of the weights and ó is the scaling function, for which show the pseudo-code our method using the global scaled utility in Algorithm I, where F; con- tains first derivatives and S contains decond-derivative approximations (see Appendix F for more details on the GetDer i vat ves function). We focus here on weight-wise UPGD and provide similar pseudo-codes for feature-wise UPGD in Appendix E.
sigmoid. We
UPGD update rule can be related to some existing update rules. When we perturb all weights evenly, that is, when all scaled utilities zero, UPGD reduces to well-known class of algorithms
4

---

Published as a conference paper at ICLR 2024
called Perturbed Gradient Descent (PGD) (Zhou et al. 2019). The PGD learning rule is given by Wi,i.j < W1,i.j a [oC/ow..,, + E. It has been shown that a PGD with weight decay algorithm, known Shrink and Perturb (S&P) (Ash & Adams 2020), can help maintain plasticity in continual classification problems (Dohare et al. 2023a) since maintaining small weights prevents weights from over-committing, making them easy to change. The learning rule of Shrink and Perturb can be written as wt,i.j < pu], a (0Lfawr.., + E), where P 1 -- da and À is the weight decay factor. When no noise added, the update reduces to SGD with weight decay (Loshchilov & Hutter 2019), known as SGDW. Incorporating the useful role of weight decay into UPGD, we can write the UPGD
with weight decay (UPGD-W) update rule as Wt,s.j + punt - a [hoe, + 6) (1 U1,32).
3.3 FORGETTING AND PLASTICITY EVALUATION METRICS
Here, we present two metrics to characterize plastic- ity and forgetting in streaming learning. First, we introduce a new online metric to quantify plasticity. Neuroplasticity is usually defined the ability of biological neural networks to change in response to some stimulus (Konorski 1948, Hebb 1949). Simi- larly, in artificial neural networks, plasticity can be viewed as the ability of neural network change its predictions in response to new information (Lyle et al. 2023). We provide definition that captures the existing intuition in the literature. We define the plasticity of learner given a sample as the ability to change its prediction to match the target. The learner achieves plasticity of 1 given a sample if it can ex- actly match the target and achieves plasticity of 0 if it achieves zero or negative progress toward the target compared to its previous prediction given the same sample. Formally, we define the sample plasticity L(w 22) to be p(Z) max (1- mmx max(C(W.Z),e) Z)e)+ E [0. where Wi is the set of weights after performing the update and is a small number to maintain numeri-
Algorithm 1 UPGD
Given a stream of data D, network f with weights { (W,.... Wr)- Initialize step size a, utility decay rate B, and noise standard deviation Ø.
Initialize (WI..., Wr.).
Initialize U2 v/ and time step / to zero
for (a. y) in D do
f t +
for 7 in {I, L - 1,....11 } do
"1 x -00
Fr, S) +GGtDerrvaaiieesfffe y.
Mi < 1/2.t o W Fi Wi
U. - +U,+
U, + Ui/(1-91
if max((i) then r max(Ui)
for / in (L, L.,... 1} do
I,..,
Sample& & eleemetss from NN 0²)
U. + 6 (Úi/n)
Wi + Wi - a [ + -0)) Ui)
E (-00,1] has an upper bound of 1, since C(W, Z) E [0, oo). VW, Z for cross-entropy and squared-error losses. We use this metric to mea- sure plasticity directly, especially since most measures that are introduced (e.g.. weight norm) to show loss of plasticity do not often correlate with plasticity (see Lyle et al. 2023). Our metric can be viewed as Axseline-normalized version of the plasticity metric by Lyle al. (2023), where the baseline is the loss prior to the update. We measure loss of plasticity as Apk+1 F - where p& is the average plasticity over all samples in the k-th evaluation window. Note that this metric ranges from -1 1 to 1, APr E 1,, Vk. Negative values indicate the learner has gained plasticity, whereas positive values indicate the learner has lost plasticity. In our experiments, we report the overall loss of plasticity on T evaluation windows: ΣKE Apr+1 = P1 pr-
cal stability. Note that the term (1- max( C(W,Z).c)
LON", (bw".))
Existing metrics for catastrophic forgetting predominantly based on offline evaluations. In streaming learning, forgetting previously learned useful features leads to future learning deterio- ration rather than affecting past performance. If the learner keeps improving its representations with each task, the average online accuracy on new tasks should improve continually. On the other hand, if it cannot improve representations, its average online accuracy on new tasks may stay the same or even decrease (see Fig. ((a)). Hence, we propose measuring forgetting using different online evalua- tion windows for streaming learning. Inspired by the windowed worretttn metric introduced by De Lange et al. (2023), we propose the metric Fk+1 = A - As+1. where A₁ is the accuracy averaged over all samples on the k-th evaluation window. This metric assumes the learned representations in one task remain relevant for future tasks, and all tasks have the same complexity. Note that this metric ranges from -1 to 1, F& E [-1, 1]. Vk, where negative values indicate the learner can im- prove on previous representations, and positive values indicate re-learning, hence forgetting. In our experiments, we report the overall forgetting on T eevallation windows, Σt-1 FEt1 A Ar-
5

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
In this section, we begin by studying the quality of our approximated utilities. Then we study the effectiveness of UPGD in mitigating loss plasticity and catastrophic forgetting. For the latter, we non-stationary streaming problems based on MNIST (LeCun et al. 1998), EMNIST (Cohen et al. 2017), CIFAR-10 (Krizhevsky 2009), and ImageNet (Deng et al. 2009) datasets with learners that multi-layer perceptrons, convolutional neural networks (LeCCn et al. 1998), and residual neural networks (He et al. 2016) (however, see Appendix G for validating UPGD on stationary tasks). We also validate UPGD in extended reinforcement learning experiments. UPGD is compared to suitable baselines for our streaming learning setup, that is, without replay, batches, or task boundaries.
The performance of continual learners is evaluated based the average online accuracy for classi- fication problems. In each of the following experiments, a thorough hyperparameter search is con- ducted (see Appendix I). Our criterion was to find the best set of hyperparameters for each method that maximizes the area under the online accuracy curve. Unless stated otherwise, we averaged the performance of each method over 20 independent runs. We focus on the key results here and give the full experimental details in Appendix I.
4.1 QUALITY OF THE APPROXIMATED UTILITIES
A high-quality approximation of utility should give a sim- ilar ordering of weights to the true utility. We use the ordi- nal correlation measure of Spearman to quantify the qual- ity of our utility approximations. An SGD learner with
0.4
Spearman's 'ss Cocfficient
0.3
small neural network with ReLU activations is used on a simple problem to minimize the online squared error.
0:2-
0.1-
At each time step, Spearman's correlation is calculated
for first- and second-order global utility against the random ordering, the squared-gradient utility, and the wegnht-me utility. We report the correlations be- tween the true utility and approximated global weight utilities in Fig. 2. The correlation the highest for the second-order utility throughout learning. On the other hand, the first-order utility becomes less correlated when the learner plateaus, likely due to zigzagging gradient el- ements near the solution. The that gets smaller. The correlation of the squared-gras -gradeet utility increases with time steps but re- mains smaller than that the first-order utility. We use random ordering as a baseline, which maintains zero correlation with the true utility, as expected. We also show the correlation between approximated local utility and true utility in Appendix in addition to results with other activations.
0.0
500
1000
1500
Number of Samples
Loss (Reference)
First-order
Second-order
Weight-magnitude
Random
Squared Gradient
Figure 2: Rank correlation between the
true utility and approximated utility.
utility shows small correlation to the true utility
4.2 UPGD AGAINST LOss OF PLASTICITY
In this section, we use Haput-Permutedd MNIST, a problem where only loss of plasticity is present, and answer two main questions: 1) how UPGD and the other continual learning methods perform on this problem, and 2) whether performance alone is indicative of plasticity in this task. Performance has been used to measure plasticity on various problems and settings (Dohare et al. 2021, Nikishin et al. 2023, Kumar et al. 2023, Abbas et al. 2023). The decaying performance is usually attributed to loss of plasticity. Here, we question using performance to measure plasticity in arbitrary problems since performance can also be affected by other issues such as catastrophic forgetting (see Fedus et al. 2020). However, we hypothesize that if the only underlying issue is loss of plasticity, performance might actually reflect plasticity. Therefore, it is necessary to plasticity is present to test our hypothesis. This approach also allows us to study the effectiveness of UPGD against loss of plasticity, isolated from catastrophic forgetting. In Input-permuted MNIST, we permute the inputs every 5000 steps where the time step at each permutation marks the beginning of a new task. After each permutation, the learned features become irrelevant to the new task, so the learner is expected to overwrite prior-learned representations as soon as possible. Thus, the input-permuted MNIST is a suitable problem to study loss of plasticity.
problem where only loss of

---

Published as a conference paper at ICLR 2024
We compare SGDW, PGD, S&P, which addresses loss of plasticity, Adam with weight decay (Loshchilov & Hutter 2019) known as AdamW, UPGD-W, and Non-protecting UPGD-W. We also introduce and com- pare against Streaming Elastic Weight Consolidation (S-EWC), Streaming Synaptic Intelligence (S-SI), and Streaming Memory-Aware Intelligence (S-MAS). These methods can be viewed as natural extension of EWC (Kirkpatrick et al. 2017). SI (Zenke et al. 2017), and MAS (Aljundi et al. 2018), respectively, which are regularization-bant based methods for mitigating forgetting to the streaming learning setting. Finally, we introduce and compare against Streaming RWalk (S- (S-RWalk). This can be seen natural extension of RWalk (Chaudhry et al. 2018), a method that addresses both issues, adapted for streaming learning. We write the update rule of the last four methods in streaming learning as wi,c. + Wi,ii where x is the regularization factor, Qa,a. is the weight importance, and Wi, the trace of weight i at the [-th layer. The weight importance is estimated as trace of squared gradients in S-EWC, whereas it is estimated
Average Online Accuracy
0.78
0.76
0.78
0.72
0.70
50
100
150
200
Task Number
UPGD-W
SGDW
PGD
AdamW
Non-protecting UPGD-W
Shrink and Perturb
S-EWC
S-MAS
S-RWallk
Figure 3: Performance of methods on the Input-permuted MNIST problem.
[0C/own... + KSRt,.. (wr.i.j Wr,s.) )
a trace of gradient magnitudes in S-MAS. S-RWalk is different from RWalk since it uses trace of previous weights instead of the instantaneous previous weights. We omit the details of S-SI and RWalk weight importance estimation here and defer Appendix I.4. Lastly, since we compare against methods that use first-order information, we use global first-order utility traces in this and subsequent experiments for a fair comparison (however, see Appendix H for an experiment using second-order utilities).
Fig. 3 shows that methods that only address catastrophic forgetting (c.g., S-EWC) continue to decay in perfor- mance whereas methods that address loss of plasticity alone (e.g., S&P) together with catastrophic forget- ting (e.g., UPGD), except S-RWalk, maintain their perfor- mance level. We plot the average online accuracy against the number of tasks. The average online accuracy is the percentage of correct predictions within each task, where the sample online accuracy is 1 for correct prediction and o otherwise. The prediction of the learner is given by argmax over its output probabilities. The learners are pre- sented with total of million examples, one example per time step, and are required to maximize online accuracy using a multi-layer (300 x 150) network with ReLU units.
Average Online Plasticity
0.50
0.45
0.40
0.35
0.90
0.675
0.700
0.725
0.750
0.775
Average Online Accuracy
UPGD-W
SGDW
PGD
AdamW
Non-protecting UPGD-W
Shrink and Perturb
S-EWC
S-MAS
S-51
S-RWalk
Figure 4: Each method's average plas- ticity against average accuracy of on Input-permuted MNIST.
In order to answer the second question, we measure on- line plasticity as defined Section 3.3 for each method and plot performance against measured plasticity in Fig. 4. The average online plasticity of each method is shown against the average online accuracies of the last 100 tasks. We notice that plasticity and accuracy strongly correlated, indicating that when the learner loses plasticity, its accuracy also decreases. This result corroborates that when the only underlying issue is loss of plasticity, performance indeed reflects plasticity. Note that such correlation not expected when loss of plasticity is not the only issue, and hence, the plasticity metric can be generally more reliable.
Finally, in Fig. 5, we use diagnostic statistics to further analyze the solutions each method achieves. More statistics for learners on this problem and next ones are reported in Appendix K.1. Notably, the results show that, for all methods, the fraction of zero activations increases and & gradient norm decreases substantially except for UPGD-W W along with Non Non-protecting UPGD-W and S&P.
4.3 UPGD AGAINST CATASTROPIIIC FORGETTING
Now, study how UPGD and other continual learning methods address forgetting, and for that we use Label permutde CIFAR-1O. CIFAR-10 dataset contains 60,000 RGB images of size 32 x 32 belonging 10 classes; each class has 6000 images. The labels are permuted every 2500 time step.
7

---

Published as a conference paper at ICLR 2024
of Zeo Ratopptonn
Narm Gradeert
of Gedkeet
Norm Wosto
1.001
1200
-001
100
Task Number
Tase Number
Tank Number
Task Number
UPGD-W
SGDW
PGD
AdamW
Non-protecting UPGD-W
Shrink
Perturb
S-EWC
S-MAS
S-RWalk
Diagnostic statistics Input-permuted MNIST. The percentage of zero activations, tu- norm and l fn-norm of the gradients, and & norm of the weights are shown. We stacked the elements from the network gradients weights into vectors to compute each norm at every sample.
Figure 5:
Each learner is trained for 1M samples, one sample each time step, using a convolutional neural network with ReLU activations. Such permutations should not make the learner change its learned representations since it can simply change the weights of the last layer to adapt to that change. This makes the Label-permuted CIFARIO problem suitable for studying catastrophic forgetting. It is not clear, however, whether the issue of loss of plasticity is present. We hypothesize that the issue of loss plasticity does not occur in this problem mainly due to its small number of classes (Lesort et al. 2023), leading to a probability of 10% for the same label re-occurrence after label permutation, resulting in less amount of non-stationarity that causes loss of plasticity.
Average Online Accuracy
Average Online Accuracy
Average Online Accuracy
0.4-
0.3
014-
0.3-
0.3
5.22-
0.2
0.1-
100
200
300
120
300
330
400
0.0
100
230
Task Number
Task Number
Task Number
(a) pebel-permuted CIFAR-10
(b) Label-permuted EMNIST
(c) Label-permuted minilmageNet
UPGD-W
SGDW
PGD
AdamW S-MAS
Non Non-protecting UPGD-W
Shrink and Perturb
S-EWC
S-RWalk
Figure 6: Performance of methods on Label-permuted CIFAR-1O, Label-permuted EMNIST, and Label-permuted mini-lmageNer. The higher the online accuracy, the better.
Fig. 6(a) shows that methods addressing catastrophic forgetting continually improve their perfor- mance. We
loss of plasticity metric to check whether learners experience any loss of plas- ticity. Fig. 7(b) shows that the majority of methods have negative values, reflecting no loss of plas- ticity, which also indicates that catastrophic forgetting is the major issue in this problem. Although all learners can improve their performance, some can improve more than others, according to their forgetting metric (see Fig. 7(a)). We observe that learners without an explicit mechanism for ad- dressing forgetting (c.g., Adam W) can reach a maximum accuracy of slightly above 40%, compared to learners addressing catastrophic forgetting that keep improving their performance.
4.4 UPGD AGAINST LOSS OF PLASTICITY AND CATASTROPHIC FORGETTING
In this section, we study the interplay of catastrophic forgetting and loss of plasticity using the Label-permuted EMNIST problem. The EMNIST dataset an extended form of MNIST that has 47 classes, of both digits and letters, instead of just 10 digits. We permute the inputs every 2500 time steps and present the learners with total million examples, one example per time step, using the same network architecture from the first problem. Hence, this problem is also suitable for studying catastrophic forgetting. Since EMNIST has more classes, label re-occurrence probability becomes significantly smaller, leading to more non-stationarity. Thus, expect that loss of plasticity might be present this problem. Fig. 7(b) shows that most learners indeed suffer from loss of plasticity.
Fig. 6(b) shows that methods addressing catastrophic forgetting, including UPGD-W but except S-RWalk and S-SI, keep improving their performance and outperform methods that only address loss of plasticity. Notably, we observe that S-RWalk, which addressed catastrophic forgetting in the previous problem, struggles in this problem, likely due to the additional loss of plasticity. On the other hand, the performance of S-SI and the rest of the methods keeps deteriorating over time.
8

---

Published as a conference paper at ICLR 2024
CIARRS 42.35---22:3300
-2.22222222 12 22.05-33
-1.52
41.25
16.37
41.33
GANRIE
+0.7
2:04
-130
40.7
L/ EMNIT
1 EXWSST
2.28
0.30
13
12:00
1.20
0.50
10
1.54
h/ mine Inogeher
11.00
19.11
12:19
1.78
1.4
12
$2.90
16
L/P misiiaggeee Ret
6:46
6.2
0.52
UNGD
12.
S.ewe
5.5
S-RWikk
5.9
(a) Forgetting (x100)
(b) Loss of Plasticity (x (x100)
Figure 7: Forgetting and loss of plasticity of each method in different problems. I/P is short for inputtpermuted, and L/P is short for label-permuted. (a) forgetting is reported using our overall forgetting metric, where positive values indicate forgetting. The metric is computed starting from the sample number using windows of twice the task length. (b) loss of plasticity is measured based on the overall loss in plasticity, meaning that positive values indicate loss of plasticity. The metric is computed starting from the sample number 0.5 million using windows of twice the task length.
Next, we perform a large-scale experiment using the Label-permuted mini-ImageNer problem, which has large number of classes; hence, loss of plasticity is expected along with catastrophic forgetting. The mini-ImageNee (Vinyals et al. 2016) is subset of the ImageNet dataset. The mini-Imagenn ImageNet dataset contains 60,000 RGB images of size 84 84 belonging 100 classes; each class has 600 images. In Label-permuted mini- Imti-lm the labels are permuted every 2500 time step. Each learner uses a fully connected network of two layers on top of pre-trained ResNer-50 (He et al. 2016) on ImageNet with fixed weights.
Fig. 6(c) exhibits the same trends manifested the previous problem, where methods address- ing catastrophic forgetting (e.g., S-EWC) performed the best, whereas methods addressing loss of plasticity (e.g.. S&P) only maintained their performance at a lower level. Fig. 7(a) and Fig. 7(b) demonstrate that both metrics of UPGD-W W are among the best overall values across all problems, indicating diminished forgetting and loss of plasticity. We refer the reader to Appendix for an ablation study on the components of UPGD.
4.5 UPGD AGAINST POLICY COLLAPSE
Lastly, we study the role of UPGD in preventing a gradual performance drop or policy collapse (Dohare al. 2023b) in reinforcement learning (RL). Dohare al. (2023b) showed that non- stationarities in RL might exhibit both continual learning issues and demonstrated that the PPO algorithm (Schulman et al. 2017) can experience policy collapse when trained for an extended pe- riod. Since UPGD helps against catastrophic forgetting and loss of plasticity, we test whether UPGD can address policy collapse in PPO. We devise an adaptive variant of our method based on Adam normalization (Kingma & Ba 2015), which we call Adaptive UPGD (AdaUPGD), making its step size robust several RL environments without tuning. We refer the reader to Algorithm 5 for the full description and Appendix 166 for experimental details. Fig. 8 shows that AdaUPGD continually improves its performance, unlike Adam, which suffers from policy collapse.
Avesage Epoodie Retum
Walker2d-v4
Avtrage Epicdic Betum
Autsage Epicodic Retum
Humanoid-vi
HumanoidStandup.vw www
Avtrage Looodic Rosur
300-
12000
3014 Step
181
152
159
250
Step
Step
Step
Figure 8: Performance of Adaptive UPGD (blue) and Adam (red) with PPO in different MuJoCo
environments. The results are averaged over 40 independent runs.
5 CONCLUSION
In this paper, we introduced a novel approach to mitigating loss of plasticity and catastrophic for- getting. We devised learning rules that protect useful weights and perturb less useful ones, thereby maintaining plasticity and reducing forgetting. We performed a series of challenging streaming learning experiments with many non-stationarities alongside reinforcement learning experiments. Our experiments showed that UPGD maintains network plasticity and reuses previously learned useful features, being among the only few methods that address both issues effectively. Our work endeavors to pave the way for a new class of methods for addressing these issues together. Finally, we discuss the limitations of our method and future works in Appendix N.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We gratefully acknowledge funding from the Canada CIFAR Chairs program, the Reinforcement learning and Artificial Intelligence (RLAT) laboratory, the Alberta Machine Intelligence Institute (Ami), and the Natural Sciences and Engineering Research Council (NSERC) of Canada. We would also like to thank the Digital Research Alliance of Canada for providing the computational resources needed. We especially thank Shibhansh Dohare and Homayoon Farrahi for the useful discussions that helped improve this paper. We thank Nishanth Anand, Alex Lewandowski, and Khurram Javed for the helpful discussions. We also thank all of the Lifelong Reinforcement Learning Barbados Reinforcement Learning Workshop 2023 participants for their feedback and discussions.
REFERENCES
Abbas, Z., Zhao, R., Modayil, J., White, A., & Machado, M. C. (2023). Loss of Plasticity in Continual Deep Reinforcement Learning. arXiv preprint arXiv:2303.05507.
Ash, J., & Adams, R. P. (2020). On warm-starting neural network training. Advances in Neural Information Processing Systems, 33, 3884-3894.
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., & Tuytelaars, T. (2018). Memory aware synapses: Learning what (not) to forget. European Conference Computer Vision (pp. 139-154).
Aljundi, R., Kelchtermans, K., & Tuytelaars, (2019). Task-free continual learning. Conference Computer Vision and Puttern Recognition (pp. 11254-1 11263..
Ammar, H. B. (2020). The optimisation series: The story of Adam. Machine Learning and AI Academy.
Carpenter, G. A., & Grossberg, S. (1987). ART 2: Self-organization of stable category recognition codes for analog input patterns. Applied Optics, 26(23), 4919-4930.
Chaudhry, A., Dokania, P. K., Ajanthan, T., Torr, P. H. (2018). Riemannian walk for incremental learning: Understanding forgetting and intransigence. European Conference on Computer Vision (pp. 532-547).
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, H., & Ranzato, M. A. (2019). On tiny episodic memories in continual learning. arXiv preprint arXiv: /902. 10486.
Chen, S., Hou, Y., Cui, Y., Che, W., Liu, T., & Yu, X. (2020). Recall and Learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Cohen, G., Afshar, S., Tapson, J., & Van Schaik, A. (2017). EMNIST: Extending MNIST to handwritten letters. International Joint Conference on Neural Nenworks (pp. 2921-2926).
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., & Tuytelaars, T. (2021). A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7), 1385-3385.
10

---

Published as a conference paper at ICLR 2024
De Lange, M., van de Ven, G. M., & Tuytelaars, T. (2023). Continual evaluation for lifelong learning: Identifying the stability gap. International Conference on Learning Representations.
Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. IEEE Conference Computer Vision and Pattern Recognition (pp- 248-255).
Dohare, S., Hernandez-Garcia, J. F, Rahman, P., Sutton, R. S., & Mahmood, A. R. (2023a).
Maintaining Plasticity in Deep Continual Learning. arXiv preprint (it.::200.1181222
Dohare, S., Lan, Q., & Mahmood, A. R. (2023b). Overcoming policy collapse in deep reinforcement learning. European Workshop on Reinforcement Learning.
Dohare, S., Sutton, R. S., & Mahmood, A. R. (2021). Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint XXiv:2108.00325.
Dong, X., Chen, S., & Pan, S. (2017). Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems, 30.
Elsayed, M. (2022). Investigating Generate and Test for Online Representation Search with Softmux Outputs. M.Sc. thesis, University of Alberta.
Elsayed, M., & Mahmood, A. R. (2022). HesScale: Scalable Computation of Hessian Diagonals. arXiv preprint arXiv:2210.11633
Fedus, W., Ghosh, D., Martin, J. D., Bellemare, M. G., Bengio, Y., & Larochelle, H. (2020). On catastrophic interference atari 2600 games. arXiv preprint arXiv:2002.12499
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., & Bengio, Y. (2013). An empirical investiga- tion of catastrophic forgetting in gradient-basee neural networks. arXiv preprint 132iv:1112.6211.
Ge, Y., Li, Y., Wu, D., Xu, A., Jones, A. M., Rios, S., & Itti, L. (2023). Lightweight Learner for Shared Knowledge Lifelong Learning. Transactions on Machine Learning Research.
Gurbuz, M. B., & Dovrolis, C. (2022). NISPA: Neuro-inspired
adaptation for continual learning in sparse networks. International Conference on Machine Learning (pp. 8157-8174).
Hebb, D. O. (1949). The organization of behavior: A neuropsychological theory. Wiley.
Hetherington, P. A.,.& & Seidenberg, M. S. (1989), Is there 'catastrophic interference' in connectionist networks? Conference of the Cognitive Science Society (pp. 26-33).
Han, S., Pool, J., Tran, J., & Dally, W. (2015). Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28.
He, T., Liu, J., Cho, K., Ott, M., Liu, B., Glass, J., & Peng. (2021). Analyzing the forgetting problem in pretrain pretrain-fietuning finetuning of open-domain dialogue response models. In Conference of the European Chapter of the Association for Computational Linguistics.
11

---

Published as a conference paper at ICLR 2024
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Conference on Computer Vision and Pattern Recognition (pp. 770-778).
He, K., Zhang, X., Ren, S., & Sun, (2015). Delving deep into rectifiers: Surpassing human-level performance imagenet classification. IEEE International Conference Computer Vision (pp- 1026-1034).
Hassibi, B., & Stork, D. (1992). Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5.
Hayes, T. L., Cahill, N. D., Kanan, C. (2019). Memory efficient experience replay for streaming learning. International Conference on Robotics and Automation (pp. 9769-9776).
Hayes, T. L., Kafle, K., Shrestha, R., Acharya, M., & Kanan, C. (2020). Remind your neural net- work to prevent catastrophic forgetting, European Conference on Computer Vision (pp. 466-483).
Hayes, T. L., Krishnan, G. P., Bazhenov, M., Siegelmann, T. Sejnowski, J., Kanan, C. (2021). Replay in deep learning: Current approaches and missing biological elements. Neural Computation, 33(11) 1), 2908-2950.
Hayes, L.,, Kanan, (2022). Online continual learning for embedded devices. Conference on Lifelong Learning Agents, PMLR 199:744-766.
Isele, D., & Cosgun, A. (2018). Selective experience replay for lifelong learning, AAAI Conference on Artificial Intelligence (pp. 3302-3309).
Jung. D., Lec, D., Hong, S., Jang. H., Bac, H., & Yoon, (2022). New insights for the playtictyy dilemma in online continual learning. International Conference on Learning Representations.
Karnin, E. D. (1990). simple procedure for pruning back-propagation trained neural networks. IEEE transactions neural networks, 1(2), 239-242.
Kemker, R., McClure, M., Abitino, A., Hayes, T., & Kanan, C. (2018). Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence 32:1.
Kingma, D. P. Ba, J. (2015). Adam: A method for stochastic optimization. International Conference on Learning Representations.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A.... & Hadsell, R. (2017). Overcoming catastrophic forgetting in neural networks. National Academy of Sciences, / (1((1)) 3521-3526.
Kim, S., Noci, L.,, Orvieto, A., & Hofmann, T. (2023). Achieving Better Stability-Plasticity Trade- off via Auxiliary Networks in Continual Learning. Conference on Computer Vision and Pattern Recognition (pp. 11930-11939).
Krizhevsky, A. (2009) Learning Multiple Layers of Features from Tiny Images. Ph.D. dissertation, University of Toronto.
12

---

Published as a conference paper at ICLR 2024
Konorski, J. (1948). Conditioned Reflexes and Neuron Organization. Cambridge University Press.
Kumar, S., Marklund, H., & Van Roy, B. (2023). Maintaining Plasticity via Regenerative Regular- ization. arXiv preprint arXiv:2308.1190. 11958..
Lan, Q. Pan, Y., Luo, J., & Mahmood, A. R. (2023). Memory-efficient reinforcement learning with value-based knowledge consolidation. Transactions Machine Learning Research.
LeCun, Y., Denker, J., & Solla, S. (1989). Optimal brain damage. Advances in neural information processing systems, 2.
LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 1), 2278-2324.
Lesort, T., Ostapenko, O., Rodriguez, P.. Arefin, M. R., Misra, D., Charlin, L., & Rish, I. (2023).
Challenging Common Assumptions about Catastrophic Forgetting and Knowledge Accumula- tion. Conference Lifelong Learning Agents.
Liu, V., Kumaraswamy, R., Le, L., & White, M. (2019). The utility of sparse representations for control in reinforcement learning. AAAI Conference Artificial Intelligence (pp. 4384-4391).
Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. International Confer- ence on Learning Representations.
Lyle, C., Zheng, Z., Nikishin, E., Pires, A., Pascanu, R., Dabney, W. (2023). Understanding plasticity in neural networks. International Conference Machine Learning (pp. 23190-23211),
Mahmood, A. R., & Sutton, R. S. (2013). Representation search through generate and test. AAAI Workshop on Learning Rich Representations from Low-Level Sensors (pp. 16-21).
Mozer, M. C., & Smolensky, (1988). Skeletonization: A technique for trimming the fat from network via relevance assessment. Advances in neural information processing systems, 1.
Molchanov, P., Mallya, A., Tyree, S., Frosio, I, & Kautz, (2019). Importance estimation for neural network pruning. IEEF/CVF Conference on Computer Vision and Pattern Recognition (pp. 11264-11272).
Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, (2016). Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440.
McCloskey, M., & Cohen, N. J. (1989), Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24, 109-165.
Nair, V. & Hinton, G. E. (2010). Rectified linear units improve restricted boltzmann machines.
International Conference on Machine Learning (pp. 807-814).
Nikishin, E., Oh, J.. Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., & Barreto, A. (2023). Deep reinforcement learning with plasticity injection. Advances Neural Information Processing Systems, 36.
13

---

Published as a conference paper at ICLR 2024
Park, S., Lee, J., Mo, S., & Shin, J. (2020). Lookahead: far-sighted alternative of magnitude- bbssed pruning. arXiv preprint arXiv:2002.04800
Ratcliff, R. (1990). Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2), 285-308.
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., & Hadsell, R. (2016). Progressive neural networks. arXiv preprint arXiv: 1606.0467.
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., & Wayne, G. (2019). Experience replay for continual learning, Advances in Neural Information Processing Systems, 32.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.03347
Schwarz, J., Czarnecki, W., Luketina. J., Grabska-Barwinska, A., Tch, Y. W., Pascanu, R., & Had- sell, R. (2018). Progress & compress: A scalable framework for continual learning. International Conference on Machine Learning (pp. 4528-4537).
Tresp, V., Neuneier, R., & Zimmermann, H. G. (1996). Early brain damage. Advances in Neural Information Processing Systems, 9.
Todorov, E., Erez, T., & Tassa, Y. (2012). Mujoco: A physics engine for model-based control. International Conference Intelligent Robots and Systems (pp. 5026-5033).
Tanaka, H., Kunin, D., Yamins, D. L., & Ganguli, S. (2020). Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33, 6377-6389.
Van de Ven, G. M., Siegelmann, H. T., & Tolias, A. S. (2020). Brain-inspine replay for continual learning with artificial neural networks. Nature communications, I1(1), 4069.
Vinyals, O., Blundell, C., Lillicrap, T., & Wierstra, D. (2016). Matching networks for one shot learning. Advances in neural information processing systems, 29.
Wang, Y+* Vasan, G. & Mahmood, A. (2023). Real-time reinforcement learning for vision- based robotics utilizing local and remote computers. International Conference on Robotics and Automation.
Wortsman, M., Ramanujan, V., Liu, R., Kembhavi, A., Rastegari, M., Yosinski, & Farhadi, A. (2020). Supermasks in superposition. Advances in Neural Information Processing Systems, 33, 15173-15184.
Zaheer, M., Reddi, S., Sachan, D., Kale, S., & Kumar, S. (2018). Adaptive methods for nonconvex optimization. Advances in Neurul Information Processing systems, 31.
Zenke, F., Poole, B., & Ganguli, S. (2017). Continual learning through synaptic intelligence. Inter- national Conference on Machine Learning (pp. 3987-3995).
Zhou, M., Liu, T., Li, Y., Lin, D., Zhou, E., & Zhao, T. (2019). Toward understanding the importance of noise in training neural networks. International Conference on Machine Learning (pp. 7594- 7602).
14

---

Published as a conference paper at ICLR 2024
A CONVERGENCE ANALYSIS FOR UPGD AND NON-PROTECTING UPGD
In this section, provide convergence analysis for UPGD and Non-protecting UPGD in noncon- vex stationary problems. We focus on the stochastic version of these two algorithms since we are interested in continual learners performing updates at every time step. The following proof shows the convergence to a stationary point up to the statistical limit of the variance the gradients, where || f()) v ö represent a d-accurate solution and is used to measure the stationarity of 0. Noncon- vex optimization problems can be written as:
min f(0) = Es-P S)), ERR
where f is the expected loss, L is the sample loss, S is random variable for samples and 0 is vector of weights parametrizing We assume that L is L-smooth, meaning that there exist constant L that satisfy
[VC(0,,8) VC-(2 8) s 0| - 01ll, vO1, O2 E RR E.
(4)
We further assume that L has bounded variance in the gradients E[1 VLO, S) (
<
o², 4O E Rd. Note that the assumption of L-ssmotthness on the sample loss result in L-smooth expected loss too, which is given by [[,) ( < L 02l1- We assume that the perturbation noise in both UPGD versions has bounded variance |||| S o For the simplicity of this proof, we use the true instantaneous weight utility, not an approximated one. We assume that the utility of any connection in the network is upper bounded by number close to one ū - 17
A.1 NON-PROTECTING UTILITY-BASED PERTURBED GRADIENT DESCENT
Remember that the update equation of the Non- con-protecting UPGD be written follows when the parameters are stacked in a single vector 0:
B++1,; or, a(g., + Et.iPt,))-
where is the step size, 9t is the sample gradient vector at time t, Pt (1 uz) is the opposite utility vector, and & is the noise perturbation. Since the function f is T-smooth, we can write the following:
L f(0++1) (V (0.+1 0.) +1)++1100 -110++1
-0112
(5)
= =((0)) - (V((()))) + (esP))s) + Laz
> (gr. + Et,i,r,)
(6)
2 im
Next, we take the conditional expectation of f(0+1)) as follows:
La² d
V[s(0.),, + ExiPe.))
Σ
Eelige. + ()
2
1=1
1=1
-(0.) ([f( La²
Eg
La
((0,) a Σ
(1)) +
Σ
[Eg
Er (Et.sPe,))))
La d
= f(0.) a
Σ
(Er[g2.) + E. +.EE (Er.spt..))))
2
i=1
La
La²
s -100)-//1/00))
Eg
1
2 on
2
i=1
15

---

Published as a conference paper at ICLR 2024
Note that 3 [ B, 0, and PE((E.,)t,;) I E[G,,,, since 0 I Pt, S 1 Vt, i.
From the bounded variance assumption, know that the [B[]| gt [|2] is bounded as follows:
E B1llg111
by + IVF
where by the batch size at time step can now bound E, ( as follows:
La²
o²
E.( - f(,) a V
+
by
=( 2a La²
La²
nn
Rearranging the inequality, taking expectations both sides, and using the telescopic sum, can write the following:
2a L
La²T 2
0²
E EVVI(00)1 s ((1) ([f(0r+))] +
00 +
2
b
Multiplying both sides by La and using the fact that f is the lowest at the global minimum 0": f(0++1) 2 f(0"), we can write the following:
T
EIV 22 [(0)) - s(0))
La²2(o + o²) -(2a- - Laz) 62(20- - La²)
Therefore, the algorithm converges to stationary point. However, in the limit T - oo, the algo- rithm has to have an increasing batch size or a decreasing step size to converge, which is the same requirement for convergence of other stochastic gradient-based methods at the limit (see Zaheer et al. 2018, Ammar 2020).
A.2 UTILITY-BASSE PERTURBED GRADIENT DESCENT
Remember that the update equation of UPGD can be written as follows when the parameters are stacked in single vector A:
0141, 0 a(g,, + E,,))
where is the step size, It is the sample gradient vector at time f, Pt uz) is the opposite utility vector, and &, is the noise perturbation. Since the function f is L-smooth, we can write the following:
L f(0(+1) $ f(0.) (VF()))) (0.+1 - -0,)+ 11 0,112
=((0)) - V[f(O:)).P...l. tc,)) + La² Σ(en (gr, + 2r..)
(8)
2
Next, we take the conditional expectation of f(0(+1) as follows:
La
E, E. s f(O.) V( Cx,i)
Eel(ge, P
Lo² d
= f(0,) a
v(s(0.) +
Σ
E,
(of. Ex + Ee Ee((e..pp..))
2
i=1
Lad
x²
< s(())) ap Σ
vI +
0
2
+
+ | (0.))))
2
b;
1=1
La²
La²
o²
-(0))
op
|v
0
b
16

---

Published as a conference paper at ICLR 2024
Note that 1 ú, 3 [V E, 0, and ((((E,)Pt;)) Pt.i S 1 vt,i.
since 0 Σ
Rearranging the inequality, taking expectations both sides, and using the telescopic write the following:
20p La
La B B f(01) - B[f((r 2
o²²
o +
2
bi
Multiplying both sides by T(208-La' La and using the fact that f the lowest the global minimum 0": f(0r+)) 2 f(0"), we can write the following:
1
Σ E V
La + 0²) T(2ap- La?) (2ap La²)
f(0))-((0)) +
T
Therefore, the algorithm converges stationary point. However, in the limit T - oo, the algo- rithm has to have an increasing batch size or a decreasing step size to converge, which is the same requirement for convergence of other stochastic gradient-basee based methods at the limit (see Zaheer et al. 2018, Ammar 2020).
B UTILITY PROPAGATION
The instantaneous utility measure can be used in a recursive formulation, allowing for backward propagation. We can get recursive formula for the utility equation for connections in neural network. This property is result of Theorem 1.
Theorem 1. If the second-orde order off off-diagonal terms in all layers in neural network except for the last one are zero and all higher-order derivatives are zero, the true weight utility for the weight ij at the layer can be propagated using the following recursive formulation:
(Z) = -Jass + 81,6,5
where
d((ass) 14,
larial
fl.3
h 11 W1.s
Σ fit1ks
kml
1 1, hi 1.j Wiis
daa,)))
d"(ass)
fLtIKS)
$1,6,1
2
28
h?
his
Proof. First, we start by writing the partial derivative of the loss with respect to each weight in terms of earlier partial derivatives in the next layers as follows:
la:+1l
|a:+1l
dC
OC da Shu, o' o'(as))
Oc
E
W4116
(9)
day
0a1+1,k Thu
day,
001+1
Oc
OL day OL h4-15"
(10)
aW.,.
Dag,
aWi,s
dani
Next, we do the same with recond-orded partial derivatives. We use the hat notation for approximated second-order information (off-diagonal terms are dropped) as follows:
lat+1l
JLL
LL
OL
W2 W duf
o (a,,))
W W4 WH11
(11)
daj
Da141,k
8L
0PL
1/2 12-1..3
(12)
dw?,
da7
17

---

Published as a conference paper at ICLR 2024
Now, we derive the utility propagation formulation the sum of two recursive quantities, fi,y and SL,f-- These two quantities represent the first and second-order terms in the Taylor approximation. Using Eq. 9, Eq. 10, Eq. 11, and Eq. 12, can derive the recursive formulation follows:
Uz) acWw, Z)
1 10 LOW Z)
W1.5.5 +
W
aWiss
2 Ow?
AC(W, 2)
1 0 L(W, Z)
W1,5.5
+
W? T,
2
OW
0L
10 L(W, Z)
11-15 Wi,i.s +
11 1, Wh,s
d0,,
0aj
fi. + $1,
(13)
From here, we can write the first-order part ft. and the second-order part st,a, as follows:
larial
0C
f1. -o' -o, Wis5
Wi-1.
(14)
004+18
R=l
1 1 1
a141
UPL
OC WI-LI.,0
$1,i,j
Wii
Σ
0af+1.k
W
D0111
(15)
Using Eq. 14 and Eq. 15, we can write the recursive formulation for Jús and 81,88 as follows:
lacel
dC
fL. = - -d,s)))) Wiis
W4114
001+1.k
d'(,)))
lactll
dc
h4- 14
h1, h1,:W441.K
hu.
80111.
d'(a),)
h1-1, Wi,s
f1+1,k,8
(16)
h4,5
JL
OC
1/ W
W
141, (ala)²
W (o.""
81..
W.i.)
Ouf+1k
day o"(aa,)
lazill
19, 1+1,ks JLC
d'(a,,)? W?
2 1/ 16-1, W?
0as+1A
la:-1|
d'a,,,)2
o"(a1s) fitIKS ).
h 1 W2,
28141,k.1
(17)
h7.
hi,
L
C APPROXIMATED FEATURE UTILITY
We define the true utility of a feature as the change in the loss after the feature is removed. The utility of the feature in the 1-th layer and sample Z is given by
(Z) = L(W, Zi/u,j 0) L(W,Z),
(18)
where h = 0 denotes setting the feature output zero (e.g., by adding mask set to zero).
We refer to it as the true feature utility to distinguish it from its approximations, which are referred either approximated utilities simply utilities. Note that this utility is global measure, and
18

---

Published as a conference paper at ICLR 2024
it provides total ordering for features according their importance. However, computing it is prohibitive since requires additional Ny forward passes, where Ny is the total number of features.
Since the computation of the true feature utiliity is prohibitive, aim approximate it such that no additional forward passes are needed. We approximate the true utility of features by a second- order Taylor approximation. We expand the true utility u, around the current feature at layer and evaluate it at the value of that feature output set to zero. The quadratic approximation of (Z) can be written
(Z, L(W, Zh1.j =0) =
Oc 1 1
alu, hu, 2 O 1.2.5-
(19)
We refer to the utility measure containing the first term as the first-order feature utility, and the utility measure containing both terms the second-order feature utility. We use the approximation by Elsayed and Mahmood (2022) that provides a Hessian diagonal approximation in linear complexity.
When we use an pagssin activation function, (0) O, we can instead compute the feature utility by setting its input to zero. Thus, the feature utility can be approximated by expanding the loss around the current activation input dr as follows:
W1,5 L(W, Z|ar, -0) -0)-
L(W,
OC
1 1
Day. 01 +
2 daf, ni.
(20)
The gradient of the loss with respect to pre-activations or activations is not readily available using most available deep learning frameworks. To have an easily implemented algorithm, we create an equivalent rule to Eq. 18 for general activations by adding mask top of the activation at each layer. This mask acts as a gate on top the activation output: h = gi o h₁. Note that the weights of such gates are set ones and never change throughout learning. The quadratic approximation of (Z) 2) can be written as
(2) = L(W, Zlg.s = =0) L(W,Z)
~ L(W,2) + OL (0 - 91.)) + 2g.
L22
(0 (0-9)))² C(W,2)
0g..
OC
826
(21)
0gi, 0gi.
For feature-wise UPGD, the global scaled utility is given by ūt, o(u,i/n), where n is the maximum utility of the features and á the scaling function, for which we use sigmoid, with its corresponding Algorithm 3, whereas UPGD with the local scaled utility is given by ūt ((u1s/ E, ui,)) in Algorithm 4.
The approximated feature utility can be computed using the approximated weight utility, which gives rise the conservation of utility property. We show this relationship in Appendix D.
D FEATURE UTILITY APPROXIMATION USING WEIGHT UTILITY
Instead of deriving feature utility directly, we derive the utility feature based on setting its output weights to zero. Equivalently to Eq. 18, the utility of feature at layer / is given by
u (Z) = L(W Z) Z),
(22)
where W is the set of all weights, L(W, Z) is the sample loss of a network parameterized by W on sample Z, and W-(.aa is the same as W except the weight W4115., is set to for all values of i.
Note that the second-order Talyor's approximation of this utility depends on the off off-diagonal ele- ments of the Hessian matrix at each layer, since more than one weight is removed at once. For our analysis, drop these elements and derive feature utility. We expand the difference around the
19

---

Published as a conference paper at ICLR 2024
current output weights of the feature at layer / and evaluate it setting the weights zero. The quadratic approximation of u,,(Z) (Z) can be written as
Wnj (2) [(W-) 2) 2(W,2)
ا!:-م|
ac
0C
Σ
W4 W41.5 +
W
WW(413.5
2aw (+1, [+1,ú
lar=1l
0C
Σ
2
Wi+1is W441,
OW i OW- 1-1,i
iml
j
a:-1l
ac
8C w(+1.1)
W441.5 +
@W4110
2 20W(41.0 (41.6)
2
1=1
U141,55
(23)
Alternatively, we can derive the utility of feature i layer / by dropping the input weights when the activation function passes through the origin (zero input leads to zero output). This gives rise to the property conservation of utility shown by Therom 2.
Theorem 2. If the second-order off off-diagonals in all layers one are zero, all higher-order derivatives are zero and an origin- passspassnng activation function is used, the sum of ousput-weight utilities to a feature equals the sum of its input-weight utilities.
neural network except for the last
0041
Jad
U1114
Σ ULJs-
Proof. From Eq. 23, we can write the utility of the feature j at layer I by dropping the output weights. The sample feature utility is given by
Ja2=1|
m (2)(Z))
Σ 1=1
(Z)..
Similarly, write the utility of the feature j at layer by dropping the input weights when the activation function passes through the origin (zero input leads to zero output) as follows:
aa
u ( (Z)
(1).
Therefore, we can write the following equality:
a++1
lail
U
Σ Uiis-
This theorem shows the property of the conservation of instantaneous utility. The sum of utilities of the outgoing weights to a feature equals the sum of utilities of the incoming weights to the same feature when origin-pat geggiin activation functions are used and the off-diagonal elements dropped. This conservation law resembles the one introduced by Tanaka et al. (2020).
20

---

Published as a conference paper at ICLR 2024
E WEIGHT-WISE UPGD, FEATURE-WISE UPGD, AND ADAPTIVE UPGD
Algorithm Weight-wise UPGD with local utility
Given a stream of data D and neural network f with weights { WL}. Initialize step size a, utility decay rate 3, and noise standard deviation d. Initialize {W1, ..., WL}-
Initialize U₁, v/ and time step / to zero.
for (ae, y) in D do
t + for in {L, L- 1, ..... 1 do
F, S: Set-Geeeeiiaaiieee ii
Mi - 1/25, o W? F Wo
0,, (1-
U
Sample & elements from o2)
U - 6 (ÚÚ)
b Dú = 1/ and Diy 0, Vi レ
W - W/ ( (1 Ui)
Algorithm 3 Feature-wise UPGD w/ global utility Algorithm 4 Feature-wise UPGD w/ local utility Given stream of data D and neural net- work f with weights { (W1,... Wz]-
Given stream of data D and a neural net- work f with weights (Wi.... W.). Initialize step size utility decay rate B, and noise standard deviation a. Initialize { W1,.... ..., W randomly. Initialize (g1..., 9L-11 to ones.
Initialize step size a, utility decay rate 8, and noise standard deviation Ø. Initialize (W1,-. Wz} randomly. Initialize (91,.... IL-1) to ones.
Initialize ut, 1/ and time step f zero
Initialize u, V/ and time step to zero
1+ 1
t t- ++ for (x: y) in D do + ++1
for (xx, y) in D do
1 +- [-[++]
for l in [ I.... 1) do
for / in {L, L 1,.., 1) do
- -00
F, GetDeri vatirivvatio i vetives
F. +GetDerrvvativee(...... 1)
if = then
fi, 81 --G.aaeereaateeeeeeeeeee
Wt + Wz aFz
m: t 1/28: - fi
else
W. + 3w: (1 B)m
fú, 81 -Gtteeerriatiiess(....r.
ú + u/( (1- 3t)
m. t 1/28: fi
if < max(ū)) ( then 17 max(ū:)
ui + Su: (1 8)m,
for in [L, 1,..,, 1) do
új + w/ (1 34)
if = L then
u, o(ún/ àill)
Wr + W, oF.
Sample elements from N/O, (0,
else
Wi + Wi a(F +) (1 1u,
ūt + Φ( (1/1)
Sample & elements from (2, Wi + W - (F, + o - lz,) a
F THE GETDERIVATIVE AND GATEDERIVATIVES FUNCTIONS
Here, we describe GetDerivatives used in UPGD and GateDerivatives used in the feature-wise variation of UPGD. Each function takes four arguments: the neural network f. the input æ. the target y. and the layer number The Get.Der vvat ive function returns the gradient of the loss with respect to the weight matrix Wi given by F; and the approximated second-order in- formation of the loss with respect to the matrix W; given by S. The GateDer inndeeiivaee vat ives function is similar, which returns the gradient of the loss with respect to the gate gi given by fi and the ap- proximated recond-order information of the loss with respect to the gate given by st. The matrix S and the vector store the diagonal approximation of Diag( 2 C) and Diag( V2 C) reshaped to be of the same size as W, and gt, respectively. While different methods can be used to get the derivatives, UPGD uses HesScale for it, which is given by Algorithm 6.
21

---

Published as a conference paper at ICLR 2024
Algorithm 5 AdaUPGD: Adaptive Utility-based Gradient Descent
Given a stream of data D, network ff with weights { [W...., WE..
Initialize utility decay rate Bu, momentum decay rate 31, and RMSprop decay rate 82, where 11. 32 [0. 1) and set a small number c for numerical stability (e.g., 10 10-$).
Initialize step size and noise standard deviation Initialize {W,.... W.).
Initialize UI, Mr, Vi, VI and time step /
for (æ: y) D do
t +
for in [L, L 1,.,, do
7 f ---
F, S: --eeeeeriiaatieeee 2, 1.
R, 1/2S, W.2 Fi W
U1 < 3,,U+ + (1-B.)
Mi + 8, + -31)
Vi < 3/ (1- 32) FP2
Ü < U1/(1 8
M: + M1/(1 - 81)
Vi + Vi/(1- -
if < max(Ui) then
17 max(Ui)
for in (T., I, 1) do Sample eeeemenss from (0)
U s(úin))
W, Wi (Mr 2 (vo + + 01- ( Un)
Algorithm The HesScale Algorithm in Classification (Elsayed & Mahmood 2022)
Require: Neural network and layer number
Require: D5 and 82 unless 1= L 0ai 0a?_1"
Require: Input-output pair (x, Compute preference vector aı f f(x) and target one-hot-encoded vector < onehot(y). Compute the predicted probability vector t- (ar, and Compute the loss L(p, q). / L then 0L da + q - p ac using Eq. 10 Compute aw:
Compute
8 daf
qoq
Compute
C aw?
using Eq. 12
else
Compute ac
and Oc ac/ow; using Eq. 9 and Eq. 10 Ba,
Compute da? OW?
using Eq. 11 and Eq. 12 OC _ ac and ØCC DW. OW
CC and 0
Return
Da. da
G UPGD ON STATIONARY MNIST
We use the MNIST dataset to assess the performance of UPGD under stationarity. A desirable property of continual learning systems is that they should not asymptotically impose any extra per- formance reduction, which can be studied in stationary task such as MNIST. We report the results in Fig. 9. We notice that UPGD improves performance over SGD. Each point in the stationary MNIST plots represents average accuracy non-overlapping window of 10000 samples.
22

---

Published as a conference paper at ICLR 2024
The learners network of 300 x 150 units with ReLU activations. We used the hyperparameter search space shown in Table 1. The utility traces are computed using exponential moving averages given by U,- B,00-1+ (1- W, where Ug is the utility trace at time t and U2 is the instantancous utility at time t.
100
Duagged Oreen Rouus
Dewgetaree Accurey
bonged O4ee Recurry
6
Bumber Samples
Mumbor Sandde
Marbe Samples
Number Sample
(a) Weight Global Utility (b) Feature Global Utility
(c) Weight Local Utility
(d) Feature Local Utility
UPGD
SGDW
PGD
Non-protecting UPGD
Shrink and Perturb
Figure 9: Performance of Utility based Perturbed Gradient Descent with first-order approximated
utilities stationary MNIST. The results are averaged over 20 independent runs.
H UPGD ON NON-STATIONARY Toy REGRESSION PROBLEM
We study UPGD's ss effectiveness in mitigating catastrophic forgetting and loss of plasticity in simple toy regression problem that is easy analyze and understand. The target at time is given by V/ [S] Eies It.i. where T is the i-th entry of input vector at time 1, S is a set of some input indices, and f-1,1). The inputs are sampled from NN 1).
In this problem, the task is to calculate the average of two inputs or its negative out of 16 inputs. We introduce statiinitnn using two ways: changing the multiplier or changing the input-index set S. The learner is required to match the targets by minimizing the online squared error. The learner uses a multi-layer (300 x 150) linear network, where the activation used is the identity activation (a(x) = x)) We use linear activations to see if catastrophic forgetting and loss of plasticity may occur even in such simple networks.
The first variation of the problem focuses solely on loss of plasticity. We can study plasticity when the learner is presented with sequential tasks requiring little transfer between them. Here, |S] and the input indices change every 200 time steps by a shift of two. For instance, if the first task has S = {1,2}, the next would be {3.4) 4} and so on. Since the tasks share little similarity between them, expect the continual learners to learn quickly as possible by discarding old features when needed to maintain their plasticity. We compare UPGD against SGD, PGD, S&P, and Non- protecting UPGD. We also use baseline with one linear layer mapping the input to the output.
The second variation of the problem focuses on catastrophic forgetting. Here, the sign of the target sum is flipped every 200 time steps by changing a from tasks share high similarities, we expect continual learners to initially learn some features during the first 200 steps. Then, after the target sign flip, we expect the learner to change the sign of only the output weights and keep the features intact since the previously learned features are fully useful. The frequency of changing a high to penalize learners for re-learning features from scratch.
-1 and vice versa. Since the two
For the input-index changing problem, Fig. 10 shows that the performance of SGD degrades with changing targets even when using linear neural networks, indicating SGD loses plasticity over time. Each point in the toy problem figures represents average squared ervor of 20 tasks. The results are averaged over 20 independent runs. Empirically, we found that the outgoing weights of the last layer get smaller, hindering the ability to change the features' input weights. S&P outperforms the linear- layer baseline. However, PGD and Non-protecting UPGD perform better than S&P, indicating that weight decay is not helpful in this problem, and it is better to just inject noise without shrinking the parameters. UPGD cannot only maintain its plasticity but also improve its performance rapidly with changing targets compared to other methods.
For the output-sign changing problem, Fig. 11 shows that the performance of SGD degrades with changing targets, indicating that it does not utilize learned features and re-learn them every time
23

---

Published as a conference paper at ICLR 2024
the targets change. S&P, Non-protecting UPGD, and PGD do not lose plasticity time but perform worse than UPGD, indicating that they are ineffective in protecting useful weights. UPGD is likely protecting and utilizzng useful weights in subsequent tasks. Moreover, the performance keeps improving with changing targets compared to the other methods.
In both variations, we use utility traces (e.g., using exponential moving average) instead of instan- taneous utility, as we empirically found utility traces to perform better. We found that the second- order approximated utility improves performance over the first-order approximated utility in these problems. A hyperparameter search, given in Table I, was conducted. The best-perforning hyper- parameter set was used for plotting (see Table for weight-wiss wise UPGD alongside other algorithms).
0.200
330
8.204
Bumpee
120
100
Narker
Taus Numder
Telk
(a) Weight Global Utility (b) Feature Global Utility
(c) Weight Local Utility
(d) Feature Local Utility
UPGD with FO Utiiyy
UPGD with so Utitiy
Nor-pratacing UPGD with Utilty Reference (Linea Layer)
SGD
PGD
Shrink and Perturb
Non-protecting UPGD with so Ubittyy
Figure 10: Performance of UPGD on the toy problem with a changing input-index set against SGD, PGD, and S&P. First-order and second-order approximated utilities are used.
Aorogg Oyten Lone
Aecogg (-4ke) Len
-290 Numder
Namker
(a) Weight Global Utility (b) Feature Global Utility (c) Weight Local Utility
(d) Feature Local Utility
with Uilty
UPGD with
Non-protactin UPGD
Reference (Linear
SGD
PGD
Pertur
Non-protecting with so Ubilty
Figure 11: Performance of UPGD the toy problem with changing output sign against SGD, PGD, and S&P. First-order and second-order approximated utilities are used.
I EXPERIMENTAL DETAILS
In each experiment, the utility traces computed using exponential moving averages given by U, = -,,ū,-1+ + (1 -1-28 U,, where U, is the utility trace at time t and U, is the instantaneous utility at time l. Each learner is trained for million time steps, except for the experiment in Fig. 1(b) and its closer look in Fig. 1(c), which we run for 15 million time steps.
1.1 EXPERIMENTS IN FIG.
In the problem of Label-permuted EMNIST, the labels are permuted every 2500 sample. For the offline variation, we form held-out set comprised of 250 samples of the test set of EMNIST. The performance is measured based the learner's prediction on this held-out set after the end of each task in an offline manner. We permute the labels of the held-out set with the same permutation in the task presented to the learner to have consistent evaluation. We observe the same phenomenon indicated by Lesort et al. (2023), which the performance SGD decreases on the first few tasks, matching the vast majority of works catastrophic forgetting. However, when scaling the number of tasks, SGD recovers and maintains its performance. The online variation uses the same experi- mental details in Section 4.4. The parammearsettess used given in Table 3.
In the problem of permuted MNIST, the inputs are permuted every 60000 sample. In Fig. 1(c), show the online accuracy in 4 tasks where each point represents average of over 600 samples.
24

---

Published as a conference paper at ICLR 2024
The step-size used for Adam is 0.0001 using the default 31 0.9, 32 0.9999, and 10 We used for UPGD-VV W a step size a of 0.01, a utility trace decay factor S. of 0.999, a weight decay A of 0.001, and noise standard deviation a of 0.01.
1.2 INPUT-PERMUTED MNIST AND LABEL--ERMUUE EMNIST/MINI-MMAGENN
Each point in the Input-permuted MNIST and Label-permuted EMNIST figures represents an aver- age accuracy one task. The leamers use network of two hidden layers containing 300 and 150 units with ReLU activations, respectively. The results are averaged over 20 independent runs, and the shaded area represents the standard error.
1.3 LABEL-PERMUTED CIFAR-10
The learners network with two convolutional layers with max-pooling followed by two fully connected layers with ReLU activations. The first convolutional layer uses kernel of 5 and outputs 6 filters, whereas the second convolutional layer uses kernel of 5 and outputs 16 filters. The max- pooling layers use a kernel of 2. The data is flattened after the second max-pooling layer and fed to fully connected network with two hidden layers containing 120 and 84 units, respectively. The results are averaged over 10 independent runs, and the shaded area represents the standard error.
1.4 S-SI AND S-RWALK DETAILS
We compute weight importance Rr,i, in S-SI by first maintaining a trace of the gradient multiplied by the weight change from its previous value (with a decay factor B1). We then maintain a trace of the weight change from its previous value (with a decay factor B1). The weight importance for S-SI is defined as the ratio of the first trace and the squared value of the second trace as follows:
W1.i.
S4 (Arag)2 +
where Wrii. moving average of the gradient Gr.i multiplied by the weight change from its previous weight Wi, as: (.aa(WT... w(,,,),, År,. a moving average of the weight change from its previous weight as (1T, w1,i,)), and is small number for numerical stability which set 10-3, Note that the update rule uses of weights with a decay rate B
On the other hand, the weight importance estimation in S-RWalk is computed as:
W1.s.j
$1,,3 1 Ft. FL. w ) +
Qu. = $1,4.j + Fi.ij
where Fi... is moving average of past squared gradients (with decay factor 3y). Note that Br,id estimated with a fast-moving average with decay factor of 0.5 based on $1,i.j-
1.5 HYPERPARAMETER SEARCH SPACE
In this section, we present the hyperparameter search space conduct for each method in each problem in Table 1. Our grid search is quite exhaustive and can reach up to 100 configurations per algorithm for a single run. Table and Table show the best hyperparameter configuration for each method on each problem.
1.6 POLICY COLLAPSE EXPERIMENT
We evaluate AdaUPGD using various MuJoCo (Todorov et al. 2012) environments in comparison to Adam. We use the CleanRL (Huang al. 2022) implementation for PPO with its default best hyper- parameters. For AdaUPGD, we used the same values of 31, 82, and e used Adam, which are 0.9, 0.999, and 10 10,5, respectively. We used noise with standard deviation of 0.001 and a utility trace with a decay rate of 0.999 all environments except for Humanoid-ve and HumanoidSStaddup.vk in which
utility trace with decay rate of 0.9.
25

---

Published as a conference paper at ICLR 2024
Table 1: Hyperparameter search spaces. N-UPGD is short for non-protecting UPGD.
Problem
Space
Method
a
{0.1, 0.01, 0.001, 0.0001
All
Toy Problems
Bu
{0.0, 0.9, 0.99, 0.999)
UPGD, N-UPGD
Stationary MNIST
{0.001, 0.01, 0.1, 1.0}
S&P, UPGD, N-UPGD S&P
X
(0.1,0.01, 0.000,0,,.00,..000000000
a
0.0,0.00,00001,,
All
Input-permuted MNIST
Bu
(0.9,0.99, 0.999, 0.9999)
UPGD, N-UPGD
{0.001, 0.01, 0.1, 1.0}
S&P, UPGD, N-UPGD
Label-permuted EMNIST
X
(0.0, 0.1, 0.01, 0.001, 0.0001)
S&P, UPGD, N-UPGD
31
0..0
Adam
82
(0.99, 0.999, 0.9999}
Adam
Label-permuted CIFAR-10
{10-4, 100 1.016)
Adam
K
{100, 10.0, 1.0, 0.1, 0.01}
S-EWC, S-SI, S-MAS, S-RWalk
Label-permuted mini-Image Imageee
Br
(0.9, 0.99, 0.999, 0.9999)
S-EWC, S-SI, S-MAS, S-RWalk
(0.9, 0.99, 0.999)
S-EWC, S-SI, S-MAS, S-RWalk
Table 2: Best hyperparameter set of each method on Label--prmuute CIFAR10.
Problem
Method
Best Set
SGDW
0.01,
0.001 0.001
S&P
0.01, 0.001
PGD
a 0.001, 0.01
AdamW
0.001, 0.0, = 0..999. 10 0.01
Label-permuted CIFAR-10
S-EWC
0.01, 31 0.9999, 8... = 0.999, K 10.0
S-MAS
0.01, 81 0.9999, 8. 0.999,
10.0
S-SI
a = 0.001, 81 = 0.99, 8 = 0.99, K 0.01
S-RWalk
0.001, 31 0.9, 8,, 0.999,
10.0
UPGD-W
a = 0.01, = -0.001, 3. = 0.999, À = 0.0
N-UPGD-W
0.01, 0.01, 3. 0.99, X 0.001
26

---

Published as a conference paper at ICLR 2024
Table 3: Best hyperparameter of each method in the toy problems, Input-permuted MNIST, Label-permuted EMNIST, and Label-permuted mini-ImageNett.
Problem
Method
Best Set
SGD
= 0.01 0.01, 0.1
PGD
S&P
= 0 1.0, 0.01, = 0.0001, 0.9
Toy Problem
UPGD
N-UPGD
0.01, 0. 0.999
Reference
a = 0.01
SGD
a= 0.01
PGD
a= 0.01, 0.1
S&P
a = 0 1.0,, 1.0,A. 0.1 0.1
Toy Problem
UPGD
0.01, a 0.1, 8n 0.9
N-UPGD Reference
Q = 0.01, 0 = 0.1, 8. 0.999
0.01
SGDW S&P
a=
0.001
a = 0.001, 0.1, 0.01
PGD
0.001,
AdamW S-EWC
Q 0.0001, 31 0.0, 32 = 0.99. 10 10-8,11110 X = 0.0
Input-permuted MNIST
0.001, 31 0.9999, B. 0.99, K 0.001
S-MAS
0.001, 3] = 0.9999, 3 0.999, 0.1
S-SI S-RWalk
0.001, 5 0.9999, 3 0.999,
0.001, 3T -0.99, 3. 0.999, K 10.0
UPGD-W
a 0.01, 0.1, = 0.9999. 0.01
N-UPGD-W
0.001,
0.1, S 0.9, À 0.01
SGDW
a=
0.0001
S&P
0.01,
0.001
PGD
a a= 000.1 0.01
AdamW
Q = 0.0001, 00 0.0, -,0,,,;00,999,, 10 = 0.1
Label-permuted EMNIST
S-EWC
0.01, = 0.999, S. = 0.999, = 1.0
S-MAS
a 0.01, 31 = 0.999, B,, = 0.999, K = 10.0
S-SI
0001,, 019, 0.9,
0.1
S-RWalk
a = 0.01, =0.9, 3w = 0.999, K = 0.1
UPGD-W
0.01, 0.001, 3, 0.9. 0.0
N-UPGD-W
a 0.01, = 0.01, 3. 0.999, 0.001
SGDW S&P
& = 0.01, 0.001 0.001
0.001
PGD AdamW
a 0.0.1 0.01
0.0001, = 0.9, =.9.892009999..44
10 X 0.1
Label-permuted mini-Ine Metteemt
S-EWC
a 0.01, 31 = -0.999, 8.. = 0.999, K = 1.0
S-MAS
a 0.01, 0.999, 8. 0.999, K 10.0
S-SI S-RWalk
a 0.001, = =0.9, = 0.99, K 0.01
a = 0.01. -0.99,
B,, = 0.999, K = 0.1
UPGD-W
0.01, = -0.001, 3, = -0.9. À 0.0
N-UPGD-W
a = 0.01, 0 = 0.01,3. = 0.999, = 0.0001
27

---

Published as a conference paper at ICLR 2024
J THE QUALITY OF THE APPROXIMATED UTILITY
Fig. 12 shows the Spearman correlation at every time step with different activation functions. An SGD learner with step size of 0.01 used single hidden layer network containing 50 units with ReLU activations (Nair & Hinton 2010) and Kaiming initialization (He et al. 2015). The network has five inputs and a single output. The target of an input vector is the sum two inputs out five inputs, where the inputs are sampled from U 0.0.
Spaaron Couteoee
Speanar's Codtire
Syearut Codfiorss
Narbber Sample
Namber of Sandiia
Numasr of Samples
Namber of Sumples
(a) Weight Global Utility
(b) Weight Local Utility (c) Feature Global Utility
(d) Feature Local Utility
Spea man Corfien
Spea war' Ceefi
Namber C/B
Samplia
Samples
Sampli
(e) Weight Global Utility
(I) Weight Local Utility
(g) Feature Global Utility
(h) Feature Local Utility
Sowrear Cortreet x
Colfeet
die
Tarnter Samples
Namber Somplee
Numosr
Mumber
Weight Global Utility
( Weight Local Utility
(k) Feature Global Utility
Feature Local Utility
(Reference)
Fist-order
Second-order
Random
Weight-magnitude
Squared Gradient
Spearman correlation between the true utility and approximated utilities. The activations used the first, second, and third rows are ReLU, LeakyReLU, and Tanh. The shaded area repre- sents the standard error.
Figure 12:
K ADDITIONAL EXPERIMENTS
K.1 MORE DIAGNOSTIC STATISTICS CHARACTERIZING SOLUTION METHODS
Here, we provide more diagnostic statistics for our methods Fig. 13, Fig. 14, Fig. 15, and Fig. 16.
Auceage Oring Must dit
Nocm Wegghe 5
f Rorm of
Taak Number
Task Number
Task Number
Task Number
Figure 13: Additional diagnostic statistics of methods on Input-permuted MNIST. The average plas-
ticity, the average online loss, the -orrm of gradients, and the form of weights are shown.
28

---

Published as a conference paper at ICLR 2024
Butoag Onise Flaat
Noro of Grodletts
1200
15 E1
Task Number
Task Number
Task Number
Task
Aurrgg Oneire Les
Gracierts
Marm Wagtts
500
2004
30
10
20
20
20-
Namber
Tonk
UPGD
50DW
PGD
Adomni
Memprotecting UPGD
Shrisk and Perturt
EWC
5-MAS
5:1
5 RRMa
Figure 14: Diagnostic statistics of methods on Label-permuted CIFAR1O. The average plasticity, percentage of zero activations, lo. & and f--norm of gradients, the average online loss, the -nomm and f--morm of the weights are shown. The shaded area represents the standard error.
Ausmae Ontre Pluardyy
of Actiotions
Norn Grstlett
Nom of Gradiert
Task Namber
Task Number
Task Number
Tesk Number
Anrrge Oniree Lows
Nom of Gradierts
Norm Weggtt
20
Number
Maspprotecting
Figure 15: Diagnostic statistics of methods Label-permuted EMNIST. The average plasticity, percentage zero activations, lo. & and fg-norm of gradients, the average online loss, the mn-mrrm and form of the weights shown. The shaded represents the standard error.
Asttag Crina Flusbihy
Nor Crodaees
Nem Graclert
2001 2200
10
Task Number
Number
Teak
Avenge Onirre Low
1, Nam Cesderts
Marm Wa
Nom We
20000
E
180
20
300
Tonc
UPGD-W
SCOW
PGD
AdamTi
Mon-grotacting UPSD-WW
Shrisk and Perturs
SWC
S-AAS
5:tt
S-RWalk
Figure 16: Diagnostic Statistics of methods on Label-permuted mini- ImageNe. The average plas- ticity, percentage of zero activations, fo, & and form of gradients, the average online loss, the L --rormm and norm of the weights are shown. The shaded area represents the standard error.
29

---

Published as a conference paper at ICLR 2024
K.2 COMPUTATIONAL TIME FOR MAKING UPDATES
Here, we conduct a small experiment to determine the computation overhead by each method. We interested in the computational time each algorithm needs perform single update. Fig. 17 shows the computational time needed to make a single update using a batch of 100 MNIST samples. The results are averaged over 10000 updates. Each leamner used a multi-layered network of 1024 x 512 units. The error bars represent one standard error.
10.
8-
Time (ms)
6
2.
SGp
Adam
PGD
upop
HUJGG
S.w
5AVak
5-645
Figure 17: Computational time in milliseconds needed to make single update. N-UPGD is short for Non-protecting UPGD.
K.3 FEATURE-WISE AND WEIGHT-WISE UPGD ON THE INPUT-PERMUTED MNIST
We repeat the experiment on Input-permuted MNIST but with feature-wise approximated utility. In addition, show the results using the local approximated utility for both weight-wise and feature- wise UPGD in Fig. 18. The weights are initialized by Kaiming initialization (He et al. 2015). A hyperparameter search was conducted (see Table 1). and the best-peerorniig hyperparameter set was used for plotting.
Averge Urtee Agunog
Averge Onlee hounin
ko
Averge Odees
20
10
60
10
20
201
Namber
(a) Weight Global Utility (b) Feature Global Utility
(c) Weight Local Utility
(d) Feature Local Utility
UPGD-W
PGD
Adami
protecting UPGD-W
Shriek and Perturs
SEWC
5-2
s-awaik
Figure 18: Performance of methods on Input-Permutee MNIST. First-order utility traces are used for UPGD and Non-protecting UPGD. Results are averaged over 10 runs.
K.4 FEATURE-WISE AND WEIGHT-WISE UPGD ON LABEL-PERMUTED EMNIST
We repeat the experiment Label-permuted MNIST but with feature- approximated utility. In addition, we show the results using the local approximated utility for both weight- wise and feature- wise UPGD in Fig. 19. The weights initialized by Kaiming initialization (He et al. 2015). A hyperparameter search was conducted (see Table 1), and the best-performing hyperparameter set was used for plotting.
K.5 LOCAL WEIGHT-WISE UPGD ON LABEL-PERMUTED CIFAR-10
We repeat the experiment on Label-permuted CIFAR-IO 10 but with the local approximated utility. Fig. 20 shows the results for UPGD using local and global approximated utilities. The weights
30

---

Published as a conference paper at ICLR 2024
Aeerge Oalvee Salee..nooo
Task Numser
Task Namber
Task Number
Task Namber
(a) Weight Global Utility (b) Feature Global Utility
(c) Weight Local Utility
(d) Feature Local Utility
UPGD-W
PGD
AdaeiW
Mon-pratacting UPGD-W
Shitek and Partura
S-WWC
Figure 19: Performance of methods on Label-permuted EMNIST. First-order utility traces used for UPGD and Non-protecting UPGD. Results are averaged over 10 runs.
initialized by Kaiming initialization (He al. 2015). A hyperparameter search was conducted (see Table 1), and the best-performing hyperparameter set was used for plotting.
Averue Online Rocuracy
Tank Namber
Tark Number
(a) Weight Global Utility
(b) Weight Local Utility
LPGDD-W
NGOW
PGD
Adaiot
Heseig Monegeatannnn www.ttiteiiinggooooooooooeeoe UPGD-W
Shutieb and Parturk
SMAS
Figure 20: Performance of methods on Label-permuted CIFAR-10. 10. First-order utility traces are used for UPGD and Non-protecting UPGD. Results averaged over 10 runs.
L ABLATION ON THE COMPONENTS OF UPGD-W
We conducted an ablation study on the components of UPGD-W: :: weight decay, weight perturbation, and utility gating. Starting from SGD, we add each component step by step until we reach UPGD-W. Fig. 21 shows the performance of learners Labell-erruted EMNIST, and cubbll-errmted mini- Imageeet. We notice that both weight pertur- bation and weight decay separately improve SGD performance. Still, the role of weight decay seems to be more important in Input-permuted MNIST and Label-permuted mini-ImageNet. Notably, the combination of weight decay and weight perturbation makes the learner maintain its performance. When utility gating is added on top of weight decay and weight perturbation, the learner can improve its performance continually in all perbuude problems and slightly improve its performance on input-permuted MNIST.
input-permuted MNIST, Label-permuter CIFARIO 10,
We also conducted an additional ablation in Fig. 22 where we start from UPGD-W and remove each component individually. This ablation bolsters the contribution of utility gating more. Using utility gating on top of SGD allows SGD maintain its performance instead of dropping on input- permuted MNIST and improves its performance continually on label-permuted problems. The role of weight decay and weight perturbation is not significant in label-permuted problems, but including both with utility gating improves performance and plasticity on input-permuted MNIST.
M RELATIONSHIP TO GENERATE-AND-TES. METHODS
The generate-and-test method (Mahmood & Sutton 2013) is method that finds better features using search, which, when combined with gradient descent (see Dohare et al. 2023a), is similar to feature-wist wiit variation of our method. However, this method only works with networks with Lingle-hidde layers in single-output regression problems. uses the weight magnitude to determine feature utilities; however, has been shown that weight magnitude is not suitable for other problems,
31

---

Published as a conference paper at ICLR 2024
Aeruue Oelee Reasmeyy
Awer Oeles Acsumayy
Aeeaae Ondre Nqu
Avergg
Task Number
Task Namber
Task Number
Task Namber
(a) MNIST
(b) CIFARIO
(c) EMNIST
(d) mini Imagele
SGD
SGD+WP
SGD+WD
SGD+WD-WP
WGD-WD-WP-UG (UPGD-W)
Weight Decay (WD), Weight Perturbation (WP), and Utility Gating (UG) shown on Input-permuted MNIST, Label-permuted EMNIST, Label- permuted 10, and Label- cabel-permuted mmai--mmaeeeet. A global first-order utility trace is used. Results are averaged over 10 runs. The shaded area represents the standard error.
Figure 21: Ablation on the components of UPGD-W:
Duange dave kaayy
Jenge Oake Ararayy
Tolk Norker
Tysk Naweher
Tosk Number
Task farster
(a) MNIST
(b) CIFARIO
(c) EMNIST
(d) mini- Mmine-lmaeeNt
MHDDUG-WP-NG (UPGD-W)
SGD-UG-WD
SGD+UG+WP
SCD+UG
SGD-WD
SGD-WP
Figure 22: Ablation on the components of UPGD-W: Weight Decay (WD), Weight Perturbation (WP), and Utility Gating (UG) shown on Input-permuted MNIST, Lebel-permuted EMNIST, Label- permuted CIFAR10, and Label-permuted mini- Inage,,,,,,,,, starting from UPGD-W and removing each component individually. SGD+WD and SGD+WP added as baselines that do not use utility gating. A global first-order utility trace is used. Results are averaged over 10 runs. The shaded area represents the standard error.
such as classification (Elsayed 2022). On the contrary, our variation uses a better notion of utility that enables better search in the feature space and works with arbitrary network structures objective functions so that it can be seen as a generalization of the generate-and anntrett-nnd-est method.
N FUTURE WORKS
Arguably, one limitation of our approach is that it measures the weight utility, assuming other weights remain unchanged. A better utility would include interaction between weight variations, which is left for future work. One desirable property of continual learners is the ability to modify their hyperparameter, allowing for greater adaptation changes. Although our method does not re- quire intensive hyperparameter tuning, it still requires some level of tuning, similar to other methods, which may hinder its ability in true lifelong learning. A promising direction to use a generate-and- test approach that continually learns the best set of hyperparameters. In our evaluation, we assumed that the non-stationary target function is piece-wise stationary. Studying the effectiveness UPGD against hard-to-din todiitiingiu non con-stationarities would be an interesting future direction.
32


Reviews:
[{"review_id": "DvFEu4D6aC", "reviewer": "Reviewer_XNx6", "summary": "This paper proposes Utility-based Perturbed Gradient Descent(UPGD). A modification to the vanilla gradient descent update rule that helps the model to operate in a more challenging scenario of streaming learning. The authors introduced their utility function as an importance weight for each parameter of a neural network. The authors show the effectiveness of their contribution compared to common importance assignment methods in the continual learning literature.", "questions": "**Q1:** Have the authors tried to use an scaled version of the perturbation that takes the magnitude of the parameters into account? (Other than the unprotected version). Also I would appreciate the if you could elaborate on the effect of perturbations.\n\n**Q2:** Could you also explain about the average online accuracy? it is stated that \"The average online accuracy is the percentage of correct predictions within each task.\" I cannot see the average part here. Is it calculating the accuracy on each task separately then averaging over the number of tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**Clear Structure and Writing:** The paper benefits from a clear structure and concise writing style.\n\n**Addressing a Complex Issue:** The authors tackle an underexplored yet challenging problem, and I appreciate their efforts to address online continual learning.\n\n**Mathematical Foundation:** The definition of utility introduced in the paper is based on simple and sound mathematical derivations. \n\n**New metric:** The introduction of a new plasticity metric is a nice contribution to the relatively uncharted territory of streaming learning.", "weaknesses": "**Unscaled perturbations:** My main point of issue is the reasoning behind the perturbations in the update rule. The authors claim that by adding the perturbation we are making the unimportant weights more plastic however I am not really convinced by this explanation I believe it requires elaboration both in the rebuttal and in the paper. \n\nAnother related issue with the proposed perturbation is the fact that all of them are getting drawn from the same standard normal distribution. This design choice is strange to me since the parameters of a neural network usually differ in magnitude from layer to layer. By adding an unscaled random perturbation to all of the weights we are ignoring this scale difference which I believe is sub-optimal. I know that in the unprotected version, they are getting weighted by different values but this particular scaling is more correlating with changes in the loss value rather than the parameter magnitudes.\n\nHighly relevant to the above issue, I believe it is also necessary to have an additional ablation study, investigating the role of having and not having the perturbations in the update rule. I also want to disentangle the effect of weight decay. The only time that UG is added in the ablation is in the presence of WD. More specifically I am curious about the following scenarios in Figure 8: \n\n* Added ablations:\n    + SGD + UG + WP + WD (present in the paper)\n    + SGD + UG + WP \n    + SGD + UG + WD\n    + SGD + UG\n\n    \n**Including more diverse experiments:** Moreover, in the experiments section I believe the authors need to include more diverse experiments. All of the streaming tasks are permutations of the same task. Whether in the label or in the input space. It is not as obvious as the authors' claim that after the permutation of the input space the previously learned representations are not relevant anymore (end of page 6). In the input-permuted scenario, only the first layer needs to have significant change. This is especially true for the label-permuted tasks as the network does a good job of clustering the data up to the final FC layer. I encourage the authors to use the Cifar100 superclass dataset (or any similar sequence of tasks that does not simply rely on the permutation).\n\n**Visualization:** Finally, I believe the visualization needs several improvements: the legends on the plot are very hard to read (Fig 2, 3, 4, 5). Some colors are similar to each other and the width of the lines in the legends is too thin. (Especially in figure 4). In Figure 7, some numbers in dark blue cells are almost impossible to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xHSKCcJ5Or", "reviewer": "Reviewer_y5kB", "summary": "The paper proposes a measure of weight utility of weights in neural networks for given loss and using it to modify a gradient-based weight update in networks to alleviate the problem of catastrophic forgetting.  The authors identify two fundamental aspect of catastrophic forgetting - the forgetting aspect (not losing what the network already know) and plasticity aspect (ability to learn new concepts).  The proposed method is meant to address two problems at the same time, preseving high utility weights with no modifications (to prevent forgetting) while randomly perturbing low utility weights to \"encourage\" them to participate in the computations related to new tasks (plasticity).  Empirical evaluations show solid performance of the proposed method according to the forgetting and plasticity metrics newly defined by the authors.", "questions": "Though I understand (and like) in principle what the utility-based update is supposed to do, I can't quite understand why it actually works.  The proposed measure of the utility of parameters is a measure with respect to the loss on the new input/output pair.  If this pair comes from a new task, how does measuring utility of the model parameters with respect to the loss of this new task have bearing on the utility of the parameters for the old tasks?  Just because utility of a given weight is, say, low for the current sample, it doesn't mean it's low for previous samples.  It seems to me that the proposed method would score high on plasticity (it finds available weights for new task)...but I don't see how it protects against forgetting, in principle, though if we are to talk about empirical evidence...  I don't understand how 4.3 measures catastrophic forgetting.  Permuting labels of CIFAR10 with the new tasks suggests to me that it's all about plasticity again.  Shouldn't it be an experiment, where labels are kept intact, but new tasks are added...and previous tasks examples are not used?  Am I missing something about how experiments reported in 4.3 are done?\n\nWhy are the accuracy results of training on CIFAR-10 and EMNIST so poor in Figure 6?  State of the art CIFAR-10 is close (or above) 90%.  Something close to 80% would be probably still acceptable...but 60% is quite poor.  I am not exactly sure what EMNIST variant entails, but is 70% accuracy a good accuracy for this dataset?  It is often easy to shown improvements of something at the low end of the models' performance, but that doesn't always translate to same effect at the high (or close to) end of the models' performance...and in the end, the latter is what we really care about.  So, does the proposed method prevent forgetting at the high end, when model is performing at or reasonably close to state of the art?\n\nThis is not a massive issue, but does the per batch normalisation of utility make the performance of the method variable with different  mini-batch size settings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed rule is straight forward.  \n\nComputational complexity of the evaluation of true utility is well addressed making the method practical.\n\nEmpirical evidence provided shows the proposed rule is effective for alleviation of catastrophic forgetting.\n\nDecomposing the catastrophic forgetting problem into two aspects: forgetting and plasticity, seems very sensible.\n\nProposed measures of plasticity and forgetting seem sensible.\n\nThe paper is well written.", "weaknesses": "Though empirical evidence provided in the paper suggest it does (in that it works), I am not sure that the proposed definition of weight utility make sense.  The power of neural networks (and the problem of the interpretation of its computation) is its distributed computation.  Utility of an individual weight is almost always nothing - in fact, quite often any particular weight, sometimes even large number of weights, can be taken out of the network, with little impact on performance.  So, it's more about combinations of weights working together...and the proposed utility doesn't measure that.  I understand that evaluating utility of combinations of weights is intractable, but I worry that this simplification, of judging utility of each weight in isolation, is encouraging less distributed representation, which might come with a penalty in performance.\n\nFundamentally, on the forgetting front, the proposed method is just another weight consolidation method, and it's a bit hard to believe it beats Elastic Weight Consolidation.  It am not 100% sure that the proposed method doesn't favour plasticity over forgetting nor that the forgetting evaluation isn't biased towards methods that favour plasticity (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dc9aLBYCdY", "reviewer": "Reviewer_3zCE", "summary": "This work proposes to modify stochastic gradient descent (SGD) to overcome forgetting and promote plasticity in continual learning. These goals are achieved by masking out the parameters with high utility and perturbing gradient direction by Gaussian noise. For utility computation, the authors propose an approximate but efficient scheme based on second-order Taylor expansion of the loss. Experiments demonstrate that (i) the proposed utility approximation is more accurate than simple baselines such as weight magnitude, (ii) it maintains plasticity, (iii) plasticity and accuracy are in general correlated, (iv) the method tends to forget less than baselines, and (v) it simultaneously promote plasticity and prevents forgetting.", "questions": "Here I list my questions as well as suggestions:\n\n- It would be better if Label-Permuted EMNIST was described before the results are discussed in paragraph 5.\n- _Although a few methods address both issues simultaneously, such methods expect known task boundaries, maintain a replay buffer, or require pretraining, which does not fit streaming learning._ <--- reference needed for this claim.\n- What does \"a Hessian diagonal approximation in linear complexity\" mean? Linear in the number of parameters?\n- It would be better if the main text included details on the \"utility propagation theorem\".\n- It would be better if the descriptions of the tasks/datasets (e.g. Input-Permuted MNIST in section 4.2) were given before the details.\n- Does \"each learner is trained for 1M samples, one sample each time step\" mean gradient descent using one sample only? Is this realistic?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 1, "strengths": "This paper is written well. The notation is okay and the mathematical derivations seem correct. The baseline methods are clearly outperformed and the experiments verify the central claim of the paper.", "weaknesses": "Although continual learning is an important machine learning challenge, I feel the paper suffers from significant weaknesses:\n\n  - First and foremost, I do not think this paper makes a significant contribution. The methodology is incremental in that it combines two well-known ideas (perturbed gradient descent + keeping active neurons unchanged). \n  - Second, it is tested on very toy setups. The experiments are not convincing enough to show the applicability of the method to interesting real-world setups. For instance, I am not sure the networks achieve similar plasticity if tested on, e.g., webcam data instead of MNIST, where the feature space is a lot richer and hence plasticity is much more difficult.\n  - Third, theoretical properties/implications of the method should be carefully examined. \n    - For instance, the Taylor expansion would only hold if $W_{l,i,j}$ are infinitesimally small. We do not know in general if this holds or not. I suggest the paper should include a (preferably rigorous) discussion on this.\n    - Likewise, gradient descent is no longer steepest descent but some approximation to it. Investigating why it works is important. As shown by the results, no collapse occurs but again, I wonder how this translates into more challenging settings where utilities of most parameters are high.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2KZ92F5dGM", "reviewer": "Reviewer_KmBd", "summary": "This paper introduces a novel approach called Utility-based Perturbed Gradient Descent (UPGD) to address catastrophic forgetting and loss of plasticity in neural networks. UPGD combines gradient updates with a mask to protect useful weights from being forgotten and reuse less useful weights. The paper also proposes metrics to evaluate loss of plasticity and catastrophic forgetting. Empirically, the method outperforms existing methods in streaming learning problems in terms of retaining plasticity and avoiding catastrophic forgetting.", "questions": "Is it possible to show theoretical performance guarantees for UPGD? For instance, can the approximation error of equation 2 be bounded? Alternatively, if the true utilities are used in equation 3, is it possible to derive some guarantees against forgetting or loss of plasticity?\n\nHow much more significantly does UPGD improve upon baselines S-EWC and S-MAS?\n\nHow does UPGD-W perform with WP and WD removed individually?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**\nConceptually, the problem addressed by the authors of avoiding forgetting while retaining plasticity in streaming learning settings remains underexplored. The specific method proposed by the authors is relatively straightforward and conceptually similar to prior approaches; however, it empirically outperforms prior methods as the experimental results demonstrate.\n\n**Quality**\nThe convergence guarantee results are valuable. The experiments are generally comprehensive and well-conducted. Assessing the quality of the approximated utilities in section 4.1 is \nof critical importance, and the results are convincing. Conducting miniImagenet scale experiments is a solid addition to the experimental section. The ablation study in Figure 8 is also insightful.\n\n**Clarity**\nThe writing is generally clear and the figures are well-illustrated.\n\n**Significance**\nOverall, the paper addresses a major issue in the field of streaming learning. Given that the paper doesn't investigate the theoretical properties of UPGD, the significance of the paper hinges on the strength of the empirical results.", "weaknesses": "Since the proposed method lacks theoretical performance guarantees, its empirical performance is critical. The authors have generally done a good job demonstrating that UPGD avoids forgetting and maintains plasticity; however, a few concerns remain:\n\n- It appears that S-EWC does not have too much of a gap with UPGD judging from figure 7: it entirely avoids catastrophic forgetting, and the only setting where it loses plasticity where UPGD does not is on MNIST\n- S-MAS outperforms UPGD on miniImagenet at the end of training, and does not have a large gap overall\n- The ablation of figure 8 checks the contribution of each component of UPGD sequentially as they are added to regular SGD. Ideally, the ablation would study how each component affects UPGD when they are *individually* removed (e.g. UPGD without WP).\n\n**Minor comments**\nFigure 7 is referred to before Figure 6; ideally, their order would be swapped.\nI see in Section 4 that the results are averaged over 20 trials, but the meaning of the error margins in some of the figures is not made clear (e.g. figure 2). I would also suggest increasing the number of trials to smooth out the curves if possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)