Continual Backprop: Stochastic Gradient Descent with Persistent Randomness
Authors: Shibhansh Dohare
Richard S. Sutton
A. Rupam Mahmood
Venue: ICLR 2022 Submitted (2022)
Keywords: Continual Adaptation
Reinforcement Learning
Continual Learning
Online Learning

Abstract: The Backprop algorithm for learning in neural networks utilizes two mechanisms: first, stochastic gradient descent and second, initialization with small random weights, where the latter is essential to the effectiveness of the former. We show that in continual learning setups, Backprop performs well initially, but over time its performance degrades. Stochastic gradient descent alone is insufficient to learn continually; the initial randomness enables only initial learning but not continual learning. To the best of our knowledge, ours is the first result showing this degradation in Backprop’s ability to learn. To address this issue, we propose an algorithm that continually injects random features alongside gradient descent using a new generate-and-test process. We call this the Continual Backprop algorithm. We show that, unlike Backprop, Continual Backprop is able to continually adapt in both supervised and reinforcement learning problems. We expect that as continual learning becomes more common in future applications, a method like Continual Backprop will be essential where the advantages of random initialization are present throughout learning.

Under review as a conference paper at ICLR 2022
CONTINUAL BACKPROP: STOCHASTIC GRADIENT DE- SCENT WITH PERSISTENT RANDOMNESS
Anonymous authors Paper under double-blind review
ABSTRACT
The Backprop algorithm for learning in neural networks utilizes two mechanisms: first, stochastic gradient descent and second, initialization with small random weights, where the latter is essential the effectiveness of the former. We show that in continual learning setups, Backprop performs well initially, but over time its performance degrades. Stochastic gradient descent alone is insufficient learn continually; the initial randomness enables only initial learning but not continual learning- To the best of our knowledge, ours is the first result showing this degra- dation in Backprop's ability to learn. To address this issue, we propose algo- rithm that continually injects random features alongside gradient descent using a new generate-and-test process. We call this the Continual Backprop algorithm. We show that, unlike Backprop, Continual Backprop is able to continually adapt in both supervised and reinforcement learning (RL) problems. We expect that continual learning becomes more common in future applications, a method like Continual Backprop will be essential where the advantages of random initializa- tion are present throughout learning.
INTRODUCTION
In the last decade, deep learning methods have been successful and become the sate--of-the-ars in many machine learning problems and applications, including supervised classification, reinforce- ment learning (Silver et al., 2016), computer vision (Krizhevsky al., 2012), and natural language processing (Brown et al., 2020). These methods learn the weights of an artificial neural network using Backprop, which primarily applied to stationary problems. However, a primary challenge to leveraging the strengths of deep learning beyond current applications is that Backprop does not work well in non-stationary problems (McCloskey and Cohen, 1989; French, 1997; Sahoo et al., 2018), for example, problem that consists sequence of stationary problems.
Leveraging the strengths of deep learning methods in non-stationary problems is important as many real-world applications of machine learning like robotics involve non-stationrities. Non- stationarities can arise due to changes in the environment (Thurn, 1998), high complexity (Sutton et al., 2007), partial observability (Khetarpal al., 2020), other actors (Foerster et al., 2018). Many works have tried to make deep learning work in non-stationary problems. Some works have pro- posed methods that can remember previously learned information (Kirkpatrick et al., 2017: Aljundi et al., 2018; Riemer et al., 2019). While, others have proposed methods that can adapt fast to non- stationarities (Rusu al., 2016; Al-Shedivat al., 2018; Finn al., 2019).
A limitation of prior works on non-stationary problems is that they did not study cases where there are many stattiitionaii.... Number of non-stationarities refers the number of times the data distribution changes. Most works only look at problems with less than ten non-stationarities (Rusu et al., 2016; Kirkpatrick et al., 2017; AI-Shedivat al., 2018). Finn et (2019) studied problems with up to a hundred non-stationarities. Dealing with many non-stationarities is important for systems that continually interact with the real world as non-stationarities frequently occur in the world.
In this work, study problems with large number (thousands) of non-stationarities. We start with a special class of problems that we call semi-stationary problems. These are online super- vised learning problems where the input distribution is non-stationary while the target function is stationary. The target function the function being approximated, for example, the true regression function. These problems natural way study problems with many ton-stationariiies: slowly

---

Under review as a conference paper at ICLR 2022
moving through large input space cause thousands of non-stationarities. These problems also important for systems in the real world as inputs from the real world often depend on previous inputs, making the input distribution non-stationary. Finally, we study non-stationary RL problems. These are full non-stationarity problems as the input distribution changes when the agent's behaviour changes and the target function, optimal policy, changes as the dynamics of the environment change.
Semi-stationary problems reveal a new difficulty with Backprop(BP) in non-stationary problems; they help us clearly understand one way in which Backprop fails. We show that in non-stationary problems, Backprop performs well initially. but surprisingly, its performance degrades substantially over time as Backprop loses its ability adapt. Backprop relies on proper random initialization for its effectiveness (Glorot et al., 2010: Sutskerver et al., 2013; He et al., 2015). However, randomiza- tion in Backprop only happens in the beginning. We hypothesize that Backprop's s ability to adapt degrades because the benefits of initial random distribution are not present at all times.
To extend the benefits of initial random distribution throughout learning, we propose the Contin- ual Backprop (CBP) algorithm. CBP uses generate-and- method to continually inject random features alongside SGD. We show that unlike BP, CBP can continually adapt in non-stationary prob- lems. Our generate-and-test algorithm consists of two parts: the generator, which proposes new features,and the second is the tester, which finds and replaces low utility features with the features proposed by the generator. Our generate-and-et test algorithm is built on the one proposed by Mah- mood and Sutton, (2013) and our algorithm is compatible with modern deep learning. We overcome three significant limitations of their work. First, algorithm is applicable to feed-forward net- works with arbitrary shape and size while theirs was only applicable to networks with a single hid- den layer and one output. Second, our algorithms works with modern activations and optimizers like Adam while theirs was limited to LTU activations, binary weights, and SGD. And third, we combine generate- senerate-and-test with SGD and show that the resulting algorithm, CBP, is significantly better than BP in complex non-stationary problems while their work was an isolated study of generate-and-test
The first contribution of our work is to show that in non-stationary problems with many non- stationarities BP loses its ability to adapt over time. In other words, we contribute to the under- standing of why BP and its variants fails in non-stationary problems. Our second contribution is that propose the CBP algorithm that extends the benefits of the initialization in BP to all times.
2 NON-STATIONARY PROBLEMS
We study Backprop, Continual Backprop, and other learning algorithms in semi-stationary and Re- inforcement Learning (RL) problems. First, we consider novel idealized semi-stationary problem. The strength of this problem is that in this problem can study continual learning algorithms ex- tensively and yet in a computationally inexpensive way and without the confounders that arise in more complex problems. Then we study the permuted MNIST problem, an online image classifi- cation problem, and two non-stationary RL problems. We demonstrate on these problems that the findings from the idealized problem scale to large neural networks in more realistic settings.
Performances measure in semi-stationary problems
In supervised learning, the task is learn function using examples of input-output pairs. This function is called the target function. In online supervised learning (Orabona, 2019), there is a stream of samples (I1, Mt), and the predictions have be made sequentially. The performance measure is the loss on the next sample. Thus. learning and evaluation happen simultancously. This fundamentally different from offline supervised learning, where there are two separate phases, one for learning and another for evaluation. Another common measure in non-stationary problems the performance on previously seen data. However, measuring performance on previously seen data is only meaningful when studying the catastrophic forgetting aspect of BP. As we do not study the forgetting problem, we do not measure the performance on old data.
2.1 BIT-FLIPPING PROBLEM
Our first problem is the Bit- Flipping problem. It differs slightly from most supervised learning in two ways. First, it is conventionally assumed that samples are independently and identically distributed, whereas we foeus the case where the sample at the current time-step depends the previous
2

---

Under review as a conference paper at ICLR 2022
sample. Second, it is often assumed that the learner has sufficient capacity to closely approximate the target function, whereas we assume that the target function is more complex than the learner. The best approximation continually changes in this problem the imput distribution is non non-stationary, and the target function has high complexity. Therefore, there a need for continual adaptation.
The target function in the Bit-Flipping problem is represented by multi-layered target network. The target network has two layers weights. We limit the learning networks to networks with the same depth. This allows us to control the relative complexity of the target function and learner. If the target network has a lot more hidden units than the learner, then the target function is more complex than the learner. We set the target network be wider.
Output
Weights U(-1, 13
The input at time step t. Xt, is binary vector of size m. Here, X; E {0, 1)"h where each element, Fi,t E {0. 1} for i in 1.... m. After every T time-steps, one of the first bits is randomly selected and its value is flipped; the values of these / bits do not change at other times. We refer to the first ʃ bits slipping bits. Each of the next m f f bits is randomly sampled from U' 1} at every time- step. The value of T allows us to control the correlation among the consecutive values of the input. Note that when a flipping bit flips, the input distribution changes. We use m = 20, f 15, and T 1e4.
X x,
X/41 -. m Input
Nipping bits i..d bits
Figure The input and target function generating the output in the Bit-Flipping problem. The input has m m+1 1 bits. One of the flipping bits is chosen after every T time-steps, and its value is flipped. The next f bits are i.i.d. at every time-step. The target func- tion is represented by a neural network with a single hidden layer of LTUs.
In the target network, all weights are randomly sampled from U[-1,1). The activation function is Linear Threshold Unit (LTU), McCulloch (1943). The output of an LTU, with input x; is I if m U, > 0g else 0. Here, v is the input weight vector. We set 8, (m + 1) * 3 S,, where S, the number of input weights with the value of - and B [0, 1]. This form of LTU is taken from Sutton and Whitehead, (1994). We use B 0.7. The output of the network at time-step I is a scalar Y/ E R. Figure 1 shows the input and the target network. The Bit-Flipping problem is regression problem; we use the squared error to measure performance.
2.2 PERMUTED MNIST
10
We use an online variant of the Permuted MNIST problem (Zenke et al., 2017). They used this problem with just 10 non-stationarities. This is an image classification problem with 10 classes. The images in permuted MNIST are generated from the images in the MNIST dataset by randomly permuting the pixels in the images. We present the images sequentially and measure online classification accuracy. The MNIST dataset has 60,000 images. We present these 60k im- ages in random order, and after all the images have been presented, we use a single permutation to change all the images. This cycle presenting all the 60,000 images and then changing the permutation of pixels can be continued indefinitely, which allows us to create a long-term continual learning problem.
Figure 2: Target selection in Semi-stationary Reacher is a two-step process. First, one of the 100 sub-squares is se- lected for 30 episodes. Sec- ond, for each episode, a new target point is randomly se- lected inside the current sub- square. Here, the target se- lected from sub-square (2,8).
2.3 NON-STATIONARY RL PROBLEMS
We use two non-stationary RL problems and both are continual variants of a corresponding Pybuilet (Coumans and Bai, 2016) problem. Non-stationary variants needed all the problems in Pybullet are stationary. In our problems, either the environment or the distribution of input changes after pre-specified time, making it necessary for the learner to adapt.
Slippery Ant: In this problem, change the friction between the agent and the ground of the standard Pybullet Ant problem. In the standard problem, the value of the friction is 1.5. We change the friction after every 10M time-steps by log-uniformly sampling it from [1e [1e-4, 4, 1e4].
3

---

Under review as a conference paper at ICLR 2022
Semi-stationary Reacher: This problem is modification of PyBullet's Reacher problem. In Reacher, the goal is to get the tip of an arm to a target point in a square. At the beginning of each episode, new target point is uniformly randomly chosen inside the square. Each episode lasts for 150 time-steps. In our problem, we change the distribution of input after every 30 episodes (4500 time-steps). We do so by dividing the square into 100 sub-squares, shown in Figure 2. At the beginning of each episode, the target point is randomly sampled from within a sub-square. This sub-square is changed after every 4500 time-steps when a new sub-square randomly selected.
BACKPROP LOSES THE ABILITY TO ADAPT UNDER EXTENDED TRACKING
In the Bit-Flipping problem, the learning networks is the network that is used to predict the output. This network has a single hidden layer with 5 hidden units, while the target network has one hidden layer but 100 hidden units. So, the target network is more complex than the learning network. Because the input distribution is changing over time and the target function is more complex than the learner, the best approximation continually changes and thus, there is need to track the best approximation.
We first used Backprop track the best approximation. Tracking with Backprop the learning process in an online supervised learning problem consists of randomly initializing the weights and updating them after every example using the gradient with respect the loss on the current example. That is, we used Backprop in an incremental manner without any mini-batches.
We studied learning networks with six different non-linear activations: tanh, sigmoid, ReLU (Nair and Hinton, 2010), UU (Mass et al., 2013) with a negative slope of 0.01, ELU (Clevert et al., 2015), and Swish (Ramachandran al., 2017). We studied all these networks when they learned using SGD or its variant Adam (Kingma and Ba, 2015).
We used uniform Kaiming distribu- tion (He et al., 2015) to initialize the learning network's 's weights. The dis- tribution is (--,,)), with bound, b
1.4
step-size 0.003
step-size 0,003
step-size 0.01
Online
1
1.2
Squared
Error 0.8
1.0
gain V xum_impptee where gain is chosen such that the magnitude of inputs does not change across lay- ers. For tanh, Sigmoid, ReLU, and Beaaky-eeLU gain is 5/3, 1, /2, and 22/(1 a?) respectively. For ELU and Swish, we used gain = v2, as was done in the original works by Clevert et al. (2016) and Ramachan- dran et (2017).
3
(100 runs)
0.6
step-size 0.001
0.8
step-size 0.01
step-size 0.001
0.6
0
0.5M
1M
0.5M
1M
tanh
ReLU
Example Number
Figure 3: The learning curve on the Bit-Flipping problem using Backprop. Surprisingly, after performing well ini- tially, the error goes up for all step-sizes and activation func- tions. Backprop's 's ability to track becomes worse under tended tracking on the Bit flipping problem. For Relu, its performance gets even worse than the linear learner.
We used a linear tracker as a baseline in the Bit- Bit-Flipping problem. The linear tracker predicts that output as linear combination of the inputs. The weights for the linear combination are learned with SGD. We chose the step-size parameter that had least total error.
We ran the experiment on the Bit-Flipping problem for 1M examples. For each activation and value of step-size, performed 100 independent runs. For different activations and values of the step-size parameter, the same 100 sequences of data (input-output pairs) were used.
Figure 3 shows the squared error of different learning networks with different step-sizes during the first 1M examples. We bin the online error into bins of size 20,000. The shaded region in the figure shows the standard error of the binned error. To display the degradation for step-size of 0.001 more clearly, we ran the experiment for 5M examples, and the results are presented in Appendix A. In Appendix A, we also show similar results when used Adam instead of SGD.
In the permuted MNIST problem, we used a feed-foree ferw-forward network with three hidden layers with 2000 hidden units and ReLU activations and SGD for updates.
Figures and 10 show that in the Bit-Flipping problem, for all reasonable step-sizes, the error with SGD either increases significantly. Figures 11 and 12 in the appendix show similar results with
4

---

Under review as a conference paper at ICLR 2022
Adam. Figure 4 shows that in the permuted MNIST problem, the performance for all steps-sizes gets worse over time. The degradation in performance is slower for smaller step-sizes. From these results, we conclude that Backprop and its variants are not suitable for extended tracking they continue to lose their ability to adapt over time.
The weights of the learning network different at the be- ginning and after a few gradient-bare based updates. In the begin- ning, they were small random weights; however, after learning for some time, the weights became optimized to reduce the error for the most recent input distribution. Thus, the start- ing weights to get to the next solution have changed. As this difference in the weights the only difference in the learn- ing algorithm across time, the initial weight distribution must have some special properties that allow the learning network to adapt fast. The initial random distribution might have many properties that allow fast adaptation, like diversity of features, non-saturated features, small weight magnitude etc.
0.96
step-size size 0.003
Online 0.35 Classification Accuracy 0.34
step-size 0.03
0.93
step size 0.001
0.32
32M
Example Number
The online classification accuracy of a deep ReL ReL.Unnetwork on Permuted MNIST. The online accuracy is binned among bins of size 60,000. The performance of Backprop gets worse over time for all step-sizes, meaning that Back- prop loses its ability to adapt under extended tracking.
Figure 4:
4 CONTINUAL BACKPROP
The benefits of initializing with small random numbers in Backprop are only present initially, initialization with small random weights only happens at the start. This special initial- ization makes Backprop temporally asymmetric it does special computation the beginning which is not continued later. But continually learning algorithms should be temporally symmetric as they should do similar computations at all times.
One way to extend the benefits of initialization throughout learning is by continually injecting ran- dom features. feature refers to hidden unit in a network. However, we are interested in contin- ually learning methods that have similar memory requirements at all times because methods whose memory requirements grow with the amount of data are not scalable. So we also need to remove some features without affecting the already learned function too much. The features to be replaced can be chosen either randomly or using some measure of utility.
Continually replacing low utility features with new random features is natural way to extend the benefits of initialization to all times while using constant memory. We use generate-and-ter pro- cess to continually replace low utility features with new random features. This process has two parts. First is a generator that provides new random features from the initial distribution. The second is a tester that finds low utility features and replaces them with features proposed by the generator. Our generate-and-tess process is applicable to arbitrary feed-forward networks, modern activations, and optimizers. While the prior algorithm (Mahmood & Sutton, 2013) was limited to learning networks with a single hidden layer, only one output, LTU activation and could only use SGD to learn.
The generator creates new features by randomly sampling from the distribution that was used to initialize the weights in the layer. When a new feature is added. its outgoing weights are initialized zero. Initializing as zero ensures that the newly added feature does not affect the already learned function. However, initializing the outgoing weight to zero makes it vulnerable to immediate re- placement. The new features are protected for maturity-thresholl m, number of updates.
The tester finds low utility features and replaces them. At every time-step, replacement-ran P. fraction of features are replaced in every layer. Our utility measure has two parts: first measures the contribution of the feature to the next features, and second measures features" ability to adapt.
The cirributtoo-uttlliny is defined for each connection or weight and for each feature. The basic intuition behind the contribution part is that magnitude of the product of feature activation, and out- going weight gives information about how useful this connection to the next feature. the features contribution to the next feature is small, it can be overwhelmed by contributions from other features. Then the current feature is not useful for the consumer. The same measure of connection utility was proposed by Hu al. (2016) for the network pruning problem. We define the contribution-utilin of feature as the sum of the utilities of all its outgoing connections. The contribution-utin utility is measured running average with a decay rate, 7). In feed-forward neural network, the contribution-utilit
5

---

Under review as a conference paper at ICLR 2022
C1ii,t, of the ith features in layer at time t is updated as
CLitt = (1 - 11 11
Σ
w. + 17 C2,,t-11
where Mesa is the output of the ith feature in layer at time t, Wr,,k, is the weight connecting the ith unit in layer to the kth unit in layer 1 + at time n141 is the number of units is layer /
In the network pruning problem, the connections are removed after learning is done. However, generate-and-test is used during learn- ing, so we need to consider the effect of the learning process on the contribution. Specifically, we need to consider the part of the contribution that will be transferred to the bias when a feature is re- moved. When feature is removed, SGD will transfer the average part of the contribution to the bias unit over time. We define, the mean-corrected contribution utility, 21,i, is the magnitude of the product of connecting weight and input minus its average value,
W
Lik
h,
W --
fL. (1- n) h + 17 f1,11-1,
(2)
f1.10-1
fu.. = 1- --rpaia
(3)
hu, fi,l
*
Ek=1
2j=1 1,j.1
21, = =(1 (1-n) * Ihi,a - fuseel Σ
luraae + 21,,1-1, (4)
Figure 5: A feature/hidden- unit in a network. The util- ity of a feature at time t is the product of its contri- bution utility and its adapta- tion utility. Adaptation util- ity is the inverse of the sum of the magnitude of the incom- ing weights. And, contribu- tion utility is the product of the magnitude of the outgo- ing weights and feature acti- vation (hz,i) minus its average (fi.)). fú, is running average of ht,4-
where h is features' output, W...kkk. e is the weight connecting the feature the ktth unit in layer ( + 1, n4+1 the number of units is layer + 1, onia is the age of the feature at time t. Here, fi.,, is a running average of hi,it and ft.i, is the bias-corrected estimate.
We define the adaptation -utliitttttttttu as the inverse of the average magni- tude of the features' input weights. The adaptation-utiliyy captures how fast a feature can adapt. The inverse the weight magnitude reasonable measure for the speed of adaptation for Adam-type optimizers. In Adam, the change in a weight in single update is either upper bounded by the step-size parameter or a small multi- ple of the step-size parameter (Kingma and Ba, 2015). So, during each update, smaller weights can have a larger relative change the function they are representing.
We define the overall utility of feature is the product of its contribution-utiliy and adaptation- utility. The overall utility, it, becomes
11LS fueol Eesl |.L.keel
U1 (1 (1-1)
17 u1i
(5)
Σn |11-1.j..1
11,,1-11
01,4, 1 1- 7/
(6)
Figure describes the utility for feature. Our utility measure is more general than the one proposed by Mahmood and Sutton (2013), as ours applies to networks where features have multiple outgoing connections. In contrast, theirs was limited the case when features with one outgoing connection.
The final algorithm combines Backprop with our generate-and-test algorithm to continually inject random features from the initial distribution. We refer to this algorithm as Continual Backprop (CBP). CBP performs gradient-descent and a generate-and-test step at every time step. Algorithm specifies the CBP algorithm for a feed-forward neural network. We describe CBP with Adam in Appendix B which contains adam specific details. The name "Continual" Backprop comes from an algorithmic perspective. The Backprop algorithm, as proposed by Rumelhart et al. (1987), had two parts, initialization and gradient descent. However, initialization only happens initially, Backprop is not a continual algorithm as it does not do similar computation at all times. On the other hand, CBP is continual performs similar computation all times.

---

Under review as a conference paper at ICLR 2022
Online Squered Error
Unwer
12
Mean Squared Error runs)
Unner
(100 runs)
10-
00
BP
L0
Linear
BP
/vawy Online
3.
04-
17
Online form
0.4
36-
CDP
CDP
CBP
62 TRoK raesi aam
com tanh
X1
Tean saan 8.31 10l
tanh
ReLU
ReLU
Example Number
Tunable parameter (repaacemeen--ttteweegg---eae-
Figure 6: The learning curves and parameter sensitivity plots of Backprop(BP), Backprop with 1.2, Backprop with Online Normalization, and Continual Backprop (CBP) on the Bit-Flipping problem. Only CBP has non-increasing error rate in all cases. Continually injecting randomness alongside gradient descent, CBP, is better for continual adaptation than just gradient descent, BP.
Algorithm 1: Continual Backprop (CBP) for a feed forward neural network with L hidden layers
Set: step-size a, replacement rate p. decay rate 7, and maturity threshold (e.g. 10-4, 10-4, 0.99, and 100)
Initialize: Initialize the weights Wo, Wr. Let, wt be sampled from a distribution di Initialize: Utilities u uy average feature activation fi,..., f. and ages a ay. to 0
for each input 24 do
Forward pass: pass input through the network, get the prediction. ÿt Evaluate: Receive loss (3a, ût)
Backward pass: update the weights using stochastic gradient descent
for loyer 1 in 1 L. do
Update age: a +=1
Update feature utility: Using Equation 5
Find eligible features: Features with age more than m
Features to replace: + of eligible features with smallest utility, let their indices be r
Initialize input weights: Reset the input weights wi-1(r] using samples from di
Initialize output weights: Set wy[r] to zero
Initialize utility, feature activation, and age: Set Utrr, fr,,,t and ai.r, to 0
Continual Backprop in semi-stationary problems
0.96
OnlineNorm
CBP (p=1o-5)
The most common ways to keep the weight distribution close to the initial distribution are 1.2 regularization and BatchNorm. So, we use these two methods along with BP and compare them to CBP on semi-stationary problems. As we study online problems and we consider incremental updates, we used Onli- neNorm (Chiley et al., 2019), an online variant of BatchNorm.
CUP (p=
0.95
Online Classification Accuracy
DP
CEP (p-1e-4)
0.94
Figure 6 shows the online squared error for various learning algorithms with tanh and Relu networks. The online error is binned into bins of size 20k. The results are averaged over 100 runs. For all configurations of parameters for all algo- rithms, we used the same data: they had the same sequences of input-output pairs. All algorithms used SGD with step- size 0.01. For CBP and BP+L2, we used replacement-rate and weight-decay values that had the least total error over 1M examples. Figure also shows the sensitivity of CBP and BP+1. to replacement-race and weight-decay respectively. OnlineNorm has two main parameters; they are used for computing running averages. The de- fault values of these parameters are 0.99 and 0.999. We chose the best set of parameters from (0.9, 0.99, 0.999, 0.9999} * {0.9, 0.99, 0.999, 0.9999}. In Appendix A, we show the performance of these algorithms with other activation functions. In Appendix B, we also show the performance of these algorithms using Adam. For all the experiments with CBP in this paper, we use = 0.99. And, for all experiments where use CBP with SGD, used m 100.
Example Number
The online classifica- tion accuracy of various algorithms on Permuted MNIST. The perfor- mance of all algorithms except CBP degrade over time. CBP maintains a good level of perfor- mance for a wide range of replace- ment rates, p.
Figure 7:
Figure shows that all three algorithms: CBP, BP+L2, and BP+OnlineNorm,, perform significantly better than BP. However, NnoOnlineNorm is not stable for any of the activations as its performance
7

---

Under review as a conference paper at ICLR 2022
degrades over time. In comparison, BP+L2 regularization is stable for the tanh network but not the ReLU network. The purameter-sensitivity plot in Figure 6 shows that CBP and BP+L2 perform better than BP for most values of replacement-rate and weight-decay. And that CBP is much less sensitive to the parameter values than 1.2. For ReLU,, CBP significantly outperformed BP+L.2.
In Appendix C, we do an ablation study for different parts of our utility measure and compare it to other baselines like ran- dom utility and weight magnitude based utility (Mahmood and Sutton, 2013). We find that all the components of our utility measure are needed for best performance.
0.1
BP
Mcan
output
weight 0.06
OnlineNorm
magnitude
L2
CBP (p=1e-6)
Figure 7 shows the performance of various learning algorithms on Permuted MNIST. We used SGD with step size of 0.003.
0.02
We binned the online accuracy into bins of size 60,000. The results are averaged over ten runs. And, for Online we did a parameter sweep for {0.9, 0.99, 0.999, 0.9999) {0.9, 0.99, 0.999, 0.9999). For L2, we chose the best weight- decay among {le 3, le- 4, le- 5, le (e--6). 6}.
6Mt Numberr
12M
Example Number
Figure 8: Evolution of the mean magnitude of the weights in the last layer with various learning algorithms on Permuted MNIST. One of the benefits of initial dis- tribution is small weight magni- tude, however, the weight magni- tude continues to increase for BP. L.2 is able to stop the increase in the weight magnitude, but its perfor- mance still gets worse. This sug- gests that small weight magnitude is not the only important benefit of the initial distribution. Other fac- tors like diversity of features could also be playing an important role.
Figure show that among all the algorithms, only CBP is sta- ble its performance does not degrade over time. The perfor- mance of all the other algorithms degrades over time.
Figure 8 shows that the average weight magnitude in the last layer increases over time for BP. Large weight magnitudes are problematic for both SGD and Adam. For SGD, large weights lead to blown-up gradients for the inner layer, which leads to unstable learning and saturated features. On the other hand, Adam has an upper bound on the weight change in a single update. Due to this upper limit on the effective change in each weight, the relative change in the function is smaller when the weights are large, which means slower adaptation. Both Adam and SGD on their own are unsuitable for Continual learning.
Continual Learning in non-stationary RL problems
We used the PPO algorithm as described by Schaulman et al. (2017). We used separate networks for policy and value function, and both had two hidden layers with 256 hidden units. We used tanh activation as it performs the best with on-policy algorithms like PPO (Andrychowicz et al., 2020).
To use CBP with PPO, we use CBP instead of BP to update the networks' weights. Whenever the weights are updated using Adam, we also update them using generate-and-test. call the resulting algorithm Continual PPO, we describe it in Appendix D.
Slippery Ant
Semi stationary Reacher
2:00
Continual-PPO
Continual-PPO
1400
We
Undiscounted episodic return (30 runs)
PD+L2
6-
PPO
PPO
Figure 9 shows the performance of PPO, PPO+L2, and Continual PPO (CPPO) for Lecun initializa- tion (Lecun et al., 1998) on our non-stationary RL problems. For L2, we chose the best weight-decay, and for CPPO we chose the best replacement-rate, maturity-threshol pair. All other parameters were the same in PPO, PPO+L2, and CPPO. We describe the param- eters in Appendix D. In the figure, the episodic return is binned among bins of 40,000 time-steps.
50M
103M
25M
Time step
Figure 9: Performance of PPO, PPO+L.2, and Continual- PPO on non-stationary RL problems. In both problems, the performance of PPO degraded rapidly. In contrast, the per- formance of Continual-PPe and PPO+L2 either degraded slowly or remained at a high level.
Figure shows that the performance of PPO degrades a lot both of our non-stationary RL prob- lems. This degradation is similar to the degradation of Backprop's 's performance on semi-stationary problems, where it performed well initially, but its performance got worse over time. Figure shows that both CPPO and PPO+L2 perform substantially better than PPO in both non-stationary
8

---

Under review as a conference paper at ICLR 2022
RL problems. On Slippery-ant CPPO performed best, where it continually performed as well it did initially. While on cemi-stationary Reacher, all algorithms fail.
RELATED WORKS
Continual learning or lifelong learning has been studied for long time from several different per- spectives. Ring (1998) considered it as reinforcement learning problem in Man-Markovian en- vironment. On the other hand, Caruana (1998) considered it as a sequence of tasks. There are two major goals in continual learning: memorizing useful information and adapting to the changes in the data stream changes by finding new useful information. Memorizing useful information is difficult for current deep-learning methods as they tend to forget previously learned information (McCloskey and Cohen, 1989; French, 1997; Parisi et 20.,2 Various ingenious methods have been proposed to tackle the forgetting issue (Kirkpatrick et al., 2017: Aljundi et al., 2018; Riemer et al., 2019; Javed et al., 2019); however, it still remains an unsolved problem. In this work, we focused on continually finding useful information but not on remembering useful information.
Multiple recent works have focused on adapting to the changes in the data stream by finding new useful information (Rusu et al. 2016; Finn al., 2017; Wang et al., 2017; Nagabandi et al., 2019; Finn et al., 2019). However, none of the prior works studied problems with thousands of non- stationarities. The problem settings in most works were offline as they had two separate phases, one for learning and the other for evaluation. And in the evaluation phase, there were only a handful of non-stationarities. On the other hand, Finn et al. (2019) studied the fully online problem setting- Although this works investigated the interesting, fully online case, there were still a small number of non-stationarities. Also, the algorithms proposed by Finn et al. (2019) are not scalable as they require storing all the data seen so far. The closest work to ours is by Rahman (2021), where they studied learning in supervised learning problems where the target function changes over time.
Prior works on the importance of initialization have focused on finding the correct weight magnitude to initialize the weights. Glorot et al. (2010) showed that it is essential to initialize the weights so that the gradients do not become exponentially small in the initial layers of network with sigmoid activations. Sutskever et al. (2013) showed that initialization with small weights is critical for sig- moid activations they may saturate if the weights are too large. More recent work (He al., 2015) has gone into ensuring that the input signal's 's scale of magnitude is preserved across the layers in a deep network. Despite all these works on the importance of initialization, the fact that its benefits are only present in the beginning but not continually has been overlooked as most of these works focused on offline learning, where learning has to be done just once.
6 CONCLUSION AND LIMITATIONS
In this work, we studied learning problems that have large number of non-stationarities. We showed that in these problems, the performance of Backprop degrades over time, and it loses its ability to adapt. In other words, we showed that neural networks have a "decaying plasticity" prob- lem. Ours is the first result showing this degradation in Backprop's 's ability to adapt. This result means that Backprop can be unsuitable for continual adaptation.
We proposed the Continual Backprop (CBP) algorithm to get a robust continual learning algorithm where both components of Backprop are present continually. CBP extends the benefits of special random initialization throughout learning using generate-and-te algorithm is the first one that compatible with modern deep learning. We showed that CBP could continually adapt even after thousands of non-stations non-stationarities. Additionally, we showed that Backprop with 1.2 regularization is also able to adapt continually in some problems, but it fails in many others.
process. Our generate-and and-test
The main limitation of our work is that our tester is based on heuristics. Even though it performs well, future work on more principled testers will improve the foundations of Continual Backprop. Continual learning is becoming more and more relevant in machine learning. These systems can not use Backprop or its variants in their current form, as they are not stable in non-stationary problems. We expect that by establishing Backprop's 's degrading adaptivity, our work will invite many solutions similar to how many works have addressed "catastrophic forgetting". We foresee that our work will open path for better Continual Backprop algorithms.

---

Under review as a conference paper at ICLR 2022
Ethics Statement: In this work, we did fundamental research in continual learning. In the long term, our work may help develop continually learn systems for the real world. Continual learning can enable learning systems that scale with data and experience under constant memory requirements. This scaling will make learning systems accessible to a large part of the population, who lack easy access to massive computation, which is essential for current Al systems. Continual learning can also enable learning in remote locations where current expensive methods face difficulty due to limited on-site connectivity and memory resource, As for any potential negative impacts, they are not unique to our work, and instead, they will be shared with other works on continual learning.
Reproducibility Statement: To ensure the Reproducibility of the experiments in this paper, we will make the source code for all the algorithms and environments publicly available after acceptance. To ensure that our work is fully accessible to the public, we used a publicly available dataset (MNIST) and simulator (PyBullet) in our experiments. Finally, to replicate the experiments, we have given the details of all the hyperparameters in the main text and Appendix D.
REFERENCES
Andrychowicz, M., Raichuk, A., Stanczyk, P., Orsini, M., Girgin, S., Marinier, R., ... & Bachem, O. (2021). What matters for on-policy deep actor-critic methods? a large scale study, In International Conference on Learning Representations.
Al-Shedivat, M., Bansal, T., Burda, Y., Sutskever, I. Mordatch, I., & Abbeel, P. (2018). Continuous adapta- tion via meta-leaming in nonstationary and competitive environments. International Conference Learning Representations.
Aljundi, R., Lin, M., Goujaud, B., & Bengio, Y. (2019). Gradient based sample selection for online continual learning. Advances in Neural Information Processing Systems.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems (pp. 1877-1901).
Caruana, R. (1997). Multitask learning. Machine learning. 28(1),41-75. 41-75.
Chiley, V. Sharapov, Kosson, A., Koster, U., Reece, R., Samaniego Fuente, S., ... James, M. (2019). Online normalization for training neural networks. Advances in Neural Information Processing Systems, 32, 8433-8443.
Clevert, A., Unterthiner, T., and Hochreiter, S. (2016). Fast and accurate deep network learning exponen- tial linear units (elus). In International Conference on Learning Representations.
Coumans, E., and Bai, Y. (2016-2021). Pybullet, python module for physics simulation for games, robotics and machine learning. bttp: /py.ulllpppuulttt... org
Finn, C., Abbeel, P., & Levine, S. (2017, July). Model- -gnost-c meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (pp. 1126-1135) PMI.R.
Finn, C., Rajeswaran, A., Kakade, S., and Levine, S. (2019, May). Online meta-learning. In International Conference on Machine Learning (pp. 1920-1930) PMLR.
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, (2018, April). Counterfactual multi-agent policy gradients. In Proceedings of the AAAT Conference on Artificial Intelligence (Vol. 32, No. 1).
French, R. M. (1999). Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4), 135.
Glorot, X., & Bengio, Y. (2010, March). Understanding the difficulty of training deep feedforward neural networks. In Proceedings the thirteenth international conference on artificial intelligence and statisties (pp. 249-256). JMLR Workshop and Conference Proceedings.
He. K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human level performance imagenet classification. In Proceedings of the IEEE international conference computer vision (pp. 1026- 1034).
Hengyuan Hu, Rui Peng, Yu- Wing Tai, and Chi-Keung Tang. Network trimming: data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.0333 2016. Hoi. S. C., Sahoo, D., Lu, J.. and Zhao, P. (2018). Online learning: A comprehensive survey. arXiv preprint 1.02:2880200877
Javed, K., & White, M. (2019). Meta-leaming representations for continual learning. Advances in Neural Information Processing Systems.
10

---

Under review as a conference paper at ICLR 2022
Kaelbling, L. (1993). Learning in embedded systems. MIT press.
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2020). Towards Continual Reinforeement Learning: Review and Perspectives. arXiv preprint arXiv:2012.1340
Kingma, D. P. and Ba, J. (2014). arXiv:1412.6995
Adam: A method for stochastic optimization. ar preprint
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J. Desjardins, G., Rusu, A. A., & Hadsell, R. (2017). Overcoming catastrophic forgetling in neural networks. Proceedings of the national academy of sciences, 114(13), 3521-3526.
Krizhevsky, A., Sutskever, I., and Hinton, E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 1097-1105.
LeCun, Y., Bottou, L.. Bengio. Y., and Haffner, P. (1998). Gradient-based learning applied to document recog- nition. Proceedings of the IEEE, 86( 1), 2278-2324.
Maas, L., Hamnun, Y., Ng. Y. (2013, June). Rectifier nonlincarities improve neural network acoustic models, Proc. ieml (Vol. 30, No. 1. 3).
McCloskey. M., and Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation (Vol. 24, pp. 109-165). Academic Press.
Mahmood, A. R., and Sutton, R. S. (2013, June). Representation Search through Generate and Test. In AAAI Workshop: Learning Rich Representations from Low-Level Sensors.
McCulloch, W. S., Pitss W. (1943), A logical calculus ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5(4), 115-133.
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., and Kavukcuoglu, K. (2016, June), Asynchronous methods for deep reinforcement learning. In International conference machine learning (pp. 1928-1937) PMLR.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A.. Veness, J., Bellemare, G., ... Hassabis, D. (2015).
Human-leve control through deep reinforcement learning. nature, 518(7540), 529-533.
Nair, V., & Hinton, G. E. (2010, January). Rectified linear units improve restricted boltzmann machines. In Icml.
Nagabandi, A., Clavera, Liu, S., Fearing, R. S., Abbeel, P., Levine, S., Finn, C. (2019). Learning adapt dynamie, real-world environments through neta-reeinfrreement learning. International Conference on Learning Representations.
Orabona, F. (2019). A modern introduction to online learning. arXiv preprint arXiv: 1912.131...
Parisi, G. I.. Kemker, R.. Part, L., Kanan, C., & Wermter, S. (2019). Continual lifelong learning with neural networks: A review. Neural Networks, 113, 54-71.
Rahman, P. (2021). Toward Generate-and-Ter Algorithms for Continual Feature Discovery (Masters Thesis, University Alberta).
Ramachandran, P., Zoph, B., and Lc, Q (2017). Scarching for activation functions. arXiv preprint 170.0:171
Ring, M. B. (1998), CHILLD: A first step towards continual learning. In Learning to learn (pp. 261-292). Springer, Boston, MA.
Rumelhart, D. E., Hinton, G. E., Williams, R. (1986), Learning representations by back-propggating eITors. nature, 323(6088), 533-536.
Rusu, A. A., Rabinowitz, N. C. Desjardins, G.. Soyer, H., Kirkpatrick, J., Kavukcuoglu, K.... & Hadsell, R. (2016). Progressive neural networks. arXiv preprint 1rXivv1606.046711
Sahoo, D., Pham, H. Q., Lu, J., & Hoi, S. C. Online deep learning: Learning deep neural networks the (2018). In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence DICA 2018, July 13-19, Stockholm (pp. 2660-2666).
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.063:1
Schwarz, Czareekki, W., Luketina, J., July). Progress and compress: A scalable framework for continual learning. In International Conference Machine Learning (pp. 4528-4537). PMLR.
A., Tch, Y. W., Pascanu, R., & Hadsell, R. (2018. 88
11

---

Under review as a conference paper at ICLR 2022
Sebastian Thrun. Lifelong learning algorithns.Leariing learn, 8:181-209, 1998.
Silver, Huang, A., Maddison, C. Guez, A., Sifre, Van Den Driessche, G., ... Hassabis, D. (2016). Mastering the game Go with deep neural networks and tree search. nature, 529(7587), 484-489.
Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013, February). On the importance of initialization and momentum in deep learning. In International conference on machine learning (pp. (139-1147).
Sutton, R. S., Koop, A., and Silver, D.. On the role of tracking in stationary environments.n In Proceedings of the 24th international conference Machine learning, pp. 871-878. ACM, 2007.
Sutton, S., Whitehead, D. (2014, May). Online learning with random representations. Proceedings the Tenth International Conference on Machine Learning (pp. 314-321).
Wang, Y. X., Ramanan, D., & Hebert, M. (2017). Growing brain: Fine-tuning by increasing model capacity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2471-2480).
Whiteson, S.. and Stone, P. (2006). Evolutionary function approxination for reinforcement learning. Journal of Machine Learning Research, 77May):877-917.
Zenke, Fricdemann, Ben Poole, and Surya Ganguli. "Continual learning through synaptic intelligence." In International Conference on Machine Learning, pp. 3987-3995. PMLR, 2017.
Zhang, H., Dauphin, Y. N., and Ma, T. (2019). Residual Learning Without Normalization via Better Initializa- tion. In International Conference on Learning Representations.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
A MORE EXPERIMENTS WITH BACKPROP ON THE BIT-FLIPPING PROBLEM
In Figure 3 showed that the performance of Backprop for ReLU and tanh activations gets worse over time. In Figure 10, we show that the performance of Backprop also degrades for other activation functions.
In Figure and 10 we ran the experiment for 1M examples. For this length of experiment it may seem that the error for smaller step-size 0.001 reduced or stayed the same for some activation functions. In Figure 11, we show that even for step-size of 0.001, if we let Backprop track for longer (SM), the error for all activations increases significantly or stays at a high level. Again, in Figure 11, we bin the online prediction enror into bins of size 20,000 and the shaded region in the figure shows the standard error the binned error.
Next, we evaluate the performance of BP when using Adam instead of SGD. In all the experiments, we used Adam with betas (0.9, 0.999). First, in Figure 12, we show that the performance BP gets worse over time for wide range of step-sizes for all activation functions. Then in Figure 13, show that even for a smaller step-size of 0.001, after tracking for longer (5M examples), the error for all activations increases significantly.
1.2
step size 0.001
1.2
step-size 0.001
1.4
step-size
1.4 step-size 0.01
Online Squared Error (100 runs)
Linear
1.2
0.0031.0
step-size 0 003
0.6
step-size 0.003
0.6
step-size 0.003
0.8
size 0.01
step-size 0.01
stop
step
0.4
0.5M
IM 0.4
0.5M
1M
0.6
0.5M
IM
0.6
0.5M Swish
1M
Sigmold
ELU
Leaky-ReLL
Example Number
Figure 10: Learning curve of BP on the Bit-Flipping problem for various activation functions using SGD. After performing well initially, the performance of Backprop gets worse over time for all activation functions.
B MORE EXPERIMENTS WITH CBP ON THE BIT-FLIPPING PROBLEM
First, we look at the performance of CBP with the remaining activation functions to complement the results shown in Figure The first half Figure shows the online performance various algo- rithms, and the second half shows the average loss over the experiment's full length (IM examples) for various values hyperparameter replacement-res (for CBP) and weight-decay (for BP+L2).
To properly use Adam with CBP, need to modify Adam. Adam maintains estimates of the average gradient and the average squared gradient. CBP resets some of the weights whenever a feature is replaced. Whenever CBP resets weight, set its estimated gradient and squared gradient to zero. Adam also maintains a 'timestep' parameter to get an unbiased estimate the average gradient. Again, whenever CBP resets weight, set its timestep parameter to zero.
13

---

Under review as a conference paper at ICLR 2022
1.2
1.2
1.2
step-size 0.001
step-size 0.001
(Averagec over 103 runs!
0.8
0.8
0.3
Yill y
Online Squared Error
step-size 0.001
0.6
0.5
tanh
Sigmoid
ELU
1.2
1.2
1.2
step-size 0.001
step-size 0.001
Linear
Linear
0.8
Linear
0.3
step sizeeizee 0.001
1M
2M
3M
4M
SM
0.6
0
1M
2M
3M
4M
5M
0.5
1M
2M
3M
4M
5M
ReLU
Leaky ReLU
Swish
Example Number
Figure 11: SGD with smaller step-size the Bit-Flipping problem. After running for 1M steps in Figure 3, it may seem that performance for step-size of 0.001 does not worsen over time. However, after rumning the experiment for longer, 5M steps, found that the error for step-size of 0.001 either increases significantly or remains significantly higher than larger step-sizes.
Linear
step-size 0.01
step-size 0.001
Lirgar
Line WWW step-size
MoN
(Averaged over 100 runs)
0.6
stepisize 0.001
0.6
step-size 0.003
0.6
Online Squared Error
step see-sizee 003
step-size 0.003
step-size size 0.01
step-size 0.02
0.4
500000
1000000
0.4
500000
1000000
D.4
1000000
tanh
Sigmaid
step-size 0.01
step-size 0.003
1.4
step-size 0.01
step-size 001
1.2
3.2
step-size
Linear
0.8
step
incar
0.8
step size 0.001
Linear
0.8
step-size
step-size 0.003
0.5
500000
1000000
500000
1000000
500000
1000000
ReLU
Leaky-ReLL
Example Number
Figure 12: The learning curve on the Bit-Flipping problem using Adam. Again, after performing well initially, the error goes up for all activation function. Backprop's ability to track becomes worse under extended tracking even with Adam.
14

---

Under review as a conference paper at ICLR 2022
1.2
step-size 0.001
1.2
1.2
Linear
Lineer
(Averaged over 100 runs)
Online Squared Eiror
step-size 0.001
step- Stap- 0.001
1M
2M
314
4M
0.6
1M
2M
3M
4M
SM
0.5
1M
2M
3M
4M
SM
tanh
Sigmoid
ELU
step-size 0.001
step-size 0.001
step 0.001
0.8
0.8
0.8
0.6
0.6
ReW
Leaky- ReLu
Example Number
Figure 13: Adam with smaller step-size on Bit-Flipping problem. Like SGD, after running for 1M steps in Figure 11, it may seem that the performance for step-size of 0.001 does not worsen over time. However, after tracking for 5M examples, the error for step-size of 0.001 increase too.
Sigmoid
ELU
Leoky-ReWJ
Swish
Linoe
Lnner
BP
12
BP
Unline
OnilooNom
Linear
10
Linear
Squared
Norm
Error
(100 runs!
8P+12
BP Up-OniinNonn
044
0.
Example Numoor
Unear
Lincar
0P
BP
Meen
BP
Squared
Enor
BP412
0.6
3.9
1100 runs
CBP
CBP
1a-4d lade 0.031 6901 5.1 01
Na-92 Tedd 0.922 0.01
6O1
6.1
1.000 8.001
Tunable parameter (Hap aciment-rate weicht-decav)
Figure 14: Parameter sweep and online performance of various learning algorithm using SGD for step-size of 0.01 on the Bit-Flipping problem. Only CBP and BP+L2 are suitable for continual learning, the performance of all the other algorithms degrade over time. For all activation functions, CBP and BP+L2 perform significantly better than BP for almost all values of hyperprameters. CBP less sensitive to its hyperparameter than BP+L2.
15

---

Under review as a conference paper at ICLR 2022
Linear
Linear
0.8
0.8
0.8
Online
BP
CBP
BP
CBP
BP+L2
Squared
BP+L2
Error
0.2
0
0.5M
1M
0.2
0.5M
1M
0.2
0.5M
IM
(100 runs)
tonh
Sigmaid
ELU
WWW Linear
BP
BP+L2
Linea
0.8
0.8
0.8
BP+
CBP
CDP
0.5M
B.5M
0.5M
Leaky-ReLU
Example
Figure 15: The learning curves of Backprop(BP),, Backprop with L2(BP+L2) and Continual Back- prop (CBP) on the Bit-Flipping problem using Adam with step-size of 0,01. Both BP+L2 and CBP performed robustly on this problem their performance did not degrade over time. The best parameters for CBP and BP+L.2 are chosen separately for each activation. For all activations, CBP either significantly better than BP+L2 or just slightly worse
Linear
Linear
Linear
0.7
gP
0.7
0.7
BP
CBP
Mean
CBP
Squared 0.3 10:06 16:05 0.0001 0.001
0.01
01
1a45
C.0001 0.501 0.11 01
0.5
16:06
18:05
0.0001
0.01
0.01
01
Error
(100 runs)
Sigmoid
ELU
DP
B2
Lineer
Linear
Ineer
BP+L2
577
CBP
CBP
CBP
0.5 16
1e 05 0.0001
0.00
0.01
03
3.5
1e:06
1e05
0.0001
0.001
0.01
01
0.5
1e:06
1e:5
0.0001 DO01
0.01 01
ReLU
Leaky-ReLU
Swish
Tunable paraametetreplacceeent rete/weiigtt
Figure 16: Parameter sweep for CBP and BP+L2 when used with Adam for step-size of 0.01 on the Bit-Flipping problem. Again, for all activation functions both CBP and L.2 perform significantly better than BP for most values of replacement-rate and weight-decay. For some activations like Sigmoid and ELU, BP+L2 performs better than CBP, while for other activations like ReLU and Leaky-ReLU, CBP performs better than BP+L2.
16

---

Under review as a conference paper at ICLR 2022
Algorithm 2: Continual Backprop (CBP) with Adam for feed forward neural network with hidden layers
Set: step-size a, replacement rate p. decay rate 77, and maturity threshold (e.g. 10 10-4, 10-4, 0.99, and 1000)
Initialize: Set moment estimates 31, and Bg (e-g., 0.9, and 0.99)
Initialize: Randomly initialize the weights Wo, ... Wr. Let, W; be sampled from a distribution
di
Initialize: Utilities ui...., u.. average feature activation f1....,fr f. and ages A1,..., ag to 0 Initialize: Moment vectors mi.... my, and vi,... VL to O, where mi Vi are the first and second moment vectors for weights in layer
Initialize: timestep vectors f,, tr to 0, where & is the timestep vector for weights in layer | for each input 222 do
Forward pass: pass input through the network, get the prediction, Dt Evaluate: Receive loss (xx, ùt)
Backward pass: update the weights using stochastic gradient descent
Update timestep: Increase ti,..., tr by
Update moment estimates: Update m1. my, and Vi..., VL, using the newly computed gradients
for layer I in I.
Update age: a; +=1
Update feature utility: Using Equation
Find eligible features: Features with age more than
Features to replace: x of eligible features with smallest utility, let their indices be r
Initialize input weights: Reset input weights w/- using random samples from di Initialize output weights: Set w/[r, to zero
Initialize utility, feature activation, and age: Set Utrr, fi,r, and a r to 0 Initialize moment estimates: Set m;-1 [;, r). mi(r;)) Vi-i rj. and vi(r. :] to
* Initialize timestep: Set 4-1 r]. and f [r. to
These are Adam specific updates in CBP The inner for-loop specifies generate-and-test based updates
In Figures 15 and 16, we show that CBP with Adam performs significantly better than BP with Adam for wide range of replacement-rates.
C ABLATION STUDY FOR THE UTILITY MEASURE
The final utility measure that we used in this work consists of two parts, the contribution-utlitt and the adaptation-utilie Now, we will compare various parts of the utility measure on the Bit-Flipping problem. We use learning network with tanh activation, and we use the Adam optimizer with step-size of 0.01. Additionally, we also compare our utility measure with random utility and weight- magnitude-baret based utility (Mahmood and Sutton, 2013). The results for this comparison presented in Figure 17a We compare the following utility measures.
Random utility: Utility, at every time-step is uniformly randomly sampled from U|0,, 1] Ti = rand(0, 1)
Weight meggntt- based utility: wm at every time-step is updated as:
ni+1
WTRLii = (1 -
Σ
W...kke + um1
k=
Contribution utility, every time-step is updated described in Equation 1
Mean corrected contribution utility, 2, at every time-step is updated as described Equa- tion 2
Adaptation utility, a, at every time-step is updated as:
01,it (1 17 Σn Wessie + 17 a1,i1111
17

---

Under review as a conference paper at ICLR 2022
2100
0.9
Mean-corrected contribution
Overall
1800
Mcan Squared
0.7
Adaptation
1400
Weight-mag
Undiscounted Episodic
(100 runs]
Rendom
nuttin
Return (30 runs)700
PPO
cera-co contribution
Overall
0.3
1e-05
0.0001
0.001
0.01
30M
60M
90M
Replacement-rate
Time-step
(a) Parameter sweep for various utility measures on the Bit-Flipping problem. The overall utility measure performs better than all other measures of utility including the mright-magnitude based utility (Mahmood and Sutton, 2013) and random utility.
(b) Continual PPO with overall utility and mean- corrected contribution utility Slippery Ant. Both util- measures performs lot better than PPO and overall utility perform better than mean-corrected contribution utility.
Figure 17: CBP with different utility measures different problems
Overall utility, at every time-step is updated as described in Equation 5
Next we compared the two best performing utility measures, overall and correccore contribu- tion, on Slippery Ant. The results are presented in Figure 17b. The results show that both utility measures perform significantly better than PPO. But, the overall utility performs significantly better than the other utility measure. This difference in more pronounced near the end, when Continual PPO with the overall utility measure performs almost as well as it did at the beginning.
D CONTINUAL PPO
We used the following values of the parameters for PPO, PPO+L2 and CPPO in our experiments on non-stationary RL problems. For CPPO and PPO+L.2 specific parameters, we chose the best performing values.
Policy Network: (256, tanh, 256, tanh, Linear) Standard Deviation variable
Value Network (256, tanh, 256, tanh, linear)
iteration size: 4096
num epochs: 10
mini-batch size: 128
GAE, A: 0.95
Discount factor, "y: 0.99
clip parameter: 0.2
Optimizer: Adam
Optimizer step-size: le 4
Optimizer As: (0.9, 0.999)
weight-decay (for 1.2): (10-3, 1001 10-5, 10-6)
replacement-rate, maturity-threshae (for CPPO): (10-3,1c)) 1e2),, (10-3 5c2), (10- 5e3). (10-1, 1,52), (10-5, 1e4), (10-5, (10-5,544) 5e4))
18

---

Under review as a conference paper at ICLR 2022
Algorithm 3: Continual PPO
Initialize: All the required parameters for PPO
Initialize: All the required parameters for CBP with Adam in Algorithm 1 in the appendix
for teration 2 2.. do
Collect data: Run the current policy to collect a set of trajectories
for epochs I, 2 do
Divide and shuffle the collected trajectories into mini-batches for each mini-batch do
Compute the objectives for policy and value networks Update the weights of both networks using Adam Update the weights of both networks using senerate-and-test
COMPUTE USED
All the experiments in this work were performed on Intel Gold 6148 Skylake CPUs or NVIDIA Tesla P100 GPUs. On the Bit-Flipping problem, all the algorithms took around 30 minutes for single run with IM examples. The total compute used for all the experiments on the Bit-Flipping problem around 12960 CPU-days took about 8 GPU hours, which totals 110 GPU days for all experiments. Finally, the experiments for non-stationary RL tasks took up to 90hours for 100M time-steps, the total compute used for all experiments on the non-stationary RL problems was 1980 CPU-days or 5.4 CPU years.
36 CPU-years. On the Cermuted-MNIST problem, each
F AFFECT OF THE SPEED OF THE DISTRIBUTION CHANGE ON DECAYING PLASTICITY
Figure 18 shows the online classification ac- deep ReLU-network with
curacy of BP different speeds of distribution change. In the fastest changing distribution, the distribution changes after every 10k examples, while in the slowest changing distribution, it changes after every 1M examples. We used SGD with step- size of 0.01. The network has three hidden lay- ers with 2000 units each.
0.99
change-after
0.95
Online Classification Accuracy
change- after 100k
0.9
change-after 10k
The Figure 18 shows that the drop in perfor- mance is fastest for the fastest changing distri- bution.
0.85
10M
20M
Example Number
Figure 18: The online classification accuracy of a deep ReLU-network on Permuted MNIST with different speeds of distribution change. The time after which the distribution changes varies from 10k to 1M. The online accuracy is binned among bins of size IM. It shows that the speed of the de- cay in performance increases with the increase in the speed of distribution change.
G A DEEPER LOOK IN THE NETWORKS: FEATURE SATURATION AND VANISHING GRADIENTS
Figure 19a shows the number of saturated features for tanh nethnketorr on the Bit-Flipping problem. We call a feature is saturated when
the magnitude of its output more than 0.9. Figure 19b shows the average magnitude of gradients for the weights in the input layer. For different algorithms, the same values of hyper-pann pperrparaeeters used as we did in Figure In both Figure 19a and Figure 19b, the data is binned into bins 20,000 examples.
Figure 19a shows that an increasingly large fraction of features saturate over time for Backprop. When a feature gets saturated, the network loses some of its representation power which causes worse performance. With Backprop, almost 90% of the features are saturated. Figure 19b shows that with BP, the magnitude of the gradient of the input weights becomes significantly smaller after
19

---

Under review as a conference paper at ICLR 2022
0.2
1.0
BP
0.9
Average magnitude the gradient of Input welghts (100 runs)
0.15
Online Norm
Praction saturated features (100 runs)
0.7
CBP
0.1
CBP
0.05
0.3
BP
Online Norm
0.1
0
0.5M
1M
0.5M
1M
Example Number
Example Number
(a) Fraction of saturated features for different algo- rithms. Despite having the least number of saturated features, Online Norm doesn't perform well. This means that feature saturation not the root cause decaying plasticity.
(b) The average magnitude of the gradient for the in- put weights for different algorithms, Again, despite hav- ing a large gradient for the input weights, Online Norm doesn't perform well. This means that the decreasing gradient of the input weights is not the full explanation of the decaying plasticity of BP.
Figure 19: Evolution of feature saturation and the gradient of the input weights for various learning algorithms.
some time. A small gradient means that the weights change less, and the network loses its ability to adapt. This increase in saturated features and the small gradient of the input weights partially explain why BP performs worse after extended tracking. However, the evolution of the level of feature saturation does not correlate with the performance of all algorithms. Figure 6 shows that CBP and L2 have the least online error, but the online eтror with of OnlineNorm much larger than CBP or L.2. However, in Figure 19a, OnlineNorm has the least number of saturated features. Similarly, in Figure 19b, OnlineNorm has a higher input gradient than CBP, but performance is worse than that of CBP. This means that neither feature saturation nor the magnitude of input gradients are fully explaining the behavior of all algorithms.
H AFFECT OF NETWORK SIZE ON DECAYING PLASTICITY
Figure shows the online classification accu- racy of BP with different deep U-Lttwotorrss. Both networks had three hidden layers. The smaller network had just 100 hidden units, while the wider network had 1000 hidden units. We used SGD with step-size of 0.003. Figure 20 shows that the drop in performance is fastest for the smaller network.
0.95
0.93
network:widthh11000
0.91
Classification Accuracy
0.89
network-width- 100 Mil Ma
0.87
0
6M
12M
Example Number
Figure 20: The online classification accuracy of deep Rel.U-networks with different widths on Permuted MNIST The distribution changes after every 60k examples, and the online accuracy binned among bins of size 60k. It shows that the speed the decay in performance larger for smaller network.
20


Reviews:
[{"review_id": "rCb0o_Fzaa4", "reviewer": "Reviewer_Evj3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong Points\n\n* The paper takes one of the most import issues in continual learning: non-stationary online CL setting. For me, the problem itself is real and practical.\n\n* The proposed approach is reasonable and addresses the problem of weight randomnization in CL setting.\n\n* Overall, the paper is well written. In particular, the Related Work section has a nice flow and puts the proposed method into context. Despite the method having limited novelty, the method has been well motivated by pointing out the limitations in SOTA methods.\n\nWeak Points\n\n* The proposed generate-and-test algorithm is built upon previous work by Mahmood and Sutton 2013. So the novelty is limited.\n\n* Although the paper focuses on fast adaptation to new tasks in CL. It is important to evaluate how the proposed method deals with catastrophic forgetting. I would like to see experimental results on the catastrophic forgetting evaluation.\n\n* \"Continual backprop\" in the title and paper does not reflect the attributes of the proposed algorithm. I would suggest changing \"Continual backprop\" to \"Continual reinitialization\" as a more appropriate summary of the paper and the algorithm.\n\n* \"feature\" in the paper is used to refer to a hidden unit in a network (See page 5). However, \"feature\" commonly refers to a vector extracted from an embedding layer in a network. I would suggest that the authors change \"feature\" to \"hidden unit\" in order to avoid misunderstanding.\n\n* Fig. 9, in semi-stationary reacher setting, PPO+L2 performs better than the proposed continual PPO. It's worth discussion the reasons for the results.\n\n", "summary_of_the_paper": "This paper investigates the problem of fast adaptation in a non-stationary online continual learning(CL) setting. It argues that keeping weight randomnization is important to fast adaptation in CL. However, current CL methods only performs weight randomization in the beginning of the algorithm; the weights loss randomness overtime, leading to degraded model performance. The paper presents a continual weight reinitialization algorithm to overcome the issue. In particular, it proposes to evaluate the utility of each hidden unit -- including importance to the current task and adaptation capability. Then selects a set of hidden units with low score and resets their incoming and outgoing weights. The authors conduct experiments to evaluate the performance of the proposed method.\n\n", "summary_of_the_review": "Overall, I vote for marginally accepting. I like the idea of continual reinitialization and handling it by the proposed generate-and-test method. My major concern is about the limited novelty of the paper -- its build upon previous work by Mahmood and Sutton 2013, and some misleading terms such as \"continual backprop\" and \"feature\".\nHopefully the authors can address my concern in the rebuttal period.\n\n[After rebuttal]:\nAfter reading the other reviewer's comments and authors' feedback, I would like to lower my score due to the same concerns as other reviewers as mentioned.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mlMvODZ7x0s", "reviewer": "Reviewer_gcKs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* The shortcomings of SGD have motivated many works on its adaptation for continual learning and several adaptations such as Orthogonal Gradient Descent [1], Stable-SGD [2] and alternative local learning approaches [3,4] have been proposed. A comparison with these methods would help with better assessing the proposed approach.\n\n* Other benchmarks such as split-CIFAR100, split-IMAGENET, CoRe50 are frequently used to benchmark the continual supervised learning approaches. It would be fair to expand the comparison to a subset of these datasets.\n\n* Classification accuracy might not be the best metrics to evaluate the continual leaning potential. Other metrics such as the backward and forward transfer are better suited [5].\n\n[1] Farajtabar, Mehrdad, et al. \"Orthogonal gradient descent for continual learning.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.\n[2] Mirzadeh, Seyed Iman, Mehrdad Farajtabar, and Hassan Ghasemzadeh. \"Dropout as an implicit gating mechanism for continual learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.\n[3] Lindsey, Jack, and Ashok Litwin-Kumar. \"Learning to learn with feedback and local plasticity.\" arXiv preprint arXiv:2006.09549 (2020).\n[4] Madireddy, Sandeep et al. “Neuromodulated Neural Architectures with Local Error Signals for Memory-Constrained Online Continual Learning”, arXiv preprint arXiv:2007.08159 (2021)\n[5] Mai, Zheda,  et al. \"Online Continual Learning in Image Classification: An Empirical Survey.\" arXiv preprint arXiv:2101.10423 (2021).\n", "summary_of_the_paper": "This work highlights the shortcomings of the backpropagation algorithm for continual learning applications and proposed continual backprop that used a generate-and-test method to continually inject random features alongside SGD which enables better learning on non-stationary data streams. This approach is evaluated with the semi-stationary scenario of Bit-Flipping and continual supervised learning with permuted-MNIST as well as on non-stationary RL problems.", "summary_of_the_review": "The approach looks promising, but is limited in terms of the experiments and the comparison with existing approaches that makes it difficult to assess the true potential of this approach", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5n7CigYHgi0", "reviewer": "Reviewer_kzUn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTH\n\nThe topic of learning online, in continual, incremental or any other online setting, is certainly of high and growing interest in the community. Works trying to tackle the problem in its fundamental aspects, starting from the weight adjustment algorithms are very much interesting for the community. In this respect, this paper certainly takes an interesting and somewhat original view into the difficulties of learning online.\n\nAdditionally, the empirical analysis covering both supervised learning and reinforcement learning scenarios is certainly a strong aspect of the paper, at least for what pertains breadth of application (though, as discussed in the following, it comes at the cost of depth of the results).\n\nWEAKNESSES\n\nThe paper has some substantial issues, ranging from solidity of the claims, scholarly value, novelty as well as empirical validation of the paper ideas.\n\nIn terms of claims, I strongly disagree with the sentence in the concluding paragraph of the introduction about “…we contribute to the understanding of why BP and its variants fail in non-stationary problems”.  The paper does not help understanding the root causes of the problem. It  provides empirical results showing that a certain amount of randomness reintroduced dynamically into the model helps in keeping the model able to learn. This is far from explaining why this happens. For instance, I would have expected to see at least experiments to rule-out the fact that the inability to learn is due to gradient vanish (this can be easily plotted by monitoring the gradient magnitude). I had the feeling that the departure from the small weight initialization might cause reduced learning due to larger weights bringing the neurons into the saturated parts of their activation functions. My first intuition was to think about regularizing on the norm of the weights, which is in fact providing effective results in the empirical analysis. Apart from the permuted MNIST case, the L2 regularization seems to be providing nearly the same advantage as the proposed CBP algorithm. \n\nIn terms of scholarly value, the paper is often confusing with respect to the community it is referring to: it is somehow linking the paper both to the continual learning community as well as to the community of learning in non-stationary environments. The paper needs to make a clear choice as the underlying assumptions (e.g. forgetting Vs remembering), the related literature and the related benchmarks differ substantially. Currently, the paper is in between the two worlds, and unsatisfactory in the way it addresses the point above with respect to both. For instance, in the continual learning community, one might wonder on what are the relationships between the proposed method and other weight-constraining/regularization approaches such as EWC and LWF.\n\nIn terms of novelty, the solution proposed in the paper builds heavily on existing approaches, mostly on generate-and-test extended on non-major aspects, such as support for multi-layer perceptrons and non-LTU activations. Hence, I do not see much original content in the proposed CBP algorithm, whereas there might be some truly original insights coming from the intuition of the root causes of BP inability in learning online (but as said, these are underdeveloped in the paper).\n\nThe empirical analysis, as anticipated, could make good use of experiments showing the behaviour of the gradient magnitude across learning iterations. Current experiments, although commendable in the fact that they cover both supervised and reinforcement learning, do not provide substantial proof of CBP being better than existing weight regularization techniques.\n", "summary_of_the_paper": "The paper discusses a modification of the BP algorithm, the Continual BP (CBP), that founds on the intuition that  standard BP cannot work in online learning scenarios due to weights departing from the initial condition of being small and random, after some training experience. The paper contributes with an empirical analysis of the phenomenon as well an algorithm to address the issue. ", "summary_of_the_review": "The topic of the paper is of potential interest for the community and the empirical analysis sufficiently broad. On the negative end, the depth of technical discussion and of the empirical analysis does not support the claims. Originality of the contributed CBP algorithm is minor.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3FVxxURkAZ5", "reviewer": "Reviewer_jf4G", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This paper proposes and tackles a novel important problem in continual learning that has not been observed in prior work.\n- They demonstrate on both proposed solution is able to perform significantly better on this problem and does not degrade in performance over time compared to the vanilla back propagation on feed-forward networks on CV, RL, and bit-flipping problems\n- They show empirical results on a variety of different Feed-forward networks with different hyperparameters and activation functions. They compare against \n\nWeakness:\n- The paper does not compare the tradeoff of their problem and method with the prior work on catastrophic forgetting. It is possible that the problems and their solutions are opposites of each other and would be very useful to study.\n- The paper would benefit from more directly exploring their hypothesis of \"decaying plasticity\" of neural networks by exploring the change in weights, gradients, and loss landscape over time.\n- This paper only demonstrated their algorithm on Feed-forward networks. Since the paper is largely empirical in demonstrating this new problem, it would be significantly stronger if it could demonstrate this problem on Deep convolutional models, and attention networks. The model sizes and datasets are also relatively smaller. Does this problem continue with overparameterized models or other regularization techniques.\n\nQuestions/Suggesiton:\nIt may be useful to position the work compared to \"Neural Rejuvenation\" [1] since the reinitialization method is somewhat similar.\n\nAre the errors shown test errors or training errors? Could this be a problem of overfitting and the proposed method is acting as regularization?\n\nHave you experimented with even lower or higher learning rates? The accuracy for the lowest in the current graphs seems to still converge quite quickly.\n\nCould we compare to a baseline of fully reinitializing the model when the data distribution changes to understand the upper-bound on the expected accuracy?\n\n[1] Qiao, Siyuan, et al. \"Neural rejuvenation: Improving deep network training by enhancing computational resource utilization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.", "summary_of_the_paper": "This paper demonstrates and proposes a solution for a new problem in continual learning which is the inverse of catastrophic forgetting. Compared to prior work, they study problems where the data distribution changes much more rapidly. They demonstrate that backpropagation based optimization loses its ability to adapt when tracking these rapidly changing continual learning problems.  They show a degradation in performance over time on permuated MNIST, non-stationary RL problems, and the bit-flipping problem. They propose a solution to this problem by reinitializing some portion of the weights of every layer. They propose a utility function to choose which layers to reinitialize based on a combination of adaptation-utility and contribution-utility. They demonstrate that utilizing this method, they can achieve better performance that does not degrade over time and that it works in more cases than l2 weight decay.", "summary_of_the_review": "This paper proposes and tackles a very novel problem in continual learning that would have great significance to the field and is important to explore. They are able to demonstrate a somewhat novel effective algorithm that significantly improves performance, and does not degrade over time. Unfortunately, this paper is largely empirical and does not have sufficiently diverse experiments to characterize this problem. Given this, I currently recommend weak rejection for this paper. While it explores a large variety of activation functions and hyperparameters, all the networks are relatively small Feed-forward networks. They compare several methods and demonstrate that regularization is somewhat effective in some cases, but does not fully explore why regularization works or other methods. It currently falls short of fully explaining the root cause of the problem.\n\n=================================================================\nPost Rebuttal:\nI would again like to thank the authors for including the additional experiments on model size and dataset change speed. I believe this paper is borderline, and am updating my review to marginally above the acceptance threshold. The additional experiments improved understanding and I believe this is a very useful direction to explore. I also agree with reviewer gcKs and Evj3, that this should be compared to existing continual learning methods and papers. The other datasets should also be useful to experiment with since they should still demonstrate at least some loss in Neural plasticity with 20 stationaries according to the other experiments. I also agree with reviewer kzUn on the depth of understanding this paper provides. This paper highlights a very useful novel direction to explore continual learning problems and a reasonable solution, but would drastically be improved if it provided much further depth.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies an important newly identified problem in continual learning of rapid adaptation, and proposes the use of a generate-and-test method to continually inject random features alongside SGD, enabling better learning on non-stationary data streams.
Unfortunately the paper remained borderline in the discussions. While reviewers liked the overall research direction and contributions, they also agreed the paper in current form still would benefit from deeper insights into the proposed method, stronger empirical evidence.
Experiments cover broad applications including RL, but don't seem to give very clear advantages over other weight regularization schemes, and other metrics of quality could be added. We appreciate the authors have added additional experiments testing it both for the two important regimes of under- and over-parameterized networks, though those can be expanded.
We are sorry that this good paper remained narrowly below the bar in this case, and hope the detailed feedback helps to strengthen the paper for a future occasion.
Decision: Reject