Self-Normalized Resets for Plasticity in Continual Learning
Authors: Vivek Farias
Adam Daniel Jozefiak
Venue: ICLR 2025 Poster (2025)
Keywords: Continual Learning
Plasticity
Lifelong Learning
Neuron Resets
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Plasticity Loss is an increasingly important phenomenon that refers to the empirical observation that as a neural network is continually trained on a sequence of changing tasks, its ability to adapt to a new task diminishes over time. We introduce Self-Normalized Resets (SNR), a simple adaptive algorithm that mitigates plasticity loss by resetting a neuron’s weights when evidence suggests its firing rate has effectively dropped to zero. Across a battery of continual learning problems and network architectures, we demonstrate that SNR consistently attains superior performance compared to its competitor algorithms. We also demonstrate that SNR is robust to its sole hyperparameter, its rejection percentile threshold, while competitor algorithms show significant sensitivity. SNR’s threshold-based reset mechanism is motivated by a simple hypothesis test we derive. Seen through the lens of this hypothesis test, competing reset proposals yield suboptimal error rates in correctly detecting inactive neurons, potentially explaining our experimental observations. We also conduct a theoretical investigation of the optimization landscape for the problem of learning a single ReLU. We show that even when initialized adversarially, an idealized version of SNR learns the target ReLU, while regularization based approaches can fail to learn.

Published as a conference paper at ICLR 2025
SELF-NORMALIZED RESETS FOR PLASTICITY IN CONTINUAL LEARNING
Vivek F. Farias
Adam D. Jozefiak"
Sloan School of Management
Operations Research Center
Massachusetts Institute of Technology Cambridge, MA 02139, USA vivekf@mit.
Massachusetts Institute of Technology Cambridge, MA 02139, USA joze fozefiaken fiakemit.eeeee
ABSTRACT
Plasticity Loss is an increasingly important phenomenon that refers to the empir- ical observation that as a neural network is continually trained on sequence of changing tasks, its ability to adapt to new task diminishes over time, We intro- duce Self-Normalized Resets (SNR), simple adaptive algorithm that mitigates plasticity loss by resetting a neuron's weights when evidence suggests its firing rate has effectively dropped to zero. Across battery of continual learning prob- lems and network architectures, we demonstrate that SNR consistently attains su- perior performance compared to its competitor algorithms. We also demonstrate that SNR is robust to its sole hyperparameter, its rejection percentile threshold, while competitor algorithms show significant sensitivity. SNR's threshold- based reset mechanism is motivated by a simple hypothesis test that we derive. Seen through the lens of this hypothesis test, competing reset proposals yield subopti- mal error rates in correctly detecting inactive neurons, potentially explaining our experimental observations. We also conduct a theoretical investigation of the op- timization landscape for the problem of learning single ReLU, We show that even when initialized adversarially, an idealized version of SNR learns the target ReLU, while regularization based approaches can fail to learn.
INTRODUCTION
Plasticity Loss is an increasingly important phenomenon studied broadly under the rubric of con- tinual learning (Dohare et al. 2024). This phenomenon refers to the empirical observation that as a network is continually trained on sequence of changing tasks, its ability to adapt to a new task diminishes over time. While this is distinct from the problem of catastrophic forgetting (also studied under the rubric of continual learning (Goodfellow et al.) 2013; Kirkpatrick et al., 2017), it is of significant practical importance. In the context of pre-traiming language models, an approach that continually trains models with newly collected data is preferable to training from scratch (lbrahim) jet al., 2024 Wu et al. 2024). On the other hand, the plasticity loss phenomenon demonstrates that such an approach will likely lead to models that are increasingly unable to adapt to new data. Simi- larly, in the context of reinforcement learning using algorithms like TD, where the learning tasks are inherently non-stationary, the plasticity loss phenomenon results in actor critic networks that are increasingly unable to adapt to new data (Lyle et al. 2022). Figure T illustrates plasticity loss in the 'Permuted MNIST' problem introduced by Goodfellow et al. (2013).
One formal definition of plasticity measures the ability of a network initialized at a specific set of parameters to fit random target function using some pre-specified optimization procedure. In this sense, random parameter initializations (eg- Lyle et al. (2024)) are known to enjoy high plasticity. This has motivated two related classes of algorithms that attempt to mitigate plasticity loss. The first explicitly 'resets" neurons that are deemed to have low "utility' (Dohare et al.] 2023 Sokkr et al.) 2023). A reset re-initializes the neurons input weights and bias according to some suitable random initialization rule, and sets the output weights to zero; algorithms vary in how the utility of a neuron defined and estimated from online data. A second class of algorithms perform this reset procedure
" "Corresponding author. Code: http://ss..//tttbb.coomm
Se1 FNormal zedkesets

---

Published as a conference paper at ICLR 2025
implicitly via regularization (Ash & Adams 2020 Kumar al. 2023b). These latter algorithms differ in their choice of what to regularize towards, with choices including the original network initialization; new randomly drawn initialization; even zero. The aforementioned approaches to mitigating plasticity loss attempt to adjust the training process; other research has studied the role of architectural and optimizer hyperparameter choices. Across all of the approaches to mitigating plasticity loss described above, no single approach is yet to emerge as both robust to hyperparameter choices, and simultancously performant across benchmark problems.
Given some point process consider the task of distinguishing between the hypotheses that this point process has a positive rate (the null hypothesis), or a rate that is identically zero with a penalty for late rejection or acceptance. An optimal test here takes the following simple form: we reject the null hypothesis as soon as the time elapsed without an event exceeds some percentile of the inter- arrival time under the null hypothesis and otherwise accept immediately upon an event. Viewing the firing of a neuron as such a point process, we propose to reset neuron based on a rejection of the hypothesis that the the neuron is firing at positive rate. We use the histogram of past nnter-firing times as a proxy of the inter-arrival time distribution under the null hypothesis. This exceedingly simple algorithm is specified by single hyperparameter: the rejection percentile threshold. We refer to this procedure as self normaazzd resets (SNR) and argue this is a promising approach to mitigating plasticity loss:
1. We demonstrate superior performance on four benchmark problems classes studied in (Do- hare et al. 2023: Kumar et al. 2023b). Interestingly, there is no single closest competitor to SNR across these problems. Many competing approaches also show significant sensitivity to the choice of hyperparameters: SNR does not. We introduce a new problem to eluci- date similar plasticity loss phenomena in the context of language models, and show similar relative merits for SNR.
2. We conduct theoretical investigation of the optimization landscape for the problem of learning a single ReLU. We show that while (an idealized version of) SNR learns the target ReLU, regularization based approaches can fail to learn in this simple setting.
1.1 RELATED LITERATURE
The phenomenon of plasticity loss was discovered in the context of transfer learning (Ash & Adams, 2020 Zillyy al. 2021 Achille al. 2017). Achille et al. (2017) showed that pre-training network on blurred CIFAR images reduces its ability to learn on the original images. In similar vein, [Ash Adams 2020) showed that pre-training a network on 50% of a training set followed by training on the complete training set reduces accuracy relative to network that forgoes the pre-training step. More recent literature has focused on problems that induce plasticity loss while training on sequence of hundreds of changing tasks, such as Permuted MNIST and Continual ImageNet in Dohare et al. (202T), capturing the necessity to learn indefinitely.
Correlates of Plasticity Loss. The persistence of plasticity loss across swathe of benchmark problems has elucidated a search for its cause. Several correlates of plasticity loss have been well observed, namely neuron inactivity, feature or weight rank collapse, increasing weight norms, and loss of curvature in the loss surface (Dohare et al., 2021: Lyle et al.) 2023) Sokar et al.] [2023] Lewandowski al. 2023] Kumar et al. 2020). The exact cause of plasticity loss remains unclear Lyle et al. (2023) have shown that for any correlate an experiment can be constructed in which its correlation with plasticity loss is negative. Nonetheless, these correlates have inspired a series of algorithms and interventions with varying degrees of success in alleviating the problem. However, none consistently performant across architectures and benchmark problems.
Reset Methods. Algorithms that periodically reset inactive or low-utility neurons have emerged as a promising approach (Dohare et al., 2023] Sokar al., 2023, Nikishin et al.) 2022). Continual Backprop (CBP) (Dohare et al., (023) is one such method which tracks utility for each neuron, and according to some reset frequency r, it resets the neuron with minimum utility in each layer. CBP's utility is a discounted average product of neuron's associated weights and activation, heuristic inspired by the literature on network pruning. Another algorithm is ReDO (Sokar et al. 2023), where every 1/1th mini-batch ReDO computes the average activity of each neuron and resets those neurons whose average activities are small relative to other neurons in the corresponding layer, according to a threshold hyperparameter. Two defining characteristics of CBP and ReDO
2

---

Published as a conference paper at ICLR 2025
fixed reset rate and that neurons are reset relative the utility of other neurons in their layer. As we will see, these proposals result in sub-optimal error rates, in a sense we make precise later.
Regularization Methods. L2 regularization has been shown to reduce plasticity loss, but is insuf- ficient in completely alleviating the phenomenon (Dohare et al., 202I; Lyle et al., 2023). While L.2 regularization limits weight norm growth during continual learning, it exacerbate weight rank collapse due to regularization towards the origin. One successful regularization technique is Shrink and Perturb (S&P) (Ash & Adams 2020), which periodically scales the network's weights by a shrinkage factor followed by adding random noise to each weight with scale o. Another approach is perform L2 regularization towards the initial weights referred as L2 Init (Kumar et al., (023b). These methods can be viewed as variants of L2 regularization that regularize to- wards a random initialization and the original initialization, respectively. These methods limit the growth of weight norms while maintaining weight rank and neuron activity by regularizing towards a high-plasticity parameterization.
Architectural and Optimizer Modifications. Architectural modifications such as layer normaliza- tion (Ba et al.) 2016) and the use of concatenated ReLU activations have been shown to improve plasticity to varying degrees across network architectures and problem settings (Lyle et al. [023] Kumar et al.) 2023b). Additionally, tuning Adam hyperparameters to improve the rate at which second moment estimates are updated has been explored with some success in Lyle et al. (2023
2
ALGORITHM
To make ideas precise, consider a sequence of training examples (X,, E X x y, drawn from some distribution f4-- Denote the network by f X' x θ + V/ and let 1 > x y TR be our loss function. Denote by He E H4, the history of network weights and training examples up to time t, and assume access to an optimization oracle O, H4-1 + θ that maps the history of weights and training examples to a new set of network weights. As a concrete example, O, might correspond to stochastic gradient descent.
Let 0r minimize Ep, ((((((;;; 0), Yi)l, denote O = -, and consider average expected regret
T E E(f((((X)))))) O.), Y)) E,,, (XX 02), YO)
1
Plasticity loss describes the phenomenon where, for certain continual learning processes O,, such as those corresponding to SGD or Adam, average expected regret increases over time, even for benign choices of | To make these ideas concrete, it is worth considering an example of the above phenomenon reported first by Dohare et al. (202I).
Example 2.1 (The Permuted MNIST problem). Consider sequence of 'tasks' presented sequen- tially to SGD, wherein each task consists of 10000 images from the MNIST dataset with the pixels permuted. SGD trains over single epoch on each task before the subsequent task is presented. Fig- ure D measures average accuracy on each task; we see that average accuracy decreases over tasks. The figure also shows a potential correlate of this phenomenon: the number of 'dead' or inactive neurons in the network increases as training proceeds, diminishing the network's 's effective capacity.
One hypothesis that seeks to explain plasticity loss is that the network weights obtained from min- imizing loss over some task yield poor initializations for a subsequent task, leading to the inactive neurons we observe in the above experiment. On the other hand random weight initializations are known to work well (Glorot & Bengio 2010), suggesting a natural class of heuristics: re-initialize inactive neurons. Of course, the crux of any such algorithm is determining whether neuron is inactive the first place, and doing so as quickly as possible.
To motivate our algorithm, SNR, consider applying the network (;) hypothetical sequence of training examples drawn i.i.d. from Ht indexed by s. Let Z indicate the sequence activa-
'Specifically, one canonical choice of the sequence of measures μrr considered in all of the literature on this topic is dividing 7' into intervals, each of length. say A. and having jr be constant and equal to μa the ith such interval. If μ4 is itself drawn randomly from some distrihution of measures and A scales faster than constant with T, we would expect average expected regret to scale like a constant; this is certainly the case the optimization problem defining 07 is convex; in which case that constant is zero.
"this notion is formalized in Section 2.1
3

---

Published as a conference paper at ICLR 2025
0.90
0.85
Purcent of Neurons that are Dead
50
Online Task Accuracy
0.80
0.75
0.70
SGD
0.60
SGD
SNR
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Task
Task
Figure 1: Illustration of plasticity loss and its mitigation by SNR during training of multilayer perceptron on the Permuted MNIST problem for a single random seed. For this figure, a neuron is declared dead if it has not fired for the last 1000 consecutive training examples.
Algorithm 1: SNR: Self-Normalized Resets
Input: Reset percentile threshold
Initialize: Initialize weights Og randomly. Set inter-firing time 0; for each neuron for each training example 2; do
Forward Pass: Evaluate f(z;; 0e). Get neuron activations 2t, for each neuron i
Update inter-firing times: For each neuron i, a, + a, 1 if žt, 0. Otherwise, a, Optimize: 0++1 + (H,))
Resets: For each neuron i, reset if ((A 2 aa) S
end
tions of neuron i, and let Al be random variable distributed the random time between any two consecutive activations over this hypothetical sequence of examples. Now turning the actual sequence of training examples, let at count the time since the last firing of neuron i prior to time t. Our (idealized) proposal is then exceedingly simple: reset neuron at time iff P(A!" 2 a S for some suitably small threshold y). We dub this algorithm Self-Normalizer Resets and present it as Algorithm [[] The algorithm requires a single hyper parameter, 77. Of course in practice, the distribu- tion of A!" unknown to and implementable version of Algorithm I simply approximates this distribution with the histogram of inter-firing times of neuron i prior to time
2.1 MOTIVATING SNR AND COMPARISON TO OTHER RESET SCHEMES
Here we motivate the SNR heuristic and compare it to other proposed reset schemes. Consider the following simple hypothesis test: observe a discrete time process Z, {0, 1} which under the null hypothesis Ho is a Bernoulli process with mean > 0. The alternative hypothesis Hi is that the mean of the process is identically zero. A hypothesis test must, some stopping time т, either reject (X, = 1) or accept (X, 0) the null; an optimal such test would choose to minimize the sum of type-1 and type-2 errors (the 'error rate'), and a penalty for delays:
P(X, 1 Ho) + IP(,, = 0) + X (E| Ho] + B[7(H1))
Here the multiplier > 0 penalizes the delay in a decision. X < p/2, the optimal test takes simple form: for some suitable threshold T. reject the null iff Z, for all times up to T: Proposition 2.1. Let T' be the - A(p 2) percentile of a Geometric(p) distribution. Then the optimal hypothesis test takes the form X, 1[2, 0} where min Z, 1) A..
2 As opposed to tracking the histogram itself, we simply track the mean inter firing time, and assume A geometrically distributed with that mean. This requires tracking just one parameter per neuron. Our mean esti- mate is itself computed over fixed length trailing window motivated by the change-point detection approach Besbes & Zeevi 201T); the length of this window is hyper-paramcter.

---

Published as a conference paper at ICLR 2025
Notice that if À p, the percentile threshold above is independent of p. Applying this setup to the setting where under the null, we observe the firing of neuron under i.id. training examples from M4 and neuron considered 'dead' inactive if the alternate hypothesis is true, imagine that p IP(Z 1). Further, we assume À = op (a < 1 1/2); reasonable assumption which models larger penalty for late detection of neurons that highly active. It is then optimal to declare neuron i 'inactive" if the length of time it has not fired exceed the 1 - a(1 - a) percentile of the distribution of A This is the underlying motivation for the SNR heuristic.
Neuron reset heuristics such as Sokar et al. (2023) define (some- times complex) notions of neuron "utility" determine whether not to re-initialize a neuron. The utility of every neuron is computed over every consecutive (say) minibatches, and neurons with utility below a threshold are reset. To facilitate a comparison, consider the setting where neurons that do not fire at all over the course of the mini batches are estimated to have zero utility, and that only neurons with zero utility are re-initialized.
Comparison with Reset Schemes:
This reveals an interesting comparison with SNR. The schemes above will re-initialize neuron after inactivity over a period of time that is uniform across all neurons. On the other hand, SNR will reset neuron after it is inactive for period that corresponds to fixed percentile of the inter-firing time distribution of that neuron. Whereas this percentile is fixed across neurons, the corresponding period of inactivity after which neuron is reset will vary across neurons: shorter for neurons that tend to fire frequently, and longer for neurons that fire less frequently.
We can make this comparison precise in the context of the hypothesis testing setup above: specif- ically, consider two neurons with null firing rates P1 and P2 respectively (P1 < P2), and delay multipliers, X, of apı and ap2 respectively. By Proposition 2.1] under SNR, the first is reset if is inactive for time at least log(o(1-aa)-) -)) loll (lgg) and the second if is inactive for time at least log(a(1 1 (1 1)/ log(1 P2). ). In contrast, for fixed threshold scheme such as Sokar et al. (0023), either neuron would be reset after being inactive for some fixed threshold, say r". Assume r" is set minimize the sum of the error rates of the two neurons while keeping the total delay identical to that for SNR. The proposition below compares the error rate between the two schemes:
-x)
Proposition 2.2. The ratio of total error rate with fixed threshold r to that under SNR scales like
Ω exp log(a(1 log(a(1-a)-) aa) (-2 1
log(1 -P1) log(1 p2)
Now recall that a 1/2 and P1 < P2. The result above then shows that: (a) the error rate under an (optimal) fixed threshold can grow arbitrarily larger than the error rate under SNR the penalty for delay shrinks to zero and (b) the rate at which this gap grows itself scales with the difference the nominal firing rates of the neurons under consideration. This provides insight into the relative merits of using a scheme like SNR in lieu of existing reset proposals: resets under SNR detect changes in the firing rate of neuron faster and more accurately: this matters particularly in situations where there is wide disparity in the nominal firing rates of neurons across the network.
EXPERIMENTS
We evaluate the efficacy and robustness of SNR on a series of benchmark problems from the contin- ual learning literature, measuring regret with respect to prediction accuracy [(y, s/) 1 # v'). As an overview, we will seek to make the following points:
Inactive neurons are an important correlate of plasticity loss: This is true across several archi- tectures: vanilla MLPs, CNNs and transformers.
Lower average loss: Across a broad set of problems/ architectures from the literature, SNR consis- tently achieves lower average loss than competing algorithms.
No consistent second-best competitor: Among competing algorithms, none emerge as consistently second best to SNR.
Robustness to apper-parameter:: The performance of SNR robust to the choice of its single hyper parameter (the rejection percentile threshold). This is less so for competing algorithms.
5

---

Published as a conference paper at ICLR 2025
3.1 EXPERIMENTAL SETUP
Each problem consists of tasks T1, T2...., TN. each of which contains training examples in X' V. A network is trained for a fixed number of epochs per task to minimize cross-entropy loss. We perform an initial hyperparameter sweep over seeds to determine the optimal choice of hyperpa- rameters (see Appendix []. For each algorithm and problem, we select the hyperparameters that attain the lowest average loss and repeat the experiment on 5 new random seeds. random seed determines the network's parameter initialization, the generation of tasks, and any randomness in the algorithms evaluated. We evaluate both SGD and Adam as the base optimization algorithm. as earlier literature has argued that Adam can be less performant than SGD in some continual learning settings (Dohare et al. 2023 Ashley et al. 2021). We evaluate on the following problems:
Permuted MNIST (PM) (Goodfellow et al., 2013; Dohare et al., 2021; Kumar et al. (023b): A subset of 1 10000 image-label pairs from the MNIST dataset are sampled for an experiment. A task consists of a random permutation applied to each of the 10000 images. The network is presented with 2400 tasks appearing in consecutive order. Each task consists of a single epoch and the network receives data in batches of size 16.
Random Label MNIST (RM) (Kumar et al., 2023b; Lyle et al. 2023): A subset of 1200 images from the MNIST dataset are sampled for an experiment. An experiment consists of 100 tasks. where each tasks is a random assignment of labels, consisting of 10 classes, to the 1200 images. A network trained for 400 epochs on each task with a batch size 16.
Random Label CIFAR (RC) (Kumar et al. 2023b; Lyle et al., 2023): A subset of 128 images from the CIFAR-10 10 dataset are sampled for an experiment. An experiment consists of 50 tasks, where each tasks is a random assignment of labels, consisting of 10 classes, to the 128 images. An agent is trained for 400 epochs on each task with a batch size 16. Continual Imagenet (CI) (Dohare et al., 2023; Kumar et al., 2023b): An experiment consists of all 1000 classes of images from the ImageNet-32 dataset (Chrabaszez et al.) 2017) containing 600 images from each class. Each task is a binary classification problem between two of the 1000 classes, selected at random. The experiment consists of 500 tasks and each class occurs in exactly
one task. Each task consists of 1200 images, 600 from each class, and the network is trained for 10 epochs with batch size of 100.
Permuted Shakespeare (PS); We propose this problem to facilitate studying the transformer ar- chitecture in analogy to the MNIST experiments. An experiment consists of 32768 tokens of text from Shakespeare's ' Tempest. For any task, we take a random permutation of the vocabulary of the Tempest and apply it to the text. The network is presented with 500 tasks. Each task consists of 100 epochs and the network receives data in batches of size with a context widow of width 128. We evaluate over 9 seeds.
This experimental setup, for all but Permuted Shakespeare, follows that of (Kumar et al. 2023b), with the exceptions of Permuted MNIST which has its task count increased from 500 to 2400, Random Label MNIST which has its task count increased from 50 to 100, and Random Label CIFAR which has its dataset reduced from 1200 to 128 images. Lyle al. (2023) consider variants of the Random Label MNIST and CIFAR problems by framing them as MDP environments for DQN agents. During training, the DQN agents are periodically paused to assess their plasticity by training them on separate, randomly generated regression tasks using the same image datasets.
3.1.1 ALGORITIIMS AND ARCHITETCURES
Our baseline in all problems consist simply of using SGD or Adam as the optimizer with no further intervention. We then consider several interventions to mitigate plasticity loss. First, we consider algorithms that employ an explicit reset of neurons: these include SNR. Continual Backprop (CBP) (Dohare et al.] 202T). and ReDO (Sokar et al. 2023). Among algorithms that attempt to use reg- ularization, we consider vanilla L2 regularization, L2 Init (Kumar et al.) (023b), and Shrink and Perturb (Ash & Adams, 2020). Finally, as a potential architectural modification we consider the use of Layer Normalization (Ba et al. 2016).
We utilize the following network architectures:
MLP: For Permuted MNIST and Random Label MNIST use MLP identical to that in Kumar ct al. (2023b) which in turn is a slight modification to that in Dohare et al. (2023). CNN: For Random Label CIFAR and Continual ImageNet we use CNN architectures identical to
6

---

Published as a conference paper at ICLR 2025
that in Kumar et al. (203bb) which in turn is slight modification that in Dohare et al. (2023)
Transformer: We use a decoder model with a single layer consisting of heads, dimension 16 for each head, and with 256 neurons in the feed forward layer with ReLU activations. We deploy this architecture on the Permuted Shakespeare problem using the GPT-2 BPE tokenizer (limited to the set of unique tokens present in the sampled text).
3.2 RESULTS AND DISCUSSION
We separately discuss the results for the first four problems (PM, RM, RC, CI) followed by Per- muted Shakespeare; we observe additional phenomena the latter experiment which merit separate discussion.
Optimizer
SGD
Adam
Algorithm
PM
RM
RC
CI
PM
RM
RC
CI
No Intv.
0.71
0.11
0.18
0.78
0.64
0.11
0.15
0.58
SNR
0.85
0.97
0.99
0.89
0.88
0.98
0.98
0.85
CBP
0.84
0.95
0.96
0.84
0.88
0.95
0.33
0.82
ReDO
0.83
0.72
0.98
0.87
0.85
0.67
0.74
0.80
1.2 Reg.
0.82
0.80
0.95
0.83
0.88
0.95
0.97
0.80
1.2 Init
0.83
0.91
0.97
0.83
0.88
0.96
0.98
0.83
S&P
0.83
0.92
0.97
0.85
0.88
0.96
0.97
0.81
Layer Norm.
0.69
0.14
0.96
0.82
0.66
0.11
0.96
0.58
Table 1: Average accuracy the last 10% of tasks the benchmark continual learning problems over 5 seeds. Standard deviations are provided in the extended Table 6
Table shows that across all four problems (PM, RM, RC, CT) and both SGD and Adam, SNR consistently attains the largest average accuracy on the final 10% of tasks. For each competitor al- gorithm there least one problem on which SNR attains superior accuracy by at least 5 percentage points with SGD and at least 2 percentage points with Adam. We also see that there is no consistent second- algorithm with the SGD optimizer while L2 Init consistently the second-best with the Adam optimizer.
Another notable property of SNR its robustness to the choice of rejection percentile threshold. In contrast, its competitors not robust to the choice of their hyperparemter(s). On all but RM with SGD, SNR experiences a decrease of at most 2 percentage points in average accuracy when varying its rejection percentile threshold across the range of optimal thresholds found across experiments. On the other hand, increasing the hyperparameter strength by single order of magnitude, as is common hyperparameter sweep, from the optimal value for L2 Init, S&P, and CBP results in decrease in average accuracy by at least 72 percentage points. See Appendix [[ detailed tables.
Next, we turn our attention the Permuted Shakespeare problem. For the no intervention network with Adam, we see dramatic plasticity loss, the average loss increases from about 0.15 on the first few tasks to 3.0242 on the last 50 tasks; see Figure 2 In Figure g and Figure we see that this plasticity loss is correlated with increasing weight norms in the self-attention and feedforward layers, persistent neuron inactivity, and collapse the entropy of the self-attention probabilities.
Camustrie Mengge Lo23
25
0.3
0.2
SMR
Comulative
S2
0.1
0.0
100
200
300
400
Figure 2: Cumulative average of the loss measured on the final epoch of each task on the Permuted Shakespeare problem. Right panel zooms in to L2 and SNR+L2

---

Published as a conference paper at ICLR 2025
Algorithm
All Tasks
First 50 Tasks
Last Tasks
L2
0.2762
(0.0309)
0.1560 (0.0236)
0.3101 (0.0425)
SNR+L2
0.2177
(0.0196)
0.1370 (0.0169)
0.2551 (0.0454)
No Intv.
2.7397
(0.0140)
1.8164
(0.0295)
3.0147
(0.0250)
L2 Init
1.5052
(0.0437)
1.2931
(0.0486)
1.5262
(0.0420)
SNR
2.6872
(0.0222)
1.7338
(0.0295)
3.0242
(0.0408)
CBP
2,4922
(0.0171)
1.3732
(0.0234)
2.9410
(0.0188)
L2
0.1506
(0.0279)
0.1874
(0.0348)
0.1092
(0.0429)
SNR+L2
0.1402
(0.0246)
0.1549
(0.0107)
0.0909
(0.0177)
ReDO
2.3258
(0.0356)
1.3689
(0.0686)
2.8119
(0.0373)
Table 2: Average loss measured on the final epoch of each task with standard deviations over
seeds on the Permuted Shakespeare problem. Note, L2 denotes L2 regularization applied only to attention weights.
We see that resets are by themselves insufficient in mitigating plasticity loss, providing most marginal improvement over no intervention. This unsurprising since neurons are only present in the feedforward layers, unlike the MLP and CNN architectures in the earlier experiments. As such, regularization appears necessary and we see that, over the last 50 tasks, L2 regularization attains average loss of 0.3101 in contrast 3.0147 and 3.0242 for no intervention and SNR. This improvement in performance coincides with stable weight norms and non-vanishing average entropy of self-attention probabilities for L2 regularization. In contrast to the earlier problems, L2 Init fares worse than L2 regularization and experiences substantial loss of plasticity, although to lesser extent than the no intervention network.
While L2 regularization addresses weight blowup, neuron death remains present; see Figure 3 The average loss with L2 increases from 0.1560, over the first 50 tasks, 0.3101, over the final 50 tasks. This prompts to consider using SNR in addition to L2 regularization. This largely eliminates neuron death (see the right panel Figure 3 while stabilizing weight norms and maintaining entropy of self-attention probabilities, providing the lowest loss (0.2551) over the final 50 tasks. As an ablation, we evaluate performance when applying L2 regularization only to the attention weights, which we denote by L2 and SNR+L2 in T
25
iractive Reurers
200
20
L2 Init
No Intv.
15
SNR
SNR4112
2000
4000
5000
10000
12000 14000
16000
2000
4000
6000
8000
10000
12000
14000 16000
Figure 3: Number of inactive neurons for 5 consecutive training steps the PS experiment.
Attention Entropy
O Projection Matrix
Projection Matrix
Feedforward Layer
40
SNit
100
Rous aqa SHtOSp
SNR-12
W0-9ht Koom
30
20
20
15-
10-
20
200
400
200
400
203
402
200
Trsk
Tisk
Bak
Figure 4: Parameter norms and attention-scort entropy on the Permuted Shakespeare experiment.
8

---

Published as a conference paper at ICLR 2025
3.2.1 SCALED PERMUTED SHAKESPEARE
While the scale of our Permuted Shakespeare problem serves as simple benchmark problem for evaluating a series of continual learning algorithms and hyperparameter choices for language mod- els, it is also of interest to investigate the effect of model and dataset scale on plasticity. To this end, we scale the number of non coneemdddi weights in our transformer network by a factor of N = 16, increasing the number of heads to and number of neurons to 1024. line with scaling laws (Kaplan et al. 2020). we increase the size of our dataset by a factor of 169.74, specifically to 254 976 tokens per task. The rest of the problem setup remains unchanged; we train the network for 100 epochs on 500 tasks in sequence. To facilitate the larger token count, train a sample of 254'976 tokens worth of text from the complete set of plays by William Shakespeare.
We limit experiment to 4 random seeds, scales and 16, evaluating only SNR+L2 and L2 with hyperparameters 7) = 0.05 and À = 10- presenting our results in Table We first note that as model and dataset size grow, the gap in average loss between L2-regulanzztion and SNR+L2- regularization grows substantially. Simultaneously. we see a dramatic increase in the proportion of inactive neurons with I.2-regularivation. At any time step, on average 0.06% of neurons inactive at scale N = 1 while 32.9% are inactive at scale N = 16. These results suggest that resets can play critical role in maintaining plasticity in large-scale language models. See Appendix which shows a scale 256 experiment, but run only up to 50 tasks, showing similar merits.
Algorithm
Loss (All Tasks)
Loss (Last 50 Tasks)
Dead Neuron Rate
1.2 - Scale
0.159
(0.025)
0.152
(0.016)
0.06%
SNR+I.2 Scale
0.154
(0.008)
0.141
(0.031)
0.00%
L2- Scale 16
0.377
(0.016)
0.410
(0.052)
32.9%
SNR+L2 Scale 16
0.324
(0.028)
0.332
(0.073)
1.41 10 66
Table 3: Average loss on the final epoch of each task for the scaled Permuted Shakespeare experi- ments, with means and standard deviations reported over 4 seeds.
3.2.2 GENERALIZATION
Lee et al. (2024) recently propose separating training loss from test loss in studying plasticity; the extant literature and the present work focuses largely on measuring training loss on each task. As complement, we briefly consider measuring test loss
holdout set for each task. We consider
three setups: first, we consider the PM problem, and measure test loss on a set of 10000 image label- pairs for each task; we annotate this setup PM-GI. Our second setup is modification of PM-GI: in each task we add noise to the labels by randomly re-labeling a fraction of the training images. We decrease the fraction of images with noisy labels from 50% on the first task to 0 on the last; annotate this task PM-G2. Finally in analogy to PM-G2, we consider a variant of the RC problem (RC-G1) where we decay the fraction of random labels from 50% to O linearly across tasks. PM-G2 and RC-G1 are analogous to the setup in Lee et al. (2024).
Detailed results of these experiments are presented in Appendix A.2 we draw the following con- clusions: first, SNR displays similar relative merits to competing algorithms as in our main body of experiments on training loss. Second, through careful study of variants of the setup in PM-G2 where we alter the number of passes through the dataset relevant to each task, we point out an im- portant issue that requires careful attention one is to study test loss: specifically, confounding the effects of overfitting with plasticity loss.
4 THEORETICAL ANALYSIS OF LEARNING A SINGLE RELU
One common hypothesis that seeks to explain plasticity loss is that the network weights obtained from minimizing loss over some task yield poor initializations for a subsequent task; these initial- izations effectively lead to poor local minima. This section attempts to make this effect transparent through the analysis of the simplest non-trivial task: learning a single ReLU. Vardi et al. (202T) have already established that for random initializations ReLU can be efficiently learned. Here ask what happens when the initialization is picked adversarially; one can think of this adversarial initialization as corresponding optimal weights for an earlier ReLU learning task.

---

Published as a conference paper at ICLR 2025
Theorem 4.1. [Informal restatement of Theorems E.2 and E.I) There exists a general class of target ReLUs, a distribution over example inputs, and a distribution over initial weights for which:
Gradient descent with L2 Init or L2 regularization fail to learn the target ReLU with positive probability with respect to the distribution over initial weights. Specifically the average expected MSE exceeds positive constant the number of training examples T grows with a probability bounded from below by a positive constant.
Gradient descent with oracle resets achieves an average expected MSE of O(1/T) with probability one.
The oracle resets in the second result above effectively know precisely whether or not a neuron will or will not fire over the input distribution and in this sense oracular. Reset methods must come as close as possible to mimicking such an oracle; Propositions 2. and and 2.2 make the case for why SNR might be good such candidate and why competing reset proposals may fall short. See Appendix for formal setup, statement of results, and proofs.
5 DISCUSSION AND LIMITATIONS
common explanation for plasticity loss that the network weights obtained from minimizing loss over some task yield poor initializations for subsequent task this explanation continues to motivate the vast majority of algorithms that attempt to combat plasticity loss. Specifically, regu- larization based algorithms can be viewed as regularizing towards a 'good" initial set of weights, whereas reset based algorithms can be viewed as explicitly reinitializing weights to good random initializations.
Theoretical Motivation: Our theoretical development showed that regularization methods them selves are insufficient combatting the plasticity loss problem, at least in the case of learning Re- LUS (Theorem 4.T). Further we showed that existing proposals to detect whether it was appropriate to reset ReLU (based on ad hoc notions of neuron's 's utility) could have high type error rates relative to an optimal resetting mechanism (Proposition 2.2 SNR was designed to minimize the error rate in solving this detection problem (Proposition
SOTA Performance: Across range of experiments we see that neuron inactivity is an important correlate of plasticity loss, and that SNR consistently outperforms other reset based methods as well as regularization methods (Table (2. We also observed that SNR was robust to its hyper- parameters while the adhoe utility schemes employed by other reset methods tended to make them somewhat brittle (Appendix(C) [ Finally, the extant literature has focused on measuring plasticity loss through the lens of training loss, but is also natural to ask about test loss. We see that SNR enjoys similar relative merits in this context as well (Table 5). but one has to be careful to not confound issues of overfitting with plasticity loss (Figure 6).
Attention Menchanisms: Whereas neuron death is an important correlate of plasticity loss, in the case of language models, a crucial component of the architecture - the attention mechanism - does not involve neurons that the notion of resetting is irrelevant that component. We have shown that plasticity loss occurs nonetheless and that is associated with a collapse in the entropy of the attention layer along with neuron death in the feedforward layers (Figures and 4). As such, we see that combining our resets along with L2 regularization of the attention mechanism is important to preserve plasticity (Table 2 Language models also give natural substrate to consider the plasticity loss phenomenon vis-a-vis scaling laws, and we see similar relative merits across several orders of magnitude of model scales (Table 3 Appendix 4
Limitations: We recognize few limitations in the present work. First, whereas we have shown the plasticity loss phenomenon over several orders of model scale our largest models have up to 5M pa- rameters. This is large in the context of the literature on plasticity loss but small in practical sense. We believe that overcoming this requires understanding ways of exploring this phenomenon without the brute force effort of feeding a model a large sequence of tasks (which is fundamentally serial and hard to accelerate). A second limitation is the theoretical characterization of the phenomenon itself; the approach in the literature has been largely phenomenological fixing conjectured root causes for what is really an increase in dynamic regret over time. While Section 4 took a first step. future work ought to connect the plasticity loss phenomenon tightly with the optimization landscape.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. arXiv preprint arXiv:1711.08855 2017.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in neural information processing systems, 33:3884-3894, 2020.
Dylan R Ashley, Sina Ghiassian, and Richard S Sutton. Does the adam optimizer exacerbate catas- trophic forgetting? arXiv preprint arXiv:2102.07686, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1671v:160..06450, 2016.
Omar Besbes and Assaf Zeevi. On the minimax complexity of pricing in a changing environment. Operations research, 59(1):66-79, 2011.
Patryk Chrabaszez, Iya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint (rXiv:/707008819 2017.
Shibhansh Dohare, Richard Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108 0:Xi::2108.06325, 2021.
Shibhansh Dohare, J Fernando Hernandez-Garcia, Parash Rahman, Richard S Sutton, and A Rupam Mahmood. Maintaining plasticity in deep continual learning. arXiv preprint arXiv:2306. 13812, 2023.
Shibhansh Dohare, Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mah- mood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-774, 2024.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint 1rXiv:1312.6211,, 2013.
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. arXiv preprint wxXiv:2403.08766 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-335 2017.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameeterizaion inhibits data eeff-efficient deep reinforcement learning. arXiv preprint arXiv:2010. 14498, 2020.
Saurabh Kumar, Henrik Marklund, Ashish Rao, Yifan Zhu, Hong Jun Jeon, Yueyang Liu, and Ben- jamin Van Roy. Continual learning computationally constrained reinforcement learning. arXiv preprint ar X3V:22077.04345 2023a.
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprint a338v:2308. /958, 2023b.
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. In Forty-first International Conference Machine Learning, 2024.

---

Published as a conference paper at ICLR 2025
Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C Machado. Curvature explains loss of plasticity. arXiv preprint arxiv:2312.00224, 2023.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in rein- forcement learning. arXiv preprint arXiv:2204.09560 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, pp. 201900232211 PMLR, 2023.
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss neural networks. arXiv preprint arXiv:2402.18762, 2024.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias deep reinforcement learning. In International conference on machine learning, pp. 16828-16847. PMLR, 2022.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference Machine Learning, pp. 32145-32168. PMLR, 2023.
Gal Vardi, Gilad Yehudai, and Ohad Shamir. Learning a single neuron with bias using gradient descent. Advances in Neural Information Processing Systems, 34:28690-28700, 2021.
Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey. arXiv preprint arXiv:2402.01364, 2024.
Julian Zilly, Alessandro Achille, Andrea Censi, and Emilio Frazzoli. On plasticity, invariance, and mutually frozen weights in sequential task learning. Advances in Neural Information Processing Systems, 34:12386-12399 2021.
A ADDITIONAL EXPERIMENTAL RESULTS
A.1 EVEN LARGER SCALE EXPERIMENT FOR PERMUTED SHAKESPEARE
We increase the scale further for the Permuted Shakespeare experiment from our original model by a factor of 256, resulting model with 5.1 million non-embedding parameters; results are Table In accordance with scaling laws, we simultaneously increase the number of tokens or examples per task by a factors of 2560.74 resulting 1.9 million tokens or 1,5 500 training examples, with a context window of 128. for the 256-scale model. Due to computational constraints, we reduce the number of tasks from 500 to 50, but keep training regime of 100 epochs per task. At the largest scale, over the 50 tasks, our model is trained for 100 epochs over 99 million tokens or 775 '00 distinct 228-token-lomg training examples.
The scale of our largest model is comparable the scale of the largest transformers considered in recent literature on plasticity loss, namely ViT Tiny (5 million parameters) in [ee et al. (2024). The largest dataset on which Lee al. (2024) train is the Tiny Imagenet dataset consisting of 1000000 training examples, on which models are trained for 100 epochs; our largest experiment consists of 775'000 training examples. It is important to consider the scale of datasets used continual learning experiments as the phenomenon of plasticity loss has been shown to be correlated with the amount of data on which network trains (Kumar
2023a Dohare et 2021
Algorithm
Train Loss at Scale 256
SNR+L2
0.4576
(0.0017)
L2
0.4681
(0.0025)
Table 4: Average training loss on the final epoch over all 50 tasks of the 256-scale Permuted Shake- speare problem with standard deviations reported for 5 random seeds.
12

---

Published as a conference paper at ICLR 2025
Accuracy
Test Accuracy for G2
0.930
Accursey
0.925
Cumilator fuccoe lest
0.920
Dumuletvv
0.88
No
12 Iiit
12
0.90
12 Reg
0.87
Strink and Perturb
L2 Reg
ReD0
SNR
0.86
SNR
20
40
50
80
100
20
40
80
100
Epoch
Epoch
Figure 5: Cumulative average test accuracy for PM-G1 and PM-G2 experiments.
It is important to note that relative to Table 3 where we show plasticity loss for 500 tasks, the above experiment has only been run to 50 tasks (due to computational budget constraints) and already shows gap between SNR+L22 and L.2
A.2 GENERALIZATION RESULTS
Table 5 shows test loss the last 10% of tasks for PM-G1, PM-G2 and RC-GI. Figure 5 shows the test accuracy for PM-G1 and PM-G2. We see that SNR enjoys similar relative merits to its competitors in PM-G1 and PM-G2. For RC-G1, it appears that there is no plasticity loss and such all of the different algorithms perform similarly. Notice that this is not contrary to our observations RC there the labels across tasks were entirely unrelated.
We consider next a slight tweak on RC-GI. Specifically, each task in RC-GI consists of 16 epochs; what happens if we more? The bottom three curves of Figure show the results of doing 100 epochs for each task. We see here that test loss for no intervention and SNR begin to suffer. As it turns out this is misleading - specifically, we see that the test accuracy for all three algorithms is significantly below test accuracy when one does only 16 epochs (as reported in Table 5). That is, the decay in test performance likely attributable overfitting in each task, and while 1.2 overfits well, it is unsurprisingly more resilient. Put different way, this overfitting by the usual tools to combat overfitting. The top three curves of Figure show that SNR performs equivalently to L2 showing that early stopping would accomplish this comfortably in this context.
be controlled
Algorithm
PM-G1
PM-G2
RC-G1
No Intv.
0.8999
(0.0041)
0.8931
(0.0013)
0.5526
(0.0034)
L. Init
0.9320
(0.0037)
0.9284
(0.0032)
N/A
L2
0.9297
(0.0034)
0.9258
(0.0032)
0.5651
(0.0023)
S&P
0.9313
(0.0033)
0.9281
(0.0027)
N/A
ReDO
0.9086
(0.0028)
0.9083
(0.0021)
N/A
SNR
0.9318
(0.0001)
0.9325
(0.0029)
0.5650
(0.0013)
Table Test accuracy over the last 10% of tasks (mean and standard deviation) for PM-GI, PM-G2, and RC-GI.
B COMPLETE RESULTS FOR BENCHMARK PROBLEMS
To maintain brevity in the main body of the paper, include here the complete results for the benchmark problems PM, RC, RM, and CI which include both the mean and standard deviation of terminal task accuracies over random seeds.
13

---

Published as a conference paper at ICLR 2025
RC- G1: 16 vs 100 Epochs
0.55
0.50
Test Accuracy
0.45
0.40
0.35
SNR 16
L2 16
No Intv. 16
0.30
SNR 100
L2 100
0.25
No Intv. 100
0
2
4
6
8
Task Index
Figure 6: Test accuracy for the RC-GI experiments evaluated with 16 and 100 epochs per task.
Optimizer
SGD
Algorithm
PM
RM
RC
CI
No Intv.
0.710 (0.007)
0.113(0.004)
0.180 (0.011)
0.784 (0.019)
SNR
0.851(0.002)
0.975 (0.001)
0.987 (0.002)
0.888 (0.010)
CBP
(0.00().002)
0.951 (0.007)
0.961 (0.011)
0.840 (0.015)
ReDO
0.831 (0.013)
0.716 (0.024)
0.981 (0.003)
0.869 (0.038)
L2 Reg.
0,818 (0.001)
0.803 (0.011)
0.952 (0.006)
0.833 (0.011)
L2 Init
0.829 (0.001)
0.913 (0.001)
0.966 (0.002)
0.832 (0.010)
S&P
0.826 (0.002)
0.920 (0.009)
0.971 (0.004)
0.853 (0.006)
Layer Norm.
0.687 (0.009)
0.143 (0.015)
0.959 (0.005)
0.819 (0.009)
Optimizer
Adam
Algorithm
PM
RM
RC
No Intv.
0.641 (0.007)
(0.005)
0.151 (0.005)
0.581 (0.081)
SNR
(0.89(0.001)
0.982(0.001)
0.976 (0.002)
0.847 (0.005)
CBP
0.876 (0.001)
0.948 (0.003)
0.331 (0.312)
0.818 (0.005)
ReDO
0.846 (0.002)
0.671 (0.021)
0.744 (0.131)
0.803 (0.063)
L2 Reg.
0.806(0.002)
0.948 (0.002)
0.967 (0.011)
0.803 (0.009)
L2 Init
0.883 (0.002)
0.961 (0.003)
0.976 (0.002)
0.827 (0.008)
S&P
0.876 (0.002)
0.955 (0.006)
0.971 (0.005)
0.814 (0.005)
Layer Norm.
0.662 (0.001)
0.113 (0.005)
0.955 (0.005)
0.651 (0.053)
Table 6: Average accuracy the last 10% of tasks the benchmark continual learning problems with standard deviations over 5 seeds.
14

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EXPERIMENTAL DETAILS AND EYPERPARAMETER SWEEP
With SGD we train with learning rate 10-2 on all problems except Random Label MNIST, for which train with learning rate 10-1 With Adam train with learning rate 10 all problems, including Permuted Shakespeare and we use the standard parameters of 31 = 0.9, 32 = 0.999, and 10 For Permuted Shakespeare train networks solely with Adam. The learning rates were selected after an initial hyperparameter sweep.
For each algorithm vary its hyperparameter(s) by an appropriate constant over 7 choices, effec- tively varying the hyperparameters over log scale. With the exception of the Permuted Shakespeare experiment, limit over hyperparameter search to 5 choices. In Table 7 we provide the hyperpa- rameter sweep for the 4 benchmark problems. CBP's replacement rate is to be interpreted as one replacement per layer every r- training examples, as presented in Dohare et al. (2024). ReDO's reset frequency determines the frequency of resets in units of tasks, as implemented and evaluated in Kumar al. (203bb).
Hyperparameter Strength
Algorithm
2
3
4
5
L2 Reg. (A)
10-6
10-5
10-4
10
10
10-1
100
L2 Init (A)
10-6
10-5
10-4
10
10
10-1
10
S&P (1-p)
10-3
10-7
7
10-
10-5
10"
10
10-2 2
S&P (o)
10-6
10-b
10-4
3
10-3
10-2
10
10"
CBP (r)
10-7
10-6
10-5
10-4
10-3
10-2
10-1
ReDO (T)
0
0.01
0.02
0.04
0.08
0.16
0.32
RcDO (r)
64
32
16
8
2
SNR (n)
0.08
0.04
0.02
0.01
0.005
0.0025
0.00125
Table 7:
Hyperparameter sweep for the Permuted MNIST (PM), Random Label MNIST (RM),
Random Label CIFAR (RC), and Continual ImageNet (CI) problems.
Hyperparameter Strength
Algorithm
3
4
L2 Reg. (A)
10-6
10-5
10-1
10-3
10-2
L2 Init (A)
10-6
10
5
10
10-3
10-2
SNR (n)
0.1
0.05
0.03
0.01
0.001
SNR L2 Reg (n)
0.1
0.05
0.03
0.01
0.001
Table 8: Hyperparameter sweep for the Permuted Shakespeare problem. For the combination of
SNR and L2 regularization we use the regularization strength of 10-4, the best performing regular- ization strength for 1.2 regularization, and vary the rejection percentile threshold
Hyperparameter Strength
Algorithm
(
2
3
4
L2 Reg. (A)
1.284
0.883
0.348
0.853
4.269
(0.037)
(0.059)
(0.093)
(0.061)
(0.013)
L2 Init (A)
1.791
1.481
1.641
2.429
5.598
(0.056)
(0.050)
(0.067)
(0.124)
(0.030)
SNR (1)
3.034
3.024
3.012
3.027
3.025
(0.022)
(0.041)
(0.025)
(0.029)
(0.045)
SNR + 1.2 Reg (n)
0.315
0.276
0.325
0.293
0.269
(0.084)
(0.078)
(0.068)
(0.065)
(0.025)
Table 9: Average loss of the final epoch of each task in Permuted Shakespeare, averaged over the final 50 tasks, reported as mean (standard deviation) over random seeds.
15

---

Published as a conference paper at ICLR 2025
Permuted MNIST Performance with SGD Optimizer
Permuted MINIST Performance with Adam Optimizer
SWR
0.8
12 1n 00
0.716
0.731
5.190
0.827
5.824
0,758
0.288
L2 1m
0.837
0.864
0.884
0.870
0.704
0.276
L2 Reg. 01
0.713
0.729
0.797
0.815
0.790
0.110
0.184
Rog.
0.792
0:837
0.858
0.878
0.859
0.365
0.363
0.716
0.770
0.6
11-pt
0.876
0.768
S6P to
0.823
0.823
0.825
0.825
0.784
0.310
0.110
0.5
SAP tof
0.874
00.875
0.875
0.8.7
0.877
0.583
0.115
Reb0
6.832
0.817
ReDO
6.636
0.847
0.867
0.4
RiDO 01
0.815
0.824
0.830
0.832
3.819
0.77
0,830
0.817
0.842
0.8:5
0.867
0,841
0.825
CBP
0.726
0.762
0.823
0.842
0.844
0.648
0.635
CBP (/)
0.723
0.773
0.817
0.857
0.876
0.733
0.720
0.3
Layer
0.2
SGD
0.110
0.730
0.100
0.730
0.710
0,710
0.710
Adom
6.636
0.0
0.636
0.636
0.636
0.636
0.636
Hyperparameter
Hyperparameter Strength
Figure 7: Permuted MNIST hyperparameter sweep results over 5 seeds.
Random MINIST Performance with
Random Performance
Optimizer
0.975
0.74
12
0.115
6.134
8.181
0.881
3.412
0.101
Im
0.92%
0.958
0.980
0.905
0.114
0.8
L2 Reg. (1)
0.115
0.115
0.115
0.125
3.801
0.113
0.308
L2 Rog. 611
0.243
0.938
0.946
0.944
0.506
0.113
0.307
0.7
SSP tel
0.907
0.913
2910
0.919
0.924
0.181
0.116
3.6
S&P lot
0.945
0.941
0.942
0.943
0.955
0.243
0.121
0.6
ReDD
0.115
0.600
3.622
0.718
0.705
0.705
-355
ReDO
0.300
0.181
0.551
0.670
3.508
0.588
0.5
ReD0
0.111
0.111
0.109
0.115
0.115
0.120
0.104
0.112
0.122
CBP
0.389
0,931
0.945
0.553
0.929
0.155
0.115
CBP
0.124
0.189
0.482
0.922
0.943
0.115
0.115
SGD
0.115
0.115
0.115
0.115
0.115
0.155
0.115
Adam 0.115
0.115
0.115
0.115
0.115
0.115
0.115
Figure 8: Random LabeL. MNIST hyperparameter sweep result over seeds.
16

---

Published as a conference paper at ICLR 2025
Performance
Optimizer
Performance
0.967
0.836
0.898
0.52
0:974
0.959
0.973
0.970
0.9
12 In CO
0,207
6,182
0.268
0.194
3.968
0.543
0.15
L2 60
0.313
0,366
0.818
0.977
0,979
3.966
0.341
Rog.
11 -ll
0.155
0.158
0.165
0.341
0.973
0.245
0.148
mpl
0.148
0.391
0.837
0.972
0.147
0.149
0.147
0.7
S&P tol
0.951
0.950
0.963
0.955
0.973
0.185
0.109
S&P
0.971
0.972
0.971
0.967
0.965
0.570
0.148
0.5
ReDO 01
0.207
6.199
0.160
0.317
3.890
0.5M
0.900
ReDO 01
0:348
0.170
0.169
0.205
0.407
0.617
0.741
0.199
0.232
CBP
0.168
Layer Norm
0.952
0.958
0.95
Layer Norm
0.958
0.958
0.958
0.2
SGD
0.184
0.184
0.184
0.1 84
0.186
0.184
0.184
Adaim 0.148
0.148
0.148
0.148
0.148
0.148
0.148
0.2
0.1
Hyperparameter Strength Log Scale
Hyperparameter Strength
Figure 9: Random Label CIFAR hyperparameter sweep results over 5 seeds.
Continual Image Net Performance with SGD Optimizer
Conti nual ImageNet Performance with Adam Optimizer
(NR(+)
0.687
0.1811
0.883
0.686
C. 0.B9e
SMRR
0.540
D.240
0.834
0.839
0.644
D.540
0.540
0.85
12
0.781
1.700
0.831
0.722
0.646
0.703
0.767
0.804
0.819
0.757
0.640
Reg.
Reg.
(3 pp)
0.778
0.784
0.752
0.815
0.653
0.485
0.489
0.75
(2
0.506
0.813
0.811
0.787
0.492
0.492
0.492
0.70
SSP (s)
0.8MM
6.079
0.811
0.814
0.497
0.500
SGP (o)
0.BcK
0.807
0.604
0.833
0.194
0.500
0.70
0..5
B=00 (n
0.827
0.824
0.844
0.870
0.867
0.864
0.836
RwDO
0.763
0:775
0.788
0.796
0.801
0.802
0.790
0.60
0.60
CBP Irl
0.777
0.791
0.792
0.837
0.840
0.488
0.488
CEP
0.562
0.747
0.770
0.802
0.829
0.677
0.712
9.785
785
785
6.
785
Adam
0.582
0.582
0.582
0.582
0,582
0.582
0,582
0.50
0
1
5
1
3
Hypergaramett eter Strength Log Scale
Hyperparametr Strength Log Scale
Figure 10: Continual ImageNet hyperparameter sweep results over seeds.
17

---

Published as a conference paper at ICLR 2025
Hyperparameter Strength Log Scale
Algorithm
0
2
4
5
SNR (m)
0.849
0.849
0.848
0.847
0.850
0.848
0.847
(0.002)
(0.003)
(0.003)
(0.004)
(0.002)
(0.002)
(0.002)
L2 Init (A)
0.716
0.731
0.790
0.827
0.824
0.758
0.288
(0.001)
(0.001)
(0.001)
(0.002)
(0.001)
(0,001)
(0.017)
L2 Reg. (A)
0.713
0.729
0.787
0.816
0.790
0.110
0.104
(0.002)
(0.002)
(0.001)
(0.002)
(0.003)
(0.003)
(0.002)
S&P (1 p)
0.709
0.716
0.770
0.826
0.804
0.615
0.106
(0.001)
(0.002)
(0.001)
(0.002)
(0.004)
(0.001)
(0.003)
S&P (σ)
0.823
0.823
0.825
0.826
0.784
0.110
0.110
(0.001)
(0.002)
(0.002)
(0.002)
(0.002)
(0.002)
(0.002)
ReDO (r)
0.710
0.832
0.817
0.742
0.743
0.743
0.743
(0.007)
(0.001)
(0.011)
(0.018)
(0.018)
(0.018)
(0.018)
ReDO (r)
0.815
0.824
0.830
0.832
0.819
0.808
0.778
(0.004)
(0.002)
(0.001)
(0.001)
(0.005)
(0.006)
(0.013)
CBP (r)
0.726
0.763
0.823
0.843
0.844
0.648
0.655
(0.006)
(0.002)
(0.002)
(0.001)
(0.002)
(0.000)
(0.000)
Layer Norm
0.689
0.689
0.689
0.689
0.689
0.689
0.689
(0.003)
(0.003)
(0.003)
(0.003)
(0.003)
(0.003)
(0.003)
SGD
0.710
0.710
0.710
0.710
0.710
0.710
0.710
(0.007)
(0.007)
(0.007)
(0.007)
(0.007)
(0.007)
(0.007)
Table 10: Performance on Permuted MNIST with different hyperparameters reported as mean (stan-
dard deviation).
Hyperparameter Strength Log Scale
Algorithm
0
2
4
5
6
SNR (1)
0.887
0.885
0.880
0.876
0.872
0.867
0.872
(0.002)
(0.001)
(0.002)
(0.002)
(0.001)
(0.002)
(0.012)
L2 Init (A)
0.790
0.837
0.864
0.884
0.870
0.784
0.276
(0.001)
(0.002)
(0.002)
(0.002)
(0.002)
(0.001)
(0.008)
1.2 Reg. (X)
0.792
0.837
0.858
0.878
0.859
0.365
0.103
(0.002)
(0.002)
(0.002)
(0.002)
(0.004)
(0.008)
(0.002)
S&P (1 p)
0.638
0.858
0.877
0.876
0.768
0.105
0.102
(0.005)
(0.002)
(0.002)
(0.002)
(0.001)
(0.003)
(0.001)
S&P (a)
0.874
0.875
0.875
0.877
0.877
0.583
0.115
(0.002)
(0.002)
(0.002)
(0.002)
(0.002)
(0.003)
(0.001)
ReDO (r)
0,636
0.846
0.847
0.847
0.847
0.847
0.847
(0.009)
(0.002)
(0.001)
(0.001)
(0.001)
(0.001)
(0.001)
ReDO (r)
0.830
0.837
0.842
0.845
0.847
0.841
0.825
(0.004)
(0.002)
(0.002)
(0.002)
(0.001)
(0.001)
(0.001)
CBP (r)
0.723
0.773
0.817
0.857
0.876
0.733
0.720
(0.012)
(0.004)
(0.002)
(0.002)
(0.002)
(0.001)
(0.008)
Layer Norm
0.661
0.661
0.661
0.661
0.661
0.661
0.661
(0,001)
(0.001)
(0.001)
(0.001)
(0.001)
(0.001)
(0.001)
Adam
0.636
0.636
0.636
0.636
0.636
0.636
0.636
(0.009)
(0.009)
(0.009)
(0.009)
(0.009)
(0.009)
(0.009)
Table 11: Performance on Permuted MNIST with different hyperparameters reported as mean (stan-
dard deviation).
18

---

Published as a conference paper at ICLR 2025
Hyperparameter Strength Log Scale
Algorithm
0
2
4
5
SNR (1)
0.975
0.975
0.974
0.974
0.974
0.973
0.973
(0.001)
(0.001)
(0.000)
(0.000)
(0.001)
(0.001)
(0.001)
L2 Init (A)
0.115
0.114
0.181
0.881
0.912
0.191
0.101
(0.006)
(0.009)
(0.027)
(0.017)
(0.006)
(0,008)
(0.010)
L2 Reg. (A)
0.115
0.115
0.115
0.126
0.801
0.113
0.108
(0.006)
(0.006)
(0.006)
(0.013)
(0.010)
(0.007)
(0.009)
S&P (1 p)
0.112
0.161
0.138
0.550
0.924
0.114
0.109
(0.008)
(0.009)
(0.008)
(0.023)
(0.003)
(0.006)
(0.009)
S&P (a)
0.917
0.913
0.910
0.919
0.924
0.181
0.116
(0.005)
(0.004)
(0.007)
(0.004)
(0.003)
(0.005)
(0.006)
ReDO (r)
0.115
0.600
0.642
0.719
0.705
0.705
0.705
(0.006)
(0.023)
(0.030)
(0.021)
(0.021)
(0.021)
(0.021)
ReDO (r)
0.111
0.111
0.109
0.116
0.117
0.233
0.719
(0.008)
(0.008)
(0.008)
(0.019)
(0.019)
(0.022)
(0.021)
CBP (r)
0.389
0.931
0.945
0.953
0.929
0.115
0.115
(0.031)
(0.051)
(0.012)
(0.005)
(0.002)
(0.006)
(0.006)
Layer Norm
0.145
0.145
0.145
0.145
0.145
0.145
0.145
(0.013)
(0.013)
(0.013)
(0.013)
(0.013)
(0.013)
(0.013)
SGD
0.115
0.115
0.115
0.115
0.115
0.115
0.115
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
Table 12: Performance on Random Label MNIST with different hyperparameters reported as mean (standard deviation).
Hyperparameter Strength Log Scale
Algorithm
0
2
4
5
SNR (1)
0.979
0.981
0.979
0.978
0.976
0.969
0.966
(0.001)
(0.001)
(0.002)
(0.001)
(0.002)
(0.001)
(0.003)
L2 Init (A)
0.312
0.929
0.958
0.960
0.905
0.114
0.098
(0.050)
(0.011)
(0.007)
(0.001)
(0.003)
(0.004)
(0.009)
1.2 Reg. (X)
0.243
0.918
0.946
0.944
0.506
0.113
0.107
(0.038)
(0.008)
(0.009)
(0.004)
(0.043)
(0.007)
(0.009)
S&P (1 p)
0.115
0.955
0.946
0.931
0.114
0.109
0.102
(0.006)
(0.004)
(0.008)
(0.005)
(0.006)
(0.009)
(0.004)
S&P (a)
0.945
0.941
0.942
0.943
0.955
0.243
0.121
(0.009)
(0.009)
(0.007)
(0.013)
(0.004)
(0.015)
(0.006)
ReDO (r)
0.115
0.306
0.381
0.553
0.670
0.588
0.588
(0.006)
(0.017)
(0.023)
(0.028)
(0.019)
(0.031)
(0.031)
ReDO (r)
0.115
0.115
0.120
0.109
0.112
0.122
0.670
(0.006)
(0.006)
(0.009)
(0.008)
(0.008)
(0.028)
(0.019)
CBP (r)
0.124
0.199
0.482
0.922
0.949
0.115
0.115
(0.030)
(0.055)
(0.077)
(0.007)
(0.004)
(0.006)
(0.006)
Layer Norm
0.115
0.115
0.115
0.115
0.115
0.115
0.115
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
Adam
0.115
0.115
0.115
0.115
0.115
0.115
0.115
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
Table 13: Performance on Random Label MNIST with different hyperparameters reported as mean (standard deviation).
19

---

Published as a conference paper at ICLR 2025
Hyperparameter Strength Log Scale
Algorithm
0
2
4
5
SNR (m)
0.887
0.888
0.893
0.886
0.886
0.886
0.886
(0.009)
(0.007)
(0.006)
(0.004)
(0.004)
(0.004)
(0.004)
L2 Init (A)
0.781
0.780
0.827
0.831
0.722
0.646
0.573
(0.028)
(0.016)
(0.011)
(0.008)
(0.014)
(0.014)
(0.018)
L2 Reg. (A)
0.775
0.800
0.833
0.819
0.488
0.488
0.491
(0.010)
(0.013)
(0.012)
(0.007)
(0.000)
(0.000)
(0.001)
S&P (1 p)
0.778
0.784
0.752
0.815
0.853
0,488
0.488
(0.023)
(0.024)
(0.073)
(0.010)
(0.008)
(0.000)
(0.000)
S&P (σ)
0.848
0.839
0.849
0.853
0.844
0.497
0.500
(0.002)
(0.010)
(0.008)
(0.008)
(0.010)
(0.005)
(0.000)
ReDO (r)
0.785
0.856
0.870
0.843
0.721
0.727
0.726
(0.024)
(0.012)
(0.010)
(0.015)
(0.041)
(0.043)
(0.040)
ReDO (r)
0.827
0.824
0.844
0.870
0.867
0.864
0.836
(0.027)
(0.038)
(0.029)
(0.010)
(0.016)
(0.011)
(0.009)
CBP (r)
0.777
0.791
0.792
0.837
0.840
0.488
0.488
(0.022)
(0.016)
(0.013)
(0.012)
(0.008)
(0.000)
(0.000)
Layer Norm
0.818
0,818
0.818
0.818
0.818
0,818
0,818
(0.011)
(0.011)
(0.011)
(0.011)
(0.011)
(0.011)
(0.011)
SGD
0.785
0.785
0.785
0.785
0.785
0.785
0.785
(0.024)
(0.024)
(0.024)
(0.024)
(0.024)
(0.024)
(0.024)
Table 14:
Performance on Continual ImageNet with different hyperparameters reported as mean
(standard deviation).
Hyperparameter Strength Log Scale
Algorithm
0
4
5
SNR (1)
0,840
0.840
0.839
0.839
0.840
0.840
0.840
(0.002)
(0.005)
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
L2 Init (A)
0.703
0.767
0.814
0.826
0.819
0.757
0.640
(0.052)
(0.025)
(0.005)
(0.009)
(0.005)
(0.011)
(0.014)
1.2 Reg. (X)
0.767
0.791
0.802
0.797
0.749
0.492
0.492
(0.018)
(0.013)
(0.012)
(0.009)
(0.021)
(0.001)
(0.000)
S&P (1 p)
0.506
0.813
0.811
0.787
0.492
0.492
0.492
(0.025)
(0.010)
(0.008)
(0.008)
(0.001)
(0.000)
(0.000)
S&P (a)
0.806
0.807
0.804
0.803
0.813
0.494
0.500
(0.012)
(0.004)
(0.011)
(0.007)
(0.010)
(0.000)
(0.001)
ReDO (r)
0.582
0.802
0.610
0.730
0.725
0.732
0.730
(0.075)
(0.018)
(0.120)
(0.012)
(0.016)
(0.016)
(0.013)
ReDO (r)
0.763
0.775
0.788
0.796
0.801
0.802
0.790
(0.033)
(0.018)
(0.013)
(0.007)
(0.008)
(0.018)
(0.006)
CBP (r)
0.562
0.747
0.770
0.802
0.819
0.677
0.712
(0.087)
(0.032)
(0.014)
(0.004)
(0.003)
(0.092)
(0.039)
Layer Norm
0.709
0.709
0.709
0.709
0.709
0.709
0.709
(0.014)
(0.014)
(0.014)
(0.014)
(0.014)
(0.014)
(0.014)
Adam
0.582
0.582
0.582
0.582
0.582
0.582
0.582
(0.075)
(0.075)
(0.075)
(0.075)
(0.075)
(0.075)
(0.075)
Table 15: Performance on Continual ImageNet with different hyperparameters reported as mean (standard deviation).
20

---

Published as a conference paper at ICLR 2025
Hyperparameter Strength Log Scale
Algorithm
0
2
4
5
SNR (m)
0.975
0.974
0.969
0.973
0.970
0.971
0.973
(0.003)
(0.001)
(0.006)
(0.004)
(0.004)
(0.004)
(0.002)
L2 Init (A)
0.313
0.366
0.818
0.977
0.979
0.966
0.141
(0.076)
(0.096)
(0.165)
(0.005)
(0.003)
(0.004)
(0.006)
L2 Reg. (A)
0.221
0.237
0.342
0.957
0.968
0.148
0.149
(0.049)
(0.029)
(0.035)
(0.012)
(0.002)
(0.004)
(0.004)
S&P (1 p)
0.148
0.391
0.837
0.972
0.147
0.149
0.147
(0.004)
(0.228)
(0.170)
(0.002)
(0.005)
(0.004)
(0.005)
S&P (a)
0.971
0.972
0.971
0.967
0.966
0.570
0.148
(0.004)
(0.002)
(0.002)
(0.009)
(0.004)
(0.039)
(0.008)
ReDO (r)
0.148
0.207
0.314
0.209
0.687
0.594
0.741
(0.004)
(0.115)
(0.330)
(0.079)
(0.125)
(0.089)
(0.128)
ReDO (r)
0.148
0.170
0.169
0.205
0.407
0.637
0.741
(0.004)
(0.044)
(0.045)
(0.077)
(0.165)
(0.155)
(0.128)
CBP (r)
0.148
0.147
0,330
0.168
0.326
0.146
0.148
(0.004)
(0.007)
(0.302)
(0.026)
(0.074)
(0.006)
(0.004)
Layer Norm
0.958
0.958
0.958
0.958
0.958
0.958
0.958
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
(0.006)
Adam
0.148
0.148
0.148
0.148
0.148
0.148
0.148
(0.004)
(0.004)
(0.004)
(0.004)
(0.004)
(0.004)
(0.004)
Table 16: Performance on Random Label CIFAR with different hyperparameters reported as mean (standard deviation).
Hyperparameter Strength Log Scale
Algorithm
0
4
5
SNR (1)
0.987
0.836
0.898
0.808
0.848
0.941
0.829
(0.002)
(0.242)
(0.112)
(0.124)
(0.183)
(0.064)
(0.173)
L2 Init (A)
0.207
0.182
0.268
0.594
0.968
0.943
0.156
(0.040)
(0.012)
(0.068)
(0.135)
(0.004)
(0.007)
(0.005)
1.2 Reg. (X)
0.182
0.187
0.261
0.393
0.951
0.146
0.148
(0.012)
(0.024)
(0.097)
(0.265)
(0.005)
(0.004)
(0.004)
S&P (1 p)
0.155
0.158
0.165
0.341
0.973
0.145
0.148
(0.005)
(0.007)
(0.009)
(0.081)
(0.003)
(0.005)
(0.004)
S&P (a)
0.951
0.960
0.963
0.965
0.973
0.185
0.100
(0.007)
(0.006)
(0.004)
(0.003)
(0.003)
(0.068)
(0.014)
ReDO (r)
0.184
0.942
0.984
0.975
0.962
0.841
0.840
(0.013)
(0.054)
(0.004)
(0.020)
(0.038)
(0.039)
(0.057)
ReDO (r)
0.207
0.199
0.160
0.377
0.890
0.984
0.980
(0.057)
(0.067)
(0.025)
(0.220)
(0.121)
(0.004)
(0.001)
CBP (r)
0.190
0.199
0.232
0.528
0.964
0.142
0.142
(0.021)
(0.024)
(0.058)
(0.266)
(0.012)
(0.006)
(0.006)
Layer Norm
0.958
0.958
0.958
0.958
0.958
0.958
0.958
(0,008)
(0.008)
(0.008)
(0.008)
(0.008)
(0.008)
(0.008)
SGD
0.184
0.184
0.184
0.184
0.184
0.184
0.184
(0.013)
(0.013)
(0.013)
(0.013)
(0.013)
(0.013)
(0.013)
Table 17: Performance on Random Label CIFAR with different hyperparameters reported as mean (standard deviation).
21

---

Published as a conference paper at ICLR 2025
D PROOFS OF PROPOSITION OF 2. AND 2.2
Proposition D.1 (Restatement of Proposition 2.1 Let T be the p-d percentile of Geometric(p) distribution. Then the optimal hypothesis test takes the form X, 1(2, 0} where r = min(s: : Zu 1)A7
Proof. We begin by assuming equal priors P(H)) = P(H1) = = We note that for any time s, if
Z5 = 1 then any optimal hypothesis test must declare X, as 2, = 1 impossible under H1 and waiting to make future declaration will incur additional cost of least X. Therefore, it remains for
us to derive an optimal stopping time for the collection of states { _... Z. = 0: E+}
Let (s) be the expected total future cost at time given that have observed Z₁ -.. Z, 0.
We define
IP 21 Z, 0)
Ho 21 =... U, = 0) P(Z1 =... Z,
(HH)(HH)
(1 (())) P(H
(1 (p)
by PHH)) = P(H,)
- (1-p)* +1 +1
we stop at time and make declaration, we choose the hypothesis with higher positive probability in order tom minimize the error probability
IP(X. 1(H)) + P(X, = :
Thus, the expected cost of stopping is
Chto (s)o((s) = min mnin(TTs)) - }
We can simplify this further by noting that < 1 - T.. We note that
こ (1 p)* - 2
by 1 1-pppp e 1]. This is equivalent to
() < マ (11 (1- +1)
by adding (1 - -p)", which in turn, equivalent to
(1-p)* 1
n. (1 (1-p)- + 2
Therefore, n. S $1 n, and so we have that
(ooPp( (s)
min T.. ]T n8
This also implies that if we stop at some state {Zi ... Z. = -), is optimal to declare
X, = 1.
If we continue at time s + 1, we incur an additional delay cost of and the expected future cost depending on whether we see a Zs+1 or Z = 0.
With probability pı, we obserbes Zs+1, under Ho, and we stop the process with X = 0, incurring zero error cost since 7x41 cannot occur under H1.
With probability (1 p)n, + (1 π continues.
pT, we observe Z.+1 = and the process
22

---

Published as a conference paper at ICLR 2025
Therefore, the expected cost of continuing at time is
C (8) + (1 - pm, (s+
Then the Bellman equation for the optimal cost-to-go function is
V(s) = m (s), C (s)}
To determine an optimal stopping time, our goal is to find smallest T for which
C (T) S (
(1)
Assuming we stop at time T.
11 (I' = TT+1
Therefore,
(T) = -1
= + (1-pT,)
and establish [I it suffices show that
Tr - A + (1- pTr ) TT
(2)
First, we write TT+1 in terms of TT. Under the updating rule for the posterior probability, we have that
WT+1 P(Ho) 21 = Zr++ 0)
P(ZT+1 = =/ Ho 71 =..= 7x = 0)
P(Zr+1 01 =...= Zr =)
P(ZT+1 = 00 HOT + -p))TT
= 0|
(1 P)TT
(1 pr + (T)
p) AT
per
Returning to 2). we need to show that
(1 p)TT
TT X + (1 pxr)
X + (1 p)FT
puT
Simplifying the above inequality, we have that
nT S
Substituting in our formula for Tr, the above is equivalent to
(1-p)
p) +1 <
which after simplification is equivalent to
1 X (1-p) <
1
X p X F
Let F be the CDF of the Geometric(p) distribution. Let T be the 1 p-x percentile of the Geometric(p) distribution. Note, since X < y then 1 for any T > T we have that
- م (0, 1) and is valid percentile. Then
1 p1 = F(T)
2 F(T))
T
2
by choice of T
p-
23

---

Published as a conference paper at ICLR 2025
which is equivalent to
(1-p) S
and therefore, the optimal hypothesis test is to declare Xr for any T T if 21 Hence, the optimal hypothesis takes the form of
Zr
X, = 1 L 0} where T = min(s Z, = AT
Proposition D.2. Restatement of Proposition 2.2 r" to that under SNR scales like
The ratio total error rate with a fixed threshold
(-1 1 + 2 log(1 p2)
log(1 - P1)
Ω exp -1
Proof. For notational convenience, define ā = a(1 a)_ Notice that by Proposition 2.1 under SNR, neuron i, 2), is reset if inactive for any longer that time T = log( log(
Pc).
Consequently, the expected delay penalty, B[r + [B(r for neuron is simply
log(ā)
1 (1-a)
log(1 - p.)
pi
Letting the optimal fixed threshold be r", we must have that the expected total delay across both neurons under this fixed threshold is at least 2r*. This total expected delay can be no larger than that under SNR. Thus,
log(1 P1) + log(1 pe) ) ((1 -
(-+1
2r < log(a)
P1
P2
But the sum of the rates across the two neurons with the fixed threshold r" is at least (1 pı pr)"" a, while the total enror under SNR is precisely Dividing these two quantities and employing the upper bound derived on r* then yields the result.
E LEARNING A SINGLE RELU
In this section we prove Theorem 4.1
E. PRELIMINARIES
We aim to learn single ReLU-activated neuron with bias, equivalently, the mapping fo R > R. which we define as
f TT x
fx(x) = -
if x
We refer to = (Ü. by) as the target parameters where ò is the slope and b, is the bias of the linear map x + T x. Likewise, we denote our model's parameters as w (w. bw.) where is the slope and 4. is the bias.
We sample data x (., Uniform(-LL, x {1} such that the first coordinate sampled uniformly from the domain [-L, L] and the second coordinate is constant 1 so to model the bias term ReLU-activated neuron. We learn target neuron with respect to the squared loss and we thus define the loss to be
F(w) Ex (o x) o(v x))
(3)
Then the gradient of F is simply
VF(w) = Ex o(v (T) [) 0)z]
where the above expectations are taken with respect to x ~ Uniform(-L, x (1). We minimize F using gradient descent with constant learning rate
W1+1 w: Fw:))
(5)
24

---

Published as a conference paper at ICLR 2025
We define the regret of learning single ReLU with iterative algorithm
E
R
= E Erg
F(w))]
0=
where the randomness is over the initialization of and the potential reinitialization of due to resets. Then the average regret simply
1 B[RT]
- Bu F(w)]]
T
1=0
For some constant mmax > we suppose that i E constant Wmin < we sample wo uniformly from [-1, wp) (wo, 1| and set bwo = O, as is
[1, mmax] and by E [- 6
L 0J. While for some
customary to initialize neurons with zero bias.
E.1.1 L2 INIT AND L2 REGULARIZATION
Given a regularization strength À 2 0, we consider the loss with L2 Init regularization as
F(w) = Ex | (a(w æ) a(v xx) || woll = F(w) + クタ 1100 w woll
Then the gradient of F is simply
(www) Ex [( x) o(v' x))[(w> 0)x] A(w un) VP(w) X(w wo)
Then the L2 Init gradient descent update simply
Wr+1 = (vv)( wt - FV - 1/A w
(6)
Similarly, we can consider vanilla 1.2 regularization whose update is simply
W++1 W F(w:) n/w,,
Note, À 0 then we simply retain the update of unregularized gradient descent sufficiently small learning rate we attain non-vanishing average regret.
Then for any
Theorem E.1. Suppose that æ is sampled according to x ~ (niform(-L,A4 x {1) C R2 and that the target parameters e = (ū, b.) satisfy
and b. 0. Then applying gradient descent with L2 Init (6) or L2 regularization 77). with regularization strength À 2 0 and learning rate > such that
5 1
ye <
and with wo sampled uniformly from (.-1, W U (wmin 1]) x {0}, for any Wnin 0. then with probability & over random initializations wo the average regret is non-vanishing 2
1 RT 2 F(O)
T
Proof. The proof follows immediately by Lemma E.4 for L2 Init. A trivial modification of Lemma E.4 Bi4ydds identical result for L2 regularization, which we omit for brevity.
1.1.2 GRADIENT DESCENT WITH RESETS
For any reset threshold > 0 define reset oracle O, such that for any E IR?
True
if supsel-L,L)x(1) w To S
O.(w) =
False ([)P=e(-L,A)x(1)) x>
We consider the following gradient descent updates with resets
U1+1
(8)
U141 sample from Uniform ([-1, - Ümin) U (wmin 1)) x {0)
if O.
False
Wr+11
(9)
if (.(++)) = True
25

---

Published as a conference paper at ICLR 2025
Theorem E.2. Suppose that æ is sampled according
Uniform( -L, L) x {1} C R and the target parameters & (i, by) satisfy E [1, mmax] and - L
6 < b, I 0 and we denote Umax
(Umax:-Ba -b). Let 0 be a reset threshold and suppose that the initial parameters wo are sampled
uniformly from wo ~ [-1, -mmin) U (Wmin,) 1] and baro = such that
120maxe
Winn
L
and
5
2 e.
2
Then gradient descent with a constant learning rate of
S
Ini
3 122.. (4 Umax
1L 1)6
and with resets, i.e. and 9 attains an average regret of
C E|RT| S
T
T
where
C -=1 -=-12))(((-1100 F(
3
(Tmax
wmin) ((maa--@min))) (2
+
2112
Proof.
The theorem is restated and proven as Theorem
E.3
E.2
PROOFS
E.2.1 PROPERTIES OF THE Loss FUNCTION
Lemma E.1. The ReLU activation function (xx) = max x} is I-Lipschitz continuous.
Proof. Let x. y R be arbitrary. Without loss of generality we suppose that 2 y. We consider two cases. Firstly, if x 2 0 then we have that
la (x) - oly) = (xx ovy)
by x 2 y
ay))
by 2 0
S x - y
by oly) 2 y
< æ yl
As for the case of x < 0 then we likewise have that y < 0 and so
(a(x) - xvy) = S |a yl
Thus, it follows that (.) is I-Lipschitz continuous.
Lemma E.2. Suppose that is sampled according to x ~ Uniform(-L. target parameters (ö, b.) satisfy 7 and b, E 0. Then for S 0, (F R. (w x))).
S R² and that the
Proof. For almost every O. w wi bw 0 and so
(a(wTa x) a(u' x)) Tw x 2 0)r 0
by I(w 0)
= o w x).
by aw T x) = 0 since w T
As for Tt < O, we have that T x < since > 0 and b, $ 0, and so ( T x) = 0. Hence,
(o(w r) oo x) I(w x 2 0)r aw I) 2 0)r w x).
where the last equality follows by the fact that 1(w x 2 0) is redundant given that ow TwuTa x) = 0 wT x < 0. Therefore, for almost every
Uniform(-L,, L) x {1}. we have that
(awwT x)) -oT Tww 2 0) aw x)x
(10)
26

---

Published as a conference paper at ICLR 2025
We recall that
VF(w)= = Ex((o(w x)) - -ov x)) 2 0)æ]
and hence by (00
VF(w) Eg [ow z)x]
Lemma E.3. Suppose that is sampled according to Uniform {1} C R and that
the target parameters v = (ő, bo) satisfy ö > and by I then VF()) is (L* 1i--sscccitzz
continuous on < 0. That is, for any w,u < O, [V VF(w)]l (1,2 +]]|| ul|-
Proof. By w. u S 0 and Lemma R.2 we have that VF(w) = Ezlo(w x)x] and V = [a z))r].- Then establish the desired result as follows,
[VF(w) - VF(')ll = [E x)x au" x))]]
s Exlllor x)x au" x))=]]]]]
by Jensen's inequality
Exllo(w" x) ou (wTT)) [x)] -3]--11))) ||2l|]
Eg I|w u x| -||l||]]
by Lemma E.1
[w u| =||Ex]||=+||]
by Cauchy-Schwand incquality by x E --- x {1}
LL2 + 1)] w ull
E.2.2 CONVERGENCE AFTER NEGATIVE INITIALIZATION
Lemma E.4. Suppose that sampled according to x Uniform(-L., L.) x (1) S TR. and that the target parameters v = (ö, b,) satisfy e > 0 and b, S 0. Then applying gradient descent with L2 Init regularization with strength 2 0 according to equation (6) with wo satisfying wig E [-1,0] 0) and bmp 0 and learning rate +T where E (0, 1] such that nd < 1, then for any & 2 0
To < úr S 0,
(11)
bws 0,
(12)
F(wx) 2 F(0)
(13)
Proof. We additionally prove the following invariant
2 0, w (-
(14)
For = 0 we have that do I up S 0 and Dwn S hold trivially by assumption. As for 14 we
observe that
T wg (-1,1) Taip + bwo
LLãp
since bun = 0 since uh [[-,,
Therefore, we suppose that (11). 12 and 14 hold for some arbitrary and proceed to show that
they hold for + 1. We begin with establishing, [2)
b bw: 17 FWW)2
Dw: v)VF(W F(w)22 n/A(b.. bwo)
(1 najbe, -7VF(w))2
by bwo 0
(1 n/)be, (w, x²]
by Lemma E.2 given that we 0, 0, b, S
< (1 m/))bu.
since ow, x) 2 0, Vr
since bnc - and n < 1
27

---

Published as a conference paper at ICLR 2025
Next, for 11 have that W1+1 - if -Li Dwil W141(1L,,H) since dwei
0.
This is equivalent to showing that
(w: - VV n)(w; - mo)) ( I,, 1) 2 0
(15)
We proceed by bounding FVF((,) (-L, 1). Again invoking Lemma E.2 we have that
17 (VW(( (-L, (1,,,, (ow (-1,1)
(ww (- 1)]
S noww! (-L, +1)
by assumption that de S 0
=oolwi (-b,1)
by choice of
by assumption of the invariant (14
S nig (-I,,
since a S
Then, we have that (w, - VVF(w, - n)(w, wo))
2 m7) ( (-L, mywwa
( (-L, 1) w/ (- 2 (1 - uo (-L, + nmyli (-L, 1) - w/ (-L,1)
=0 0
where the second inequality above follows by wh いे follows that W I 0. Moreover, this also establishes (14) for t + 1. As for showing that i1t+1 vog in order to complete (11), argue follows.
0 and b I 0 bwg- Therefore,
i ui F(W))
= it (VF(()) - 17,(10 - Wp)
(1 - n))ū; + v7>in nBxlo 2) by Lemma E.2 given that 0, > 0, b, S 0
2 (1 n)) + niuu
since ă 0 = oww, x) 0 by wy S 0
2 (1 7) بية + ,7Aio
by x 2 dy and ny <
Therefore, that for any
holds for t 1. As for given that have established
and 12 it follows
1 F(w) Ez (0( (w) x)) - oo =])
2 Eg 7 (o(w, x) -olv ((TT)))21 xx))
Ex
(0(2 [
since S 0 and hence aww, 2)[[ = 0
(o x))
since of 0 < 0b ē > 0, b, S
F(0)
E.2.3 RESETS AFTER A NEGATIVE INITIALIZATION
Lemma E.5. Suppose that sampled according to x Uniform(-L,L) (1} C IR² and that the target parameters
(ö, b.) satisfy > 0 and b, < 0. Then applying gradient descent according to equation 5 with we satisfying wo 1-1,, 1, 0] and bwn and learning rate 17 < 1#41*
we have that for any threshold > 0, there exists some E 1,.,, T 16L'C, where
C, 2 F((-1, 0)), such that suP=E(-L,C)x(1) (w, x) <c
Proof. If Wg |- 0, then
sup -E([L_,[xx11
aww x) - Lioo S
28

---

Published as a conference paper at ICLR 2025
and at t 0 we immediately have that SUPzEI-L.LIx(14 (w, 2) S Therefore, consider the case of whg [ T., ---) {) and proceed follows. According Lemma E.4 and by the choice of learning rate 7/, we have that vt E 10.1...., 7), ue I 0. Therefore by Lemma E.2 (-)) (L2 + -Lpischhizz continuous on the span of {wo, w1,..., WT). Consequently, following the canonical analysis of gradient descent, we have that for any 2 0
F(w++1)= & F(ws) + ( (((e)))(mra) (wi+11 wi) + 1,2 +
W1+1 w
2
F(w)) - (1 (L2 + 1)n?
2
S ((w) V | (wx))l
where the second line above follows by u+++1 Wwt F(ws) and the last line above follows by the fact that
1,+1
(1)
17²)
11
= (1
(1? + +))r²)
2
2
2
2 (1) 11)
by 11 S T,2 +
0
Then by telescoping sum,
2 V (1:))1 < (F(wo) F(Wr+1))
7
< 2 =(((1.)))
77
where the second line follows by the fact that the loss F(wa) is maximized at up = (-1, 0) (over the space of initializations of wo) and by the fact that the loss F(WT+1) is nonnegative. Then defining
[[FF(w))] < C. More precisely, C, 7) ()))),)) we have that for some (0,1,.... 73, [ we have that
[< x)- oo (VTT())I(( Tx 2 0) C
T
Given that r ~ Unifor((LL,L) x {1}. by considering the second element of the gradient, which corresponds to the constant component of x, this implies that
[x x) of T I)) T x 2 0) C
T
Additionally, by Lemma we have that dhg S w1 S and bws S 0, and thus by Lemma E,2 we have that
Cn T
Eylo(w [0o[w>))==-- = Ex x) ov xT)[(ae 2 2)1² S
Hence,
C 7/
Ex|o(w, x))
Then, noting that vp w. S 0 and bw. O. have that E (a (w) x:) is the area of the
triangle formed by the line w. x over the r-axis with its base ranging from its x-intercept to L.
We denote the length of its base by b and its height by h. Given that w, 0 it follows that h
ow, (-L, 1)) = -. (w, z). Additionally, we have that / = | 6 S | 6 S b
given that -1 < Wn S W; I 0. Hence we have that
h2
bh
C
Eolo(w, x)
4L
4L
C. < Hence, we obtain By the choice of T have that 4 T
C ALV
sup
alwe x²) = h /
Se
x1(-L,1|x11)
T
29

---

Published as a conference paper at ICLR 2025
E.2.4 CONVERGENCE AFTER POSITIVE INITIALIZATION
Lemma E.6. Suppose that æ is sampled according to x Uniform(-LL, L, L) x [1] S E the target parameters (Ü, bu) satisfy ő E [1, maax] and -/ by S 0, and initial parameters wy satisfy dãp E (0, 1] and wwo = O. Then there exists some 8 # such that F(wo) - F(0) 8.
Proof.
F(wo)) = Ezl ( a(v x)))2 x))²
= -F()) + Bx (w 2)²1 Eu z)o
2
/
L
=(00)
x dx 2 () x)]
by Ui 2 and bero 0
4L
L
F(0)
Ey 2
12
Therefore, we define 5 = L (o x x))r xx)] and we proceed to show that ā > L.
12
We let by that (z, 1) the z-intercept the line vTT. x. Then, vTx o(x 2). Moreover,
2 E [0, | since b. -- 0] and 2 Therefore, Vă 2 2, v x 2 0 and vā < 2, v x < 0. Since
buto = 0 and ui 2 0, then vā 2 O, wa x 2 0 and VI < 0, wo x < 0. Thus,
Tnzū(ā - =) if 2
a (w x)o(v
(16)
if < 2
From here, can bound the second term of follows,
L
Ex (oloww)))ooo"" xx]] =
y² yzdy
by 16
2L
1
1,3
I,2
23
w 0
2L
3
2
Then,
LZ
8
+ E,
( 2)o(v' [TT)
12
w 2
1
L
L
+
12
2T.
3
1
33
L
by Ün
2 0, = E 10, ~/
4
Wn
12
2T.
3
12
2T.
I
12
W L
00
Wp I
2
by 2 1, ty 5
24
Thus,
F(wa): = F(0) 8 = F(0)
24
Lemma E.7. Suppose that æ is sampled according to x ~ Uniform(-L, x (1) C IR2,, the target
parameters (ū, bw) satisfy 7 E [1, [max] and L b, S 0, and initial parameters wh satisfy
wñp (0, 1] and bwn 0. Then there exists some g ". such that F F(wo) < F(0) 8. Then
30

---

Hibllahie as a conferenne paper at (CLR 2025
gt  3.12%( 3
and applyingg gradient descent according to equation s
defining 7 with learning rate r) I (1=t))) we have Laveiiie that At 2 0
W - - ull? ool s (1 - (1- (/wo - rll²
(17)
F(wi) ((wi) s 1 (12 1/10² 1)(1 1i--111
(1-1177''luu l1mo - u11
1777
(18)
E.6 there exists some 6 > Lnde such that F(wb) S F(0) 66. Then 17) follows by the latter inequaii and slight cedifficattin of the proof of Theorem 5.2 (along with Lemma D.2, Lemma D.4, and Lemmmmm 1).) of Vardi et al. (202T), to extend the result to targed
Proof. By Lemma
24
g
marammnn v modification of the porcementie proofs implics pediificatio
with arbitrary geanttudee
2 that setting 7 3.1221 Iuse Speeetiiiniee a simplee
where we take c .max || = vL² + +1 and c' = 1 due
to x Unifoomy -L. L) {1) vltrzzniie
17 Finally, (18 follows by
1 F(ux) = Fel
2
(o (w) x) - =oo
TxTx)) x))² |
by o()) L.L.ipcchite contimuuus Lemmnn. E.1
s Ea 5 ( x - v x)
by the Cauchy Schwarz aewwarz veequailily
- 2 |0 - u1|2E, |||=|| |2]
$ 1 (T. 1)]| w112
by E (1 1,,,, [.] x (1)
1
$ 2 (1²2 + 1)(1 - 1)(1-))A)111)) v/12
by 17
E.2.5 No RESETS AFTER A POSITIVE COLALLLZTTOO
Lemmmni a F. Suppose that x is sample recordingg to x ~ Unifomii - L, L) x {1} C RR, the targed
parumir v = (ū, b,) sultsty ă e [1, 0max] and
Lt by S 0. Let e > 0 such such that 3(L - 2) 2 5
(hen for any w E IR² such snchtii that
I To 15553111 Aey 2 F(0)
üL 2
sup v (0-11110
Proof.
F(w)) = Ball (o x) - o x
2 En(lo(w x) o(u x)) 2 Tzod
olv
2 Ea((o(v x) - e ) (mt [22)]
by -
sup
w x S 6
Let = = " such that T (z, 1) = O. Then u (2 g 1] = c and since 0 > 0 then Vž 2 2 + 8 we
iince?
have that v (x, 1) z e. Therefore.
B. x)) e) (0? (o
= En - E) !
1 p
(oy+ + by - =)
(ōy
21
(2+
1
(
since 0 2 0
2L
32
(1
E sp
67.
31

---

Published as a conference paper at ICLR 2025
By similar argument, have that
F(0) = [ T x)
1
(vy + bo) P
2L
(
J
2L
0²
(L - 2)3
67
Then,
8²
F(w)-F(00) > ((L
) (L =2)²)
61.
72
22
(-3(L - 2)2 ?
3(L
=)
6T.
7/2
0
(L 2)?E
since 3(L -))
2L
Hence, we have that
iL
F(w) 2 F(0)
2
Uniform(-L,L1 L) (1) B2 and the target parameters (Ü, bb) satisfy [1, Fmaxx] and S b, S 0. Let >> 0 be reset threshold and suppose that the initial parameters wo satisfy up E ( (mmin 1] and bwo = 0 such that
Lemma E.9. Suppose that is sampled according
120c
Ømin
(19)
L
and
3(L (1)=))2
Then there exists a 5 > t.th2 such that defining 7 gradient descent with resets, according to (e(w+))) = False Vt 2 0.
(1) and applying
3-12"
8 and 9 with learning rate n (1=-1)² we have that
Proof. According Lemma E.7 there exists some 8 2 w. such that
F(wo) & F(0) 5
Additionally, Lemma E.7 which utilizes modified proof of Theorem 5.2 of Vardi et al. (2021)
ensures that
F(u++1) < F(0) 8, vi 2 0 For the sake of contradiction, we suppose that for some 2 0, Oc(4++1) True, or equivalently, that
sup EE L.1
U Se
Then according to Lemma E.8
F(u+11) 2 F(0) TL.
However, we note that
wp 2
W Wmin 2
>
24
24
üL
by assumption 19
2
32

---

Published as a conference paper at ICLR 2025
Then we have that
L2 L
F(U(+1) F(O)
öL
2
ūhg La
24
which is a contradiction. Therefore, it must follow that Vt 2 0, (u(+1) = False
E.2.6 PROOF OF THEOREM
Theorem E.3 (Restatement of Theorem E.2). Suppose that a is sampled according to ~
Uniform( L, {1] IR_ and the target parameters v (ö, by) satisfy e [1, Umax] and < by 0, where denote Umax (ūmax: ()) Let >> be reset threshold and sup- pose that the initial parameters are sampled uniformly from e0p ~ [-1, (wmin) U (Wmin 1] and b bw 0 such that
12ûE L
Wmin
and
3(L -) 2
Then gradient descent with a constant learning rate of
2h min L
S 3 3.122 (2 + || Uax ( 2 1)6
and with resets, i.e. 8 and attains an average regret of
C E S T
where
C= 0= -=1
32.. 32T,2F(-1100 F((-1,0) 2n7²
(8imxx (immx-WWann Wmin) ) (123
njc
Proof. In order apply Lemmas
E.5
E.7
and E.9
verify that
1) 5
(20)
12 +1
and
(21)
(L2 + 1)2
where 7 = 3 (1 and > L2 24 and where wo is an arbitrary (re)initialization
of w given the prior distribution. For 20 we note that Wmin S 1 S 122 223 (2 + |/max |) and so
= 3. 122 (2 + Faxx (L2 + 1)6
I
(I2 + 1)6
(I.2 + 1)3
(L2 + 1)6
(L2 + 1)3
41
33

---

Published as a conference paper at ICLR 2025
As for 21 proceed follows
min
3 122 24
Umax
(L2 + 1)6
L
3 12a 24
+ ||a|| + 1)3 1)
53
W6g T.2 by 5 >
3. 12 (210ll
w00l
+ ||o||
1)+
24
(I, + 1)
We continue by upper bounding the regret as follows
E[AT] E[ resets occur]
By Lemma E.5 and Lemma E.9 and the choice of learning rate 7, we only ever reset if Wy is (re)initialized such that Wz < Conversely, if (re)initialize the parameters such that wy >> then we never reset for subsequent gradient descent updates. Additionally, since Wy is reinitialized by an independent draw from its initial distribution, then for any particular reset count r we have that
(r resets occur) = Piño < 0)" (P > 0) 2-+1
Where IP(wn < 0) P( > 0) by the fact wo is sampled uniformly from [-1, -Wmin) U
(Umin: 1]. Hence, we have that
E | |r resets occur] < (rr resets occur) occur)(rM)))) (r + M+) 2 (rM- + M+)
(22)
Where M_. is an upper bound on the total loss during any period of consecutive time steps to l' such that wy is a (re)initialization of such that W. < and l' is the earliest reset after I. While M4, an upper bound on the total loss for time periods after and including where w, is (re)initialization of w such that We > O.
By the choice of learning rate 7) and Lemma E.5 if w is (re)initialized such that w; < 0 then within
Trese 1 [322* F((-10)) gradient descent updates w is reset. Moreover, the loss F(wy) is at most F((-1,0) for time steps t' preceding reset of w as loss is maximized at = (-1,0) over initializations and choosing learning rate 7 at most I=4I. the Lipschitz constant of V (Lemma
17*
E.3h. guarantees that loss never exceeds F((-1,0)) Thus,
M_ FreseF(((111
(23)
As for M4 we can construct the following upper bound
M < Eww
F(w,) (w0 0]
2 + 1)(1 - -77) EEmo [||wo- - -111//000 0]
by Lemma E.7
t=0
T.2 + 1
Ens [1/uo -0119
by geometric series
m
1 [I2 +1)
(Fmaax (Umax Wiii)) + (4)3))
by assumption on wp,
m
1
(Umax Wmin) )² + [112)
by S I +1
2
Where the use of the geometric series in the third line above is valid by
17
34

---

Published as a conference paper at ICLR 2025
by the assumptions 17 and observe that
Then returning to goal of bounding the average regret,
Bl
B
5
E[Rr|r resets occur]
r=0
21+1 (r M+)
22
M+ + (M- + +M+)
Σ
r
2
2
==
M + (M- + M+)
by arithmetico-gcometric series
M_ M+
Then defining C = M_ + 3 M+, we have the desired result of
C S
EiRTl
T
T
35


Reviews:
[{"review_id": "iWc2lk7hgo", "reviewer": "Reviewer_HcgF", "summary": "The paper proposes a particular heuristic (SNR) to identify neurons that need to be reset in order to recover plasticity. They show that this particular choice of resetting neurons works very well. The work motivates intuitively how SNR compares to more complex utility measures proposed previously, and show results on a bunch of continual learning problems.", "questions": "1. Could you move other things to the appendix to make room for a conclusion / discussion section? The results section is meant to include the discussion, but I don't think it goes sufficiently deep into the discussion aspect of it. \n\n2. Can you run an ablation showing what happens if you only L2 the attention weights?\n \n3. Can you provide some more insights in how the heuristic works better than the others? Is it that is resets less, therefore is less disruptive. Or the amount of reset correlates better with the firing rate of the unit? Or something else? \n\n4. There has been a bit of discussion in the papers that you cite (e.g. works from Lyle et al.) on the causes of the loss of plasticity. Can you somehow connect a bit more the heuristic and the causes of plasticity? Right now it feels like the heuristic is just more reliable at predicting when a unit is actually dormant rather then firing rarely. But maybe is not as much connected to the problem it tries to resolve, that of loss of plasticity.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method is simple, easy to implement. And empirically on the problems it has been applied to, it outperforms the existing baseline, which are similar reset methods that use more complex conditions to reset (basically define complex utility functions that are used to make this selection).", "weaknesses": "I'm somewhat concerned about the significance of the results. I think the method outperforms the baseline, but most more detailed results tend to be on permuted MNIST or permuted Shakespeare. I know previous work relied on similar small scale problems, and it feels unfair to punish this work, while the others got away with it. But at some point I'm worried that we are reading too much into these numbers. \n\nThe authors have run on other tasks (Random Label Cifar, Continual ImageNet) but we can only see an aggregate result on those. And why they highlight specific conditions, they are still borderline as dataset (at least the random label cifar). I think some previous papers also showed results on atari which I think is a more complex setting. And previous works have not scaled things a lot more either. \n\nA second weakness is in my view, not sufficient analysis. E.g. trying to compare some of those utility functions with the SNR heuristic. How often they disagree. Is the intuition that some of these utility functions don't take into account the firing rate of the unit the answer (it the paper there is a suggestion in this direction). The paper argues that L2init is needed for Shakespeare because of the attention layer. Has this been ablated (e.g. apply L2 init just to attention weights?).  Or is the L2init within the MLP block important. And if so why? \n\nPutting them together, I feel that the main sales pitch is that it is a simpler utility function and it got better numbers, but there is less on understanding why this is the right thing to do. And I'm worried at such it might not have the impact on the community it should. \n\nFinally I find the lack of conclusion and discussion a bit strange. I understand that there was a page limit, but I find the solution of not providing a discussion suboptimal. I would urge the authors to find other ways to reduce the paper size.  \n\nOverall I'm happy to increase my score if some of the questions are answered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "63ipYQL2gY", "reviewer": "Reviewer_aGhR", "summary": "This paper proposes a plasticity loss mitigation method called Self Normalized Resets (SNR). It proposes a new method to decide when a neuron should be reset. Specifically, for each neuron, it tracks the mean time between non zero activations of that neuron, models a geometric distribution with that mean, and resets the neuron if the probability that a random variable with this distribution is greater than the neuron’s current time without firing is less than some hyperparameter.\n\nThe paper performs some experiments on Permuted MNIST, Random Label MNIST, Random Label CIFAR10, Continual Imagenet, and Permuted Shakespeare, showing a slight improvement or matching previous baselines.", "questions": "- How do you reset the neurons that you select? Is it the same procedure as in ReDo?\n- How does this method do at improving generalization/testing accuracy across distribution shifts?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper’s new mechanism to select which neurons to reset seems interesting. The theory and intuition behind the method need to be explained more clearly, but the results do show a slight improvement on some settings for some datasets.", "weaknesses": "- This paper feels incomplete. The entire theory section in Section 4 is hard to follow and does not feel very motivated. It’s unclear why Section 4.2 is put in the paper. The paper then immediately ends after an equation, with no explanation of how it relates to the rest of the paper, and it is missing a conclusion section.\n- The results in Table 1 are conducted over 5 seeds, but they are missing any error bars.\n- Table 2 shows results on a task introduced in this paper, Permuted Shakespeare, but several of the baselines for earlier experiments are missing for this dataset.\n- The caption for Figure 3 could be made clearer.\n- The details on the hyperparameter search could be clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lGsnvdsdsi", "reviewer": "Reviewer_GyG5", "summary": "This work presents a new method for avoiding plasticity loss in continual learning settings (where neural networks lose their ability to learn new tasks after being trained on other tasks). The proposed method is called Self-Normalized Resets (SNR) and simply involves choosing a threshold time of zero activations, so that any neuron that does not activate (has 0 output) for the threshold number of training iterations has its weights randomly reset during training. SNR is found to outperform other methods in a number of small continual learning image classification tasks, and combines with L2 regularization to outperform on a small natural language task.", "questions": "One issue with the tasks chosen is that they are entirely disjoint and do not test for robustness to distribution shifts or catastrophic forgetting - e.g. training on task A and B alternately and trying to maintain accuracy on A. Although out of the immediate scope of neural plasticity, I think these factors are important to consider for practical settings. How does SNR interact with catastrophic forgetting, i.e. does it make it better or worse?\n\nIs L2+SNR competitive with SNR alone for the tasks other than Permuted Shakespeare? If so, it would be nice to promote L2+SNR as a \"default\" method that works for all tasks.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "__Originality:__ The idea of resetting neurons to avoid loss of plasticity is not particularly novel or surprising (as evidenced by the cited related work), but the proposed method appears novel and most importantly, satisfies multiple desiderata (simple, effective, robust) well beyond previous methods.\n\n__Quality:__ Although on relatively small tasks, the experiments are thorough and convincing. The comparisons are made on a variety of datasets and consideration is made for hyperparameter tuning. A number of known covariates for plasticity loss (L2 complexity of parameters, dead neurons) are plotted to explain the effectiveness of SNR versus other methods.\n\n__Clarity:__ The paper is overall well written and easy to understand (see Weaknesses for a few areas that need improvement).\n\n__Significance:__ The results are only on small datasets, but nevertheless show promise (scaling Permuted Shakespeare gives a positive trend in terms of the method's superiority). The robustness of SNR to hyperparameter choice is very striking. To highlight this, I actually would recommend putting at least 1 figure or table in the main text to summarize one of the tables in Appendix C: for example, plotting hyperparameter strength vs accuracy on one CL task for the compared methods.", "weaknesses": "This paragraph is confusing: \"This reveals an interesting comparison with SNR. The schemes above will re-initialize a neuron after inactivity over a period of time that is uniform across all neurons. In the context of the hypothesis testing setup above, this will result in sub-optimal error rates across neurons. On the other hand, SNR will *reset a neuron after it is inactive for a period that is effectively normalized to the nominal firing rate of that neuron*, while still only specifying a single hyperparameter for the network.\" - referring to the italicized part of the last sentence, doesn't SNR also reset at a uniform rate of $\\bar{T}$? The only difference I can tell is that in the previously described method, neurons are reset all at once every $r$ minibatches, and it is possible for a neuron to be inactive for anywhere between $r$ and $2r-1$ minibatches before reset. If $\\bar{T} = r$ then SNR only differs in resetting neurons at exactly $r$ minibatches, and resetting individual neurons at different times during training. Could the authors please clarify this point?\n\nIn Permuted Shakespeare, what is the actual task being optimized (e.g. classification, next word prediction, etc.)?\n\nNotes on the presentation:\n- a discussion/conclusion section is missing\n- some figures have too much noise to be easily readable (e.g. figure 3) and could either have a lower sampling rate for the displayed points, or use a confidence interval to cover the range of variability.\n- typo on \"hyperparemter(s).\" (line 331)\n- typo in appendix B \"with the [second] element fixed as 1\" (line 723)\n- typo in appendix B: $w_0$ should be $w_{min}$ in the second part of \"We sample $w_0$ uniformly from $[−1, w_0 ) \\cup (w_0, 1]$\" (line 755)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H0KfOeoqP7", "reviewer": "Reviewer_hG3B", "summary": "The authors introduce Self-Normalized Resets (SNR) which is an adaptive algorithm that resets neuron weights when there is a drop in its firing rate in order to reduce plasticity loss. They also conduct a theoretical investigation of the optimization landscape for learning a single ReLU neuron. Their empirical results show that while existing baselines struggle in terms of sensitivity and detecting inactive neurons, SNR is robust to hyperparameter tuning, and can result in better performance.", "questions": "- How would SNR's test accuracy (on the held-out dataset) compare to other reset methods? Recent works have been focusing on generalization performance showing that while reset-based methods may result in better training accuracy, they often struggle in terms of generalizability and are outperformed by regularization methods like L2 [1].\n- Why do some of the methods in Table 1 results show a significant drop in performance compared to others? Is it dependent on the size/type of hyper-parameter search?\n\n**I am willing to increase the score if the authors address the concerns and questions raised.**\n\n[1] Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. arXiv preprint arXiv:2406.02596, 2024.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- As resetting neurons is a well-known technique in addressing plasticity loss, the main novelty of this paper lies in its exploration of the question of 'how to reset' the neurons. If I understood correctly, by focusing on the *recent* activity of a neuron rather than activity within a uniform period of time, SNR tries to answer the question by selecting individual neurons to reset by being more sensitive to the distribution shifts.\n- Their empirical results (especially Table 1) clearly show better average accuracy on a variety of benchmarks. In addition, SNR is better in terms of robustness to its hyper-parameter as compared to other baselines.", "weaknesses": "- While I understand the adaptive nature of SNR in selecting neurons, wouldn't it also result in over-dependence on the type of distribution shifts across tasks? The vision experiments, in particular, are primarily conducted on benchmarks where a task is uniformly sampled. As a result, the boundaries (or shifts) between consecutive tasks are relatively uniform as well. But what if instead of uniformly, the tasks arrive based on increasing/decreasing (or both) levels of noise and difficulty [1]? A controlled experiment could be conducted to analyze how the nominal firing rate of that neuron changes based on such distribution shifts. \n- The authors described their method SNR as an alternative way to pick neurons to reset them. However, in their experiments, they did not compare SNR with Redo and CBP in Table 2. Also, the standard deviation is crucially missing in Table 1. \n- The paper appears to be incomplete i.e., it ends abruptly after section 4 (with theorem 4.2) without providing any discussion points, limitations, and conclusion. \n\n\nThere are several grammatical/clarity issues - \n- Line 215 - tacking the histogram\n- Lines 57 - 60 can be explained better. \n- Line 269 - Shouldn't it be 1000 classes?\n- Line 377 - seems redundant after the previous para", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pUd7KhWDC5", "reviewer": "Reviewer_PBeJ", "summary": "The paper presents a reset-based approach to continual learning, where a network's neurons are reset based on a function of their inactivity. A different threshold function is applied to each neuron. The proposed method is compared to other reset schemes as well as regularization-based approaches. Experiments show promising performance on several datasets (such as MNIST-adjacent ones). Some theory is presented in a setting for learning a single target ReLU.", "questions": "Section 2 - Each (x_t, y_t) is drawn from a separate distribution \\mu_t? Is there any relationship between the different \\mu_t?\nLine 134 - This looks like dynamic regret, why is that (pessimistic) objective considered?\nLine 136 -  Is this the definition of plasticity? That the averaged dynamic regret grows faster than a constant?\nLine 144 - What is the formal definition of a 'dead' or 'inactive' neuron?\nLine 153/5 - The definitions of Z and A are unclear. What exactly is s indexing? Which two consecutive activations for A?\nLine 155 - 'let let' typo\nLine 194 - what guarantees do you have on this approximation? \\mu_t seems to be the distribution at a particular time, for which you only have one sample or a batch of samples? Why is considering 'firing times' while training on different, past, tasks/distributions a principled thing to do?\n\nHow exactly is A_i^{\\mu_t} approximated? I would like more detail than footnote 2.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method appears to outperform the selected baselines on a variety of datasets and exhibits good dependence on its sole hyperparameter. The experiments also appear to be rather thorough. For that reason, I think the proposed method may be practical.\n\nThe writing is also strong.", "weaknesses": "The experiments appear to be in a rather small setting, which is a slight problem since I believe the paper's main contribution is an experimental one.\n\nSection 4 (theoretical analysis) studies an exceedingly simplified setting. There is no notion of distribution shift or changing tasks (as the expectation is taken with respect to \\mu rather than a \\mu_t), and the algorithm is full gradient descent, which is not only impractical in real-world settings but impossible without knowledge of the underlying distribution. The second (subtracted) term in the regret also appears to be 0. The restrictions on v in Theorem 4.1 seem rather restrictive and I don't know if they are necessary. It's also unclear if the algorithm suggested in Theorem 4.2 is executable - are the variables with 'min' and 'max' known a priori? \n\nNo conclusion section.\n\nIf the authors provide strong answers to my concerns, I would be happy to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)