Learning Continually by Spectral Regularization
Authors: Alex Lewandowski
Michał Bortkiewicz
Saurabh Kumar
András György
Dale Schuurmans
Mateusz Ostaszewski
Marlos C. Machado
Venue: ICLR 2025 Poster (2025)
Keywords: plasticity
neural networks
spectral regularization
continual learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Loss of plasticity is a phenomenon where neural networks can become more difficult to train over the course of learning. Continual learning algorithms seek to mitigate this effect by sustaining good performance while maintaining network trainability. We develop a new technique for improving continual learning inspired by the observation that the singular values of the neural network parameters at initialization are an important factor for trainability during early phases of learning. From this perspective, we derive a new spectral regularizer for continual learning that better sustains these beneficial initialization properties throughout training. In particular, the regularizer keeps the maximum singular value of each layer close to one. Spectral regularization directly ensures that gradient diversity is maintained throughout training, which promotes continual trainability, while minimally interfering with performance in a single task. We present an experimental analysis that shows how the proposed spectral regularizer can sustain trainability and performance across a range of model architectures in continual supervised and reinforcement learning settings. Spectral regularization is less sensitive to hyperparameters while demonstrating better training in individual tasks, sustaining trainability as new tasks arrive, and achieving better generalization performance..

Published as a conference paper at ICLR 2025
LEARNING CONTINUALLY BY SPECTRAL REGULARIZATION
Alex Lewandowski 1,5
Michal Bortkiewicz
Saurabh Kumar
András György* Dale Schuurmans -4,56, 14,5,6
Mateusz Ostaszewski Marlos C. Machado 1,5,6
University of Alberta, "Warsaw University of Technology, Staanford University, 'Google DeepMind, $ AAmi,, "Canada CIFAR AI Chair
ABSTRACT
Loss of plasticity is a phenomenon where neural networks can become more difficult to train over the course of learning. Continual learning algorithms seek mitigate this effect by sustaining good performance while maintaining network trainability. We develop a new technique for improving continual learning inspired by the observation that the singular values of the neural network parameters initialization are an important factor for trainability during early phases of learning. From this perspective, we derive a new spectral regularizer for continual learning that better sustains these beneficial initialization properties throughout training. In particular, the regularizer keeps the maximum singular value of each layer close one. Spectral regularization directly ensures that grudient diversity is maintained throughout training, which promotes continual trainability, while minimally interfering with performance in single task. We present experimental analysis that shows how the proposed spectral regularizer can sustain trainability and performance across range of model architectures in continual supervised and reinforcement learning settings. Spectral regularization is less sensitive to hyperparameters while demonstrating better training individual tasks, sustaining trainability as new tasks arrive, and achieving better generalization performance.
INTRODUCTION
longstanding goal of machine learning research is to develop algorithms that can learn continually and cope with unforeseen changes in the data distribution (Ring 1994 Thrun 1998). Current learning algorithms, however, struggle to learn from dynamically changing targets and are unable to adapt gracefully to unforeseen changes in the distribution during the learning process (Abbas) ctt al. 2023 Dohare et al. 2024 Lyle et al., 2023 Zilly et al. 2021). Such limitations can be seen as a byproduct of assuming, one way or another, that the problem is stationary. Recently, there has been growing recognition of the fact that there are limitations to what can be learned from a fixed and unchanging dataset (Hoffmann al., (022), that there are implicit non-stationarities in many problems of interest (Igl et al. 2222) and that some real-world problems benefit from learning continually (Han et al. 2022 Janjua et al. 2023).
The concept of plasticity has been receiving growing attention in the continual learning literature. where the loss of plasticity-either a reduction neural network's ability to train (Dohare et al.
2021 [02lyElsayed and Mahmood| [2024] Lyle et al. 2022), or ability to generalize (Ash and Adams] 2020 Zilly al.] 2021) been noted as critical shortcoming current learning algorithms. Learning algorithms, and more specifically neural networks, that are performant in the non-continual learning setting, often struggle when applied to continual learning problems. Settings where a neural network must continue to learn after changes occur in the data distribution exhibit striking loss of plasticity such that learning slows down (Lyle et al.
2023) or even halts after successive changes (Abbas et al.) 2023 Dohare et al.) 2024 Nikishin et al.
2022
Several aspects of a learning algorithm have been found to contribute to or mitigate loss of plasticity. Examples include the type of optimizer (Dohare et al. 2024 Lyle et al. 2023), the step-size (Ash)
and Adams 2020 Berariu et al.) [021). the number of optimiser iterations (Lyle et al. 2023) and
the of specific regularizers (Dohare al., 2022 Kumar al. 2023 Lewandowski al. 2023
Lyle et al. (022).. Such factors hint that there might be simpler underlying optimisation principles
that govern the loss of plasticity. For example, the success of several methods that regularize neural

---

Published as a conference paper at ICLR 2025
networks towards properties of the initialization suggests that some of those properties mitigate loss of plasticity (Dohare et al. 2021 Kumar et al. 2023 Lyle et al. 2022).
The properties of neural network weights at initialization are associated with trainability in the early phases of learning. Previous analyses have demonstrated that the trainability of deep neural networks can be improved by ensuring that the initialization variance of hidden activations and gradients remains uniform across the different layers of the neural network (Glorot and Bengio 2010), which has the effect of keeping the average singular value of the layerwise Jacobians close one. A stronger condition is dynamical isometry, where all singular values are close to one, or exactly one in the case of an orthogonal initialization (Pennington et al., 2017 Saxe et al. 2014; Xiao et al., 2018). Over the course of learning, the singular values deviate from their initialization and tend to grow over time (Martin and Mahoney 2021), which can reduce trainability and impede continual learning.
Given the effectiveness of initialization in ensuring the trainability of a neural network, continual learning algorithms may benefit from sustaining the relevant properties of the weights. This is the motivation of several methods that use the initialization either through regularization (Kumar jet al. (023) or weight reinitialization (Dohare et al., 2021 2024; Sokar et al. 2023). Such methods leverage the initialization explicitly, either by regularizing parameters towards their initial values or by resetting parameters by resampling them from the initialization distribution. In this paper, we seek to more directly address the loss of trainability by sustaining the key properties present at initialization, thereby striking a better balance between trainability and performance compared to approaches that replicate the initialization explicitly.
We investigate the key properties of the initialization that ensure trainability, how these properties lost over the course of learning, and the effects this has on continual learning. We identify that deviations from the initial singular value distribution can result in low gradient diversity, thereby impeding continual learming. Based on this analysis, we then introduce a spectral regularizer to control the deviation of the singular values by keeping the maximum singular value of each layer close to one directly address the deviation of the singular values from initialization over the course of learning. Our experiments show that spectral regularization is more performant and less sensitive to hyperparameters than other regularizers across datasets, nonstationarities, and architectures. While well-tuned regularizers are often able to mitigate loss of plasticity to varying degree, learning continually with spectral regularization is robust, achieving high initial and sustained performance. In particular, we show that spectral regularization is also capable of improving generalization with both Vision Transformer (Dosovitskiy et al. 2021) and ResNet- 18 (He et al. 20166) on continual versions of Nint- (Le and Yang 2015), CIFARIO, 10, CIFAR 100 (Krizhevsky 2009), and SVHN2 (Netzer et al.) 2011). Note that these datasets and architectures encompass all continual supervised learning experimental settings considered in the loss of plasticity literature. We also show that spectral regularization can improve the performance of soft-actor critic (Haarnoja et 20188) in reinforcement learning settings where loss of plasticity occurs due to primacy bias (Nikishin al. 2022).
PROBLEM SETTING
We investigate the trainability of neural network learning algorithms in the task-agnostic setting. We denote neural network with T. layers, defined recursively as fo(z): == (x). where ho(x))
the input vector, h4+1 (x) @(W, (1y + -b)) with an element-wise wise activation function, d and parameters 0 = {Ol..... 01} } where 0; ( bi is the weight and bias parameter for layer <ا We assume that fe is trained over a sequence of tasks: the rth task is specified by a distribution pr over the observation-tuut target pairs, denoted by (z. v). For simplicity distribution) changes periodically after every T iterations. In addition, we consider the agnostto setting, where the learning algorithm does not have access to information about the task except through the data that it samples. For each iteration in the task, f. E ((T 1)T, the learning algorithm optimises the neural network's parameters to minimize the objective
assume that the task (data
(0) = B(..v)>,, [<(o(())
for some loss function f. In this paper, we consider gradient- based methods to learn the weight parameters; the basic version of gradient descent would update the parameters at iteration t as g(t+1) (()) a VoJrre) (0) (0) p where rt) = [t/T] denotes the current task number. In practice this is done by considering the empirical error instead of Jr(e) using samples from the task
distribution, (x,, y2) ~ Pr. and usually some more involved optimization algorithm.
In this notation we suppressed the presence of biases, which can effectively be included in the parameters.
2

---

Published as a conference paper at ICLR 2025
The evaluation of the continual learning algorithm is performed at the end of task Lyle et al. 2023). Specifically, the learning algorithm is evaluated at times t = TT, after being given T iterations to learn on task A useful assumption in some problems is that each task is sampled independently and identically, and that each task is equally difficult, meaning that a neural network which is randomly initialized each task is able reach a similar objective value for any particular task (Dohare et al.)
2024, Elsayed and Mahmood, [0244] Lyle et 2023)) While this assumption is useful, it is not always applicable. In class-incremental learning (Van de Ven al. 2022), the addition of more classes increases the difficulty of subsequent tasks. Another example is the transient tasks induced over the course of reinforcement learning (Igl et al.] (222]) which can have varying task difficulty and lead
to loss of plasticity due to primacy bias (Nikishin et al. 2022). We also consider such settings here.
Loss of plasticity in the continual learning literature can refer to either loss of trainability (Dohare et al.) |021]] Lyle et al. (023) or to loss of generalization (Ash and Adams] 2020). Because trainability is a requirement for learning and generalization, we focus primarily on loss of trainability.
Specifically, we use loss of trainability to refer to the phenomenon that the objective value, Jr (B((T)) increases as a function of the task т. Equivalently, the performance measures, such as accuracy. decrease with new tasks. Under the assumption that the tasks are sampled independently and identically, this would suggest that the neural network's 's trainability diminishes on new tasks.
SPECTRAL PROPERTIES AND CONTINUAL TRAINABILITY
At a high level, the reason behind loss of plasticity is simple: the solution of one task becomes the initialization for learning on the next task, and if this initialization is not sufficiently good to enable learning in the new task, we face the aforementioned problem. To avoid this issue, we need to keep the network parameters within region that can serve as good initialization at the start of task. This approach comes with two challenges: (i) determining a suitable region for parameter initialization on any given task; (ii) ensuring that training within the initialization region does not prevent effective learning on the current task.
is easy to satisfy (i) and (ii) in isolation: using standard initialization algorithm, such as those by He et al. (2015); Hinton and Salakhutdinov
and then keeping the parameters fixed at this value satisfies (i) but clearly does not satisfy (i)). Conversely, not restricting the parameter space while using standard learning algorithm addresses (ii) but not (i). We seek balance of the two requirements. Note that (i) and (ii) are not binary properties; hence there can be a trade-off (e.g., the parameter that allows for better optimization
parameter initialization for the next task). It is worth noting that such an approach has been successfully applied in the context of online convex optimization in changing environments, where several algorithms can be recognized as running a standard learning algorithm (such as mirror descent) on a carefully selected subset of the parameter space (e.g.. György and Szepesvári 2016 Herbster and Warmuth 1998 Zinkevich| 2003).
the current task may be less favourable
In this paper, to address problem (i), we first identify key properties that initialization algorithms (He et al., 2015 Hinton and Salakhutdinoy (006) impose on the starting parameters of neural network. Then, we propose spectral regularization to sustain these properties while striking good balance between (i) and (ii).
3.1 SPECTRAL PROPERTIES AT INITIALIZATION
Neural network initialization is key to trainability (He et al. 2015: Hinton and Salakhutdinov 2006). One property of the initialization thought to be important that the layerwise mapping, h+ ReI. (0,hy), ). has a Jacobian with singular values that are close to or exactly one (Glorot and
Bengio| 2010; Pennington et al., 2017, Saxe et al., 2014, Xiao et al., 2018). Writing this Jacobian
explicitly, we have that J, = ahe
sh++1 = D,O, where D, = Diag(ReLU' (10rhy))). (e,L ReLu'((8 ((Bphiaa))
We can obtain upper and lower bounds on the singular values of the layerwise Jacobian in terms of the singular values of the weight matrix. Denoting the ordered singular values of B and D, (Dr), respectively, we have (D()oo) < a (J) < (D()))a,(0.) for all E {1,... .d} (Zhagg 2011 Theorem 131. In particular, if the spectral norm (largest singular value) of the weight matrix tj increases, then the spectral norm of the Jacobian D, increases as well, potentially impacting trainability. Furthermore, the condition number
by (a)):) < ... ai (0:) and (D(DD)
<(J)) = (1(.))/a(d.)) can be bounded with the product of the condition numbers of 0 and D,, ((()))
RRelL (z)) denotes the derivative of the Re function, and it equals to 1 for 0 and O for < 0 (we define it be 1 for J. 0).
3

---

Published as a conference paper at ICLR 2025
and k(D)) k(0i)/<(D)) < k(J)) I (())) Thus, if our goal is to keep the singular values of the Jacobian close to one by controlling the singular values of the weight matrix, we should ensure that the condition number of the latter is not too large.
3.2 AN ILLUSTRATIVE EXAMPLE
To make these points more concrete, consider hingle-layer neural network fa(z) 0gReLU(012) mapping x E IR? to R. Suppose that the first task is to fit x = (1,0) and y = 0 in mean-squared
error, An optimal solution with fo(œ) = y is 01 [
and 02 (0,a) a) for an arbitrarily small
value of a 2 0. Now consider that the next task is to fit x = (o, 1) and = 1. The gradients of the loss ((f()))))) [ {(fo (x))² are. for the given new (x. pair, Vo, ((f((r),) (fo y) Re (a² ())0,a) a) and Vo, Fo,((f((z),1) v) (fo(z) y)
(a² 1) 0 Perforring updates with these gradients keeps the @(a) parameters at the same
order while maintaining a loss of approximately 1. The condition number of the weight matrix 0 is 1/a, which requires ( /a) = (1:) update steps to make significant progress during learning. (Another example showing this empirically is given in Appendix A.
3.3 TRAINABILITY AND EFFECTIVE GRADIENT DIVERSITY
We can generalize the above observation to a problem of reduced gradient diversity. By gradient diversity, we mean the spread of singular vectors in the matrix of per-example stochastic gradients, G l........... Bm] If this matrix contains a few large singular values and many small singular values, then the gradients will largely in the span of the singular vectors corresponding to the large singular values. For this analysis, we focus on the rank, which is one particular summary statistic for the set of singular values that counts the number of non-zero singular values, rank(G) I(i: (G) > 0}|- However, for practical purposes, the rank is problematic because it is unstable to perturbations (Feng et al. 2022, Theorem 1). In our experiments below we will consider the condition number, (G1/ (G) (G) and effective rank, erank G) = Σ (G log (G), where o; (G) (:/CE (G) (Roy and Vetterli 2007). These measures make explicit the issues that arise if the largest singular value grows faster than the smallest singular values (our experiments show this to be the case in Section (522). In this case, the condition number increases and the effective rank decreases. We refer to this as reduction in the effective gradient diversity.
A reduction in the rank, or erank, of the gradient matrix results in collinear gradients on different datapoints, limiting the diversity of gradient directions used in the parameter update. This can have an adverse effect on learning. In the extreme case, where the gradient matrix is rank one, every datapoint provides gradient in the same direction, even if the datapoints correspond to different classes. Consider the gradient of the loss on a particular datapoint (æ4, y:) with respect to the weight matrix of hidden layer, Oz E ddd which can be written recursively for the parameters of layer as G1, Vo, (f,ss)) us) = b, where oU.i 0f 11,10 is the error gradient from the next layer with dz,; (f(fo(r.) and D Diag o' (1.),...,,,..... d' d((0))),,)la). We can rewrite the gradient in terms of its vectorization gt. vec(Gr,,)
(Ia @ 114,) where Ia is the d x d E æº is the vectorization of data-dependent terms. The matrix of gradients for different datapoints is the concatenation of the per-example gradients, G 1g.,, gi...... gt.m] .g. (Ia @ V, where V, [v vi.,..., Vi,m). The rank of G2 is upper bounded as
identity matrix and Vi,i = vec Dr.h,T .t.
rank (Gi) < minnkan @ 0T,1), (Vn))))))) mdnn rank())))
Thus, if the (effective) rank 0.41 decreases, then the (effective) rank of the gradient matrix may decrease as well. The rank of the gradient matrix can even decrease due to rank reduction in parameters at other layers, or through the rank of the representation, which others have noted to occasionally correlate with loss of trainability (Dohare et al. 2024 Kumar al. 2023 Lyle et al., 2023).
Why Do Spectral Properties Deviate From Initialization? In large-scale self-supervised learning,
it has been observed empirically that the parameter norm grows at a rate of Vł, where f is the number of iterations (Merrill et al., (021). Similar observations have been made in continual learning, provided that the neural network does not stop learning due to loss of trainability (Dohare et al.. 2024 Lyle et al. 2024 Nikishin et al. 2022). A growing parameter norm is problematic from an
4

---

Published as a conference paper at ICLR 2025
optimization perspective. Specifically, the parameter norm that these works consider is the Frobenius norm, which is equal to the sum of squared singular values, |00 Σ, -, 02; = Σ, o, (0)².
Thus, a growing parameter norm is equivalent to an increasing sum of the squared singular values. In particular, note that |0 S / /rran (1) a (0t) where (0t) denotes the largest singular value, or the spectral norm (Golub and Van Loan 2013). If the parameter norm of layer I grows at rate of vi, then the spectral norm of the parameter matrix Or also increases at the same rate. This leads to an increase in the spectral norm of the layerwise Jacobian, ( S 01 (Jt), which can reduce effective gradient diversity and may harm trainability.
SPECTRAL REGULARIZATION FOR CONTINUAL LEARNING
If important properties of the initialization are lost during the course of learning, natural to regularize the neural network toward the initialization. This is the motivation for regularization (Kumar et al. 2023) and weight reinitialization (Dohare et al.) 2024 Sokar et al., (022) as loss of plasticity mitigators. However, our motivation is to more directly target key properties of initialization, using the insights from Section 3
We denote regularizer by (R,,) s), which is a function of (i) the parameters of the neural network, 0, (ii) the data, through the current task r(), and of (iii) auxiliary information, such as the parameters at initialization, through the state variable s. Naturally, not every regularizer takes all of these elements into account. The regularizer is optimized alongside the base objective, J+(t) (0). We write the composite objective as J210 (0) (0) + XR (0, s), where is tunable hyperparameter governing the regularization strength.
In Section 3 we argued that growing spectral norm and condition number can harm trainability by re- ducing effective gradient diversity and that this growth occurs at a rate of VI. Now we investigate regu- larization a means of controlling the spectral norm maintain the trainability of the neural network.
One commonly used regularizer both continual and non-continual learning is 1.2 regularization, (+ Ø) |0012 which regularizes the parameters towards zero. A recent alternative, L2 regularization towards the initialization, ( (()) 1 - ((0) was proposed to deal with sensitivity parameters
zero (Kumar al., 2023 Regularizing towards the particular parameters present at initialization allows the neural network to regenerate parameters, providing soft reset parameters if the gradient the base objective is zero (see Appendix [A.2 for an example and Appendix A.4 for more details). One potential problem with L2 regularization towards the initialization is that may prevent the parameters from deviating from the particular sampled value from the initialization distribution.
Our proposed spectral regularizer explicitly regularizes each layer's spectral norm (the maximum singular value) and addresses the reduced effective gradient diversity described in Section 3.3 We only minimize the maximum singular value because the smallest singular value remains relatively constant across all the architectures we consider (see Appendix ( The parameter matrix 0 in layer 7 is a concatenation of the weight matrix W, and the bias b,, giving the augmented parameter
matrix 02 = [W/ [bj] and augmented input h; = [hr, 1]. With this notation, the equation defining the neural network fo can be rewritten as 1 ) = o( (W (fx(z)) + by) = ((0,h;). As discussed in Section 3 our goal is to control the largest singular value of 0j. Using the fact that the concatenated parameters can be upper-bounded by the sum, 0n ((())) = dn ([W]lb.) I a1 (W,) + 01 (b,)). we achieve this by regularizing the spectral norm of each parameter in the layer separately: the spectral norm of the multiplicative weight parameter W, is regularized towards one and the spectral norm of the additive bias parameter by is regularized towards zero:
(0:)
Σ
(or -12² (1)) (oi(bi)))))
Σ IElayers
((o) - 1)2 + [bbi (1)
1Elayess
We add the exponent & to the spectral norm to penalize large spectral norms deviating from one. We set = in our experiments and provide an ablation study in Appendix C.I For implementation details for other layers, such normalization and convolutional layers, see Appendix A.7 The largest singular value can be computed efficiently using power iteration (Golub and Van der Vorst 2000 Householder 2013) with computational complexity scaling linearly in the number of parameters.
5

---

Published as a conference paper at ICLR 2025
Similar to previous work using spectral regularization for generalization in supervised learning, we find that a single iteration is sufficient for effective regularization (Yoshida and Miyato 2017).
In addition to the exponent k. our approach to spectral regularization is different in two important ways compared to previous work, such as spectral-norn regularization (Yoshida and Miyato 2017). First, we regularize the spectral norm of every parameter, including bias terms and normalization parameters (see Appendix A.7). This is required because every parameter experiences norm growth. Second, we regularize the spectral norm of the multiplicative parameters one rather than zero. Aggressive regularization of the spectral norm towards zero can lead to collapse issues similar to L2 regularization towards zero. These differences are crucial to improving trainability in continual learning. We em- phasize that regularizing the maximum singular value is sufficient for sustaining trainability, and that other approaches such as 1.2 can lead to over regularization and increased sensitivity. Lastly, spectral regularization is preferable over spectral normalization (Miyato et al. 2018 Zhai et al. 2023), which explicitly normalizes the weights the forward pass to be exactly one, for two reasons: (i) spectral nor-
malization is data-dependent (see Section 2.1 and Equation 12 by Miyato et al. (2118) , and Appendix A.6). which can be problematic in continual learning due to the changing data distribution, and, (ii) other forms of normalization are already used to train deep neural networks, such as LayerNorm (Ba et al. 2016), and it has been shown that spectral normalization does not improve trainability (Lyle et al.
2023 See Appendix A.5 for detailed comparison to L.2 regularization and spectral normalization.
5 EXPERIMENTS
The goal of our experiments to investigate the effect of spectral regularization on trainability in continual supervised learning, as well as reinforcement learning. We cover several different datasets, types of nonstationarity, and architectures. We compare our proposed regularizers against baselines that have been shown to improve trainability in previous work, which we detail below. Overall, our experiments demonstrate that spectral regularization (i) consistently mitigates loss of trainability on a wide variety of continual supervised learning problems, including training large neural networks for thousands of epochs across a hundred tasks, (ii) is highly robust to the regularization strength, type of statinationartty and the number of training epochs per task, (iii) achieves better generalization performance over the course of continual learning, and (iv) is generally applicable, which we demonstrate by applying spectral regularization to reinforcement learning with continuous actions.
Datasets, Nonstationarities, and Architectures Our main results uses all commonly used im- age classification datasets for continual supervised learning: Lin--ImaeNNet (Le and Yang 2015). CIFAR10, and CIFAR1O0 (Krizhevsky 2009). Experiments in the appendix also smaller-scale datasets, like MNIST (LeCun et al.] 1998). Fashion MNIST (Xiao et al.] (017). EMNIST (Cohen) jet al., 2017), and SVHN2 (Netzer et al. 2011). In addition to the dataset, we consider different types of non-stationarity: (i) random label assignments, (ii) pixel permutation, (iii) label flipping, and (iv) class-incremeettal learning. Random label assignments are commonly used to evaluate trainability (Lyle et al.) 2023) due to the large distribution shift between tasks in memorizing completely new and random labels. Pixel permutations, on the other hand, require only learning the permutation mask applied to the image; it induces loss of trainability more slowly but is useful for evaluating generalization (Kumar et al. 2023). Label flipping is a re-assignment of all the observations from one label to another label. Unlike the other two non-stationarities, a label flip distribution shift does not require learning a new representation. This is because only the output layer of neural network needs to be permuted to learn the label re-assignment, but gradients representation leading to loss of trainability (Elsayed and Mahmood] (024).. We also consider the class-incremental setting in which the network is trained on growing subset of the classes from dataset, starting with only five classes on the first task and introducing five new classes on new tasks.
still unnecessarily change the
We use both a ResNet-18 18 He al. 2016) and Vision Transformer (Dosovitskiy et al., 2021).
Loss of Trainability Mitigators In our main results, compare spectral regularization against L2 regularization towards zero, shrink and perturb (Ash and Adams) 2020). L2 regularization towards the initialization (Kumar et al. 2023), recycling dormant neurons (ReDO, [Sokar al. 2023), concatenated ReLU (Abbas et al. 2023 Shang et al. 2016), and Wasserstein regularization (Leaandowski et al. 2023). Several regularizers in the continual learning without forgetting literature rely on privileged task information, which is not applicable to the task-agnostic setting that we consider. We use the streaming conversion (Elsayed and Mahmood 2024) to transform elastic weight
*On 1080TI, training with spectral regularization is approximately 14% slower.

---

Published as a conference paper at ICLR 2025
Class incremental
Labe Plip
Pixat Permutat on
0.22
17
(evr)
0:23
Accuracy
0.23
5.00
Ploc
0.00
0.02
633
0.04
0.02
10
40
80
12
30
40
00
14
Number of tasks
Numbar of tasks
Number tasks
Class incremental
Labe: Fiip
Psooi Permutation
0.20
Accurayy
6.30
Text Accuracy
0.100
0.35
0.00
0:075
0.33
0.23
50
6c
40
80
10
30
10
Number of tasks
Number of Tasks
Number of tacks
Figure 1: Generalization across different types of non-stationarity on tiny-ImageNee using a ResNet (top) Vision Transformer (bottom). Compared the baselines, spectral regularization is consistently among the best-performing methods across class incremental, label flip, and pixel permutation non-stationarities Note that the Vision Transformer often achieves better generalization performance than the ResNet architecture.
consolidation (Kirkpatrick et al. 2017 Zenke et al. 2017), so that it no longer requires task boundary information, and include as baseline. Additional experiment details can be found in Appendix B
Overview of Results Our results demonstrate that our method (i) performant on average, and better than the best baseline in 10 out of 12 problems considered; (ii) robust across the problems considered, given that it is always the top 2 pessfrerforming methods (the best baseline is in the top 2 only 7 out of 12 problems). Furthermore, the best baseline on a reinforcement learning problem, where the best baseline is only better than method the very end of the training for the original problem and becomes superior once the problem is varied.
also show that method is at least on par with
5.1 COMPARATIVE EVALUATION
In Figure I we plot the results of training a ResNet-18 and a Vision Transformer on ting-emaeeNet with different non-stationarities. Across networks, non-stationarities, and methods considered, we see that spectral regularization is among the methods best capable of sustaining plasticity. The advantage of our approach, spectral regularization, is particularly high in class-incremental learning, but the performance of spectral regularization with a Vision Transformer was also the best on label flipping and pixel permutation. In contrast, the performance of other baselines was highly variable with respect to the specific type of non-stationarity. For example, shrink and perturb is not robust to the settings considered. Sometimes, it is among the best pereftreeronnggg or worst-performing methods. Similar results other datasets and the pixel permutation non-stationarity can be seen in Figure 2
5.2 LOOKING INSIDE THE NETWORK
We now explore how the structural properties of a neural network evolve over the course of continual learning, how these properties are affected by spectral regularization, and the different baselines considered. For this, we consider the continual learning problem in which ResNet-18 18 must memorize a set of random labels which changes from task to task using the tiny-ImageNet, CIFARIO, and CIFAR 100 datasets. We consider the average representation change measure the distance between the neural network's 's hidden activations from the beginning of one task, to the beginning of the next task. The average representation change is proxy for plasticity, allowing us to how much the behavior of the neural network has changed. In Figure 3] we see that the unregularized ReLU networks suffers from loss of trainability in all problems considered, and this coincides with an
7

---

Published as a conference paper at ICLR 2025
SVI1N2
CITAR1O
CIARLO0
0.36
0.20
0.24
0.22
044
0.20
0.52
Tec: ALCur
0.3
Spectral
6.5
0.12
Figure 2: Continual learning with pixel permutation tasks on SVHN2, CIFARIO, CIFAR100 using ResNet-18 (top) or Vision Transformer (bottom). Across different datasets, spectral regularization effective at maintaining test accuracy on new tasks. Without any mitigators, both ResNet- 18 and Vision Transformer have diminishing test accuracy, suggesting loss of plasticity.
increasing spectral norm (middle-top) and decrease the average representation change (bottom) Although only spectral regularization directly regularizes the spectral norm of the network, other regularizers do so indirectly by controlling other norms, like L.2. However, these other regularizers also regularize other parameters, preventing them from deviating from initialization and potentially leading to suboptimal use of capacity, which can be observed in the bottom row Figure 3 In Appendix C.3 we show that the Vision Transformer not able to memorize random labels, suggesting that its capabilities for generalization are offset by its relatively lower trainabilty.
5.3 SENSITIVITY ANALYSIS
FARRDD
The effectiveness of regularizer depends on the regularization strength. Too much regularization can slow down training, leading to suboptimal performance on any given task. However, too little regularization can lead to loss of trainability and suboptimal performance on future tasks. In continual learning problem, hyperparameter tuning is particularly costly (or infeasible), especially if the distribution tasks is not known in advance (Meshahi al., 2024). Thus, should favour regularizers that are less sensitive to hyperparameters. Our results in Figure show that the sensitivity of spectral regularization to its hyperparameter is much lower than other regularizers.
Aeguastcatinn Stergh
Degusniiatinn Stregh
Inguntrrairr Streggt
Figure 4: Sensitivity analysis on regularization strength. Compared to other regularizers, spectral regularization is insensitive regulariza- tion strength while sustaining higher trainability for any given regular- ization strength.
Appendix C.4 we also present additional results showing the robustness of spectral regularization when varying (i) the regularization strength using other network architectures, (ii) the type of notsstationarity, and (iii) the number of training epochs per task.
'In Appendix C.10 provide table showing the final performance after training the sequence of tasks.
8

---

Published as a conference paper at ICLR 2025
Bry mep-mettl
CTAR100
0.
Acc/oC
0.
62-
12.5
Overral Max Sm1
11.0
3.0
Rausove Rup. Change (loa)
25
25
Nur ber of Tasks
Number of Tasiss
Number of Tisks
105
0.12
124
0.10
05r. Rooh
140
6.00
30
20
15
30
Trainability and neural network properties across ImageNet, CIFAR1O, and CIFAR100. Baselines that suffer from a loss of trainability (top) also have an increasing average spectral norm (middle-top), and a decrease in their average representation change (middle-bottom).
Figure 3:
5.4 FROM SUPERVISED LEARNING TO REINFORCEMENT LEARNING
In addition to supervised learning, evaluate spectral regularization in reinforcement learning. Specifically, we investigate the control tasks from the DMC benchmark (Tassa et al.] 2020), with soft actor-critic (SAC, |Haamoja et al.) 2018), in the high replay-ratio (RR) regime (D'Oro et al. 2022) with 16 gradient updates for every environment step. The high replay-ratio regime leads to primacy bias (Nikishin et al., 20022). phenomenon related loss plasticity. Recent work has demonstrated that only full network resets with layer normalization (Ba et al. 2016) serve as an effective mitigation strategy in this setting (Ball et 2023
Nauman | (024).
As shown in Figure 5 spectral regularization yields substantially higher returns compared to the baseline that uses layer normalization with either an unbounded replay buffer size limited replay buffer size. In aggregate, spectral regularization is competitive with the final performance of the strongest baseline, full network resets combination with layer normalization. However, in five out of seven environments, spectral regularization substantially improves over the reset baseline in terms of sample efficiency (see Appendix C.8). Spectral regularization is also less sensitive to its regularization strength than resets are for their reset frequency (Figure 5 middle-top and botdle-botttom.. Moreover, we used a single regularization strength for every network. Better perfor- mance may be achieved by individually tuning each regularization strength for the value and policy networks. In addition, a combination of spectral normalization and layer norm aggressively reduces Q-value overestimation (Figure 5 top-left), which is one of the well-established proxies for RI.
9

---

Published as a conference paper at ICLR 2025
Agoregate DMC Suite (Peplay Butter 3M)
ovanasiinaact
Scedta Iegudartaation
410
550
100
120
200
06
300
0.0
0.2
3.0
Stagos
0.6
0.0
1.0
1r1/08
Te 2E5. Te77/900.
200
DMC Subte Imaplay Buffer 2501)
150
200
250
200
-00
0.0
2.2
0.5
0.5
14
130
30
100
Steps
Pypersarnnte walue
3. 360
Figure 5: Spectral regularization enhances plasticity in reinforcement learning in the DMC suite. Spectral regularization competitive with the network reset layernorm (Reset) even when the replay buffer is unbounded (Top Left). When the replay buffer size is bounded to 250k steps, spectral regularization improves over the Reset baseline (Bottom Left). In both cases, Spectral regularization significantly outperforms layernorm (Baseline). Compared to the hyperparameter governing the reset frequency, spectral regularization is less sensitive to regularization strength. Spectral regularization also prevents both parameter and gradients from exploding, and reduces value overestimation (Righ().
training destabilization (Hasselt 2010 Nauman al., 2024). Lastly, spectral normalization prevents exploding gradients and keeps weights norms small during training, which is key component for learning continually and preserving the plasticity of neural networks (Lyle et al. 2022 2023).
6 CONCLUSION
In this paper, we investigated the connection between initialization, trainability, and regularization. We identified that deviations of the maximum singular value of each layer can lead to low gradient diver- sity, preventing neural networks from training on new tasks. To directly preserve trainability properties present at initialization, we proposed spectral regularization as a way of controlling the maximum singular value of each layer so that it does not deviate significantly from 1. Our experiments show that spectral regularization is more performant and less sensitive hyperparameters than other methods across datasets, nonstationarities, and architectures. While several methods mitigate loss of trainabil- ity to a varying degree, learning continually with spectral regularization is robust, achieving high initial and sustained performance. We also showed that spectral regularization is capable of improving generalization with both Vision Transformer and ResNet- 18 on continual versions of tiny- ImageNe,, CIFAR10, CIFARION 100, amongst others. Lastly, we showed that spectral regularization learned effective policy in reinforcement learning by avoiding early loss of plasticity due to primacy bias.
ACKNOWLEDGMENTS
The research is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada CIFAR AI Chair Program, Alberta Innovates, and the Digital Research Alliance of Canada. Michal Bortkiewicz and Mateusz Ostaszewski were partially supported by the National Science Centre, Poland, under grant 1/23/5////T6/011099
REFERENCES
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plasticity in continual deep reinforcement learning. In Conference Lifelong Learning Agents, 2023.
10

---

Published as a conference paper at ICLR 2025
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 2019.
Jordan T. Ash and Ryan P. Adams. On Sarm-Starting Neural Network Training. In Advances in Neural Information Processing Systems, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. 0as/1607.0440v.. 2016.
CoRR,
Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient Online Reinforcement Learning with Offline Data, 2023.
Martin Benning and Martin Burger. Modern regularization methods for inverse problems. Acta numerica, 27:1-111. 2018.
Tudor Berariu, Wojeiech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks. CoRR, abs/2106.00042. 2021.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In International Joint Conference Neural Networks (IJCNN), 2017.
Shibhansh Dohare, Richard S. Sutton, and A. Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. CoRR, abs/2108.06325v3 2021.
Shibhansh Dohare, J. Fernando Srmmandez-Gariiin Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, and Richard Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-774, 2024.
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G. Bellemare, and Aaron Courville. Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier. In International Conference Learning Representations, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
Mohamed Elsayed and A. Rupam Mahmood. Addressing loss of plasticity and catastrophic forgetting in continual learning. International Conference on Learning Representations, 2024.
Ruili Feng. Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. Rank diminishing in deep neural networks. Advances in Neural Information Processing Systems, 2022.
Tomer Galanti, Zachary S. Siegel, Aparna Gupte, and Tomaso Poggio. Characterizing the Implicit Bias of Regularized SGD in Rank Minimization. CoRR, abs/2206.05790
2022.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, 2010.
Gene H Golub and Henk A Van der Vorst. Eigenvalue computation in the 20th century. Journal of Computational and Applied Mathematics, (23(1-2):35-55, 2000.
Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
András György and Csaba Szepesvári. Shifting regret, mirror descent, and matrices. In Proceedings of The 33rd International Conference on Machine Learning, pages 2943-2951 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor, 2018.
Benjamin Han, Hyungjun Lee, and Sébastien Martin. Real-time rideshare driver supply values using online reinforcement learning. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
Hado Hasselt. Double Q-learning. In Advances in Neural Information Processing Systems, 2010.

---

Published as a conference paper at ICLR 2025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In International Conference on Computer Vision, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, 2016.
M. Herbster and M. K. Warmuth. Tracking the best expert. Machine Learning, 32(2):151-178, 1998.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-5000 2006.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 2022.
Alston S Householder. The theory of matrices in numerical analysis. Courier Corporation, 2013.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient con-stationarity and generalisation in deep reinforcement learning. In International Conference on Learning Representations, 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by
reducing internal covariate shift. In International Conference on Muchine Leurning, 2015.
Arthur Jacot. Implicit bias of large depth networks: a notion of rank for nonlinear functions. In International Conference Learning Representations, 2023.
Muhammad Kamran Janjua, Haseeb Shah, Martha White, Erfan Miahi, Marlos Machado, and Adam White. Gvfs in the real world: making predictions online for water treatment. Machine Learning, 2023.
Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant self-attention. In International Conference Machine Learning, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114 (13):3521-3526,, 2017.
et al. Overcoming
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under prameterrant inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021.
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. CoRR, abs/2308.1 1958v1, 2023.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 2015.
Yann LeCun, Corinna Cortes, and Burges. MNIST handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.ccom wwt.ex/b///iiiistt.. http://www...cc....coo////////s.
1998.
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min. Jaegul Choo. and Clare
Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. In International Conference Machine Learning, 2024.
12

---

Published as a conference paper at ICLR 2025
Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C. Machado. Directions of Curvature as an Explanation for Loss of Plasticity. CoRR, abs/2312.00246v22, 2023.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference Learning Representations, 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity neural networks. In International Conference on Machine Learning, 2023.
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the Causes of Plasticity Loss in Neural Networks. CoRR. abs/2402.
01122402.1776211 18762v1, 2024.
Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1-73, 2021.
William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, and Noah A Smith. Effects of parameter norm growth during transformer training: Inductive bias from gradient descent. In Conference on Empirical Methods in Natural Language Processing, 2021.
Golnaz Mesbahi, Olya Mastikhina, Parham Mohammad Panahi, Martha White, and Adam White. Tun- ing for the Unknown: Revisiting Evaluation Strategies for Lifelong RL. CoRR, abs/2404.02113v2. 2024.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018.
URL http:::///// httpps/:///pp.eeeee........
net wttp:/iweeeeeee/rtueeeee forum? i dd= 1==1@gggi1 T
Michal Nauman, Michal Bortkiewicz, Piotr Mitoš, Tomasz Trzciński, Mateusz Ostaszewski, and Marek Cygan. Overestimation, Overfitting, and Plasticity in Actor-Critic: The Bitter Lesson of Reinforcement Learning, 2024.
Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning, 2011.
Evgenii Nikishin, Max Schwarzer, Pierluca D' D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference Machine Learning. 2022.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in Neural Information Processing Systems, 2017.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, 2018.
Mark Bishop Ring. Continual learning in reinforcement environments. The University of Texas at Austin, 1994.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In European Signal Processing Conference. IEEE, 2007.
A Saxe, J McClelland, and S Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Represenatations, 2014.
Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In international conference machine learning, pages 2217-2225. PMLR, 2016.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference on Machine Learning. 2023.
13

---

Published as a conference paper at ICLR 2025
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs/ 131.6619999 2013.
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Piotr Trochim, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. Dm__ontrol: Software and Tasks for Continuous Control. Software Impacts, 6:100022, 2020.
Sebastian Thrun. Lifelong learning algorithms. In Learning to Learn, pages 181-209. Springer, 1998.
Nadav Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu networks. In International Conference on Algorithmic Learning Theory, 2023.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. Advances in Neural Information Processing Systems, 2018.
Gido M Van de Ven, Tinne Tuytelaars, and Andreas S. Tolias. Three types of incremental learning. 2022.
Nature Machine Intelligence,
Jingfeng Wu, Difan Zou, Vladimir Braverman, and Quanquan Gu. Direction matters: On the implicit bias of stochastic gradient descent with moderate learning rate. In International Conference Learning Representations, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning. 2018
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. CoRR, abs/1 10s/17055.0941v 2017
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence.
In International Conference on Machine Learning, 2017.
Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning. PMLR, 2023.
Fuzhen Zhang. Matrix theory: basic results and techniques. Springer Science & Business Media, 2011.
Julian Zilly, Alessandro Achille, Andrea Censi, and Emilio Frazzoli. On plasticity, invariance, and mutually frozen weights in sequential task learning. Advances in Neural Information Processing Systems, 2021.
M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proc. 20th International Conference on Machine Learning (ICML-2003), Washington, DC, 2003,
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL DETAILS
A.1 ILLUSTRATIVE EXAMPLE: LARGE SPECTRAL NORM CAN IMPEDE TRAINABILITY
We show that, given family of solutions on the first task, preference should be given solutions that have a spectral norm that is closer to one. We consider 2 dimensional binary and orthogonal inputs, 21 [0. 1]" and I2 [1. O] with corresponding binary targets, 3/1 = 1/2 1. We consider the squared loss, and use mutli-layer perceptron with one hidden layer and two units, funawo (x) W2 (2) A common observation regarding solutions found for deep neural networks is that they are low rank (Arora et al. 2019. Galanti et al. 2022 Jacot 2023 Timor et al. deep network emerge
2023). In addition, it has been observed that large outheers in the spectrum
during training, meaning that the largest singular values of the weight matrices tend to be much larger than their smallest singular values (Martin and Mahoney 2021] Pennington et al. 2018). Consider
c one such solution where w1 1/c -1 and w2 [4/, E R1x2 The input-output function is
invariant under different values of c. achieving zero error for any particular choice. However, the parameter matrix for the hidden activations, W1. low rank because it only has single non-zero singular value that is not invariant. That is, the largest singular value of depends the choice of c, || |l2 == 7mar((1) = c ++ 3/c. In addition, the gradients and parameter updates depend on c. Thus, if the targets were to change, v/ = 0, the imbalance of singular values due very large or small values can impede trainability:
-1 e w1(x) =
0
%/2
2/2
3/c -1
wg((1)) -V/,
[c,0 Ving &
(2)
Vay &
where l ((fwa.wr (21), v.). and is sizp- One of the consequences of the imbalanced singular values is that the per-example gradients are poorly conditioned (Wu et al. 202T). That is, when we sample æ1, there will be large gradient for W2, [V (31 vi c but small gradient for W1. V ((fwgmn (I1), 11 l/c. Thus, sufficiently small step-size that stabilizes the update to wy leads to slow learning on wi. A similar result holds the target 1/2 were to change In Figure 66| we demonstrate that learning is more stable with smaller spectral norm in this example. For higher dimensional weight matrices, we may expect that the imbalance of the singular values leads low gradient diversity where particular per-example gradients give rise outliers with large gradient magnitudes. Thus, all else being equal, a solution with smaller spectral norm (c Ro 1) preferable for the purposes of continual learning.
Trainability (a 0.1)
Trainability (a 0.05)
Trainability (a 0.01)
Loss
10
30"
Iterations
Iterations
Iterations
Figure 6: Illustrative Example. A lower spectral norm is more trainable across different step sizes in the simple illustrative example.
A.2 DIFFERENCES BETWEEN CONTINUAL AND NON-CONTINUAL LEARNING
One critical difference between continual and non-continual learning is that, in continual learning, convergence towards fixed point is not the objective. Due to continual changes in the data distri- bution, there is fixed point that is optimal for every distribution in general. Thus, convergence to fixed point is necessarily suboptimal. If convergence to a fixed point does occur, then the norm of the gradients will be zero. The gradients converging zero is condition for loss of plasticity,
15

---

Published as a conference paper at ICLR 2025
and can be satisfied every distribution depending the choice of activation function. Focusing on the case when the activation function is ReLU(z) = max 0), the gradients can become zero if 1) the weights converge to zero or, 2) the activations converge to zero. As a rumning example, we will consider the two-layer neural network without biases denoted by (x) = 020(0rx), where 0 Rddx, and 0g E RRdouedd are weight matrices, and Φ is elementwise activation function. We will consider the squared loss, J() = g)zz.g)>>p [y fo { but a similar result holds for classification losses, such as cross-entropy. The gradient for each of the parameters given by, JJ = (y ())( and J(O) = 0 (f J (f) The neural network will reach a fixed point in continual learning, meaning that the gradients are zero for every datapoint, if either 1) both 0, and 02 are zero, or 2) the activations are zero for every input Φ(012) = O,
A.3 SPECTRAL NORM AND LIPSCHITZ CONSTANT
One other potential reason for why a growing spectral norm is problematic is from the theory convex optimisation. For the ReLU activation function, the Lipschitz constant of the layer map, h441 = (00h),), 1, is equal to the spectral norm of the weight matrix. We can thus bound the Lipschitz constant of the entire neural network above by the product of the layer-wise Lipschitz constants (Szegedy et al. 2013). Putting these results together, we conclude that during continual learning, the Lipschitz constant of each layer is increasing and thus the Lipschitz constant of the entire network also increasing. This is problematic from the standpoint of optimisation because gradient descent only converges on convex optimisation problems when the step-size is smaller than F. where L is the Lipschitz constant. Thus, if the Lipschitz constant grows then, eventually, the chosen step-size will be too large and gradient descent will not converge locally on the task.
A.4 MORE DETAILS ON REGULARIZATION AND TRAINABILITY
One other important advantage of regularization that it has well understood effects for ensuring trainability in approaches outside of deep learning, such as degenerate linear regression and inverse problems (Benning and Burger 2018).
Fach regularizer biases the parameter dynamics in their own way, such as keeping the parameters close to zero or close to initialization. But, the regularizer term is independent of the base objective, and potentially the data and/or architecture. Even if the base objective provides zero gradient, explicit regularization can provide a gradient to all weights, and reset weights if the hidden unit associated with the weight becomes saturated. For example, if the nonlinearity is ReLU, then
hidden unit (i) is inactive if for all inputs in the dataset (k), h á (Σ (Ty)) O. Then the hidden representation has collapsed for that unit, and none of the weights contributing to that unit will be updated, keeping the unit inactive. But, with regularization, the weights will be updated and the unit may become active again for some inputs.
A.5 DETAILS REGARDING SPECTRAL REGULARIZATION
Spectral Regularization vs L2 Regularization L2 regularization is not as effective as spectral regularization because it constrains the magnitude of the parameters, and can even cause rank collapse (Kumar et al. 2023). Spectral regularization constrains only the spectral norm of the (0:) = 1. This has the effect of controlling the magnitude of the maximal column of the weight matrix. That is, if 0 E Rd4 xd: 1, then 1/ < [[0:12 where ||0:]] maxig/sd, Σ E 100 This implies that Σ [[t]]]] < Vdd for every column j. This differs in two ways from L2 regularization: (i) the regularizer does not regularize all the columns of the weight matrix, only the parameters in the column with a maximal sum, and (ii) the regularizer does not regularize the individual weights of the column, but their sum. This means that the parameters can potentially be larger, and move further from the initialization, potentially enabling more effective use of the neural network's 's capacity. Lastly, we note that L2 regularization can also be viewed in terms of a matrix norm, specifically the Frobenius norm.
weight matrix close to 1, (()) ((((())
Spectral Regularization vs Spectral Normalization: The spectral norm can also be controlled via spectral normalization (Miyato et al.) 2018). For continual learning, spectral regularization is preferable over spectral normalization for reasons: (i) spectral normalization is data-dependent
16

---

Published as a conference paper at ICLR 2025
(see Section 2.1 and Equation 12 in Miyato al. (2018), 8), and Appendix A.6 for more details), which can be problematic in continual learning due to the changing data distribution, and, (ii) other forms of normalization are often already used to train deep neural networks, such LayerNorm (Ba let al. 2016). LayerNorm, in particular, regularizes the magnitude of the layerwise map by dividing by the norm of the features in the layer. However, because the spectral norm of the underlying parameters still grows at rate of vt, the normalization layer itself will suffer from loss of gradient diversity. For the purposes of ensuring continual trainability, regularization explicitly controls the spectral norm of all parameters and is thus preferable over normalization in isolation. That being said, normalization can still improve performance and optimal performance may be achieved by combination of regularization and normalization, which we show in Section 5.3
A.6 CATEGORIZING REGULARIZERS FOR CONTINUAL LEARNING
We can categorize regularizers broadly into data-dependent and data- data-indepedent regularizers. Data- dependent regularizers use the data some way, which can be probelmatic for continual learning when the data distribution changes. Regularizing maintain useful properties for learning on the next distribution of data. Examples of data-dependent regularizers include feature rank regularization (Kumar et al. 2021), auxiliary tasks and other feature- space regularizers (Lyle et al.] 2022). Other non-standard approaches that have a data-dependent regularization effect include gradient regularizers such as gradient-clipping and weight reinitialization based on dormant neurons (Sokar et al. 2023). These data-dependent regularizers regularizers control some property on data from the current task but may not control the property on data from new task, which is needed to maintain plasticity.
one distribution of data does not necessarily
Data- Inta-Independent Regularization, on the other hand, does not depend on any data. This category regularizes the parameters directly, which is particularly useful in continual learning when the data distribution is changing. Examples of iata-independent regularization include L.2 regularization towards zero, weight decay (for adaptive gradient methods), and regenerative regularization (Kumar jet al., 2023). Our proposed approach of spectral regularization is data-independent and, moreover, explicitly targets a trainability condition for continual learning. Thus, we expect that it is particu- larly effective on maintaining trainability in continual learning compared to other data- indeeeeddddd regularizers.
A.7 SPECTRAL REGULARIZATION OF OTHER LAYERS
Normalization Lavers Other layers using per-unit scaling, such as normalization layers (Ba et al.
2016; loffe and Szegedy 2015). are also spectrally regularized. For example, denote y R as the trainable scaling parameters for LayerNorm (Ba et al. 2016), then the element-wise product of the weights can be written as a diagonal weight matrix, [() = 7 o 2= Diag(7)z. The maximum singular value of a diagonal matrix is the maximum entry, maz (Diag(7)) = max, ITil. However, optimisation with a maximum is problematic because the max is not differentiable. Furthermore, differentiable surrogates, such as log-sum-exp (log (E, exp(hal)) 1)3) only give a loose upper bound on the maximum which is problematic because we want to regularize the maximum towards one. Thus, we regularize each weight towards 1.
Convolutional Layers Similar to other work, we reshape the convolutional weight tensor (with kernel size (A: x k dn filters and dout filters) into a matrix of size dout X (k - d.n) (Yshhida and Miyato 2017). The spectral norm of this reshaped matrix provides an efficient upper bound on the spectral norm of the Toeplitz matrix defining the convolution (Tsuzuku et al. 2018 Corollary 1).
17

---

Published as a conference paper at ICLR 2025
B EXPERIMENT DETAILS
All of our experiments used Adam Kingma and Ba 2015) where the default step size of 0.001 was selected after an initial sweep over [0.005, 0.001, 0.0005|. | For all of our results, we use 10 random seeds and provided shaded region corresponding to the standard error of the mean. For experiments on tiny- ImageNet SVHN2, CIFAR 10 and CIFAR 100, we used 4 seeds to sweep over the regularization strengths of (0.01, 0.001, 0.0001], and found that 0.0001 worked well on Niny-ImaeeNet, CIFARIO and CIFARI RR10 10 for all regularizers, whereas 0.001 worked best on SVHN2 for all regularizers.
Datasets and non-stationarities:
MNIST:
28x28 greyscale images, 10 classes. Only the first 12800 datapoints were used for training, with a batch size of 512 and [40, 80, 120] epochs, and a total of 50 tasks.
Fashion MNIST: 28x28 greyscale images, 10 classes. Only the first 12800 images, with batch size of 512 and [40, 80, 120] epochs per task, and a total of 50 tasks.
EMNIST: 28x28 greyscale images, 50 classes. We use the balanced version of the dataset, using the first 100000 datapoints. For random label non-stationarity, we used a batch size of 500 and 100 epochs per task, with total of 50 tasks. For both label flipping and pixel permutation non-stationarities, we used a batch size of 500 and 20 epochs per task, with a total of 200 tasks.
SVHN2: 32x32 color images, 10 classes. The first 50000 images were used for training, 5000 images from the test set were used for validation and the rest were used for testing. The batch size used was 500, we found 250 to be unreliable for learning due to high variance. For random label non-stationarity. 20 epochs per task and 25 tasks was enough to lose trainability for an unregularized network. For pixel permutation non statsttttiiiity 10 epochs and 100 tasks was enough to lose trainability for unregularized network.
CIFARIO: 32x32 color images, 10 classes. All of the 50000 images were used for training. 1000 images from the test set were used for validation and the rest were used for testing. We used batch size 250 and found this to effective. For random label non-stationarity, 20 epochs per task and 30 tasks was enough to lose trainability for an unregularized network. For pixel permutation non-stationarity, 10 epochs and 100 tasks was enough to lose trainability for an unregularized network.
CIFAR100: 32x32 color images, 100 classes. All of the 50000 images were used for training, 1000 images from the test set were used for validation and the rest were used for testing. We used batch size of 250 and found this to be effective. For random label non-stationarity, 20 epochs per task and 30 tasks was enough to lose trainability for an unregularized network. For pixel permutation non non-stationarity, 10 epochs and 100 tasks was enough to lose trainability for unregularized network.
tmageNma 64x64 color images, 200 classes. All of the 100000 images were used for training, 10000 images were used for validation, and 10000 images were used for testing according to the predetermined split. We used a batch size of 250 and found this to be effective. For random label non-stationarity, 20 epochs per task and 30 tasks enough to lose trainability for an unregularized network. For pixel permutation non-stationarity, 20 epochs and 100 tasks enough to lose trainability for unregularized network.
Reinforcement Learning: We evaluate spectral regularization in RL control tasks from DMC benchmark (Tassa et al.) 2020), with SAC method (Haarnoja et al., 2018) in demanding replay ratio (RR) regime (D'Oro al., (022) with 16 gradient updates per every new environments step. We choose this setup because high RR regime leads to significant primacy bias (Nikishin et al.] 2022) defined as a tendency to overfit initial experiences that damages the rest of the learning process.
Neural Network Architectures:
MNIST, EMNIST, and Fashion MNIST: 4-yayer MLP with 256 neurons per layer and relu activations. For experiments that use it, LayerNorm was applied after the linear weight matrix and before the non-linearity.
18

---

Published as a conference paper at ICLR 2025
Iiny--mageNet, CIFARIO, CIFARIO0, and SVHN2: We follow the architectures used by Lee et al. (2024) which include both an off-the-shelf ResNet- 18 with batch norm, as well as Vision Transformer with embedding dimension of 192, patch size of 4x4, 3 attention heads, 12 layers, layer normalization and a dropout rate of 0.1.
Reinforcement Learning: Recent studies (Ball al., 2023; Nauman et al., 2024) have
demonstrated that, in this setup, only resets with layer normalization (Ba et al. 2016) serve as an effective mitigation strategy. We compare the performance of the SAC agent with both spectral regularization and layer normalization two baseline agents: SAC with only layer normalization, and SAC with layer normalization plus resets. We use spectral regularization with coefficient le for both the actor and critic. For every method, use a single critic and architecture size of 2 layers and 256 neurons per layer for both actor and critic. Using a random policy, prefill replay buffer with 10, 000 transitions before starting the training. Replay buffer maximum size is million transitions.
Metrics reported
Figure test accuracy the end of training on task
Figure 2: test accuracy the end of training task
Figure 3: top: train accuracy at the end of training a task, middle-top: average of the maximum singular value at each lacyr, representation at the beginning of task and at the end of a task, bottom: distance between current parameters and parameters at initialization.
average distance between
Figure Area under curve for training accuracy as function of regularization strength.
Figure 5: Left: Return achieved by policy under evaluation, middle: area under curve of return as a function of the hyperparameter value. right-top: difference between estimated Q-value and Monte-carlo return under the current policy, right-midde average gradient norm for the policy network, right- bottom: average parameter norm for the policy network.
19

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EXPERIMENTS
C.1 ABLATING HYPERPARAMETER à FOR SPECTRAL REGULARIZATION
Accuracy
Accuracy
0.6
spectral regl
0.4
spectral regl
spectral reg2
spectral reg2
spectral rega
spectral rega
spectra[rrgg
spectral regB
10
20
30
40
10
20
30
40
Number of tasks
Number of tasks
Figure 7: Evaluating the choice of & for spectral regularization for 1, on MNIST (Right) and Fashion MNIST (left). We found that it = 2 balance stability with effectiveness, and use this value throughout our experiments.
C.2 RESULTS ON FASHION MNIST
Random Labe MesSI (40 (poccs)
Random Label PashionMNIST (50 (paccs)
Rendom Labe ReshicoMNiSST (220 Epochs)
03
59
0.9
Accurey
6.8
Dert
04
0.6
5.5
112
Number of tasks
Number of tasks
Number of tasks
Figure 8: Loss of Plasticity in Fashion MNIST. Although the all the newtorks in this experiment use LayerNorm, loss of plasticity occurs without regularization: the performance decreases function of tasks, even with an increasing number of iterations.
20

---

Published as a conference paper at ICLR 2025
C.3
VISION TRANSFORMER CANNOT MEMORIZE RANDOM LABELS
Random Label (Single Task)
1.00
ResNet
VIT
Train Accuracy
0.25
0.00
10
20
30
40
Number of iterations 1000)
Figure 9: ResNet and Vision Transformer on the random label memorization task using tiny- ImageNet. The Vision Transformer architecture is particularly well-suited to tasks with structure from which generalization is possible. However, we found that its trainability random labels is lacking. We were unable to get the Vision Transformer to memorize random labels even on a single task in ImageNet.
C.4 ADDITIONAL SENSITIVITY RESULTS
40 Epochs, No LayerNorm
80 Epachs, LayerNorm
40 Epochs, With LayerNorm
Epachs, With LayerNorm
83
Degubbrtntt Strengtt
Regu leguttentiion Streggh
Regulen
Reguler Strengt
Figure 10: Sensitivity analysis on regularization strength Compared to other regularizers, spectral regularization is insensitive to regularization strength while sustaining higher trainability for any given regularization strength.
21

---

Published as a conference paper at ICLR 2025
EMNIST IRandom Label)
EMNNIST (Pixel Permutation!
EMNIST CLabel flipt
0.48
6:48
Accuaatyy
0.00
0:95
0.34
0.34
Coactral
0.92
0.32
0.42
6.40
Sensitivity to type of non-stationarity on EMNIST. Spectral regularization is able to consistently maintain high trainability across random label assignment, pixel permutations, and label flipping. When applying random label assignment, L.2 (init) and L2 (zero) are unable attain high trainability as compared to spectral regularization.
Figure 11:
No Layatoor
Epocha, Loyerkare
Eyecht Wht Layatiom
120 Eoochs Logerfem
Mambar takka
Mumber
Sensitivity number of epochs per task on MNIST with random label assignment. Without Layer Norm, spectral regularization is able to maintain trainability over other regularizers even when the number of epochs low. Spectral regularization also synergizes with Layer Norm, improving initial performance on the first task, and improving over tasks.
Figure 12:
Robustness to Non-stationarity in Fashion MNIST Next, explore whether our proposed regularizers are effective across different types of non-stationarity. To this end, we use EMNIST because the number of classes large enough that label flipping induces loss of trainability (Elsayed and Mahmood, 2024). In Figure II we found that spectral regularization best maintained trainability across different non-stationarities.
Varying the Number of Epochs Per Task Given that the continual learning problem outlined in Section [ on the number of iterations, it may be the case that a higher number of epochs per task can mitigate loss of trainability. In Figure 12 we found that the number of epochs only delays loss of trainability (see Appendix C.2 for results on Fashion MNIST). Even when the number of epochs per task is high enough reach 100% accuracy the first task, loss of trainability eventually occurred without regularization. In contrast, when using spectral regularization, loss of trainabililty was consistently mitigated. We additionally found that spectral regularization particularly effective at maintaining trainability when the number of epochs per tasks is low. The evidence for this is most striking without Layer Norm, in the top row, where spectral regularization was the only method capable of maintaining its trainability. Even though Layer Norm controls the spectral norm through the activations (Kim et al. 202T), adding Layer Norm alone was not enough to mitigate loss of trainability. However, Layer Norm is synergistic with various forms of regularization, with accuracy being improved regardless the number epochs per task.
22

---

Published as a conference paper at ICLR 2025
C.5 EFFECT OF REGULARIZATION ON CAPACITY A SINGLE TASK
The regularization strength for preventing loss of trainability must be sufficiently high, and this regu- larization strength may limit the capacity of the neural network. We now show the extended training of each regularizer using the same regularization strength used to prevent loss of trainability in Figure 13 Spectral regularization was only best in preventing loss of trainability, it also achieved the highest accuracy when training to convergence on single task. This means that spectral regularization constrains the capacity of the neural network the least, while still preventing loss of trainability.
SVHN2
CIFAR10
CIRAR100
Zoactrll
0.98
0:3
Mctuney
0.96
6.96
6-96
6.94
6.94
0.4
0.32
0.32
6.32
0.00
100
150
202
0.00
50
300
0.70
0
50
150
150
202
Number Iterations (x100)
Number of (x100)
(:100)
Figure 13: Single task performance with ResNet-18. Spectral regularization is least restrictive of the neural network capacity, evidenced by its ability to better fit the randomly assigned labels compared to other regularizers. The unregularized baseline able to use its capacity fully, but the cost of reduced trainability in later tasks.
23

---

Published as a conference paper at ICLR 2025
C.6 INVESTIGATING EFFECTS OF REGULARIZATION ON NEURAL NETWORK PROPERTIES
SVHIN2
CIARRO
CIARRCO
23
Aog Value
-3
12 1980
AMg
Ave
Saxtrol
Number of Lesks
Number of lusis CIFR10
Number of tankc
SVH-22
CIFARLOD
2.5
Beg Smg Nahu
2.0
Avg Mir. Sin. Value
1.0
Avg Min Sin Wakre
2.5
1.5
0.5
25
Number of tasks SVH922
Number of tasks CIAAL10
Number of tasks
CIAANLOO
0.34
Wedghtt (.3)
6.00
(eeght Norm 0.33
2.00
0.35
0:00
0:33
0.33
0.04
6.00
0.12
Number of affsks SVAR22
Number of tasks CIPAR10
Number of Lasks
CIRAR100
0.36
0.00
0.00
0.25
6.06
Deet. Prom mnk
0.34
0.33
0.22
0.02
5.00
0.33
Number of taales 5VHM2
NJmber tasks FIRR3C
Number
CIFAR10D
125
Ave Weight tank
Aug Waight Punk
320
Weight
60
300
75
40
35
Number
Numbor
Number
100 00g Singram Gro Asporux Hessz
Graderne Rank (Strtt Tave?
4.46
Gradkent Diversty ky
0.8
0.5
8.00
NCC 207
32
10
20
Figure 14: Singular values, weight norms, stable ranks and effective gradient diversity: Left:
SVHN2, Middle: 10, Right: CIFAR100.
24

---

Published as a conference paper at ICLR 2025
C.7 INVESTIGATING GENERALIZATION ON CONTINUAL IMAGENET
We investigate the generalization performance of spectral regularization on Continual ImageNet. In Continual ImageNet, each task is distinguish between two ImageNet classes. We use the network architecture and training protocol used in Kumar et al. (023). We find that all regularization approaches tested achieve high generalization performance across tasks. The methods we compare are spectral regularization (SpectralReg Agent), L2 towards zero (L2Agent), L2 towards initialization (L.2InitAeent, recycling dormant neurons (ReDOAgent), and no regularization (BaseAgent).
agent name
BaseAgent
LAAgent
Ageitt
ReDDAgent
SpectralRegi
Task Test Accuracy
0.85
0.80
0.75
100
Task
Figure 15: Generalization performance on Continual ImageNet.
C.8 RESULTS ON INDIVIDUAL DMC ENVIRONMENTS
We report mean returns for DMC environments in Figure 16
300
Eieeto Litee Vore
Fibach Liver Morm
Eaeto Unver
200
339
200
Rosathi
Layer Stars
150
100
200
300
200
800
250
Steps
Figure 16: Mean and standard error of return for 7 DMC environments.
25

---

Published as a conference paper at ICLR 2025
C.9
COMPARING CONTINUAL BACKPROP, REDO AND SPECTRAL REGULARIZATION
tiny-ImageNet
CIFARI00
Continual Backprop
ReDO
0.4
Spectral
Test Accuracy
Test Accuracy
0.15
0.3
0.10
0.2
0.05
Backprop
0.1
SpectralReg
20
40
50
60
10
15
20
Number of tasks
Number of tasks
Figure 17: Class--ncremental Learning on tiny-ImageNet and CIFAR100 with continual back- prop. For continual backprop. we used the suggested hyperparameters: maturity threshold of 1000 and replacement rate of 10-5 Continual backprop performs similarly Recycling DOrmant neurons (ReDO). However, spectral regularization outperforms both methods.
C.10 TABLE SUMMARY OF MAIN RESULTS
Class Incremental
Label Flip
Pixel Perm
L2 (init)
0.2313 (0.0013)
0.2571 (0.0013)
0.0926 (0.0018)
L2 (zero)
0.2197 (0.0021)
0.2670 (0.0029)
0.0873 (0.0015)
ReDO
0.1726 (0.0008)
0.2478 (0.0103)
0.0754 (0.0025)
S-EWC
0.1615 (0.0016)
0.2655 (0.0011)
0.0782 (0.0018)
ShrinkPerturb
0.1778 (0.0053)
0.2451 (0.0051)
0.1085 (0.0025)
Unreg
0.1749 (0.0007)
0.2307 (0.0056)
0.0788 (0.0009)
Wass
0,2260 (0.0022)
0,2726 (0.0040)
0.1057 (0.0019)
Spectral
0.2447 (0.0016)
0.2594 (0.0016)
0.1104 (0.0029)
Table 1: Table summarizing the results in the top row of Figure 1 (mean of final test accuracy and standard deviation, ResNet on tiny-ImageNet).
Class Incremental
Label Flip
Pixel Perm
L2 (init)
0.1890 (0.0042)
0.2822 (0.0045)
0.1370 (0.0022)
L2 (zero)
0.1457 (0,0001)
0.2126 (0.0047)
0.1188 (0.0012)
S-EWC
0.1185 (0.0052)
0.2230 (0.0079)
0.0923 (0.0073)
ShrinkPerturb
0.1171 (0.0011)
0.1392 (0.0068)
0.1088 (0.0033)
Unreg
0.1226 (0.0074)
0.1936 (0.0029)
0.0813 (0.0021)
Wass
0.1727 (0.0252)
0.2184 (0.0022)
0.0800 (0.0010)
Spectral
0.2452 (0.0014)
0.2743 (0.0081)
0.1392 (0.0006)
Table 2: Table summarizing the results in the bottom row of Figure (mean of final test accuracy and standard deviation, Vision Transformer ting-ImageNee).
26

---

Published as a conference paper at ICLR 2025
SVHN2
CIFARIO
CIFARI00
L2 (init)
0.7341 (0.0032)
0,4830 (0.0038)
0.2594 (0.0037)
L2 (zero)
0.7409 (0.0027)
0,4770 (0.0035)
0.2563 (0,0023)
ReDO
0.7356 (0.0018)
0.4531 (0.0016)
0.2485 (0.0012)
S-EWC
0.7288 (0.0027)
0,4612 (0.0018)
0.2481 (0.0017)
CReLU
0.7539 (0.0016)
0,5002 (0.0028)
0.2443 (0.0013)
ShrinkPerturb
0.7548 (0.0015)
0.5032 (0.0016)
0.2546 (0.0008)
Unreg
0.7277 (0.0036)
0.4490 (0.0023)
0.2444 (0.0018)
Wass
0.7337 (0.0032)
0,4849 (0.0052)
0.2654 (0.0023)
Spectral
0.7489 (0.0026)
0,5082 (0.0052)
0.2785 (0.0043)
Table 3: Table summarizing the results in the top row of Figure 2 (mean of final test accuracy and standard deviation, ResNet on SVHN2, CIFARIO and CIFARI00 with pixel permutation nonstation- arity).
SVHN2
CIFARIO
CIFARI00
L2 (init)
0.8163 (0.0028)
0,5160 (0.0034)
0.2267 (0.0023)
L2 (zero)
0.6476 (4.1494)
0.5251 (0.0019)
0.2457 (0.0031)
S-EWC
0.8003 (0.0040)
0.4735 (0.0026)
0.2371 (0.0030)
ShrinkPerturb
0.3565 (0.1344)
0.5357 (0.0062)
0.2564 (0.0019)
Unreg
0.7870 (0.0044)
0,4610 (0.0041)
0.2004 (0.0120)
Wass
0.8098 (0.0035)
0.4811 (0.0078)
0.2132 (0.0027)
Spectral
0.8248 (0.0016)
0.5310 (0.0044)
0.2775 (0.0024)
Table 4: Table summarizing the results in the bottom row of Figure (mean of final test accuracy and standard deviation, Vision Transformer SVHN2, CIFAR10 and CIFAR100 with pixel permutation nonstationarity).
tiny ImageNet
CIFAR10
CIFAR100
L2 (init)
0,8206 (0.0172)
0.8317 (0.0094)
0.6499 (0.0287)
L2 (zero)
0.7974 (0.0102)
0.6536 (0.0143)
0.5973 (0.0244)
ReDO
0.2637 (0.0107)
0.7387 (0.0561)
0.2800 (0.0578)
S-EWC
0.3607 (0.0934)
0.7195 (0.0725)
0.1980 (0.0287)
CReLU
0.3587 (0.0086)
0.7156 (0.0335)
0.4722 (0.0121)
ShrinkPerturb
0.5832 (0.0040)
0.7451 (0.0064)
0.6483 (0.0145)
Unreg
0.2361 (0.0302)
0.7371 (0.0297)
0.2671 (0.0366)
Spectral
0.8542 (0.0040)
0.8159 (0.0072)
0.7771 (0.0128)
Table 5: Table summarizing the results in the bottom of Figure (mean of final train accuracy and standard deviation, ResNet on tiny ImageNet, CIFAR10 and CIFAR 100 with random label nonstationarity).
27


Reviews:
[{"review_id": "VCBpYoCW3w", "reviewer": "Reviewer_veM9", "summary": "The authors derive a new spectral regularizer for continual learning inspired by the observation that the singular values of the neural network parameters at initialization are an important factor for trainability during early phases of learning. Under such regularization they keep the maximum singular value of each layer close to one. \n\nUnder such spectral regularization they claim that they can directly ensure gradient diversity throughout training, which promotes continual trainability, while minimally interfering with performance in a single task. \n\nTo validate the utility of the proposed regularization and the claims they present empirical evaluation. The experimental results shows how the proposed spectral regularizer can sustain trainability and performance across a range of model architectures in continual supervised and reinforcement learning settings. They try to demonstrate better training in individual tasks, trainability as new tasks arrive, and better generalization performance.", "questions": "How different the propose approach is from L2 regularization on the same set of parameters?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The continual learning topic is important area and loss of plasticity is fundamental challenge.\n - The approach is interesting and the idea to regularize the maximum singular value of each layer close to one seems novel.\n - The presentation of the work is nice, while the motivation is appropriate.\n - The authors present results for several experiments, evaluating different properties.", "weaknesses": "- It appears like a marginal improvement and novelty with respect to L2 regularization.\n- It seems that there is a connection to L2 regularization, or L2 regularization towards initialization of parameters which is a sort of spectral regularization.\n- Deeper understanding/analysis of the proposed and the one above seems important and might be beneficial, but I was not able to find it in the paper L2 norm vs spectral norm in this context.  \n- In the experimental section, looking at the graph and the results it seems that the improvements are marginal, or not highlighted significantly.\n- In the same experimental section, I was able to see only graphs and plots, was not able to see actual table or values, to be able to apricate the achieved results more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PZSqPvabYf", "reviewer": "Reviewer_YuMr", "summary": "The paper considers the continual learning problem. In order to keep the stability of problem, the spectral regularizer is added to the objective function. Experimental results demonstrate the effective of the approach method.", "questions": "1 through the initialization is the key to the trainability, singular values of Jacobian and the condition numbers would induce the new selected parameters, e.g., rank\n2 how to effectively deal with computation of the spectral norm, i.e. spectral regularization,\n3 the performance of the spectral is not always better than that of vision transformer and resents\n4 what’s more, the datasets are not enough, please refer to the latest approach,\n5 for the figure 5, what’s the phenomenon of suddenly decrease for the y-axis.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper provide the spectral regularize to the objective function for improving the stability of the problem.", "weaknesses": "However, the added spectral regularize would bring in the cost of computation for the continual learning if facing large model and large data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QqkzR5iVOL", "reviewer": "Reviewer_uBBh", "summary": "The paper addresses plasticity loss in continual learning by proposing a spectral regularization technique. The proposed regularization technique regularizes the network’s weights at each layer towards a spectral norm of 1. This method is motivated by a spectral analysis of (continual) learning and the observation that standard neural network weight initializations exhibit good trainability and generalization. The paper’s proposed method is validated on a series of continual learning benchmark problems where it is shown to be performant and robust to the choice of regularization strength.", "questions": "- Is there any particular reason why Continual Backprop was not evaluated in the experiments?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Section 1 Introduction and Section 2 Problem Setting are well written with clarity and detail, effectively introducing the problem, the spectral regularization algorithm, and some intuition behind its formulation.\n- The spectral analysis of continual learning well motivates the algorithm. \n- The experiments consider a wide swathe of existing continual learning algorithms, both ResNet-18 and Vision Transformer architectures, and a variety of datasets and non-stationarities. Over this set of experiments, spectral regularization attains competitive performance and is robust to its choice of hyperparameter.", "weaknesses": "- Although spectral regularization is well motivated and the paper includes positive empirical results, one way to improve the paper would be to derive some theory illustrating the benefit of spectral regularization, and drawbacks of other regularization schemes, in continual learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fbyjT9W5Uv", "reviewer": "Reviewer_f6F3", "summary": "The paper introduces a new regularization to encourage neural networks to combat loss of plasticity phenomenon in a continual learning setting. The new approach is based on the observation that the maximum singular value of neural network layers grow over the course of training and avoiding this growth can help improve trainability of models in a continual learning setting. The proposed regularizer attempts to steer the maximum singular value of a layer to 1 (and 0 for the bias terms). The paper uses accepted datasets and tasks in continual learning literature to study the effectiveness of the proposed method over several baselines.", "questions": "In addition to the points noted in weakness section, I wonder if the following statement is too strong\n```\nNote that these datasets and architectures encompass all continual supervised learning experimental settings considered in the loss of plasticity literature.\n```", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper solves the problem of loss of plasticity in continual learning by addressing trainability. \n\n- Section 3 and Section 4 in the paper are nicely written and is valuable to a general audience. Also, the paper provides at least two illustrative examples as well as some analysis to make the observation that the top singular value growth may cause training issues.\n\n- The development of proposed solution via spectral regularization  is intuitive", "weaknesses": "* The proposed method to keep top singular value small bears some similarities to the following paper:\n  - https://arxiv.org/abs/2303.06296\n\n* The experimental setup used in the paper is incomplete and not very convincing in order to judge the empirical results:\n  - The models used are on the smaller side (ResNet-18 and ViT)\n  - ViT model size is not specified in the paper \n  - The dataset used in Tiny-Imagenet which is not fully described. I believe the inputs are 64x64 based on my search but this detail is missing in the paper\nWhile I am not an expert in continual learning area, there are recent papers that suggest that larger datasets (resolution and size) are available and used in continual learning studies\n\n* The presentation of the experimental results and discussion section could be improved:\n  - Figure 1 has legend only on the top left figure which makes it hard for the reader to follow how each method works across different continual learning scenarios\n  - Figure 2 has a similar issue with legend\n  - he paper does not define all the quantities plotted in Figure 3.\n  - Since the paper focuses on trainability, additional graphs that show how trainability improves in CL setting might be useful in the presentation. Figure 3 provides a hint but in this case the shrink and perturb baseline shows better trainability as defined by singular value.\n\nI'm unable to support acceptance at this point due questions on setup used in empirical analysis. The paper could benefit from an update that includes missing information that can give the reader an idea on exact metrics tracked, setup used, dataset details and other information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)