Parseval Regularization for Continual Reinforcement Learning
Authors: Wesley Chung
Lynn Cherif
Doina Precup
David Meger
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Continual Learning
Plasticity
Optimization
Primary area: reinforcement_learning
TL;DR: Maintaining orthogonal weight matrices during training improves continual reinforcement learning agents.

Abstract: Plasticity loss, trainability loss, and primacy bias have been identified as issues arising when training deep neural networks on sequences of tasks---referring to the increased difficulty in training on new tasks.
We propose to use Parseval regularization, which maintains orthogonality of weight matrices, to preserve useful optimization properties and improve training in a continual reinforcement learning setting.
We show that it provides significant benefits to RL agents on a suite of gridworld, CARL and MetaWorld tasks.
We conduct comprehensive ablations to identify the source of its benefits and investigate the effect of certain metrics associated to network trainability including weight matrix rank, weight norms and policy entropy.

Parseval Regularization for Continual Reinforcement Learning
Wesley Chung, Lynn Cherif, David Meger, Doina Precup Mila, McGill University chungves@mila, quebec
Abstract
Loss of plasticity, trainability loss, and primacy bias have been identified as issues arising when training deep neural networks on sequences of tasks tasks-all referring to the increased difficulty in training new tasks. We propose to use Parseval regularization, which maintains orthogonality of weight matrices, to preserve useful optimization properties and improve training in continual reinforcement learning setting. We show that it provides significant benefits to RL agents on a suite gridworld, CARL and Horld tasks. We conduct comprehensive ablations to identify the source of its benefits and investigate the effect of certain metrics associated network trainability including weight matrix rank, weight norms and policy entropy.
1 Introduction
Continual reinforcement learning (RL) [30 setting where a single agent has to learn in a complex environment with potentially changing tasks and dynamics, has remained a challenge for current agents. The core difficulty stems from training deep neural networks on sequences of tasks. Although the network may be able learn the first task easily, after several tasks, progress may be impeded phenomenon termed plasticity loss 12
57 I.. Other works have found that neural networks may be strongly influenced by data in the early phases of learning, leading weaker performance on later tasks [5] 43 2 29 58]. Taken together, these works have demonstrated the difficulty of learning in the presence nonstationarity, an integral aspect of reinforcement learning problems.
38
37
Optimization has historically been a barrier towards training neural networks, even on single tasks (25] 46). The development of suitable parameter initialization schemes was a key ingredient to successful training, such as Xavier and Kaiming initializations [16] (22). By prescribing suitable variance for random Gaussian weights, these initialization strategies allow gradients to propagate throughout the network and avoid the vanishing and exploding gradient problems.
This line of work led to the development of orthogonal initialization [33, a technique designed to ensure better gradient propagation by making the singular values of the weight matrices all equal to 1 and thus maintaining the singular values of the Jacobian of the output with respect to the input to also be 1-a property known dynamical isometry. Orthogonal initialization of a vanilla convolutional network with 10 thousand layers without the use of normalization layers [61. Additionally, this technique has found success RL settings with more shallow networks, being the default setting for PPO agents and improving upon Gaussian initialization strategies [28] 27]. Overall, this research direction has showcased the importance of encouraging model parameters to lie in regions amenable to optimization.
used allow training
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Taking this idea further, we can interpret the difficulty of optimization in continual RL as related to the parameters' movement during learning. At the beginning of training, the parameters are initialized such that the agent can easily adapt the policy and value functions in any direction based on the observed data. As training proceeds, the parameters may move to a part of the space that is no longer as easy to navigate 442. Due to the nonstationarity of the policy and objective in RI., this can be problematic if the agent is required to learn something that its current parameters cannot easily accommodate. These troubles may be further exacerbated when the task itself is modified, completely preventing the agent from learning a new task.
Using this intuition, we propose continual RL solution Parseval regularization encourages orthogonal weight matrices W by regularizing WWT to be close cl where c is a constant and I is the identity matrix. By preserving the orthogonal property, we hope to keep the beneficial optimization properties of the initialization throughout the training process and improve the learning speed of new tasks. We empirically demonstrate that this addition facilitates learning on sequences of RI. tasks as seen in Fig- and Fig. in Meta World [62] CARL 9 and Gridworld environments. In addition, it compares fävourably to alternative algorithms such as layer norm [6]. peii--and-perturb [5], and regenerative regularization (33
which
Algorithm Comparison
Algorithm Comparison
1.0
1.0
MAMM
PrSuccess esscess Rate x
0.8
0.8
0.6
Success
0.4
0.4
0.2
Layer
0.2
Layer
SRP
Rogen
Rigicnn
0.0
Begen
Regin
0.0
0.4
1.0
Average Success Rate
Steps
Figure 1: Performance of algorithms on Metaworld tasks. The tasks change every million steps, matching the dips in success rate in the learning curves (right). On the left, we show performance profiles showing the distribution of average success rates across tasks. Higher is better for both. Parseval regularization significantly improves on the baseline and outperforms other alternatives.
Additionally, we conduct ablation studies, separating out two parts of Parseval regularization: A regularizer on the norm of the rows of the weight matrices and a regularizer for the angles between these weight vectors. We find that, while both components improve the baseline, regularizing angles makes a larger impact and it is the combination that fares best.
Delving into the network properties, we find that Parseval regularization interacts with the diversity of neuron weights, the input-output Jacobian, the rank of the weight matrices and the entropy of the policy. Moreover, the regularization can reduce the correlation between neuron weights, increase the stable rank and maintain a tighter spread of entries in the input-output Jacobian, suggesting using these metrics as a target or diagnostic tool for future algorithms. Moving further, we explore how Parseval regularization interacts with different activation functions and network widths, finding that it can be productively used with various network architecture choices.
Finally, we investigate a few aspects linked to optimization and plasticity loss in neural networks including the role of policy entropy, initialization scale and rank of the initialization, finding that these quantities have complex relationships to performance.
2 Preliminaries
The problems we consider are defined as sequences of Markov Decision Processes (MDP), An MDP is defined by its state space, action space, transition function, reward function and discount factor. The agent aims to learn a policy и that maximizes the expected return E E>> 7 R.). this work, we will focus on sequences of tasks where the reward function changes after certain number of timesteps while the transition dynamics stay the same. The changes are not signaled the agent and thus there is nonstationarity in the environment that is difficult to model. This represents simplified

---

setting for the full continual RL problem, where the changes in the environment can be more general and less regular (3)..
3 Parseval Regularization
Orthogonal initialization was initially proposed in the context of deep linear neural networks. It was shown that if orthonorma (orthogonal would have depth-independent training times [53]. This result was expanded to show that nonlinear networks with tanh activations could also achieve better convergence times as a function of depth
(orthoogn with unit norm row vectors) weight matrices are used, one
[4[26]., (6). These theoretical results were validated by successfully training networks with thousands of layers and orthogonal initialization was also effectively employed with shallow networks in a deep
RL setting (7).
Orthogonal weight matrices useful because they can ensure that the singular values of the weight matrices are all equal and, if these are equal to 1, it would mean that the layer forms an isometry- preserving distances, magnitudes and angles of its inputs and outputs (when the dimensions matching) [48]. Since the Jacobian of a linear layer with weights W is simply WT, which is also orthogonal, the error gradients passed through in the backwards pass also maintain their structure as well, without exploding or vanishing (ignoring the activation function). This intuitively can lead to more favourable optimization.
Parseval regularization is implemented simply by adding a term to the usual objective. For each weight matrix W of dense layer, we add the following regularizer:
Parseval (W)- = M|lWW - -711/
where x= > O controls the regularization strength, > is scaling factor. I is the identity matrix of appropriate dimension and ||- || denotes the Frobenius norm. This regularizing loss encourages the rows of W to be orthogonal to each other and also have a squared f--norm equal to If these constraints are satisfied, all the singular values of W will be equal to vs. We directly add this regularization term to both the policy and value networks to every layer except the last. That is, the final objective is:
C(6) Lp()) + X.. A A Inyersi- Lrarseval (Ws)
where 0 denotes all the parameter and W₂ denotes the weight matrix of layer k. Lpp and Lu are the policy and value losses, with X, and À being weighting coefficients.
The additional computational cost of this regularizer is of order ((d) for a dense layer of width d (with d inputs). This is similar to the cost of one forward pass when the size of the minibatch is close to d. In practice, we have found that the net runtime increase ranges from 1.8% to 11.4% over the vanilla agent. Empirical runtimes and more detailed analysis can be found in appendix
3.1 Network capacity and Lipschitz continuity By restricting the weights to be orthogonal, the network may be overly constrained. In partic- ular, if the weight matrices of all the layers are orthogonal and the activation function is Lip- schitz, then the function given by the neural network is also Lipschitz. A Lipschitz func- tion f R.= - Thm satisfies: for all x. y E R. [I(()) - fwyil < L y|l for some constant L L> and norm || I|- This a fairly strong constraint, meaning that the function values can- not vary too quickly as the inputs change, which may be overly limiting the neural network's ca- pacity to express complex functions.
Parseval Regularization with Additional Capacity
1.0
0.8
PrSuucess Rate
0.6
0.4
Parsoval and Ingut
0.2
Diagonnl Layer Parsoval, Extra Parameters
Perveval, with Irout Scale
0.0
Perveval, with agggnal Layer
0.0
0.2
0.6 Average Success Rate
1.0
Thus, to relax this Lipschitz condition, we test few additions. Throughout all the experiments, we do not regularize the final layer to preserve some expressiveness. However, the network
Figure 2: Comparing performance profiles of di- agonal layers and learnale input scales on Meta- world sequences. Either addition helps with Parse- val regularization.
'The term orthogonal is often used to mean orthonormal in the literature though we distinguish the two here.
3

---

remains significantly restricted with only this change.
Adding diagonal layers. The main addition that will be used throughout the paper is to introduce additional parameters after each (near) orthogonal layer. We multiply each output of the orthogonal layer by (learnable) constant and include a bias term. This can be viewed as adding a dense layer with diagonal weight matrix. is identical to the additional parameters used by Layer Norm [6] after the normalization operation. Note that adding these parameters incurs only a small additional cost in terms of memory and compute (linear in the width of a layer).
Input scaling. The second alternative is to leave the network untouched but rescale the inputs. That is, introduce factor Gin that multiples every input. The net effect is that the input space gets rescaled, which effectively multiplies the network's Lipchitz constant by Cin- We can view this allowing the function vary more quickly across the input space. For this addition, add learnable constant Cin. which multiples the entries, giving the network additional flexibility.
Fig. 2 shows that adding diagonal layers learnable input scale improves upon the agent with Parseval regularization. Interestingly, the addition of either degrades performance of the base agent though. We will use the addition of diagonal layers in all later Metaworld experiments for consistency,
Relaxing the orthogonality constraint. Finally, the third option explored is to relax the orthogonality constraint on the layers by freeing some weights from the constraint. We consider a weaker form of Parseval regularization where we split neurons in layer into multiple groups and apply Parseval regularization only within each group. This allows the weight vectors of neurons in different groups vary freely with respect to each other. Since we are mainly interested in the optimization benefits of Parseval regularization, we are willing to give up the Lipschitz constraint in favor of quicker learning.
Parseval Regularization With Subgroups
Actor Stable Rank [Layer 2)
1.0
PrSSccess Rate xI
0.8
40
0.6
Stable
30
0.4
Subgroupa1
Subgroups2
0.2
Subgro.pas
Sahgmipsa
Subgm.pod4
10
0.0
No Rasseval
0.0
0.2
0.0
0.2
0.6
1.0
Average Success Rate
Steps
Figure 3: The left plot shows performance profiles of Parseval regularization on Metaworld sequences when dividing neurons in layer into multiple groups. There is no significant improvement from splitting into groups; using only one group is the best choice. Adding Parseval regularization with any number of groups improves on the baseline though. The right plot shows the stable rank of the actor's second layer's weight matrix. Due to the relaxed orthogonal constraint on the weights, we can observe decrease in the stable rank. Similar plots can be observed for other layers and in the critic.
As reported in Fig. 3 we do not observe any improvement from Subgroup-Parsevar regularization and the performance decreases with more groups This suggests that, this context, the optimization benefits from Parseval regularization outweigh the loss of expressiveness of the neural network.
We that the stable rank decreases to be approximately equal to the number of orthogonal weight vectors. That is, if there are 64 neurons divided into two groups, the stable rank is around 32. This may reflect neural networks' tendency decrease the stable rank until it reaches small value, just enough to perform the task 45
To conclude, in the following experiments, will use Parseval regularization in conjunction with either additional diagonal layers or learnable input scale to introduce a little more capacity when required (details in appendix C.5)

---

4 Experiments
Environments and baseline algorithms
We utilize a sequence of tasks with changes between each force the agent to learn something new. How to choose these changes is an important design decision. Some common tasks in the continual learning literature such as Permuted MNIST 17 (shuffling the inputs at each change) or changing Atari games []] have drastic changes between tasks, which may not reflect realistic variations that would face any practical agent. Due to these large changes, often leaves little room for the agent to reuse previously learned knowledge.
We focus on producing different tasks by either changing the task (reward functions) while the transition dynamics generally remain the same, or by changing context variables such as wind or stiffness affecting the simulation properties while keeping the reward function fixed. These two types of nonstationarity can be natural. Agents may need to learn many different tasks over time in single environment. e.g. robot arm can both learn to push objects and, later, to grasp them. Reinforcement learning from human feedback (RLHF) is another instance where the reward function may change due to changing preferences although the environment otherwise stays the same. From another perspective, the agent may face different environmental conditions and would want to adapt to changes in them.
We run experiments in four sets of environments:
The first is a navigation task in a 15-by-15 Gridworld where the agent has to reach varying goal location. The goal is fixed for multiple episodes and is then changed every 40 thousand steps. Each episode limited to 100 timesteps. For the gridworld, the agent is evaluated every 5000 steps by running 10 episodes. The success rate is measured as the fraction of the episodes where the agent successfully reaches the goal (within 100 timesteps). We run 50 seeds. For training, the agent receives a reward equal to the length of the shortest path to the goal state (divided by 10 for scaling purposes). This is dense, informative reward signal so the difficulty from exploration is minimized. As such, we can attribute any performance issues to the optimization aspects of training a deep RL agent.
Then, we consider two environments from the CARL suite 99: LunarLander and DMCQuadruped. To generate a sequence of tasks, we choose certain context variables (e.g. gravity, wind, stiffness) and vary them for each task. The same sequence of context variables are presented to all the agents. For LunarLander, the agent is trained for 10 million steps with task changes every 500 thousand steps. For DMCQuadruped, the task changes every 1.5 million steps up 12 million steps. For both of these environments, we generate 20 different sequences of tasks and run each sequence for 3 seeds.
As final benchmark, our experiments will be conducted. First, we run an RPO agent on all the environments and identify those where the agent can achieve high success rate after 1 IM steps of training. This preliminary selection process results in a set of 19 environments (see Appendix C.4). These roughly match the tasks with high success rates indicated in [22 (appendix B).
environments from the Meta World suite 62 where the majority of
From this set of candidate environments, we produce sequences of 10 tasks by sampling environ- ments. In total, we produce 20 sequences of tasks, where each task corresponds to one Metaworld environment. We use a stratified sampling approach to ensure that each of the 19 environments are present the same number of times (or difference of one) in all the sequences. Moreover, each sequence of tasks does not contain the same environment twice. These choices promote a diversity of task orderings and task choices. Overall, we obtain 20 sequences of 10 tasks each. We call this benchmark Metaworld20-10 Each agent is run on every sequence with 3 seeds each.
Base agent. We use the RPO agent (50) (a variant of PPO) for continuous actions or PPO for discrete actions, based on the implementation from CleanRL (28). Some adjustments to the hyperparameters were made as specified in the appendix (Sec. C.5). with one set of hyperparameters being used across all Metaworld tasks and one set for each of the other environments. For the learning curves, we plot the interquartile mean along with 90% confidence interval shaded region. Each curve is also smoothed with a window of 5. Hyperparameters were tuned around the values provided in the CleanRL implementation. For all the algorithms, small sweeps were conducted relevant hyperparameters and the best setting was chosen.
Performance profiles. Due to the high variability in performance due to the nonstationarity of tasks, we favor looking at the entire distribution of the agent's performance. To do so, we plot performance profiles (3). These plots correspond to minus the empirical CDF of the measured evaluation statistic
5

---

and help us visualize the distribution of the agent's performance in a cleaner manner. To produce one plot, we consider each task in a sequence as a separate datapoint. For each fixed task, we take the mean success rate across the learning curve to get one summary number. For example, for the Metaworld runs, this would mean the 10 tasks in a sequence are considered individually to get 10 summary numbers, Since there are 20 sequences of tasks and 3 seeds are run for each, this would give a total of 10 x 20 x 3 = 600 datapoints to form one performance profile. 90% confidence bands are formed using the Dvoretzky-Kiefer Wieler--owiz inequality (see appendix C.2)
Baseline algorithms
Shrink-and -entrur (SnP) was suggested as a solution to poor performance on a later task after pretraining [5]. It also applies to sequences of tasks by decaying the weights and then adding some small amount of random noise at every step. SnP has also been found to be beneficial even on single tasks due to benefits arising from (partial) reinitialization (63]. To implement SnP, following the original description we add a small fraction of a freshly-initialized network's weights using Xavier initialization [16] to the learning agent's weights. We use AdamW (36) to implement weight decay.
Layer norm 66] has been found to give substantial benefits for training in a continual learning context [38. It allows deep neural networks to effectively learn sequences of tasks by adjusting the internal activations to have mean O and variance 1, mitigating the effect of distribution shift in the activations when tasks change. Layer norm has been used in variety of contexts including RL [2]] and large language models [59], displaying its utility even on single tasks.
Regenerative regularization (denoted "Regen" in plots) 33 is simple strategy to attempt to maintain favourable optimization properties of the initialization scheme. It consists of fz-regularization towards the initial weights. An extension relaxes the soft constraint by using an empirical Wasserstein loss between the distribution of the initial weights and current weights, allowing the learned weights to move further from the initialization but still preserving the distribution (denoted "W-Regen") [34]
4.1 Utility of Parseval regularization
We first test the base RPO agent, the proposed addition of Parseval regularization well as the baseline algorithms on the primary continual RL setting. Metaworld results are presented in Fig. [
and other environments in Fig.
3.04
2.
1.4
06
2.0
0.4
58
5.-
5.0
02
2.4
2.5
1.5
200
400
800
300
-303
-200
-1.00
100
200
300
Ancrog: Succobs Razz
Ancrge Eval Rozumn
Assiage Evall fotuun
Figure 4: Performance of algorithms on gridworld and CARL environments. Parseval regularization yields the largest improvements although other approaches can be helpful.
Overall, we see that the addition of Parseval regularization can greatly improve the performance of the base agent on the four set of tasks. While the alternative algorithms can also improve the bascline, Parseval regularization makes the largest difference. This is most apparent in the gridworld where lets the agent make progress on almost every task while the agent often gets stuck at O success rate with the other algorithms (as indicated by the large drop near zero on the x-axis).
For the next sections, we focus our attention on the Metaworld benchmark and investigate various questions in detail, including testing varying the network architecture, ablation studies and analysing parameter properties throughout training.
4.2 Variations of architecture and algorithm
We check if the benefits of Parseval regularization carry over to different neural network architectures.

---

Activation functions
The tanh activation function is the default choice for PPO 28 and we have already seen from the baseline that adding Parseval regularization improves the agent's 's performance considerably. We validate that Parseval regularization benefits other choices of activation functions such as ReLU [4]] mish [40], concatenated ReLU (CRelLU)
i 55 and MaxMin 4
ReLU is classic choice of activation function still widely in use. Mish, a smoother version of 24 14, [S have been shown to outperform the standard ReLU activation in deep learning tasks. Concatenated ReLU (CReLU) was used in the context of reinforcement to deal with the problem of dead units M. MaxMin was proposed in the design of Lipschitz neural networks 4 and, by maintaining the norm of backpropagated gradients, may also be suitable in this continual RL setting.
ReLU, and similar functions (e.g, GELU, Swish) 40,
Activatton Function Comparison
Activatton Function Comparison
Comparing Network Wish
1.0
0.5
0.6
0.6
Pitucess
Min
Tann
No Partouat 26th 22
Chetti
Sartral 1itth G4
0.2
sameral
Faroral
0.2
dm:32
Faroooa
Cketu, Farteva
wit Parteus or d51151
Farcerd
Vaciin Pareac
Win Parneval d1:1120
0.0
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Avrerage Success
ogerage Success
age Success
Figure 5: Performance profiles for different architecture choices. (Left and center) Varying activation functions: all choices benefit from Parseval regularization. (Right) Varying the network width. Parseval regularization can benefit all three settings. Increasing the width alone does not help.
In Fig. we see that all activation functions benefit from Parseval regularization, with Tanh showing the largest difference. Interestingly, without Parseval regularization, the different activation functions perform roughly the same. Even Concatenated ReLU, which was designed tackle an aspect of plasticity loss fares no better. Only the MaxMin activation has noticeably higher performance, suggesting that preserving gradient norms across the activation function (as it was designed to do) may be useful guiding principle for architectures in a continual RL setting.
In Fig. 14 (in the appendix), as an additional check, we verify that using identity activations (and hence a linear function overall), results in poor performance, confirming that nonlinearity is crucial even with orthogonal weights. Interestingly, linear activations tend to make some progress (e.g. more than 20% success) on more tasks than the nonlinearities, but at the price of fewer tasks where the agent achieves high success rates (e.g. over 90%).
Neural network widths
We investigate what happens when the width of the network is changed. To adapt Parseval regulariza- tion to different network widths, we ensure that the regularization strength is scaled appropriately by the square of the width. If the width is doubled, the regularization strength is divided by four. Results are summarized in Fig. 5 Again, Parseval regularization can benefit networks of reduced or increased widths.
4.3 Ablation studies
Parseval regularization can split into two different components. It acts on the rows of the weight matrices and 1) encourages the angles between them to be 0 and 2) encourages the norms to be constant. We test these components separately to better identify the source of Parseval regularization's benefits.
Angles between vectors We investigate whether variant of Parseval regularization that only regularizes the angles is also helpful. This can assess the benefits of diversity in the weight vector directions. Specifically, if W denotes the weight matrix to be regularized, we first normalize the rows to have norm to get W = W/IlWWlrow where division is row-wise and ||Wll computes the fg norm of each row. Then, we apply Parseval regularization on W. This has the net effect of disregarding the norm of the row weight vectors while still regularizing the inner products towards zero.
7

---

Fig. shows that this ablation still produces weight vectors that are orthogonal to each other. While there is some benefit, the performance worse than standard Parseval regularization.
Figure 6: Parseval regulariza- tion on normalized row weight vectors, regularizing only angles, on the Metaworld sequences. The right figure shows the av- erage angle between row weight vectors, confirming it has the de- sired effect.
Perencal 18th Norma Weiglts
Crélir Coline Simbariy Layer?
0.040
0.035
0.B
20.030
0.6
0.023
0.029
na remollstic
0.4
0.013
Ro Fanoval
0.010
rermelion piter
0.005
0.0
0.000
0.2
0.4
0-5
0.3
1.0
0.2
0..4
0.6
3.5
Average Succeas Rata
Steps
Norm of the weights. Parseval regularization regularizes the row weight vectors towards the initial scale, set to 2 by default. The norm of the weights is common metric linked to plasticity loss, with growing norms potentially indicating training difficulties 38]. In this view, regularizing the norms of the weight matrix rows could be beneficial in similar fashion to weight decay.
Revisiting Fig. [] setting the number of subgroups to 64 (the width of the network) corresponds to applying only regularization to the weight norms, without regularizing angles. There is a small benefit over the baseline, but not nearly as much as full Parseval regularization.
Findings. Both ablations improve the performance over the baseline but do not match full Parseval regularization, although using regularization on the angles makes a larger difference, From this, we can conclude that both components are critical to the success of Parseval regularization.
4.4 Analysis of training
To verify the impact of Parseval regularization on network properties throughout training, we inspect two measures of diversity: the stable rank of weight matrices and the correlation between weight vectors of neurons.
Stable rank. We first check the stable rank of the matrices, defined as srank(A) Σx where A is an n X n matrix and o; (i 1, ...,. n) are its singular values. This soft version of the rank will be equal to n if all the singular values are equal and is at most the standard rank. It is less sensitive small singular values.
max, a
Justification: Having a larger rank would indicate that the weight vectors span a greater portion of the input space. A similar notion of rank has previously been found to be correlated to the performance of reinforcement learning agents 32 37 12 in certain settings. In particular, excessively low rank values be linked to poor performance. Since Parseval regularization encourages matrices to be orthogonal and thus have all equal singular values, we would expect its addition to make matrices closer to full rank, potentially boosting performance.
Findings. We find that Parseval regularization can increase the stable significantly, leading the matrices to maintain almost full rank, while the baselines all experience quick reduction in rank during training before stabilizing at a small value (often less than 10). See plots in Appendix B. for details.
Neuron weight similarity. We also verify another measure of diversity, corresponding to the average cosine similarity of the row vectors of a weight matrix. Symbolically, for given weight matrix W where w,, E (L....,n) denotes the i-th [w
with n rows, it is given by: m(n-1 Eits row of W.
Justification: Weight similarity measures how different the directions of the weight vectors are from each other. So. the lower the weight correlation, the greater the diversity amongst the neurons. With orthogonal initialization, we expect direction correlation to be zero at the start of training (or near-zero if the weight matrix has more rows than columns so there are more vectors than dimensions in the space). This measure was also used by [0] under the name forward correlation.
One may expect that smaller neuron weight correlation would be favourable since there will be greater diversity the "active" regions of the activation function and cover the space of pre-activations more fully. This could be especially important when there is nonstationarity in the data (as in RL), so state inputs or intermediate activations may lie previously uncovered regions of the space. Maintaining some diversity pre-emptively could yield faster learning when change occurs. There is
8

---

some evidence that weight vectors in vanilla MLP networks will be oriented in the same, redundant directions if they are sufficient to perform the task [52]. Larger weight correlation may be a symptom of networks that have overly specialized to task and would also be reflected in reduction in stable rank.
Findings. We find that Parseval regularization indeed maintains a near-zero neuron weight correlation while also having nearly full stable rank. The other algorithms manifest decreasing stable ranks and increasing neuron weight correlation over time (see Appendix B.I for plots).
Additional experiments. In Appendix A we have included other exploratory experiments. We find that Parseval regularization can help reduce the variance of entries in the input-output Jacobian, a quantity which may be linked to training stability [9]] compared to the baseline (see Fig. [TP. In a different experiment, to help understand how low weight stable ranks and large parameter norms affect training, we perturb these initialization but it does not necessarily cause the agent to fail to learn a task (see appendix [A.1]. Thus, we can conclude it is these properties in conjunction with other aspects of the optimization process which can be problematic, painting more complex picture and echoing the findings of Lyle et al. (38)
5 Related works
Previous works have tackled the problem of loss of plasticity from different angles. Many algorithms in this line of work focus on injecting new randomness into the weights to restore some of the initial randomness present in the usual Gaussian weights. Aside from Marink-and-Pertuun discussed previously, other algorithms algorithms focus on identifying useful neurons and resetting the weights of those deemed unnecessary dependent on using ReL.U activations, which is not directly applicable to other nonlinearities such as the tanh activation. Another approach is reset the weights entirely for certain layers [8] [ to fresh weights. While this can solve trainability issues, it also removes any possibility of using learned representations from previous tasks to speed up future learning. Moreover, the frequency of the resets can be an important hyperparameter. Methods like Parseval regularization that act on the network architecture (parameters, activations or normalization layers), naturally avoid having to specify the timescale at which change occurs. Expanding the network by adding more neurons to the network during training can also be used to improve trainability 44 This comes with increasing compute and memory costs as the agent interacts with the environment, making it less appealing for continual learning settings where, in principle, an infinite number of task changes may occur.
112 57 These algorithms
notion of usefulness that is
In the continual learning community, many works have tackled how to learn efficiently from sequences of tasks, although they have mainly focused on the problem of catastrophic forgetting [13 35 31 60),
that is, how to remember previous solutions to previous tasks without overriding them when learning on new tasks. Many of these investigations have focused on supervised learning in the past. In this paper, we focus mainly on improving the plasticity of neural networks rather than the stability, which may be priority in RL settings where the agent is focused on improving its current policy rather than remembering all past policies.
Parseval regularization was originally proposed in the context of maintaining Lipchitz constraints to improve adversarial robustness [T] 4 with little focus on the optimization aspects. Many other techniques have been proposed to maintain orthogonal weight matrices, including regularization (18), parameter scaling [49] and specialized parameterizations [4] 556.. In this work, we have focused on the simplest peehod-ddrectte regularizing the weight matrices to be orthogonal-althou future investigations may find some benefit in using more advanced techniques.
Orthogonal initialization [53] and later developments in initialization schemes such as ReZero ? and Fixup (64) have used similar design principles such as maintaining an input-output Jacobian near identity matrix to maintain gradient magnitudes across layers and avoiding exploding and vanishing gradients. These later initializations are tailored to Resnet-based architectures featuring skip connections (23] and have focused the optimization properties at initialization whereas, in this paper, we emphasize the entire training process.
6 Discussion and Limitations
We take an optimization viewpoint on the continual RL problem and suggest improving performance by considering the optimization properties of network layers. Starting from the classic idea of using proper weight initializations, Parseval regularization aims to keep the weights in regions of the

---

parameter space where the network has well-conditioned gradients throughout the entirety of training even across task changes. potential weakness of Parseval regularization is that reduces the capacity of the network by constraining optimization to lie within the space of orthogonal matrices. We have seen that this can be offset by introducing a few parameters (or only one) through diagonal layers or a learnable input scale. The ingredients of deep neural networks were initially developed for single tasks in the supervised learning setting and we believe revisiting the basic building blocks may lead to other fruitful developments in reinforcement learning and continual learning.
Acknowledgements
We would like to thank the reviewers for their feedback and Harley Wiltzer and Jesse Farebrother for helpful discussions. The authors acknowledge funding from NSERC. Lynn Cherif supported by FRQNT Master's s Training Scholarship.
References
[1] Z. Abbas, R. Zhao, J. Modayil, A. White, and M. C. Machado. Loss of plasticity in continual deep reinforcement learning. arXiv preprint urXiv:2303.07500 2023.
[2] A. Achille, M. Rovere, and S. Soatto. Critical learning periods in deep neural networks. arXiv preprint arXiv: (rcXiv:1711.00 1ee.111111.8666 2017.
[3] R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in neural information processing systems, 34:29304 24422204-29320, 2021.
[4] C. Anil, J. Lucas, and R. Grosse. Sorting out lipschitz function approximation. In International Conference on Machine Learning, pages 291-301. PMLR, 2019.
[5] J. Ash and R. P. Adams. On warm-starting neural network training, Advances in neural information processing systems, 33:3884-3894, 2020.
[6] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization, arXiv preprint arXiv:1607.0645 2016.
[7] Bachlechner, B. Majumder, Mao, G. Cottrell, and McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pages 1352-1361. PMLR, 2021.
[8] D. Balduzzi, M. Frean, L. Leary, J. Lewis, K. W.-D. Ma, and B. McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In International Conference on Machine Learning, pages 342-350, PMLR, 2017.
[9] C. Benjaminss, T. Eimer, F. Schubert, A. Biedenkapp. B. Rosenhahn, F. Hutter, and M. Lin- dauer. Carl: A benchmark for contextual and adaptive reinforcement learning. arXiv preprint arXiv:2/ 10.02 /02, 2021.
[10] Y. Blumenfeld, D. Gilboa, and D. Soudry. Beyond signal propagation: is feature diversity necessary in deep neural network initialization? In International Conference on Machine Learning, pages 960-969. PMLR, 2020.
[11] M. Cisse, P. Bojanowski. E. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness adversarial examples. International conference on machine learning, pages 854-863. PMLR, 2017.
[12] S. Dohare, R. S. Sutton, and A. R. Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint 108, 06325500066....80 2021.
[13] P. D'Oro, M. Schwarzer, E. Nikishin, P.-.. Bacon, M. G. Bellemare, and A. Courville. Sample- efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NewrIPS 2022, 2022.
10

---

[14] S. Elfwing, E. Uchibe, and K. Doya. Sigmoid-weighted linear units for neural network function approximation reinforcement learning. arxiv e-prints (2017). arXiv preprint arXiv: 172.v1170..03318, 1702, 2017.
[15] R. M. French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 344: 1999.
[16] X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
[17] J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio. An empirical investigation of catastrophic forgetting in gradient-basee based neural networks. arXiv preprint arXiv:1312.6211, 2013.
[18] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
[19] B. Hanin. Which neural net architectures give rise to exploding and vanishing gradients? Advances in neural information processing systems, 31, 2018.
[20] B. Hanin and D. Rolnick. How to start training: The effect of initialization and architecture. Advances in Neural Information Processing Systems, 31, 2018.
[21] N. Hansen, H. Su, and X. Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16822 2023.
[22] K. Hc, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[23] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[24] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv: (ee::6066000 18 2016.
arXiv preprint
[25] Hochreiter, Y. Bengio, P. Frasconi, J. Schmidhuber, et al. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001.
[26] W. Hu, L. Xiao, and J. Pennington. Provable benefit of orthogonal initialization in optimizing deep linear networks. arXiv preprint arXiv:2001105992, 2020.
[27] S. Huang, R. F. J. Dossa, A. Raffin, A. Kanervisto, and W. Wang. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023, 2022.
[28] Huang, R. F. Dossa, C. Ye, Braga, D. Chakraborty, Mehta, and J. G. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23 2022. http:///:t org. orggpapers/vv3/
21-1342.0 html
[29] S. Jastrzebski, D. Arpit, O. Astrand, G. B. Kerg, H. Wang, C. Xiong, R. Socher, K. Cho, and
K. J. Geras. Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4772-4784 PMLR, 18-24 Jul 2021. URL httptt:::ppoceeed.iiote
mlr. prw.s/rssssseec wttp://asttteeas.coo. html
[30] K. Khetarpal, M. Riemer, Rish, and D. Precup. Towards continual reinforcement learning: review and perspectives. arxiv. arXiv preprint arXiv:2012.13490 2020.
[31] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, Quan, T. Ramalho, A. neural networks. Proceedings of the national academy of sciences, 114(13):3521-352 2017.
et al. Overcoming catastrophic forgetting in

---

[32] A. Kumar, R. Agarwal, D. Ghosh, and S. Levine. Implicit pondderparammtiiiiatttoo inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010.14498, 2020.
[33] S. Kumar, Marklund, and B. Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprini arXiv:2308. 17958, 1958, 2023.
[34] A. Lewandowski, H. Tanaka, D. Schuurmans, and M. C. Machado. Curvature explains loss of plasticity. arXiv preprint arX1vv:3122.00246, 2023.
[35] D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[36] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05/01,220 2017.
[37] C. Lyle, M. Rowland, and W. Dabney. Understanding and preventing capacity loss in reinforce- ment learning. arXiv preprint arX1v:220.005600, 2022.
[38] C. Lyle, Z. Zheng, F. Nikishin, B. A. Pires, R. Pascanu, and W. Dabney. Understanding plasticity in neural networks. arXiv preprint arXiv:2303.01440 014866 2023.
[39] W. Masarczyk, M. Ostaszewski, E. Imani, R. Pascanu, P. Mitoš, and T. Trzcinski. The tunnel effect: Building data representations in deep neural networks. Advances in Neural Information Processing Systems, 36, 2024.
[40] D. Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint
Xr:iv: /tt.//9888000...1 2019.
[41] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-I0), pages 807- 814, 2010.
[42] B. Neyshabur, Sedghi, and C. Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
[43] E. Nikishin, M. Schwarzer, P. D'Oro, P.-. Bacon, and A. Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pages 16828-16847. PMLR, 2022.
[44] E. Nikishin, J. Oh, G. Ostrovski, C. Lyle, R. Pascanu, W. Dabney, and A. Barreto. Deep reinforcement learning with plasticity injection. arXiv preprins arXiv:2305.1555. 2023.
[45] V. Papyan, X. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 14652-24663, 2020.
[46] R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318. Pmlr, 2013.
[47] J. Pennington, S. Schoenholz, and S. Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in neural information processing systems, 30, 2017.
[48] Pennington, S. Schoenholz, and Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, pages 1924-1932. PMLR, 2018
[49] B. Prach and C. H. Lampert. Almost-orthogonal layers for efficient ceneral-purposs lipschitz networks. In European Conference on Computer Vision, pages 350-365 363.6....0 Springer, 2022.
[50] M. M. Rahman and Y. Xue. Robust policy optimization in deep reinforcement learning. arXiv preprint 222.0:2. 2022.
[51] Ramachandran, B. Zoph, and Q. Le Swish: self-gated activation function. arXiv preprint 17X1v:1710.05941, 7(1:5, 2017.
12

---

[52] A. Saxe, S. Sodhani, and S. J. Lewallen. The neural race reduction: Dynamics of abstraction in gated networks. In International Conference on Machine Learning, pages 19287-19309. PMLR, 2022.
[53] A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6122 2013.
[54] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 10X.v:7777.06344 2017.
[55] W. Shang. K. Sohn, Almeida, and H. Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In international conference on machine learning, pages 2217-2225. PMLR, 2016.
[56] Singla and Feizi. Skew orthogonal convolutions. International Conference on Machine Learning, pages 9756-9766. PMLR, 2021.
[57] G. Sokar, R. Agarwal, P. S. Castro, and U. Evei. The dormant neuron phenomenon in deep reinforcement learning. arXiv preprint wrXi:2230..12902, 2023.
[58] A. A. Team, J. Bauer, K. Baumli, S. Baveja, F. Behbahani, A. Bhoopchand, N. Bradley-Schmiee M. Chang, N. Clay, A. Collister, et al. Human-timescale adaptation in open-ended task space. arXiv preprini urXiv:2301.0760
2023.
[59] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[60] E. Verwimp, S. Ben-David, M. Bethge, A. Cossu, A. Gepperth, T. L. Hayes, E. Hüllermeier, C. Kanan, D. Kudithipudi, C. H. Lampert, et al. Continual learning: Applications and the road forward. arXiv preprint 1xiv:23311.1108, 2023.
[61] L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In International Conference Machine Learning, pages 1393-5402. PMLR, 2018.
[62] T. Yu, D. Quillen, 7. He, R. Julian, K. Hausman, C. Finn, and S. Levine Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100. PMLR, 2020.
[63] S. Zaidi, T. Berariu, H. Kim, Bornschein, C. Clopath, Y. W. Teh, and Pascanu, When does re- re-initialization work? In Proceedings pages 12-26. PMIR, 2023.
[64] H. Zhang, Y. N. Dauphin, and T. Ma. Fixup initialization: Residual learning without normaliza- tion. arXiv preprint
2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly specify what the contributions are and to which area it belongs to. An overview of the paper is presented.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
This is found in the Discussion and Limitations section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
These details provided in the Experiments section along with the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//guiii wwwticccguie//eeeee CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
These are outlined in the Experiments section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The plots do contain error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This is mentioned the appendix section on Compute.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The topic studied is foundational for training continual reinforcement learning
agents. Any risks that arise would only tied to broad risks of Al agents.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Citations are provided throughout the paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19

---

Appendix
A
Additional experiments
We investigate other aspects of plasticity including the role of entropy and experiments with perturbing standard initializations.
Policy entropy. We find that higher policy entropy (the average entropy of the policy averaged across states) is often correlated with better performance across tasks but directly adding entropy regularization does not bring any significant benefits (see Appendix A..).. This suggests that policy entropy may be correlated but does not have a causal link to performance improvements.
Initialization properties. To better understand the favourable optimization properties of initializations, perturb the rank and the scale of the standard orthogonal initialization (see Appendix (A.I) and evaluate the performance on single tasks from the Metaworld suite. By doing so, we can better understand how large weights and low rank weight matrices impact agent's performance. We find that using a larger initialization scale does not degrade performance. Reducing the rank of the initialization has nonlinear relationship with performance, sometimes improving and sometimes reducing it. Similar to the findings of Lyle et al. [38]. we conclude that the relationship between these quantities and performance is tricky. Further investigations would be required elucidate these results.
Parameter scale. Theory dictates that of each row is 1) to maintain true isometry. In practice, using norm of v2 is more common. Comparing the two settings in Fig. A.2 we do not observe a significant difference although we may expect these conclusions to change with deeper networks.
an orthonormal matrix for the parameters (the norm
A.1 Single Task Investigations
In this subsection, we scrutinize two metrics often linked to plasticity loss: large weight norms and small weight matrix ranks. To be precise, we alter the parameter initialization of the agent to induce these two conditions and inspect the effects on training a single task. The success rate is averaged over the 19 base Metaworld environments identified previously with seeds run on each. Fig. contains the findings.
Large weight norms. To induce large parameter norms, choose larger setting of the initialization scale. This has the effect of multiplying the weight matrix by constant. The default value is v2 and we consider scales up to 8, which span the weight norms we typically see during training.
Findings. The initial scale has little effect on the performance of the agent. This suggests that the link between weight norms and trainability more nuanced. this case, training can be successful despite much larger parameter norms.
We reduce the rank of the weight matrices by projecting the initial random weight matrices onto subspace of the chosen dimension, m. The subspace is chosen uniformly at random by taking the first m basis vectors from a random (full-rank) orthonormal matrix. The projected weight vectors are then rescaled to their original length times 1 + c, with being small random number, ensure that there are no duplicate weights.
Small weight matrix rank.
Findings. We can see there is a complex relationship between the dimension of the projection subspace and performance. An intermediate value of the dimension yields the best performance and the relationship is not monotonic. At first, this seems contradictory to the results concerning using subgroups with Parseval regularization (see Fig. 3 since the best setting was not using subgroups all. These findings can be reconciled by the fact that a larger rank might be more beneficial when is maintained throughout the entire training process while, here, it is only the initial rank that is modified.
A.2 Initialization scale
In orthogonal initialization, the weights of the matrix can rescaled by a constant as an additional hyperparameter called the initialization gain. For different initialization schemes, this gain can vary
20

---

Intialization
Projection Subspece
Saliz
Scaln:3
0.8
Salec
Scalndl
S4CCC0 1185
Ratt
0.6
0.4
64
Dkm:37
Dmm:16
0.7
07
0mm
Steps
305
Lub
Parameter Norms (Critic Layer 2)
Parameter Projection Stable Rank for Critic Layer 2
Scale #2
Scale:2
Dem:16
Dem:8
scale:e
Dem:4
Scalert
Dem:
59.0066
20-
10
0.2
10
0.0
0.8
Steps
266
50pps
166
Figure 7: These plots present learning curves on a single task. On the left, we compare different initialization scales. On the right, compare the rank of the initial weight matrix. See Sec. A.l for details.
intiidiiatiinn Scale Compaatioon
0.8
PrBDucess hale
2.6
0.4
Scales1, Persevel
Scalecgp11)) no Rarsnvel
Scalt: 1. wil Parzeval
Scalecugrt22, with Ranarcal
0.0
1.0
Ansrape
Figure 8: Performance profile for two settings of the initialization scale. With Parseval regularization, they both perform similarly.
based on the activation function used. The baseline RPO agent recommends using an initialization scale (also called "gain") of /2 (28).
Orthogonal initialization is usually presented with an initialization scale of since this gives the dynamical isometry which is important for avoiding exploding and vanishing gradients in deep networks [47 61. Our previous experiments used the recommended gain of v2 and test if changing this to 1, as prescribed by theory, makes a difference.
Empirically, in Fig. 8 we find that the two settings are about equal when combined with Parseval regularization. This may be due to the shallow network used (3 layers) and choosing a gain of might be more important with deeper networks.
21

---

A.3 The role of entropy
In many environments, well well-performing agent seems to correlate well with a high level of entropy in the policy. For example, see Fig. (right).
Entropy regularization has been found to be theoretically beneficial for policy gradient algorithms by improving the curvature properties of the policy optimization objective. In practice, the baseline PPO agent does not use entropy regularization by default as it did not produce any benefit in continuous control tasks [27 We revisit this finding and verify the use of entropy regularization in conjunction with PPO. Entropy regularization is added to the usual policy optimization loss as follows: ((T policcy (TT) dent (TTe) where H(x)) = J x (x) In π (2) dx is the entropy of the
policy ग., Aent > O is a weighting term.
Entropy Regularization Compariiioiinnooco
Policy Entropy
10
tntrepyo
Pb seval
Entrogy
70-
Raese
Intragysled
Norra
0.8
Entropy
18-
ShP
Tetmayy
Pegon
12-
0.0
Average
Staps
167
Figure 9: The first plot shows learning curves for different values of entropy regularization. The second plot shows curves for the policy entropy throughout training for different algorithms. We observe that larger entropy can be better when set properly btu can also have a detrimental effect if too large.
In Fig. 9 we find that some amount of entropy regularization can be beneficial in the Metaworld tasks. For larger values of the entropy regularization coefficient, the performance is markedly worse though.
We can conclude that, while larger policy entropy seems correlated to better performance, is not entirely causal relationship. A possible explanation is that agents have high entropy in states that are new and better agents can visit larger parts of the state space as the policy improves. Together, this would lead to the observation of higher entropy on average in the states visited Sonlyddollyyaa a side-effect of better policy optimization
A.4 Input-output Jacobian
The
Jacobian is defined as the matrix of derivatives of outputs with respect to the inputs. It has been useful tool to derive initializations to train very deep neural networks through the concept of dynamical isometry, the property that input-output Jacobian has singular values equal to 1 This property allows gradients propagate backwards while avoiding the vanishing and exploding gradient problems 200). [19] more directly studied the entries of the input-output Jacobian and found that the variance among the (square of the) entries was also an important quantity, with a low variance being crucial to favourable optimization properties.
533 47
Balduzzi et al. 8 found that the input-output Jacobian could also be used to study the optimization benefits of Resnets [3). They identify the shattered gradients problem which is indicated by the onput-output Jacobian being sensitive perturbations in the input.
We examine the input-output Jacobian matrix at various points training to glean insight into the trainability of the network. Specifically, we save a batch of test states for an environment obtained by the policy during training. We compute the Jacobian matrix of the outputs with respect to the inputs of the network and inspect its entries these test states. We are interested in inspecting the
22

---

Trpat autp pa
Vakaa wtg.eed....... NerZAS
Fput oudpul juceban Eudes. Metoworki sequecze
TOLE ouput tandion Values Haur Veltoakii Sequncce
3050
2040
Figure 10: The distribution of the squared entries of the input-output Jacobian along training for the first three sequences Metaworld tasks. The solid line is the mean while the shaded region denotes the 5th and 95th percentiles of the distribution. Parseval regularization reduces the spread of the entries.
magnitudes and thus take the square of each entry. This gives us empirical distribution of Jacobian entry magnitudes at regular checkpoints throughout training.
indeed that using Parseval regularization promotes tighter spread of magnitudes of the Jacobian entries whereas the base agent contains few very large magnitudes ones. The presence of these larger entries may reflect average magnitude is larger with Parseval regularization, this may be acceptable since it the spread of the magnitudes that matters more.
Findings. In Fig. 10
difficult loss landscape. While the
23

---

B Additional Plots
B.1 Cosine Similarity and Stable Rank
For cosine similarity, see Fig. and, for stable rank, see Fig. 12
Actor Cosine Similarity
Actor Cosine Sinilarity for Layer
Baac
0.00
Layer Norm
5tiP
UeBegen
0,03
Rogen
0.00
0.04
0.01
0.62
Regnn
0.6
1.0
Cribic Cosine similarity for Layer 1
Similarity Layer
0.20
Layer
SKi
0.08
Iegen
ReRgge
venseval
Base
bant
0.06
0.055
0.10
0.2
0.005
0.000
0.5 0.5
1.0
Steps
767
Figure The cosine similarity of row vectors the weight matrices the first two layers. Parseval regularization keeps the cosine similarity near ( whereas most of the other agents have increasing cosine similarities throughout training. Regenerative regularization also maintains relatively low cosine similarity.
B.2
Parameter norms with subgroup Parseval regularization
See Fig.
B.3 Linear neural network
As sanity check, we look at using identity activations to produce a linear neural network and test whether nonlinearity is needed for the tasks. See Fig.
24

---

Layer
Layer
30.
25
Parseval
Bank
Layer Norm
Layer horm
Sopp
Wheggen
Pegen
Regen
10-
1.0
0.0
Stops
Layer
tic Stable Layer
60-
50
Parseval
40
Rank
Layer Nor
Layer Norm
Spp
Weggen
30
Witggen
15
Pegen
Pargen
70-
5-
10
0.0
Steps
Figure 12: The cosine stable rank of the weight matrices for the first two layers. Parseval regularization keeps the stable rank near its maximum value while it decreases quickly for the other algorithms settling at small value far from the initial rank.
C Experimental Details
C.1
Gridworld
See Fig.
15 for information about the gridworld's layout.
C.2 DKW Confidence Bands
The DKW inequality can be used to construct a confidence band around the estimated CDF. Let F(x) be the true CDF value at x and (x())) be the estimated value from n samples. Precisely, Fa(x) = 1 Σ 1(x; < x) where ((A) is the indicator for the event A. Then, with probability at least 1 - a, we have [F(x) F,())) V
log (2/a) for all x.
2n
This is a simultaneous confidence band since holds for values at once, meaning that the region denoted by the shaded area of the plots will contain the entire curve with high confidence. This is opposed to pointwise confidence intervals, which only guarantees that a particular point on the curve is within the interval with high probability but not simultancously for different points the curve. These pointwise confidence intervals were originally reported with performance profiles 3 Here, confidence bands since they provide stronger statistical guarantee.
25

---

Layer
Layer
70
340groups 1
Subgroups..1
segmapec
30.0
bubgroapuce
Sdaguuap
Subgroupass
10
27.5
Subgroupo:8
Gubgreaputto
farseval
Ko Fartewal
20.0
31.5
15.0
Cribc Farameter Norms Layer
Critic Paramater Norms Layer
Subgroups
Subersups
Subgroups
Subgroups
Subgroups
Subgroups:
Subgroups
Subgroupe
Subgroups
14
14
13
0.6
0.9
10
04
1.0
lw/
Figure 13: Parameter norms when using subgroups for Parseval regularization. Subgroups:64 corresponds to only regularizing the row weight vector norms. Parseval regularization any setting can maintain constant norm for the parameters.
Act valion Function Comperisan
Activation Function Comparisor
tanth
0x
018-
PECuSCS111 T1BC -1
RelAi Po bevel
Nish, Forsovol
idertRe. Parsevol
kanh
Relli
0.4
ReWU,
Idenkity Perseval
Average
Figure 14: Performance of linear neural networks compared to using activation functions. The linear neural network denoted by "Identity" activations. We see nonlinearity is crucial.
26

---

Figure 15: Gridworld layout. There are nine rooms with doorways indicated by the orange lines. The agent starts at the green square at the beginning of each episode. The goal location is randomly generated in one cells shaded blue. This is kept fixed until the task changes at which point is resampled.
C.3
CARL environments
To produce a sequence of contexts, we generate deterministic sequence based on the seed. We select some context variables and either randomly sample then or cycle through few settings. We ensure that enough change occurs between subsequent contexts that the return of the agent will dip after change (to force the agent to learn a new policy). We set the seeds from to 20 to generate 20 different fixed sequences of context variables.
The code used to generate the sequences are outlined below for Lunar Lander and DMCQuadruped.
Iaport suxpy as 9p
def generate_ iettt___aaraaadee
(dask_idx,
seed):
rng up. gravicy_loo
RandonStatn
gravicy_hige _high = 1.5
angine_lov 1.0
wngine_high
initial_lov
initinl_high 1.5
def sasple_grav_xe (i)):
10
toggle settinge:
grav_x -
13
elif -- grav_s
14
grav_x
17
return
18
earple_context,
return up array (Irgay(rrr onfforn gravity.high), rng. (ng.nni engine_high)
20
rag, reg.uniford (anginnnnnnnnnae (ggine_high).
www.ornnoonnnnnn (nttonniinttsll.loo initial..ighh])
21
23
anaure aubasquent
snough from each
change 1.0 in 1.I distance =
24
25
context, _iictxt_,iettt_ee.te [ap concatenate (carpaaccccaasppcccocooot (), ap. atrayy(eaxpie.ggrr..x tack_1dx:
wbile Lenteentert
27
context__vra._ randon sanpla__ontextt
78
abbncpp.bb(cooooottt randor min_change:
29
contort_wars + concasonate ([costert, rard_r, apray
31
contest_liet (ppens(
32
context_var_ce car_ ['GRAVITY_Y', MAINNEGGINE, POVER', (VVIYY_XX)
34
(var:valus
context, 1ist (task_idx 11)
35
Listing Lunar Lander sequence generation
27

---

iapozt suzpy
ganeraie, www.:aawwwwnnguaauueeeoooo seud_offaet seed a seed seed_offset rag np.readde. h.t.:mmin..eclntttttaaaaa
seed):
actuator_lov 0.3
astuator_bight x 1.3
gravicy_lor 0.3
gravicy.high 1.3
aeed X wind_liat - [(O, 0), (-1, -1), (1,1), a,, -1,, (-1, 1)1
11
def sasple,context().
12
vind_x, wind.y [ind_ I (1+1) 2
15
ap.array (Irng uniforn (actuator_loo actnator_high), rag.anifarm gravity.bigh),
16
wind.x, vind.y])
17
48 ansure that aubaequent dravs are far anough from nach other
18
change 0.0 a in 1.1 distance
19
costext, list [sarple_conters taxk_1dr;
whila lan(eanters, suaple_context.f
consext_ (bontarrtttnt contexs.list (-13)) nin_change:
22
(p..aannp..abrccc. append (ootttttga
23
context_ var_na ['actustor, "'actaator.atrenghh 'gravity', twind_x', "uind_y
26
return (varivalus for valus in coonteertt, contest contest_liat [takk
27
Listing 2: DMC Quadruped sequence generation
C.4 Metaworld environments
Generally, every environment Metaworld uses the same robotic arm controlled by a four- dimensional action, with various tasks such opening door or sliding plate. This makes switching between environments easy since the observation space and action space is common.
The following table specifies which tasks from the metaworld suite were used in the Metaworid20-1B 10 benchmark. These were chosen because RPO could achieve good enough score after million steps on these tasks. They roughly correspond to the casiest tasks indicated by Yu et al. [62] in Fig. 11 of Appendix B. Note that the training protocol different in that paper and they allow the agent train for significantly longer (20M steps).
Task Names
door-close-v2
sweep-into-v2
coffee-bution-v2
window-open-v2
reach-wall-v2
drawer-close-v2
button-press-v2
platesslide-bacc-ssde--2
reach-v2
plate-slide-sid--22
coffce-push-ve
plate-slide-bakk-
soccer-v2
window-close-v2
handle handlle-ull-side-ve
hand-insert-v2
door-lock-v2
push-v2
peg-unplug-side-vv
Table 1: Metaworld Tasks
To construct the sequences of tasks, we use stratified sampling approach and sample 20 different sequences 10 tasks with some constraints. Each sequence does not contain any duplicate tasks and every task is represented the same number of times (plus minus 1) in total across all 20 10 200
28

---

tasks, This ensures more variation in the tasks and that sequences contain nonstationarity. Once these tasks are sampled, they are fixed for all experiments to ensure consistency.
The 20 sequences of tasks are specified below (the "-v2" versions are used for all of them):
1. handle-pull-side, peg-un peg-unplug-side, coffee-push, soccer, drawer-close, reach-wall, plate-slide- back, window-open, slate-slde-ssdde plate-slide-back-S50
2. window-close, window-open, hand-insert, door-lock, reach, button-press, sweep-into, coffee- button, door-close, push
3. window-close, reach-wall, sweep-into, reach, soccer, coffee-push, plate-slide-side, drawer- close, hand-insert, door-close
4. plate-slide-be back, reach-wall, door-lock, peg-unplug-side push, button-press, plate-slide- back-side, coffee-push, coffee-button, handle-pull-side
5. push, coffee-button, sweep-into, door-close, drawer-close, soccer, peg-unpen reg-unplu--sid hand- insert, door-lock, reach
6. button-press, plate-slide sitte--lidebbac-siidee. window-close, plate-slide-side, peg- peg-un un-luuupuuusie side, plate- blcke-back, coffee- button, window-open, aandle-pull-sine door-close
7. push, button-press, plate-slide-back, drawer-close, soccer, plate-slide-side, reach-wall, coffee-push, window-close, door-lock
8. plate-slide-side, hand-insert, handle-pull-side, plate-slide-bcck-sdde,, window-open, sweep- into, relch-wall, reach, soccer, peg- seg-unplug-side side
9. hand-insert, reach, window-close, drawer-close, window-open, coffee-button, plate-slide- back, coffee-push, push, plate-slide-back-side
10. sweep-into, peg-unplug-side, window-close, door-lock, hand-insert, handle-pulle pull-side,
window-open, door-close, button-press, reach-wall
11. reach, door-lock, sweep-into, push, button-press, coffee-push, handle- handle-pull-side pull-side, plate-slide- side, door-close, drawer-close
12. plate-slide-back-sid,, soccer, sweep-into, handle-pull-side, plate-slide-side, peg-unplug-side, door-lock, reach, plate-slide-hack coffee-button
13. reach-wall, plate-slide-back, drawer-close, hand-insert, coffee-push, coffee-button, window- close, plate-slide- seatt--lid--bac--sddeee door-close, button-press presss
14. soccer, drawer-close, push, sweep-into, window-open, reach-wall, door-lock, window-close,
reach, hand-insert
15. plate-slide-back, plate-slide-side, door-close, push, peg-unplug-side, plate-slide-back-side, coffee-push, coffee-button, button-press, soccer
16. hand-insert, coffee- button, soccer, window-open, push, reach, drawer-close, handle-pull-side dock-lock, slute-slide-back---e
17. coffee-pust push, door-close, handle-pull-side window-close, plate-slide-back, reach-wall,
sweep-into, window-open, plate-slide-side, peg-unplug-side
18. coffee-push, button-press, reach, peg-unplug-side reach-wall, door-close, window-open, handle-pull-side suil-se blutt-sliddebbac--side soccer
19. sweep-into, plate-slide-side, button-press, drawer-close, push, coffee-button, door-lock, hand-insert, plate-slide-back, window-close
20. reach, button-press, plate-slide-side, door-close, plate-slide-back-sidee plate-slide-back, coffee-button, sweep-into, reach-wall, drawer-close
Note that each of these tasks additionally have goal location that can be varied. So due to the seed, the same environment (e.g. door-close) may not have the same goal location across different sequences.
29

---

Environment
Metaworld 0.0003
Gridworld
Quadruped
Lunar Lander
Learning rate
0.00025
0.0003
0.0003
Number envs
Minibatch size
64
32
64
32
Number minibatches
32
Update epochs
10
4
10
4
GAE lambda
0.95
0.95
0.95
0.95
Max grad norm
0.5
0.5
0.5
0.5
Entropy regularization
0.0
0.01
0.0001
0.0001
RPO alpha
0.5
0.0
0.5
0.0
Network width
64
64
Number of hidden layers
2
2
2
Additional parameters
Diag Layer
None
Input Scale
None
Table 2: Hyperparameters for RPO and PPO
C.5 Agent Hyperparameters
Hyperparameters for RPO and PPO on benchmark tasks.
Hyperparameters values checked for algorithms:
Purseval regularization Regularization strength: 1003 10.1100-1000 10-1,
10-5)
Layer Norm
None
Regenerative regularization (and Wasserstein version)
Regularization strength: (10-?, 10-3, 10-4, 10-b)
Strink-and-perturff
Perturb scale: (10-2, 10-3 10-4, 10-5)
Weight decay: (10-2, 10-3 10-5
C.6 Compute Utilized
We run the experiments CPUs given the small size of the networks a combination of Intel Gold 6148 Skylake at 2.4 GHz, AMD Rome 7532 at 2.40 GHz 256M cache L3 and AMD Rome 7502 at 2.50 GHz 128M cache L3 CPUs as part of cluster. Each single of algorithms used one CPU with at most 4GB of memory.
C.7 Runtime analysis
Table showw the runtimes of Parseval regularization, the standard baseline and and-euuurr on the various benchmark tasks. We can see that the additional computational cost from Parseval regularization is modest (up 11.4%) and less than that of SnP.
From theoretical standpoint, can compare the computational complexity of Parseval regular- ization and contrast it to the complexity of a forward pass. Parseval regularization requires ( operations for single dense layer of width d (and d inputs). On the other hand, forward pass through the same dense layer will cost (ndd) operations where n is the size of the minibatch.
As such, the relative cost of adding Parseval regularization depends on the ratio of d and n. In our experiments, the width is set to 64 with minibatches size 64. In this case, the cost of computing the Parseval regularizer and forward pass is roughly equal. Since backward pass takes about twice the computation forward pass, we expect Parseval regularization to require approximately 33% more computation compared the base agent alone.
The additional runtime observed in practice can significantly smaller than the theoretical value of 33% since substantial part of the total runtime comes from interacting with the environment to
30

---

collect data, both doing inference to generate actions and for the simulator to step forward. These costs are unaffected by Parseval regularization.
Moreover, we were to apply Parseval regularization to convolutional layers, then the relative cost would be even smaller the number of parameters for that layer would be small: A convolutional kernel only has ck parameters due to parameter-sharing, where is the number of channels and is the kernel width. For convolutional layers, the additional computational cost of Parseval regularization is much smaller compared to the cost of a forward pass.
Environment
No Parseval
With Parseval
Shrink-and-Pertur)
Metaworld sequence
607.5
634.7 (+4.5%)
658.3 (+8,4%)
Gridworld sequence
28.0
28.5 (+1.8%)
31.7 (+13.2%%
CARL DMCQuadruped
348.7
388.4 (+11,4%)
394.0 (+13.0%)
CARL LunarLanndrr
206.4
213.0 (+3.2%)
213.1 (+3.2%)
Table 3: Runtimes for Parseval regularization and baseline methods in minutes. Adding Parseval regularization only leads a mild computational cost over the standard agent.
31


Reviews:
[{"review_id": "Xeu0MnTArZ", "reviewer": "Reviewer_f6b5", "summary": "This work studies the problems of plasticity loss in continual reinforcement learning. Parseval regularization is proposed as a solution to plasticity loss. Parseval regularization encourages the weight matrices in all layers to remain orthogonal, which ensures that useful learning properties are preserved. Empirical evaluation is performed in various RL environments, and it is shown that parseval regularization outperforms many existing methods.", "questions": "1. What is the task used in Figure 5? The task should be specified in the figure caption of all figures.\n2. There are some typos. On line 272, the \"... between them to be 0 ...\", 0 should be 90. On line 71, \"tha\" -> \"that\".\n3. The title of the paper is misleading. \"Building network architectures ..\". There is nothing in the paper about network architectures. The paper is about a learning algorithm. I suggest the authors reconsider the title of the paper. Just \"Continual RL with Parseval Regularization\" might be good.", "limitations": "The authors need to discuss the high computational complexity of their method in the last section of the main paper.\n\n---------------------------------------------\nUPDATE: The authors satisfactorily address the concerns about computational complexity. I have raised my score to reflect that. This paper is a good contribution to the community and should be accepted to NeurIPS.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written. The proposed method is evaluated in a wide range of environments, and the results are statistically significant. The proposed solution is well-justified, found to be useful in many cases, and it is easy to use.", "weaknesses": "The paper has one major problem:\n\nComputational complexity. The computational complexity of the proposed method seems too large. It is O($l*d^3$) , where $d$ is the width of the network, and $l$ is the number of layers in the network. This is much larger than that of both forward and backward passes, which is O($l  * d^2$). Given the large complexity, what is the run time of the proposed method? The authors should report the run times of all the algorithms for at least one experiment, ideally the one presented in Figure 1. If the run-time of the proposed algorithm is too long, the authors may need to come up with a variation with smaller computational complexity. One solution could be to calculate the Parseval loss after every $d$ updates.\n\nI will update my final rating for the paper once the computational complexity and run-time of parseval regularization are reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the clear feedback and for recognizing the qualities of the paper. \n\nAs the main concern was centered around the computational complexity of Parseval regularization, we would like to point the reviewer to the shared rebuttal statement where we have discussed this in-depth. \nWe hope this response has sufficiently addressed this issue and we are happy to answer any additional questions.\n\nTo clarify, the task in Fig.5 consists of the MetaWorld suite of environments described in the paragraph starting on line 181. We will fix the caption.\n\nWe will also modify the title as suggested and agree it is more suitable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V5ChC8Sw76", "reviewer": "Reviewer_yTUz", "summary": "The paper addresses challenges in continual reinforcement learning settings by introducing an additional term, called Parseval regularization. This regularization ensures that the weight update direction remains somewhat orthogonal to the current weight, thereby preserving beneficial optimization properties. Empirical results with ablation studies are presented on Gridworld, CARL, and MetaWorld tasks, demonstrating the effectiveness of the proposed method.", "questions": "see above", "limitations": "see above", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Important Topic: The paper targets a significant and timely topic in the field of continual reinforcement learning.\n2. Ablation Study: The inclusion of comprehensive ablation studies provides insights into the method's components, such as the importance of regularizing weight angles versus weight norms.", "weaknesses": "I have identified several primary drawbacks of this work. Overall, while the paper presents promising results, addressing these points would provide a more comprehensive understanding of its contributions and limitations.\n\n1. Additional Memory/Computation Cost: The introduced regularization method adds extra memory and computation costs. In continual learning, which is often required for large-scale tasks, such additional costs can be significant. It is critical for the authors to report a detailed comparison of the computational costs between their method and the baselines to fully understand the trade-offs involved, including metrics such as memory usage and training time.\n\n2. Missing Related Works: The paper overlooks two highly relevant works:\n\n1). “Superposition of Many Models into One” by Brian Cheung et al.: Although this work is not in the RL domain, its underlying concept of model superposition is highly similar and could be extended to RL settings. The authors should discuss the relevance of this work and provide empirical comparisons to demonstrate the advantages or limitations of their approach in comparison.\n\n2). “Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation” by Qingfeng Lan et al.: This work addresses the continual learning problem in RL, albeit in a single-task context. However, its methods can be extended to multi-task settings. Moreover, one possible effect of Parseval regularization is to mitigate strong disturbances to the current model’s outputs, making knowledge consolidation a reasonable baseline for comparison. The authors should discuss this work and consider it in their empirical evaluations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the specific and concise points. \n\nFor the first point concerning the computational complexity, we have discussed this in detail in the shared rebuttal statement and would direct the reviewer’s attention there. \n\nConcerning the memory requirement, Parseval regularization only requires at most $d^2$ additional memory coming from the computation of the Frobenius norm of the product of parameter matrices, where $d$ is the width of a dense layer. Note that we compute this norm sequentially per-layer so we only require a single matrix of memory overall.  In our experiments, since $d=64$, this amounts to $64^2=4096$ additional float32 numbers, equivalent to approximately 16 kb of memory, a negligible amount. \n\n\nConcerning the references:\n- _“Superposition of many models into one”_ by Cheung et al.\n\nThis superposition technique is interesting although it addresses a different aspect of continual learning: catastrophic forgetting. In contrast, our paper focuses on the issues of plasticity and improving learning on new tasks. Additionally, the superposition technique requires knowledge of the task identity to change the \"context\" of the superposition whereas we are working in the task-agnostic setting, where the agent does not receive a signal when tasks change or a label for the current task. It must learn new tasks as they come using only  the base observations and rewards. As such, we could not directly apply the superposition algorithm as it is unclear how to extend the method to the task-agnostic setting, although this could be a fruitful research direction. As a side note, it is interesting to see that the superposition technique also makes use of orthogonal matrices albeit in a completely different manner than Parseval regularization.\n\n- _“Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation”_ by Lan et al.\n\nThis paper also focuses on the problem of catastrophic forgetting rather than plasticity in continual learning. Inspecting the proposed algorithm (MeDQN) closer, it is tailored to DQN or could be adapted to other algorithms which utilize a target network and off-policy updates. It's main goal is to eliminate the need for maintaining a large replay buffer through a sampling technique and an auxiliary loss to matching the target network. Since we used PPO in our experiments, which does not have a target network and is on-policy, it would not be possible to apply their algorithm. Exploring the interaction between Parseval regularization and other algorithms such as MeDQN would be an interesting avenue for future work.\n\nThank you for pointing us to the references, we will mention these works in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gbaIEJBDtp", "reviewer": "Reviewer_g7ai", "summary": "The paper addresses the challenge of selecting and designing embedding methods by proposing a unified framework that treats these methods as RL problems. This framework encompasses various embedding techniques, including VAEs, UMAP, and t-SNE, providing insights into their relationships and enabling the creation of hybrid methods. The authors illustrate how the ELBO approximation in VAEs relates to the exploration-exploitation trade-off in RL, and they demonstrate the framework's flexibility in designing novel methods and extending existing ones using RL techniques. They also present several new hybrid methods, such as a variational UMAP and a UMAP/t-SNE hybrid, which show state-of-the-art performance across multiple datasets. Finally, they offer a Python package implementing this framework for practical use.", "questions": "How are the parameters for Parseval regularization chosen and tuned? Is there a standard procedure or does it require extensive experimentation?\n\nHow does Parseval regularization perform on more complex and high-dimensional tasks beyond the gridworld, CARL, and MetaWorld environments used in the experiments?\n\nWhat is the computational cost associated with applying Parseval regularization, and how does it compare to the computational requirements of other regularization techniques?\n\nCan the benefits of Parseval regularization be generalized to other types of neural networks or machine learning models beyond those used in reinforcement learning?\n\n: How does Parseval regularization compare to a broader range of baseline methods, including more recent state-of-the-art continual learning techniques?\n\nCan the theoretical foundations of Parseval regularization be expanded upon to provide stronger guarantees or deeper insights into why it works effectively in continual RL settings?\n\nHow do the different components of Parseval regularization (regularizing row norms vs. angles between weight vectors) interact with each other, and is there an optimal balance between these components?\n\nHow does Parseval regularization perform across different network architectures and activation functions? Are there specific types of networks where it is particularly beneficial or less effective?\n\nHow does Parseval regularization specifically influence policy entropy and optimization dynamics? Are there scenarios where it might lead to suboptimal performance due to these influences?\n\nWhat are the practical implications of using Parseval regularization in real-world continual learning applications? Are there specific domains or use cases where it shows the most promise?", "limitations": "The paper primarily demonstrates its methods on relatively simple environments such as gridworld, CARL, and MetaWorld. It remains unclear how well Parseval regularization scales to more complex, high-dimensional tasks or real-world applications.\n\nImplementing Parseval regularization may introduce additional computational costs, which could be significant, especially for larger networks or more complex tasks. The paper does not thoroughly discuss these potential overheads or provide a cost-benefit analysis.\n\nThe paper compares Parseval regularization with a few alternative methods, but a more extensive comparison with a broader range of state-of-the-art techniques in continual reinforcement learning could provide a more comprehensive evaluation.\n\nThe results presented are specific to the environments and tasks used in the study. There is limited evidence to suggest that the benefits of Parseval regularization can be generalized to other types of neural networks, machine learning models, or more varied and realistic continual learning scenarios.\n\nWhile the empirical results are strong, the theoretical explanation for why Parseval regularization works is not deeply explored. A more rigorous theoretical analysis could strengthen the paper’s contributions and provide better insights into the underlying mechanisms.\n\n The paper does not extensively address how sensitive Parseval regularization is to the choice of hyperparameters or provide guidelines for selecting these parameters in practice.\n\nAlthough the paper touches on different activation functions and network widths, it does not explore a wide enough variety to fully understand how Parseval regularization interacts with different network components and architectures.\n\nThe paper does not provide extensive analysis on the long-term stability and performance of reinforcement learning agents using Parseval regularization in highly dynamic and nonstationary environments.\n\n There is a lack of discussion on the practical implications and potential challenges of implementing Parseval regularization in real-world applications, which may differ significantly from controlled experimental settings.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Introducing Parseval regularization as a novel technique to address the challenges of continual reinforcement learning is a significant contribution. By maintaining orthogonal weight matrices, the method preserves optimization properties crucial for training neural networks on sequential tasks.\n\nThe paper provides robust empirical evidence of the effectiveness of Parseval regularization across a variety of RL tasks, including gridworld, CARL, and MetaWorld. This empirical validation enhances the credibility of the proposed method.\n\nThe paper includes thorough ablation studies to dissect the impact of Parseval regularization. By isolating and analyzing different components of the regularization technique, such as the regularization of row norms and angles between weight vectors, the authors provide insights into why and how the method improves training performance.\n\nIt compares Parseval regularization with alternative algorithms like layer norm and shrink-and-perturb, demonstrating its superiority in certain contexts. This comparative analysis strengthens the paper's claims about the efficacy of Parseval regularization.\n\nThe exploration of how Parseval regularization interacts with different activation functions, network widths, and initialization scales suggests its versatility and potential for application across various network architectures and RL settings.\n\nGrounding the approach in theoretical concepts such as dynamical isometry and orthogonal initialization adds depth to the paper's theoretical framework, providing a clear rationale for the effectiveness of Parseval regularization.", "weaknesses": "Parseval regularization, while theoretically sound, may add significant complexity to the implementation of neural networks. Practitioners might find it challenging to integrate and tune the regularization parameters in practice.\n\n The paper primarily demonstrates the benefits of Parseval regularization on relatively contained environments like gridworld, CARL, and MetaWorld. It is unclear how well the method scales to more complex, high-dimensional tasks or real-world applications.\n\n Although the paper compares Parseval regularization with a few alternative algorithms, it might benefit from a broader range of baseline comparisons. Including more state-of-the-art methods could provide a more comprehensive evaluation of its effectiveness.\n\n Regularizing weight matrices to maintain orthogonality could introduce computational overhead. The paper does not extensively discuss the trade-offs between performance gains and the computational cost of implementing Parseval regularization.\n\nWhile the empirical results are strong, the theoretical analysis might be lacking in depth. A more rigorous exploration of the underlying mechanisms and theoretical guarantees could strengthen the paper’s contributions.\n\nThe findings are primarily demonstrated in controlled experimental settings. Additional experiments in more varied and realistic environments would help confirm the generalizability and robustness of the proposed approach.\n\nAlthough the paper touches on different activation functions, it might not explore a wide enough variety to fully understand how", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their comprehensive assessment, also including positive aspects. \n\nFor the concern about computational complexity, we point the reviewer to the shared rebuttal statement, where we discuss this in detail. We address the other points individually below:\n\n- _“How are the parameters for Parseval regularization chosen and tuned? Is there a standard procedure or does it require extensive experimentation?”_ \n\nTo tune the regularization coefficient, we do a coarse sweep of 4 values: $10^{-i}$ for $i \\in (2,3,4,5)$, (as outlined in Appendix C.5). We find that Parseval regularization is fairly robust to the strength of the regularization coefficient and a setting of $10^{-3}$ or $10^{-4}$ works best for the environments we used. \n\n- _“While the empirical results are strong, the theoretical analysis might be lacking in depth. A more rigorous exploration of the underlying mechanisms and theoretical guarantees could strengthen the paper’s contributions.”_\n\nWe agree theoretical results would be of great interest. Unfortunately, given the current state of deep learning theory it is difficult to give rigorous proofs of such optimization benefits for practical neural networks. One potential direction would be to consider the simplified setting of deep linear networks, where orthogonal initialization has been shown to scale better with respect to depth in a supervised learning setting on a single task [1]. Orthogonality of weights is also preserved by following the gradient flow in linear neural networks. Since we are considering the continual RL setting with nonlinear networks, this would add substantial complications to the problem setting and would likely require many new insights to develop the theory, making it more suitable as a future work.\n\n[1] “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks” Saxe et al.\n\n- _“...It is unclear how well the method scales to more complex, high-dimensional tasks or real-world applications.”_\n\nWe agree it would be interesting for future work to consider more challenging tasks and real-world applications. In this paper, we have demonstrated the utility of Parseval regularization in a variety of environments using sequences of standard benchmark tasks. These results provide evidence for its effectiveness in a broader range of settings and the simplicity of the approach makes it easy to incorporate into RL agents. \nFinally, since computational costs for running continual RL experiments are already fairly high due to having to train the agent on multiple tasks in a sequence, it is overly costly to run experiments where learning a single task may take a long time.\n\n- _“Although the paper compares Parseval regularization with a few alternative algorithms, it might benefit from a broader range of baseline comparisons. Including more state-of-the-art methods could provide a more comprehensive evaluation of its effectiveness.\"_\n\nWe compare to many baselines from the plasticity loss literature covering all the major kinds of algorithms including Shrink-and-Perturb (injecting randomness and weight decay), layer norm (normalization to improve curvature of loss surface), concatenated ReLU (dealing with dead units) and regenerative regularization (regularization towards the initialization). \nWe are open to considering other algorithms we may have missed.\n\n\n\n\n- _”How do the different components of Parseval regularization (regularizing row norms vs. angles between weight vectors) interact with each other, and is there an optimal balance between these components?”_\n\nParseval regularization encourages parameter matrices to be orthogonal, which improves the gradient propagation properties (avoiding vanishing and exploding gradients). As such, from a theoretical standpoint, it would make sense to keep the two parts (regularizing row norms and angles) as a single objective. From our ablation studies (see Fig. 6), we saw that both components contribute to the success of Parseval regularization.\n\n- _”How does Parseval regularization perform across different network architectures and activation functions? Are there specific types of networks where it is particularly beneficial or less effective?”_\n\nIn Fig. 5, we can see that Parseval regularization works well with a variety of activation functions although it seems to make the largest difference for the tanh function (which also performs best overall). \n\n- _”How does Parseval regularization specifically influence policy entropy and optimization dynamics? Are there scenarios where it might lead to suboptimal performance due to these influences?”_\n\nIn Appendix A.3, we reported results on the entropy of the policy during training for Parseval regularization and other methods. We saw that the entropy tended to be higher for Parseval regularization. To investigate this further, we experimented with adding different levels of entropy regularization but did not find a systematic link between the entropy regularization strength and the performance. Based on this, we hypothesize that higher entropy may be a byproduct of better performance but entropy does not have a causal link to performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank every reviewer for their time and valuable feedback. 

First, we would like to highlight some positive qualities the reviewers have identified, including: 
- The simplicity and effectiveness of the algorithm ( “The proposed solution is well-justified, found to be useful in many cases, and it is easy to use.” Reviewer f6b5)
- The novelty of the method ( “Introducing Parseval regularization as a novel technique to address the challenges of continual reinforcement learning is a significant contribution” Reviewer g7ai) 
- The comprehensive experiments ( “The inclusion of comprehensive ablation studies provides insights into the method's components,...” Reviewer yTUz).

We are happy to see the reviewers have appreciated the paper’s strengths and hope the following responses can address any remaining concerns preventing a higher score from being given. 


A common concern among the reviewers is the computational complexity of Parseval regularization.
A detailed runtime analysis shows the improvements of Parseval regularization come at a mild computational cost, ranging from 1.8% to 11.4% longer runtimes over the vanilla agent depending on the environment.

More specifically, we report the runtimes for our experiments with Parseval regularization, without it, and for Shrink-and-Perturb as an alternative baseline algorithm. 
Average total runtimes are reported in minutes with the additional cost over no Parseval regularization reported in parentheses.

| Environment            | No Parseval | With Parseval    | Shrink-and-Perturb  |
|------------------------|-------------|------------------|---------------------|
| Metaworld sequence     | 607.5       | 634.7 (+4.5%)    | 658.3 (+8.4%)       |
| Gridworld sequence     | 28.0        | 28.5 (+1.8%)     | 31.7 (+13.2%)       |
| CARL DMCQuadruped      | 348.7       | 388.4 (+11.4%)   | 394.0 (+13.0%)      |
| CARL LunarLander       | 206.4       | 213.0 (+3.2%)    | 213.1 (+3.2%)       |



From the table above, we can see that the additional cost for runtime is small (less than 12%) and Parseval regularization is more computationally efficient than Shrink-and-Perturb, which could be due to the fact that SnP requires generating a large number of random variables to perturb every parameter. 

From a theoretical standpoint, we clarify the computational complexity of Parseval regularization and contrast it to the complexity of a forward pass. As Reviewer f6b5 correctly points out, Parseval regularization requires $O(d^3)$ operations for a single dense layer of width $d$ (and $d$ inputs). On the other hand, a forward pass through the same dense layer will cost $O(nd^2)$ operations where $n$ is the size of the minibatch. 

As such, the relative cost of adding Parseval regularization depends on the ratio of $d$ and $n$. In our experiments, the width is set to $d=64$ with minibatches of size $n=64$. In this case, the cost of computing the Parseval regularizer and a forward pass would be roughly equal. Since a backward pass takes about twice the computation of a forward pass, we expect Parseval regularization to require approximately 33% more computation compared to the base agent alone. 

The additional runtime observed in practice can be significantly smaller than the theoretical value of 33% since a substantial part of the total runtime comes from interacting with the environment to collect data, both doing inference to generate actions and for the simulator to step forward. These costs are unaffected by Parseval regularization.

Next, we inspect existing PPO implementations for other environments from popular codebases [1,2,3,4] to compare minibatch sizes (n), the widths of dense layers in the network (d) and the ratio $n / d$. Larger ratios would indicate a smaller relative cost for adding Parseval regularization.


| Environment                   | n    | d    | ratio |
|-------------------------------|------|------|-------|
| Classic control [1]           | 128  | 64   | 2     |
| DMC/MuJoCo [1]                | 64   | 64   | 1     |
| Atari [1]                     | 256  | 512  | 0.5   |
| Procgen [1]                   | 2048 | 256  | 8     |
| MetaWorld MT-1/MT-10/MT-50 [2]| 1024 | 64   | 16    |
| PyBullet/MuJoCo [3]           | 64   | 64   | 1     |
| Minigrid [4]                  | 256  | 64   | 4     |
| MinAtar [4]                   | 128  | 64   | 2     |


[1] CleanRL repo: https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl 

[2] Garage documentation: https://garage.readthedocs.io/en/v2000.13.1/user/algo_mtppo.html

[3] StableBaselines3: https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html

[4] PureJaxRL repo: https://github.com/luchris429/purejaxrl

We can see that the minibatch size is usually comparable to the width of the network and, in many cases, minibatch sizes can be considerably larger. As such, we can expect Parseval regularization to result in only a modest increase in computation.

Moreover, if we were to apply Parseval regularization to convolutional layers, then the relative cost would be even smaller as the number of parameters for that layer would be small: A convolutional kernel only has $ ck^2$ parameters due to parameter-sharing, where $c$ is the number of channels and $k$ is the kernel width. For convolutional layers, the additional computational cost of Parseval regularization is much smaller compared to the cost of a forward pass. 

In summary, our reported runtimes indicate that Parseval regularization only incurs a modest additional cost compared to the vanilla agent and the theoretical analysis suggests that we can expect similar costs for other network architectures and environments.  We will add an analysis of the runtimes to the paper as well as discuss these limitations more thoroughly. 

We are happy to address any remaining concerns and provide clarifications.

Meta-review: This paper provides a timely study on the idea of using orthogonal regularization for retaining plasticity. The experimental results on the GridWorld, Meta-World, and CARL seem well-isolated and clearly ablate this regularization. All of the reviewers raised concerns about the computational complexity, but the authors show that in non-trivial experimental settings the overhead is not much (1-12% slower than baselines without the regularization)
Decision: Accept (poster)