Deep Reinforcement Learning with Plasticity Injection
Authors: Evgenii Nikishin
Junhyuk Oh
Georg Ostrovski
Clare Lyle
Razvan Pascanu
Will Dabney
Andre Barreto
Venue: NeurIPS 2023 spotlight (2023)
Keywords: deep reinforcement learning
continual learning
loss of plasticity
TL;DR: We present plasticity injection, an intervention that can be used for careful analysis of the plasticity loss phenomenon in deep RL and dynamically growing a network for computational efficiency.

Abstract: A growing body of evidence suggests that neural networks employed in deep reinforcement learning (RL) gradually lose their plasticity, the ability to learn from new data; however, the analysis and mitigation of this phenomenon is hampered by the complex relationship between plasticity, exploration, and performance in RL. This paper introduces plasticity injection, a minimalistic intervention that increases the network plasticity without changing the number of trainable parameters or biasing the predictions. The applications of this intervention are two-fold: first, as a diagnostic tool — if injection increases the performance, we may conclude that an agent's network was losing its plasticity. This tool allows us to identify a subset of Atari environments where the lack of plasticity causes performance plateaus, motivating future studies on understanding and combating plasticity loss. Second, plasticity injection can be used to improve the computational efficiency of RL training if the agent has to re-learn from scratch due to exhausted plasticity or by growing the agent's network dynamically without compromising performance. The results on Atari show that plasticity injection attains stronger performance compared to alternative methods while being computationally efficient.

Deep Reinforcement Learning with Plasticity Injection
Evgenii Nikishin"
Junhyuk Oh Georg Ostrovski Clare Lyle
Razvan Pascanu
Will Dabney André Barreto DeepMind
Abstract
A growing body of evidence suggests that neural networks employed in deep reinforcement learning (RL) gradually lose their plasticity, the ability learn from new data; however, the analysis and mitigation of this phenomenon is hampered by the complex relationship between plasticity, exploration, and performance in RL. This paper introduces plasticity injection, a minimalistic intervention that increases the network plasticity without changing the number of trainable parameters biasing the predictions. The applications of this intervention are two-fold: first, may conclude
diagnostic tool - if injection increases the performance, that an agent's network was losing its plasticity. This tool allows us to identify subset of Atari environments where the lack of plasticity causes performance plateaus, motivating future studies on understanding and combating plasticity loss. Second, plasticity injection can be used to improve the computational efficiency of RL training if the agent has to re-learn from scratch due to exhausted plasticity or by growing the agent's network dynamically without compromising performance. The results on Atari show that plasticity injection attains stronger performance compared to alternative methods while being computationally efficient.
Introduction
"You cannot teach an old dog new tricks" an old proverb says. While the common wisdom is not necessarily a source absolute truth, neuroscientists recognized long time ago that biological agents indeed gradually lose adaptability with age [Livingston] [966). This phenomenon is referred to loss
of plasticity in brains [Nelson] 1999 Mateos- and Rodríguez-Morest 2019] and happens for multiple reasons, including natural degradation of neurons and their connections [Mahncke et al., 2006 Kolb and Gibb [011]]
Since the biological causes of loss of plasticity do not apply to artificial agents, in principle there is no reason to expect that this phenomenon also happens in the context of machine learning. Surprisingly, several recent works show that reinforcement learning (RL.) agents that use neural networks may gradually lose the ability to learn from new experiences [Dohare et al. 2021 Nikishin al. 2022].
Lyle et al.; 2022
The precise mechanisms causing loss of plasticity in RL particularly challenging to study in this context because performance in RL is influenced by many factors. For example, an agent without plasticity issues may still struggle to learn if it fails to properly explore the environment [Taīga et al., [019]. Past literature focused on using and controlling proxy measures of plasticity such as the number of saturated rectified linear units [Nair and Hinton 2010] and feature rank [Kumar et al. 2021]. but it is unclear how well these measures manage to capture the underlying phenomenon (uulcerre et al. 2022].
well understood. The problem is
This paper complements past evidence about the existence of plasticity loss in deep RL and introduces plasticity injection, intervention that augments plasticity of the agent's neural network. The
Work done during the internship: currently at Mila, Université de Montréal.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

conceptual idea is simple: at any point in training, one can freeze the current network and create new one that is going be learning a change the predictions, whilst ensuring that the change is initially zero. Crucially, plasticity injection does not increase the number of trainable parameters and does not affect the network's predictions when it applied. Because of these properties, the intervention enables careful analysis of the plasticity loss phenomenon in RL while keeping other confounding factors aside.
We suggest two uses of plasticity injection, one as an analytic tool and another as a practical algorithmic technique. For analysis, we propose an experimental protocol that uses plasticity injection for diagnosing the problem of plasticity loss: for example, if an agent that was struggling to improve its behavior escapes a performance plateau after the intervention, we can conclude that the agent had been experiencing problems with its network plasticity. Using this protocol in the Arcade Learning Environment [Bellemare et al., 2013), we identify scenarios where loss of plasticity hinders the learning process. Furthermore, based on the intervention-enabee analysis, we provide recommendations for controlling the degree of plasticity loss.
We also propose to use plasticity injection as a way to improve computational efficiency of RL training in the following scenarios. First, when the agent loses its plasticity because its network turns out to be too small, plasticity injection can be dynamically used to increase the capacity of the agent without having to re-train the agent with a larger network from scratch. We empirically show that our method improves the aggregate score across 57 Atari games by 20% compared to other methods for dynamically addressing plasticity loss. Second, plasticity injection can be used to minimize computation by switching from small network to larger network in the middle of training without compromising the performance compared to using the larger network from scratch; we also empirically verify it on Atari games.
To summarize, our contributions include:
1. A minimalistic intervention called plasticity injection that increases plasticity of the agent while preserving the number of trainable parameters and not affecting its predictions;
2. Complementary evidence about the existence of the plasticity loss phenomenon in deep RL;
3. An experimental protocol for diagnosing loss of plasticity using the intervention;
4. A way to improve computational efficiency of RL training by dynamically expanding neural network used by the agent.
2 Related Work
Plasticity in Continual Learning. Discussions about plasticity of neural networks date back (at least) to a seminal paper by McCloskey and Cohen [1989] outlining the plasticity-stability dilemma, trade-off between preserving performance on previous tasks and maintaining adaptability to future ones. The continual learning community historically put a higher emphasis on the stability aspect, addressing catastrophic forgetting of past behaviors [French] 1999]. Recently, several works raised awareness of difficulties with learning on future tasks too. Ash and Adams (2020) demonstrated an instance of loss of generalization, when pre-training a network might unrecoverably damage generalization even if pre-training was done on a uniform subsample of the same dataset. [Berariu] al. (202T) deepened the study and conjectured that the phenomenon might happen because of the reduction of gradient noise when warm-starting the network. Dohare et al. [202] explicitly study the network plasticity in continual learning and demonstrate the reduced ability to minimize even the training error as the number of tasks increase. These works build an understanding of the problem by studying simplified settings that isolate different aspects of learning capabilities in continual learning, whereas our work aims at tackling the deep RL setting in its whole.
Loss of Plasticity in Deep RL. Issues with plasticity and related phenomena have been recently highlighted in deep RL under a plethora of different names. Lyle et al. [202]] show loss of capac- ity for fitting targets in online RL and Kumar al.) [2021] demonstrate related implicit under- parameterization phenomenon caused by bootstrapping with more emphasis on the offline RL case. Both of these works use the feature rank as a proxy measure for plasticity but later Gulcehre et al.) (2022) question the reliance on such measure by demonstrating weak correlation between the rank and the agent's performance, partially motivating our study that focuses directly on agent's performance to reason about plasticity. Works of Sokar et al. [2033 and Abbas al. 2023 focus on

---

Up Down
Value Prediction MSE
10
10-1
4000
8000
12000
16000
20000
Iterations
Reset Every Task
Reset Never
Figure I: Demonstration of plasticity loss in a sequence of policy evaluation tasks. The task (a policy evaluate) changes every 1000 iterations. The resel every lask setting shows that newly-initialized parameters are able to fit each task, whereas the reset never setting shows the diminishing capability fit the data when using the trained parameters from one task as the initialization for another task.
saturation of neurons over the course of training, but Lyyle et al. 2023 demonstrate that the saturation alone cannot fully characterize the plasticity loss phenomenon. Nikishin et al. [2022) discuss the primacy bias in deep RL, tendency to excessively train carly data damaging further learning progress, and propose to periodically reset part of the network to address the issue while relying on
the replay buffer knowledge transfer mechanism. Earlier, [g] al. [2021] had observed that deep RL agents can lose the ability to generalize due to non-stationarity and proposed to use distillation as mitigation mechanism. Plasticity injection closely relates to these approaches by leveraging newly initialized weights, but does not require re-training and directly continues learning.
Architectures. The works above mainly discuss algorithmic aspects with less focus on the network architecture, although also an important component the agent's design [Mirzadeh et al. 2022]. The closest work in this space is about progressive networks [Rusu al., 2016) that considers setting with multiple environments and adds a new network with cross-connections to the layers previous networks. A network after plasticity injection be viewed as simplified version of the architecture with motivation of increasing plasticity within single task without affecting agent's 's predictions. A line of work the mixture of experts [Shazeer et al. 2017 and modular networks [Andreas et al.) 2016 is also related, but the focus these papers typically is compositionality or handling multiple modalities. The idea of growing network layers neurons has also been investigated [Fahlman and Lebiere 1989 Chen et al. 2015); plasticity injection belongs to a family of these methods up the differences the growing strategy and in explicitly controlling for the number of trainable parameters. the context of language modeling, Hu et al. [2022] explored a similar idea of freezing a pre-trained model and fine-tuning a low-rank addition to the weight matrices on downstream task. Lastly, plasticity injection can be conceptually viewed as an instance of residual learning [He et al. 2016 and boosting [Schapire 1990..
3 An Illustration of Plasticity Loss
Plasticity of a neural network broadly defined as the ability to learn from new experiences. To provide intuition on how this ability can decrease over time, we present didactic example before investigating the case with deep RL. Figure shows the mean-squared eттor (MSE) on a sequence of supervised policy evaluation problems derived from the Up Down Atari environment. We first trained an agent on this environment for 200M frames and stored the policies occurring at every 10M frames. Then, for each stored policy, we sampled states from the corresponding stationary distribution and computed Monte- Monte-Carlo estimates of the value function for each state, resulting training set composed of states and their values. We then trained network to solve the resulting sequence of prediction problems. This sequence of related prediction problems differing in the input and target distribution aims to reproduce the scenario faced by an online RI. agent (Dabney et al.) 2021). The curve labeled "reset never" corresponds to starting each prediction problem using the

---

final parameters from the previous one, while "reset every task" corresponds to randomly initializing the network parameters at every prediction problem.
The conventional wisdom about transfer learning suggests that, if two tasks are related, pre-training on the first might accelerate learning on the second (Pan and Yang 2009]. Here observe the opposite trend: it takes longer and longer for the network to decrease training error on the subsequent policy evaluation problems if its parameters are not re-initialized. This example gives a simple
demonstration of how plasticity loss can occur; we refer to the work by Dohare et al. [202] for an in-depth study of the phenomenon in the continual setting.
After building intuition about loss of plasticity, we turn our attention to its analysis in deep RL. The key distinctive feature of RL is the presence of exploration confounder: in contrast to the continual setting with a fixed sequence of datasets, an RI. agent influences the future data it learns from. Thus, failure of RL system can be attributed not only to loss of plasticity but also to inability to explore. The next section presents a strategy to increase plasticity of an agent that addresses this difficulty with the analysis.
4 Plasticity Injection
Before describing the experimental design in detail, we list the motivating desiderata:
Unaffected predictions: the agent's 's predictions should stay the same after the intervention to avoid abrupt changes. This criterion allows isolating confounding factors related to exploration;
Preserving the trainable parameter count: the intervention should not affect the number of trainable parameters to minimize confounding factors related to representational capacity.
We now present the proposed intervention to increase plasticity of an RL agent. First, let denote the neural network approximator employed by the agent (for example, used for action-value prediction) as ho (zr), where indicates the parameters. At some point in training, where the network might have started losing plasticity, we are going to freeze the parameters @ and introduce new set of parameters O' sampled from random initialization. The key idea is to keep two copies of 0', which we denote by 0g and 02: while 0f are free parameters used to learn a residual to the old network outputs, 0g remains frozen throughout. The agent's predictions after plasticity injection will be calculated using the following expression:
ho(r)) + hog (ar)
11g (x)
frozen
trained
Since initially 0f = 0z. immediately after plasticity injection the predictions of the neural network remain unaltered. As learning progresses, 0f deviates from 02 and hg(#r) - haa (1r) serves bias term for predictions.
Note that if we apply plasticity injection to all parameters of the network, the new network will have to re-learn the representations encoded in ho(-) from scratch. Thus, we apply our intervention to only subset of the parameters and explain the idea further with slight abuse of notation. We
schematically split the network into an encoder (f.), that denotes a mapping induced by first l layers of the network, and head ha(-) where 9 now refers to parameters of the remaining layers of the network. After this relabelling, we can apply the intervention to ha(-) as outlined above. Appendix B later presents ablation of sharing the encoder.
Figure 2 illustrates the strategy to apply plasticity injection. Note that gradients from the frozen heads affect the encoder too, i.c. we do not stop the gradient propagation from any of the components of the output. It is worth noting that the proposed intervention increases the total number of parameters of the network (but keeps the same number of trainable parameters), which in turn may result in an increase of training time. However, we later discuss in Section 5.3 how plasticity injection can save computational resources.
The idea of learning with newly-initialized last layers has been explored by Nikishin et al. 2022 who suggested resetting the corresponding parameters of the network at fixed intervals and used the replay buffer [Lin] 1992] to re-learn after resets. Their experimental evidence supports the hypothesis that resets mitigate plasticity loss. However, resetting parameters of the network abruptly changes

---

(1(x))
hg(x) (x) + hgy (x) -
hgg (x)
Plasticity Injection
Figure 2: An illustration of the architecture before and after plasticity injection. Before the interven- tion, the network is schematically separated into an encoder d6.) and a head he(-), both parts learning. After plasticity injection, we freeze the parameters of the head (we use red to indicate parameters that parameters 0:: one frozen and one unfrozen. The output of the agent is obtained by first passing the input æ to the encoder o(-). next passing (zz) to all three heads, and finally combining the heads' outputs according to Expression (T).
updated in the illustration) and create two copies of randomly initialized
its predictions, which results temporary decrease in performance and induces an exploration effect. From an analysis perspective, these abrupt changes make it more difficult to isolate the effect of additional plasticity on the agent's performance. From a practical perspective, plasticity injection does not rely the buffer; Section 5.3 demonstrates how this difference can be critical.
Experiments
This section presents results for two main applications of plasticity injection: as tool for diagnosing plasticity loss and as way to dynamically grow the network to efficiently use computations.
5.1 Experimental Setup
The baseline agent is Double DQN (Van Hasselt et al. 2016) learning for 200M interactions on standard set of 57 Atari games from the Arcade Learning Environment benchmark (Bellemare et 2013). The choice of Double DQN is motivated by the relative robustness and stronger performance with double Q-learning (Van Hasselt 2010 compared the vanilla DQN agent [Mnih et al. 2015 as well as simplicity compared to later DQN-based agents such as Rainbow (Hessel et al. 2018].
The majority of the experiments use single plasticity injection after 50M frames; otherwise, explicitly specify the number and timesteps of injections. Appendix B discusses ablations on the design choices when using plasticity injection. A convolutional neural network employed by the Double DQN agent consists of 5 layers. The encoder corresponds to the first three of them (hence = 3), while the head refers the last two. Since DQN-based agents employ a target copy of the network parameters, we perform the same interventions on them.
For reliable evaluation of the performance across environments, adopt the protocol of Agarwal et al. 2021] with a focus on the interquartile mean (IQM). All experiments use random seeds.
5.2 Plasticity Injection as Diagnostic Tool
Consider the task of improving a deep RL system when an agent performs suboptimally. Practitioners know how non-trivial is the process of pinpointing exact reasons why agent might be struggling to improve the behavior. One of the reasons, as we discussed, can be loss of network plasticity throughout training.
We view the proposed intervention as a tool that can provide insight when analyzing deep RL. systems. The procedure for using it is as follows: when agent is on performance plateau has slower
5

---

Space invaders
Phoenix
Assnul
Robslank
5000
3000
Ensose
9000
15000
2000
2000
10000
30
1000
5000
1000
Laxira
2 10
25 50 75
150 175 200
25
50 75 100 125 150 200
25 50 75 100
155 175 200
25 50
100 125 130 200
Environment framess (miilions
Eintrommnttfrmm 0mi kons))
Emy Eww.lomment fromes (nill Folis
Brifronment fraimss (millions)
Figure 3: A demonstration of diverse effects from plasticity injection applied to the Double DQN agent after 25M, 50M, and 100M frames on a selection of Atari games comprising two examples where the intervention improves the performance and two examples where it does not. The baseline in Space invaders and Phoenix demonstrates the diminishing performance improvements and the performance plateau respectively, whilst the agent after the injection is capable of achieving higher returns. The stalled performance in Assault t is due to exploration challenges (see Appendix [ for further details): adding plasticity could not alleviate them. If the agent does not show signs of the diminishing ability to learn, like in Robotank, the injection would not lead to improved performance. Varying the injection timestep allows identifying the moment when plasticity loss occurs. Results for all 57 env environments are available in Figure 7
learning progress, take saved copy of the agent, perform plasticity injection, and compare the training curves with and without the intervention. This way we answer a counterfactual question: what could have been the agent's 's performance if the network had more plasticity?
Figure Bl gives set of example behaviors after following the procedure. In Space invaders, the baseline agent keeps learning but the post-injection agent improves at a faster rate towards the end of learning; we might interpret the observation as an indication of decreasing network plasticity over the course of training. In Phoenix, we see a completely stalled performance and the intervention allows doubling the final returns; such an observation point at possible catastrophic loss of plasticity, where additional interactions do not translate to improved behavior. In Assault, on the other hand, the agent has plateaued but the injection does not make a difference. Further inspection revealed that around a score of 2800, the environment transitions to a new regime where an agent needs to start using an action that was not relevant before (see Appendix D for a visualization). This observation suggests that performance stagnation is related to exploration. In Robotank, the learning progress shows no signs of pathologies, giving evidence that the agent does not have problems with plasticity.
Plasticity injection can also demonstrate when loss of plasticity occurs. The post-intervention performance in Space invaders does not differ for varying injection timestep, suggesting that the agent might not start experiencing consequences of the lost plasticity until around 100M frames. On the other hand, in Phoenix, plasticity injection improves the performance earlier, implying that the agent lost its plasticity around 25M frames. Varying the injection timestep in Assault t and Robotank does not change the performance significantly, supporting our previous conclusion about these games.
Figure 4 summarizes when and to which extent the Double DQN agent benefits from plasticity injection across 57 Atari games. The observations about improvements from injection complement evidence of the existence of plasticity loss in deep RL [Kumar et al., 2021 Lyle et al. 2022 Nikishin al. 2022]. We note that the argument here is nuanced: since the notion of plasticity defined broadly and is challenging to measure, il is our best interpretation that the post-intervention agent can learn further because it addressed plasticity issues. But because of an experimental design that strived to be careful, we believe that it is the most likely explanation.
What should we do after using the tool and observing loss of plasticity? Dohare et al. 202 Nikishin ct al. 2022). Gogianu et al. [202T) provide evidence that plasticity loss is strongly affected by the learning rate, the replay ratio (the number of gradient steps per an environment step), the network sizd"] and normalizations (such as spectral norm [Miyato et al. 2018)). We measure the sensitivity of the aggregate improvements of the final score from plasticity injection at 50M with respect to these choices of the agent specification. Results in Figure 5 are consistent with observations from previous
To make network two times larger, multiply the width of all hidden layers by V2.

---

20
10
-10
Assaut
Double Doubledunkk dunk
Robotank
Demon attack
Private eye
Hers
revends
Portig
pitrail
Solarss
Tenma
Venture
Battis zon
Centpped
Icc Ice hockey'
Bowiling
g Name this game e Obert,
Beam neer
Crazy dimber
Kung master
Atlants
Timee pilot
pacman
Boxin
Gravitar
Turankoaa
Krull
Arnidar
Skiin
Video derts
Defencer
Bank henchelse
comimand
Space invadess
Wizard aeverraa
Rlvver
down
Breaked keut
Berrzerk
Gopher
Astar Aiteeron olds
Endure
Allen
Seaques
Yars revenge Phocnix
Montezuma
Chopper
Figure 4: Percentage improvement of the average performance after adding plasticity injection across all 57 Atari games. take the maximum score among the agents with plasticity injection after 25M, 50M, and 100M steps to roughly estimate the improvement as if plasticity injection applied at a proper timestep and to demonstrate what the performance could have been plasticity loss was mitigated. Table later presents categorization of environments into buckets where the agent does and does not benefit from injection. Learning curves corresponding to each environment are available in Appendix/A A
works and suggest a recipe for controlling the degree of plasticity loss by decreasing the learning rate or the replay ratio, increasing the network size, or employing normalizations
In addition to gaining scientific insight, we now discuss how the intervention can be useful in large-scale RL.
5.3 Plasticity Injection for Computational Efficiency
Over the recent years, RL agents have been trained increasingly larger scales. For example. mastering particularly challenging environments required an equivalent of hundreds of years of human gameplay (Vinyals et al.) 2019). or obtaining a diverse set of skills required IB+ parameter
networks [Reed et al. 2022 Given the trend, computational considerations become increasingly relevant.
% vemravementt
Improvement
Improvement
40
40
30
% Improvement
20
20
20
20
10
10
0.5 RR Multiplier
1
0.5
2
0.5
1
LR Multiplier
Size Multiplier
Figure 5: Percentage improvements of the IQM scores from plasticity injection in varying regimes controlling the degree of plasticity loss. The intervention effect size monotonically increases with the replay ratio (RR) and the learning rate (LR), monotonically decreases with the size the neural network, and is smaller yet positive for agent employing spectral normalization (SN). These observations can be seen as recommendations about how to address loss plasticity.
202 and apply SN the penultimate layer: since apply injection the last two layers, issues with their plasticity might be partially alleviated SN. Given that Gogianu et al, [202]] notice that the spectral norm of other layers starts growing more and DD' Oro et all 2023 observe that the first layers benefit from partial resets, we conjecture that first layers' plasticity is still declining with SN.
"We follow the recommendation of Gogianu

---

IQM
IQM Normal Score
1.2
Width Scale SnP
1.1
Resets
Injection
1.0
Baseline
Larger Net Injection
0.9
1.04 1.12
1.20
Human Normalized Score
50
100
150 200
Wallclock Time (Hours)
Figure 6: Left: Comparison of plasticity injection to other methods that can be applied to dynamically address loss of plasticity. The difference in performance between all methods and the baseline is insignificant except for injection; agent with injection is capable of improving without having to re-train from scratch. Right: Performance over wallclock time for an agent with plasticity injection and an agents that uses network with increased width from the beginning. Because Injection switches from small network to a larger network through plasticity injection at 50M frames and uses less trainable parameters than Larger Net, it achieves similar IQM while saving equivalent of 20 hours of computational resources.
Plasticity injection can be used to improve the computational efficiency of RI. in the following ways.
Reincarnating with Plasticity Injection. Recently, Agarwal al. [2022] proposed workflow called "Reincarnating RL" which reuses computations from previously trained agents during the iterative process of agent design. For example, we trained an agent for several days or weeks and then decided to change its design (such as the network size), the workflow suggests to leverage the spent computations instead of training again from scratch. Plasticity injection can be useful from this perspective when the agent is unable to improve due to loss of plasticity and re-training from scratch expensive. To see the effectiveness of plasticity injection in such setting, we compared plasticity injection with several alternatives that address loss of plasticity dynamically during training, including Shrink-and- Pertuer (SnP) [Ash and Adams 2020), resets [Nikishin al. 2022], and naive width scaling (we describe the methods in detail in Appendix G. Figure 6 (left) shows that plasticity injection achieves a higher aggregate score across 57 Atari games compared to the alternatives. The results suggest that plasticity injection can be used to "reincarnate" agents more efficiently compared to the alternatives, without re-training from scratch.
Minimizing Computations via Dynamic Growth. Although larger networks tend to maintain plasticity longer, they require more computations or can be more challenging train [Team et al.) 2023]. We hypothesize that the full capacity of a large network may not always be necessary early in truining, even
useful to maintain plasticity later. this hypothesis true, we can save computations by starting from a smaller network and injecting plasticity during training, without compromising the final performance compared to using the large network from the beginning. To verify this hypothesis, we took network with larger layer width, matching the total number of parameters in (()), h-))), and he, (+) combined. The results in Figure (right) show that an agent with plasticity injection during training performs comparably to the alternative that uses a larger network from the start. At the same time, saves about 20 hours of wallclock time on an A100 GPU since it uses smaller network up to 50M frames and has fewer parameters that are updated after plasticity injection. These results confirm the hypothesis and suggest that plasticity injection can be used tool for minimizing computations when training RL systems at large scale.
6 Limitations
The first and foremost limitation of plasticity injection an increase in memory and training time compared to the baseline agent with standard-sized network. When using plasticity injection as a diagnostic tool, we believe the overhead is largely justified since preserving the network output makes it easier to isolate confounding factors like exploration. From the deployment viewpoint, practitioner should decide whether the increase compute and memory is justified based on how much plasticity injection improves performance. While the effect of the intervention was positive on the aggregate performance on Atari, it varied considerably across individual games: in some cases, did not change scores much, in other cases, it had significant positive effect.

---

Preserving the network outputs and keeping weights can be undesirable in case of parameter diver- gence that often occurs the deep RL experimentation (Van Hasselt et al. 2018]. Such a scenario also qualifies as loss of plasticity; addressing it without drastic tools can be challenging. Lastly, while we propose a diagnostic and mitigation tool, we do not identify causal factors driving plasticity loss in deep RI.. More research is needed here: understanding these causes could lead to avoiding plasticity loss in the first place.
7 Discussion and Conclusion
Results in this paper can serve as clear study of the plasticity loss phenomenon in deep RL and evidence that the optimization aspect in RL still leaves room for improvement. The version of plasticity injection we propose may yet not be optimal: we strived for simplicity rather than performance and view the intervention as a blueprint for future methods.
The experiments in this paper adopted the convolutional architecture from Van Hasselt et al. [016 but modern deep RL practice not rarely involves ResNets (He et al. 2016 Espeholt et al. 2018 and Transformers [Vaswani et al. 2017 Chen et al.) [021] Reed et al. 2022 we did not investigate
settings with these advanced architectures. However, the idea plasticity injection is agnostic the choice of the architecture. For example, it can be applied for residual blocks in ResNets or decoder blocks in Transformers.
An exciting avenue for future research is understanding trade-off's between architectural design decisions: RL agents typically employ networks that were originally proposed for stationary problems, but perhaps dynamically growing networks would suit the non-stationary nature of RL better.
Applications of plasticity injection focus on diagnosing RL systems and their efficiency. We compli- ment a recent opinion paper from Mannor and Tamar [2023] by arguing that if deep RL is to become technology that non-expert use, more research is needed on the process of iterating on the agent design and computational efficiency.
Although this paper attempted to understand and mitigate loss of plasticity in RL, there still remaining open questions. Can we solve the problem of plasticity loss completely? Which properties of newly initialized networks enable high plasticity? Answering these questions is key challenge for training truly intelligent agents.
Acknowledgments and Disclosure of Funding
EN thanks Tom Schaul, Greg Farquhar, John Quan, Dan Horgan, Mihaela Rosca, Angelos Filos, Diana Borsa, Luisa Zintgraf, Yash Chandak, Robert Lange, Chris Lu, David Parkes, Blanca Huergo, Rich Sutton, David Silver, Hado van Hasselt, Alexander Novikov, Julia Novikova, and especially lurii Kemaev for their help and valuable discussions. EN also thanks many other interns and the broader Deep team for the great internship experience. This work was funded by DeepMind.
We acknowledge the Python community [Van Rossum and Drake Jr 1995| Oliphant 2007 for developing the core tools that enabled this work, including JAX BBaddbry al. [018 Babuschkin ct al. 2020), Jupyter [Kluyver et al.) 2016), NumPy [Oliphant] 2006 Van Der Walt et al. 2011] SciPy Dones et al. 2014], Matplotlib
Hunter 2007], and pandas |McKinney] 2012).
References
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. arXiv preprint arXiv:2303.0753 2023.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.

---

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39-48. 2016.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in Neural Information Processing Systems, 33:3884 3394444-3894, 2020.
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacei, Jonathan Godwin, Chris Jones, Tom Hennigan, Matteo Hessel, Steven Kapturowski, Thomas Keck, lurii Kemaev, Michael King, Lena Martens, Vladimir Mikulik, Tamara Norman, John Quan, George Papamakarios, Roman Ring. Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojciech Stokowiec, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL. http:://githu com/ deepmind
Mare G Bellemare, Yavar Naddaf. Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning, pages 449-458 PMLR, 2017.
Tudor Berariu, Wojeiech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity neural networks. arXiv preprint arXiv: (ee::2106.0044 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Minn and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
httt::/ggttan www../oooggoo////eeeoo
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06811 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 5004411774, 2021.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv: /5/ 15551.05666 1.05641, 2015.
Will Dabney, André Barreto. Mark Rowland, Robert Dadashi. John Quan, Mare G Bellemare, and David Silver. The imtrovempee path: Towards better representations for reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 7160-7168, 2021.
Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.0633 2021.
Kefan Dong, Yuping Luo, Tianhe Yu, Chelsea Finn, and Tengyu Ma. On the expressivity of neural networks for deep reinforcement learning. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume I 119 of Proceedings of Machine Learning Research, pages 2627-2637. PMLR, 13-18 Jul 2020.
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G. Bellemare, and Aaron C. Courville, Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Eleventh International Conference on Learning Representations, 2023.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407-1416. PMLR, 2018
10

---

Scott Fahlman and Christian Lebiere. The cascade-correlation learning architecture. Advances in neural information processing systems, 2, 1989.
Amir Farahmand, Mohammad Ghavamzadeh, Shie Mannor, and Csaba Szepesvári. Regularized policy iteration. Advances in Neural Information Processing Systems, 21, 2008.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128-135, 135, 1999.
Florin Gogianu, Tudor Berariu, Mihaela C Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference on Machine Learning, pages 3734-3744. PMLR, 2021.
Caglar Gulcehre, Srivatsan Srinivasan, Jakub Sygnowski, Georg Ostrovski, Mehrdad Farajtabar, Matt Hoffman, Razvan Pascanu, and Arnaud Doucet. An empirical study of implicit regularization in deep offline rl. arXiv preprint arXiv:2207.02099 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen -hu,, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
John Hunter. Matplotlib: A 2d graphics environment. IEEE Annals of the History of Computing, (03):90- 95, 2007.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In International Conference on Learning Representations, 2021.
Eric Jones, Travis Oliphant, and Pearu Peterson. SciPy: Open source scientific tools for Python. 2014.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.0040 003744 2019.
Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian E Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. Jupyter Notebooks-a publishing format for reproducible computational workflows., volume 2016. 2016.
Bryan Kolb and Robbin Gibb. Brain plasticity and behaviour in the developing brain. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 20(4):265, 2011.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameterizatii inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021.
Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting. In The Eleventh International Conference on Learning Representa- fions, 2023.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint -=X.v:/509000977 2015.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3):293-321, 1992.

---

Robert B Livingston. Brain mechanisms in conditioning and learning. Technical report, 1966.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney, Understanding plasticity in neural networks. arXiv preprint arXiv:2303.01488, 2023.
Henry W Mahncke, Amy Bronstone, and Michael M Merzenich. Brain plasticity and functional losses in the aged: scientific bases for a novel intervention. Progress in brain research, 157:81-109, 2006.
Shie Mannor and Aviv Tamar. Towards deployable rl rl-what's broken with rl research and potential fix. arXiv preprint arXiv:2301.00 320, 2023.
Pedro Mateos-Aparicio and Antonio Rodríguez-Moremn The impact of studying brain plasticity. Frontiers in cellular neuroscience, 13:66, 2019.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165.
Elsevier, 1989.
Wes McKinney. Python for data analysis: Data wrangling with Pandas, NumPy, and IPython. O'Reilly Media, Inc.". 2012.
Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Architecture matters in continual learning. arXiv preprint arXiv:2202.00225. 2022.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 5 518(7540::599553 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pages 1928-1937. PMLR, 2016.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Ieml, 2010.
Charles A Nelson. Neural plasticity and human development. Current directions in psychological science, 8(2):42 45, 1999.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference Machine Learning, pages 16828-16847. PMLR, 2022.
Travis E Oliphant. A guide to NumPy, volume 1. Trelgol Publishing USA, 2006.
Travis E Oliphant. Python for scientific computing. Computing in Science & Engineering, 9(3): 10-20, 2007.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, (2110:13455 1359, 2009.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et A generalist agent. arXiv preprint arXiv:2205.06/77 2022.
Andrei A RUsU, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 0467,,,,000 2016.
12

---

Robert E Schapire. The strength of weak learnability. Machine learning, 5(2): 1990.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
Noam Shazeer, * AAaalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phe- nomenon in deep reinforcement learning. arXiv preprint arXiv:2302. /2902, 2023.
lly Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139- 1147. PMLR, 2013.
Adrien Ali Taïga, William Fedus, Marlos C Machado, Aaron Courville, and Mare G Bellemare. Benchmarking bonus-based exploration methods on the arcade learning environment. arXiv preprint arXiv: 190.000888.0 2019.
Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmiee Michael Chang, Natalie Clay, Adrian Collister, et al. Human- timesalee adaptation in open-ended task space. arXiv preprint arXiv:2301 2000:23001.0008. 2023.
Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Stefan Van Der Walt, Chris Colbert, and Gael Varoquaux. The numpy array: structure for efficient numerical computation. Computing in science & engineering, 13(2):22 30, 2011.
Hado Van Hasselt. Double q-learning. Advances in neural information processing systems, 23, 2010.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.0266 2018.
Guido Van Rossum and Fred L Drake Jr. Python tutorial, volume 620. Centrum voor Wiskunde Informatica Amsterdam, 1995.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575((782):3500-333 2019.
13

---

1d00
300
2000
Ereakoot
Crntbrds
Chapsrrcomerr
Crayy cieter
Delender
Dimeen
12500
31.00
2000
Pockure
Pork
Proriabe
Cuenerr
1300
-330
-203
1000
Obent
Kivers
haad keea
Ecastant
1500
1809
Space
Zorgumer
555
21000
410000
2000
Witrurd
13830
500
Figure 7: Performance of the Double DQN with and without plasticity injection after 25M, 50M, and 100M frames on the full Atari 57 benchmark. The potential discontinuities in the plots such as in Road runner are caused by the evaluation each 1M frames, i.e. the first moment the agent with injection contributes to the plot after learning for 1M frames.
14

---

Injection
Environments
Effect
Consistent
Alien, Asteroids, Breakout, Chopper command, Endura, Frostbite, Gopher, Phoenix, Space invaders, Surround, Wizard wor, Yars revenge (12 total)
Improvement
Minor
Amidar. Asterix, Atlantis, Bank heist, Beam rider. Berzerk, Boxing, Defender, Fishing derby, Jamesbond, Krull, Ms pacman, Road runner, Seaquest, Time pilot, Up down, Video pinball, Zaxxon (18 total)
Improvement
Negligible
Battle zone, Bowling, Centipede, Crazy climber, Double dunk, Freeway, Gravitar, Hero, Ice hockey, Kangaroo, Kung fu master, Montezuma revenge, Name this game, Pitfall, Pong, Private eye, Qbert, Riverraid, Skiing, Solaris, Star gunner, Tennis, Tutankham, Venture (24 total)
Negative
Assault, Demon attack, Robotank (3 total)
Table 1: Summary of effects from applying plasticity injection Double DQN on 57 Atari games.
A
Complete Learning Curves
Figure 7 presents the return plots over the course of Double DQN training for 200M frames on the whole set of 57 Atari games. We informally categorized environments into four buckets upon visual inspection of effects from plasticity injection in Table [T The most notable negative example is Demon attack, while on Assault and Robotank the effect is negative but minor. In the rest of the 54 games, plasticity injection either improves performance or has a negligible effect, possibly depending on the injection timestep.
B Ablations
This appendix presents an ablation analysis of the various design choices made during the study of plasticity injection. The purpose of such ablations is to build intuition on the behavior of plasticity injection under different conditions so that an RL practitioner can use it their application.
Injection Variants. The proposed modification of the network architecture is not the only one possible. In Section 4 we initially described version of plasticity injection without encoder sharing, that is. when the intervention is applied to the entire network (referred to as Injection, Whole Net create a whole new set of parameters and copy the encoder
in Figure B Another alternative parameters of the old network without sharing it (denoted as Injection, Whole Net, Copy Enc). Lastly, for all three versions, there is the possibility of not freezing the old set of parameters (weights corresponding to the third, output correction term are always going to be frozen).
Figure (left) summarizes the findings:
1. Creating a completely new encoder-head pair is the alternative with the lowest IQM scores;
2. Variants with encoder sharing or copying have comparable performance; the Injection, Whole Net, Copy Enc version has a slightly lower performance than the rest. We conjecture that might be due to the larger number of frozen parameters;
3. Unfrozen variants generally perform not worse than their frozen counterparts. The unfrozen variants introduce more trainable parameters compared to the baseline, which require more computations during learning and increase the network expressivity. Since we were interested in a careful diagnosis of plasticity loss and extra expressivity may be a confounding factor, we decided stick to the frozen version by default.
Multiple Injections. Given the improved performance from plasticity injection in the previous experiments, a natural question whether applying plasticity injection multiple times would improve performance even further. To investigate this question, we applied plasticity injection at 100M and 150M frames, in addition to 50M frames, and plotted the IQM improvements with respect to a single injection at 50M frames. As shown in Figure 8 (right), additional injections do not improve the performance over a single injection
setup with a standard network. We hypothesize that in our particular experimental setting, loss of plasticity can be largely mitigated with a single plasticity
15

---

IQM
% Improvement
Unfrozen Injection, Whole Net, Copy Enc
10
Unfrozen Injection, Whole Net
Unfrozen Injection
Injection, Whole Net, Copy Enc Injection, Whole Net Injection
1.0 1.1 1.2 1.3
0.25
0.5 1
Human Normalized Score
Size Multiplier
Figure 8: Left: Comparison between variations of plasticity injection. Whole Ner denotes injection of both the encoder o(-) and the head ha Copy Enc denotes copying the o6-) at the moment of injection without further sharing; Unfrozen denotes keeping parameters of the first term unfrozen. Relying on a new encoder leads to a lower performance; the rest of the alternatives have comparable scores. Right: Percentage improvements of the IQM score from multiple injections over a single injection for varying network sizes. Multiple injections are beneficial for smaller networks. Note that previous plots in Figure show improvements when comparing injection over injections while this plot compares multiple injections over onc.
injection. To verify this hypothesis, we applied multiple injections while varying the network size (similarly Section 5.2 to make the network 2x smaller, we divide the width of the hidden layers by v2). Figure (right) confirms that the level of improvement grows monotonically as the agent uses smaller networks. Since the results in Figure [ suggests that the degree of plasticity loss increases with smaller networks, this result indicates that multiple rounds of plasticity injection can be beneficial in situations where the agent network is too small to maintain plasticity.
No Output Correction. In the majority of the games, subtracting the initial copy of the newly intro- duced head he,()) resulted mostly similar learning curves as without the subtraction, although not always. In particular, the impact of the injection Yars Revenge is smaller without compensating for the bias. Also, we observed significant difference in high variance games (such as Berzerk and Hero). Note that removing effects the predictions from introducing the new head would be possible by modifying the initialization [Brohan et al.] 2022]. From the analysis viewpoint, we strove have clean experimental design possible and wanted to remove initialization-speeif confounders since initialization would affect network plasticity as well [Sutskever et al. 20133.. From the saving memory and computations viewpoint, it might be preferrable do plasticity injection without introducing the third network.
Optimizer. One might hypothesize that benefits from injection can be attributed to manipulations with the optimizer state. To test this hypothesis, we perform two ablations: the first resets statistics of the RMSProp optimizer [Tieleman et al. 2012 used by Double DQN after 50M steps, the second copies the optimizer state of the original head to the newly initialized head after the injection. Figure 9 (left) demonstrates that most of the effects from injection come from having additional weights rather than from interventions on the optimizer.
we presented the results for selection of environments while varying injection timestep. Figure (right) suggests that across games, changing the timestep by a
Injection Timestep. In Section 5.2
IQM
IQM
Reset Opt
Injection @ 100
Injection Copy Opt Injection Baseline
Injection @ 50 Injection @ 25 Baseline
1.04 1.12 1.20
1.04 1.12 1.20 1.28
Human Normalized Score
Human Normalized Score
Figure 9: Left: Comparison of an agent with injection, agent with injection but copied optimizer state for the newly initialized head (Injection + Copy Opr), and an agent that resets the optimizer statistics of the last two layers (Reser Opt). The results suggest that effects from interventions on the optimizer state are marginal compared to having new weights. Right: Aggregate performance for agents with varying injection timesteps. Whilst Figures 7 and 10 suggest that loss plasticity might be happening at different paces across environments, the final IQM score is relatively robust with respect the injection moment.
16

---

14
10
6-
4
2
Montazuma Venture
freeewy
Salaris
Tornic
Booling
Private eye
Pono Pong
Robotiont
Battl zone
Ice hockey
Kangaroo
Double Perall
Berzeik
233000
Wizerd pllst
Allaniis
Kuna fu master
Amioa
Chopper command Hero
frostette
Star Assoutt
Up n down
Space invader
Demon bemonattadd attac
Yors revenaa
Antelii
Riverrald
Breakaut
Allen
Enduro
Bank leess
Fishing climber
Derender
Ms pacmman
Boxing
futamidiam
Road nurnet Copher
Cemtieeee Krul
jumes
Beam
Time
Videa
Ntme thi
Figure 10: Per-game ratios of weight magnitude after learning for 200M frames and before experi- encing any data. The ratios can vary up to 10 times between games.
factor of two yields comparable aggregate performance. Note though that we measure the IQM score after 200M frames, so the transient performance would differ depending on the timestep.
Adaptive Criterion for Injection. As step towards getting rid of the need to specify the injection timestep, we also explored the option of having a criterion for triggering the intervention. If the agent has the initial weight magnitude ||wo|| (w denotes here both encoder and head weights), threshold. The IQM scores of the agent with injection after 50M steps and with this heuristic coincide, although the frame when the agent reaches the threshold differs per game significantly: for some environments, it can be as small as 20M (such Enduro), for other environments, it can be beyond 200M (such Robotank) implying that the agent will learn without injection. Figure I0 gives an overview of how much the weight norm grows over the course of training (suggesting how fast the agent reaches the woll threshold on each game). We view devising an even more powerful criterion as a promising avenue for future work.
inject
plasticity after the weight norm surpasses the u/
L2 Regularization. The observations about the norm increase made try adding L2 regularization
to the Double DQN agent. A grid search over [10-? 3 10 10 ",3 10 10-6, 10
10-1--3033 .,3 10-b]
coefficients resulted in the best coefficient of 3 10 but leaving the aggregate score mostly the same; higher values resulted in significant performance deterioration. The result gives evidence that controlling the weight norm itself does not address plasticity loss but allows multiple interpretations. We speculate that 1.2 might be prematurely encouraging weights to have zero magnitude before obtaining high rewards (the effect would be especially profound in sparse reward settings) or that L2 might have undesirable side effects of smoothing approximate value functions while the true value functions might be non-smooth [Dong et al. 20200. We are puzzled about the inefficacy of L2 in our experiments and mixed results from applying it in RL in past works: the majority of deep RL algorithms do not use it [Mnih al. 2015 Schulman al. 2015 Lillicrap et al., 2015 Mnih et al., 2016 Bellemare et al., 2017], although not without exceptions [Schrittwieser et al.] 202 12]. Some works have explicitly reported negative effects from controlling the weight norm in deep RL Niiishhnn et al. 2022], while others highlighted its benefits [Farahmand et al. 2008 Li et al. 2023 more research in needed understand its effect RI.
C Details about the Baselines
we considered three alternative ways of dynamically addressing plasticity loss during training: resets, lidink-andPeettuur (SnP), and naive width scaling. Resets re-initialize parameters of the last layers (using our notation, it corresponds to replacing he(:) with hog (-))) and rely on a replay buffer to transfer knowledge before and after the intervention. Resets require the specification of the number of last layers and the application timestep. We ran sweep over [1, 2] layers and two choices of timesteps: either once at 50M frames or trice at 50M, 100M, and 150M. Afterwards, we reported the results that attain the highest IQM score.
In Section 5.3
17

---

n01
002901
Figure 11: A demonstration of the Assault game evolution when a high-performing agent found on the Internet reaches a score of around 2800: before, the agent had to shoot only upwards; afterwards, it has to shoot up, left, and right. We interpret that the failure improve upon the 2800 score is explained by exploration.
Shrink-and- Serink-and-Peeturbb modify all network weights w as w + Aw + at the given application timesteps, where is random vector with the same dimensionality as sampled from the standard Gaussian distribution. SnP has three hyperparameters: the shrink coefficient X, the noise scale o, and the application timesteps. We performed grid search over À in [0.1, 0.3, 11, in [0.01, 0.1, 1]. and the same choices of timesteps as for resets.
The best hyperparameters ended up being the ones that somewhat minimized the effect of both resets layer, application time) and SnP (A 1, 0.01, 3 application times): other hyperparameters resulted in even worse performance. The paper on resets [Nikishin al.) 2022] demonstrates results on the Atari 100k benchmark Kaiser et al. 2019 that focuses on a data-efficient regime with 10s interactions only and contains subset of 26 (out of 57) games. In this setting, the replay buffer has all experiences encountered during the agent's 's lifetime; this data can sufficient for recovering the performance after reset. In the Atari 200M setting though, the replay buffer has only 4M frames which might not be enough to recover fast after reset. We speculate that similar reasoning applies to SnP since it can be seen soft version of resets (D'Oro et al. 2023].
For the width scaling method, we modify the last two layers by doubling their width. In detail, suppose the weight matrices are Wi N and W2 IR KR|A, where |A| is the action space dimensionality. We create columns of W/ with values of W1 and the first K rows of W with values of W2. The remaining entries sampled from the random initializer. We perform modification the bias term by 2R22 by copying values from b E R and setting the rest to zero. The width is scaled once at S0M.
matrices W/ Nx2K and W 2| and fill the first K
Such naive approach increases plasticity but its inability to improve over the standard Double DQN might be caused by adverse effects on the predictions after the intervention without output correction.
D The Assault Game Analysis
We searched for a high-scoring behavior demonstration in the Assault environment YouTube The screenshots in Figure IT demonstrate the change of the environment around the score of 2800: before, the enemies were appearing only above the controlled starship, while afterwards, they start to appear from the left and from the right. Before the transition, the algorithm learned that actions "shoot left" and "shoot right" were irrelevant, while afterwards, it has start using these actions, suggesting that the performance plateau can be attributed to exploration challenges.
We highlight that it the suggested protocol for diagnosis that led the insight: after seeing that the post-injection agent has the same performance plateau as the baseline, we decided to investigate the behavior in the game and realized that previously irrelevant actions became critical.
https://ooutubb/HHwW/Wssss
18


Reviews:
[{"review_id": "v7QOwLhj5L", "reviewer": "Reviewer_227D", "summary": "This paper provides a deep dive into the issue of plasticity loss in deep reinforcement learning and introduces a novel approach, \"Plasticity Injection\". The authors employ this technique to not only diagnose plasticity loss but also effectively dissect confounding factors, such as exploration mechanisms. Through extensive experiments, the authors verified that Plasticity Injection significantly increases the agent's performance with minimal computational overhead. The comprehensive investigation of the role of resets in the plasticity viewpoint further enriches the paper's contributions to the field.", "questions": "1. Can the authors provide the results of the experiments conducted for the Adaptive Criterion for Injection and L2 Regularization? It seems that these results are missing in the appendix section.\n2. A recent investigation from Lee et al [1] has claimed that the reset mechanism is closely tied to the non-stationarity of labels in reinforcement learning while having minimal impact in terms of the inputs' non-stationarity. I would like to see whether the Plasticity Injection (Reset without exploration) behaves similarly to the Reset.\n\n[1] Enhancing Generalization and Plasticity for Sample Efficient Reinforcement Learning., Hojoon Lee et al.\n", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Building upon the reset mechanism, this paper offers a successful strategy for tackling plasticity loss in Deep RL, while dissecting the role of exploration.\n2. Plasticity Injection module is simple and straightforward, highlighting its wide potential applicability across various algorithms and architectures.\n3. The authors conduct an in-depth analysis of the elements contributing to performance improvement.\n4. I liked the qualitative analysis of why the score of Assualt gets stuck.", "weaknesses": "1. While the Plasticity Injection module was successfully integrated into the Atari environment, it would benefit from broader testing across various environments, considering that the reset mechanism [1] has consistently performed well in the Deepmind Control Suite environment.\n2. Related to Weakeness 1, recent work from Lyle et al [2] have found that the last layer reset does not improve the model's plasticity in their synthetic experiments. Can we really confirm that the reset mechanism is improving the performance on the Atari environments because it is improving the model's plasticity?\n\n[1] The primacy bias in deep reinforcement learning., Evgenii Nikishin et al.\n\n[2] Understanding plasticity in neural networks., Clare Lyle et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our submission and for the thoughtful comments. We provide responses to your questions and concerns below.\n\nThank you for raising the point about testing plasticity injection outside of Atari. We have conducted an experiment in a simple non-Atari setting based on CIFAR10. In detail, we split the dataset into 10 parts uniformly at random and fit the network sequentially on the first part, then on the second, and so on. The default network gradually loses the plasticity as it faces subsequent parts of the data, while the network with plasticity injection is able to learn on all parts. The results are attached results in the supplementary PDF that could be found with the joint response. Because of the time and resource constraints, we provide only this simple experiment but we hope that it can serve as a piece of evidence of the applicability of plasticity injection outside of Atari.\n\nAs for your concern about the findings of Lyle et al., we would like to point out that the argument in their paper is more nuanced: indeed, the solution that they argued for in the end was not based on resets because one of their motivations was to come up with a non-drastic method for addressing plasticity loss, but the experiment in Figure 6 of their paper actually shows that resets mitigate plasticity loss.\n\nWe articulate the results of the experiments with adaptive criterion and L2 regularization in the appendix in textual form (L542, L550). We did not include a separate figure for these results because the IQM for the adaptive criterion coincides with the IQM after applying plasticity injection at 50M frames; likewise, the IQM of the agent with the best L2 coefficient coincides with the IQM of the base Double DQN agent.\n\nLastly, thank you for putting the paper of Lee et al. from 2 months ago on our radar! It is indeed an interesting question whether PI helps with the input or target non-stationarity and there’s potential evidence for both: in the CIFAR10 experiment that we introduced earlier and that resembles one of the experiments in the paper you have suggested, the target dependency is fixed and it is a set of datapoints that is changing. At the same time, since the extra network is learning a residual to the prediction of the frozen network, we could also say that PI could be helpful with the target non-stationarity. Carefully investigating this question would be an exciting avenue for future research.\n"}, {"review_id": "MbkKawl9sq", "reviewer": "Reviewer_P5kx", "summary": "The work attempts to study the role of plasticity loss in deep reinforcement learning. The paper proposes a new tool to diagnose plasticity loss. The new tool is designed to minimize the effect of other confounders like exploration. The results on Atari show that plasticity loss affects the performance of double DQN, and plasticity injection often breaks performance plateaus.", "questions": "The results with L2 regularization in the appendix are surprising. Prior results on plasticity loss have shown that L2 regularization helps mitigate it. Figure 1 in the paper by Sokar et al. (2023) shows that L2 regularization improves the performance of DQN. Why do you think there is a discrepancy between the effect of L2 regularization in your results and previous results?\nI guess that because the parameters of double DQN are not tuned for L2, we might need to tune all the parameters of double DQN to benefit from L2.\n\nSokar, G., Agarwal, R., Castro, P. S., & Evci, U. (2023). The dormant neuron phenomenon in deep reinforcement learning. arXiv preprint arXiv:2302.12902.", "limitations": "The main limitation is that other methods are not properly tuned, so the conclusions we can not draw strong conclusions from experiments in Section 5.3.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper's main contribution is a new diagnostic tool for plasticity loss. The complexity of deep RL makes it challenging to isolate phenomena like loss of plasticity and study them carefully. The proposed tool, plasticity injection, does an excellent job of isolating the effect of plasticity loss while minimizing the impact of other confounders.\nThe empirical results show that double DQN suffers from plasticity loss in Atari games. And plasticity injection improves performance in almost all games. \nThe improved performance is particularly exciting because agents like Double DQN are tuned to achieve high performance in 200M frames. My guess is that if we let these agents run for longer, their performance will plateau in most games. So, the effect of plasticity loss will be more apparent if these agents learn for longer, say 1B frames. And in such long-lived agents, loss of plasticity will become a more dominant phenomenon. So it's interesting to see plasticity loss even with parameters tuned for good performance in 200M frames. ", "weaknesses": "The proposed diagnostic tool is well thought out and works as intended. However, other contributions in the paper have some issues.\n\nThe main problem lies when comparing other methods like SnP and Width Scale with plasticity injection. It is not clear if other methods were tuned properly. The appendix said that some parameters of SnP were tuned, but the paper does not mention if other parameters, like learning rate, replay ratio, or buffer size, were tuned for any of these methods. Unfortunately, that is the downside of using only large environments like Atari; we can not perform rigorous empirical investigations, limiting the conclusion we can draw from some of the experiments, like Figure 6.\n\nA small issue with the paper is that it only studies value-methods methods and not policy gradient methods, so its title is a bit misleading. A better title would be _Value-based Deep RL with Plasticity Injection_. \n\nHowever, these issues do not affect the paper's primary contribution, so I recommend accepting this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to write the review. We are grateful for the positive evaluation of our submission. We provide responses to your questions and concerns below.\n\nWe acknowledge that due to the scale of the experiments we did some hyperparameter tuning for methods like SnP and Width Scale but not a fully exhaustive one (that would include the learning rate, the replay ratio, and the buffer size). Since we have used the default hyperparameters of Double DQN and did not tune them when using plasticity injection (i.e. leaving the LR, the RR, and the buffer size untouched), we hope that during the comparison with other methods in Section 5.3 we did not give a disadvantage to the alternative methods.\n\nAs for the title of the paper, even though we indeed used value-based RL as a testbed, the plasticity injection is agnostic to the choice of the family of algorithms and could be applied to function approximators both for values and policies. \n\nLastly, it is indeed slightly puzzling to see the discrepancy between our L2 results and the results reported by Sokar et al. After checking the details of their paper (Appendix A on Page 14), it appears that Sokar et al. also did not tune the hyperparameters of their agent to be compatible with L2, they have done only a grid search over the regularization coefficient. However, there are several details about the setting that could explain the difference: first, Sokar et al. used a subset of 17 Atari games, while our setting considers all 57 Atari games; this subsampling might have affected the conclusions. Likewise, their experiments with L2 on Atari use 10M frames, while we learned agents for 200M frames. Finally, their base algorithm was DQN while ours was Double DQN; perhaps L2 was mitigating the overestimation bias by implicitly controlling the magnitude of the outputs through controlling the magnitude of the weights. Overall, the effect of L2 in deep RL is quite mysterious and it is surprising to realize that the majority of standard algorithms like DQN, PPO, and SAC do not use it. More research is needed to understand clearly the interaction of L2 regularization with RL.\n\nThank you again for the thoughtful review and comments.\n"}, {"review_id": "C4PygSnIkN", "reviewer": "Reviewer_Foz7", "summary": "The author propose a solution to the problem of loss of plasticity (decrease in learning effectiveness over time). Their solution is to freeze the existing network and add two new heads, one free to learn, the other frozen to the negative of the free-learning new head. The result is that the intervention has zero effect on outputs at the time of plasticity injection, while allowing a freshly reset head to learn from scratch.\n\nThey suggest that the method can be used as a diagnostic tool, avoiding possible confounds from other methods that rely on resets and create large changes in outputs.\n\nThey also suggest that the method improves performance better than existing plasticity-restoring methods, while being less costly than training a larger network from time 0. \n\n\n== Update after rebuttal == \n\nConsidering the authors' reply, I have slightly updated my evaluation to lean more towards acceptance.", "questions": "- Figure 1: The successive “tasks” are not random. There is an order induced by the fact that they represent results of a slowly learning and improving policy: the policy at 20K vs 10K or 4K is not just different, it is better trained. This might be a confound if the training impacts (say) the complexity of the actual value function. What happens if you do the same figure but randomizing or inverting the order of the successive policies whose value is being predicted? (i.e. at iteration K, the task is to predict the values for K’-th frozen policy, with K’ != K)\n\n- l. 233: affect -> affected\n\n- Figure 4: “We take the maximum score among the agents with plasticity injection after 25M, 50M, and 100M steps\" - come on! Even adding random noise to the values would provide an apparent \"net improvement\" under such a selection method! At the very least least show us the results with injection at 50M for all.\n\n- In Fig 6a, at what time do we perform the interventions? 50M?\n\n- IIUC the difference between \"Width Scale\" in Fig 6a and \"Larger Net\" in Fig 6b is that the former only enlarges the networks at 50M (?), while the latter is larger from time 0. Is this correct? If so, it should be stated in the main text. \n\n- There is an alternative to the tripling of head parameters imposed by the method. Just add one new freshly initialized head, with a scalar multiplier on its output initially set to 0. Presumably this should reduce the added cost while providing equivalent results. What happens if you do that?\n\n\n", "limitations": "The authors do mention some limitations of the work, especially the cost.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method is novel. The experiments seem thorough (though some clarifications are needed, see below).", "weaknesses": "The main difficulty is that the method actually imposes a considerable computational cost and further complexity, while apparently providing a small benefit vs. the much simpler alternative of training a larger network from scratch (<10% total wallclock time, IIUC from Figure 6)\n\nIt also seems that the method introduces an additional parameter (when to inject plasticity) against the simpler method of training a larger network from scratch. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We address your concerns below (AR = author response).\n\n*“The method actually imposes a considerable computational cost and further complexity, while apparently providing a small benefit vs. the much simpler alternative of training a larger network”*\n\n**AR**: We appreciate your concern about the benefits of plasticity injection (PI) compared to the use of a larger network. Whether the additional complexity is justified depends on the scale of the setting. As an extreme example, Agarwal et al. [1] calculated that reproducing the results of the AlphaStar agent [2] would cost $3M+. We believe that saving 10% of computations in such settings would justify the additional complexity from PI. As the models get larger, it is likely that this type of scenario will become increasingly common.\n\nAnother benefit of plasticity injection is that it allows adding extra plasticity anytime whilst having a large network requires committing to a fixed network size in advance. In other words, if an RL practitioner estimated incorrectly the needed network size before launching an experiment, PI could be used to extend the network on the fly.\n\nLastly, we would like to emphasize another use case of plasticity injection we argue for: as a tool for diagnosing plasticity loss, which is one of the main contributions of the paper. In our experience, Deep RL systems are notoriously challenging to debug, and being able to isolate plasticity concerns from other confounding factors present in RL, such as exploration, would be valuable for understanding the source of performance bottlenecks. \n\n*“The method introduces an additional parameter (when to inject plasticity) against the simpler method of training a larger network”*\n\n**AR**: Thank you for raising the point about the injection timestep hyperparameter. Firstly, we have verified that plasticity injection is robust to the choice of the timestep: Figure 9 demonstrates that the IQM scores after applying PI at 25M, 50M, and 100M are roughly the same. Secondly, we have experimented with an adaptive criterion for injection that gets rid of the necessity to specify the injection timestep (see Appendix B, “Adaptive Criterion for Injection”): with such a criterion, the agent achieves the same IQM as with using PI at 50M frames, however, we decided to use the version of PI with a specified timestep for simplicity and clarity.\n\nPlease also find answers to your questions below:\n- **On the task order.** Thank you for this question, it’s indeed possible that the prediction problem difficulty is growing over time. To address this concern, we proceeded with the reviewer’s recommendation and tried to randomize the order of the policies to evaluate; the pattern was the same: if we preserve the parameters after every task change, the final value prediction error grows for each subsequent task. Crucially, please also note that resetting the network parameters after each task switch serves as a control: since the randomly initialized network attains the same final loss on each task, we can conclude that all prediction problems in the sequence are of similar difficulty.\n- **Typo.** Thanks for catching the typo, we’ve fixed it.\n- **On taking the max score after PI at 25M, 50M, and 100M frames.** We emphasize that *we do not use Figure 4 to give an unbiased estimate of the performance of PI*. We agree that such a selection strategy incurs a risk of overestimating the effectiveness of the method Figure 4 is intended to *illustrate the potential performance improvements that could be gained from eliminating plasticity loss*. We will give a more detailed clarification in a revision. In Section 5.3, where we argue that PI outperforms alternative methods, we use PI applied at 50M without taking the maximum score. Figure 7 in Appendix gives per-environment results for PI applied at 50M frames.\n- **Intervention timestep in Figure 6, left.** Injection and Width Scale are applied after 50M frames; for Resets and SnP we have performed a grid search and reported the best results after applying Resets / SnP either once at 50M frames or trice at 50M, 100M, and 150M. Appendix C gives all details about the baselines.\n- **Difference between Width Scale and Larger Net.** Your understanding of the difference is correct: the former enlarges the network at 50M steps, while the latter is larger from the start. We’ll clarify that in the revision.\n- **On having a single extra head with an initial scalar multiplier set to 0.** That’s a good point. We have tried to use a scalar multiplier on the outputs of the second network. The challenge is what to do with the multiplier after setting it initially to zero. Both learning this scalar and manually annealing it resulted in unstable training so we discarded this option. Now, your direction of thinking is still valid: it might be possible to minimize the effects of the second network by, for example, modifying its initialization scheme. From the analysis viewpoint, we strived for clarity: changing the initialization is undesirable because it would introduce a confounder for our analysis (since initialization affects the signal propagation and thus network plasticity); from minimizing the computations viewpoint, this avenue might be promising though.\n\nWe thank you again for the review and all the questions and hope that the rebuttal addresses the outlined concerns. We will happily engage in a subsequent conversation during the author-reviewer discussion stage.\n\n[1] Agarwal, Rishabh, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc Bellemare. \"Reincarnating reinforcement learning: Reusing prior computation to accelerate progress.\" Advances in Neural Information Processing Systems 35 (2022): 28955-28971. \n\n[2] Vinyals, Oriol, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu et al. \"Grandmaster level in StarCraft II using multi-agent reinforcement learning.\" Nature 575, no. 7782 (2019): 350-354."}, {"review_id": "rnt58rVQvd", "reviewer": "Reviewer_x17L", "summary": "This paper studies the loss of plasticity in deep RL and proposes a method for mitigating it at the cost of more overhead memory and computation, but doing so in such a way that does not necessitate adding more trainable parameters or affecting predictions. The latter two are key when using their approach for the diagnosis/analysis of plasticity loss in deep RL agents (as they remove confounders due to changes in exploration or representational capacity). \n\nThey also show that their method can be used to increase the size of the agent network during the course of training to better deal with plateaus in training and save computation cost (i.e. starting with a smaller network to save computation and increase the size when plateauing). \n\nThis method is also a nice addition to the Reincarnating RL framework, whereby reusing previously trained agents with plasticity injection could help both utilize what was learned in a previous iteration as well as keep the network flexible to absorb new knowledge.  ", "questions": "1. I agree that analysing plasticity in RL is confounded by exploration. But why not study this phenomenon in manageable finite environments and perform full sweeps over all states or state-action pairs? This could at least serve as a minimalistic way to fully remove the impact of exploration and solely focus on plasticity loss.\n\n2. Regarding this claim (lines 298-300) \"*However, the idea of plasticity injection is agnostic to the choice of the architecture. For example, it can be applied for residual blocks in ResNets or decoder blocks in Transformers.*\" \n\nI agree. But do we know whether such architectures still suffer from loss of plasticity when used properly in deep RL? \n\n3. Is anything known about whether the root cause of plasticity loss in deep RL would differ from that of deep learning?  \n\n4. What measure do you use to automatically identify loss of plasticity or plateaus during training?", "limitations": "Limitations have been discussed sufficiently well. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Loss of plasticity in continual deep RL is a very interesting topic and this paper focuses on a methodology to both help analysis/diagnosis of this phenomenon as well as to mitigate it. \n\n- The proposed method has other benefits such as allowing us to increase the size of the neural net during the course of training without retraining or affecting current predictions. \n\n- Paper is well written and does a good job at introducing and motivating the problem, discussing related works, and exposing the ideas\n\n- Perhaps what I'm most excited about in this paper is the following possibility with their *plasticity injection* approach:\n\"*An exciting avenue for future research is understanding trade-offs between architectural design decisions: RL agents typically employ networks that were originally proposed for stationary problems, but perhaps dynamically growing networks would suit the non-stationary nature of RL better.*\"\n\n  ", "weaknesses": "- The methodology of this paper in the short run would be beneficial, but it's a way around the real underlying problem: the real solution to this problem would likely emerge in the deep-learning toolbox. \n\n- Full-sweep experiments would have helped with illustrative experiments in smaller domains without any exploration confounding altogether (e.g. Q-learning but with a full sweep over the full state-action space.)\n\n- I like the experiment in Figure 1; helps illustrate the phenomenon nicely. \nHowever, the description of what constitutes tasks at each interval could help understand the nature of this illustrative experiment better; e.g., how were the states selected using frozen policies (from every 10M frames), how many such states were selected (as many as the tasks?), how was MC applied to estimate values. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and for the positive evaluation of our submission. We address your questions and concerns as follows:\n- We acknowledge that plasticity injection is likely not the end solution to the phenomenon of plasticity loss. Currently, the phenomenon is not clearly understood and even reliably detecting it was outside of the reach. Plasticity injection is a step towards being able to diagnose plasticity loss and thus is a step on the way to understand its causes and arrive at an ultimate solution (indeed potentially based on the deep learning toolbox).\n- We agree that experiments in smaller domains without exploration would have been helpful. We have considered the possibilities of using environments with finite state-action spaces where the Q-function would be represented as a table as well as environments where linear function approximation would suffice but figured that the plasticity loss phenomenon is most compelling to study in the case with neural networks and complex environments. Having said that, the work of Dohare et al. [1] presents experiments in controlled domains without exploration reminiscent of the setting we considered in Figure 1.\n- Thank you for pointing out that the experiment in Figure 1 could be explained better. We will clarify the setting in a revision. In particular, we construct 20 tasks using policies stored after training the Double DQN agent for 10M, 20M, …, 200M frames. To collect states for each task, we let the stored policy interact with the environment in the evaluation regime, without any policy updates. We stop the interactions after the policy collects 4M states (the default size of the replay buffer for Double DQN). Finally, since we collect rewards from the environment, we could construct Monte-Carlo estimates of the value function in state s_t by taking the discounted sum of rewards r_t + gamma r_{t+1} + gamma^2 r_{t+2} + …. The resulting dataset with states and the MC estimates is then used for one of the 20 regression tasks.\n- It is indeed compelling to consider tabular environments because of the clarity of the analysis they offer. One of the reasons why we decided to use complex environments like Atari directly is because it is not guaranteed that empirical observations about plasticity loss we would gain in tabular domains would be the same as in complex environments. However, we will think more about the possibilities of designing convincing experiments in environments with finite state-action spaces that could offer general insight on the phenomenon.\n- Whether ResNets and Transformers suffer from plasticity loss is a great question. The research is still preliminary in this space but the authors are aware of two pieces of evidence suggesting that the answer might be “yes”: first, Lyle et al. [2] demonstrated loss of plasticity for both ResNets and Transformers on small-scale constructed MDPs; second, Adaptive Agent Team [3] argued that difficulties of training their architecture (that contained both Transformer and Residual blocks) were due to plasticity loss due to overfitting to initial experiences.\n- Whether the causes of plasticity loss in deep RL and deep learning are different is another great question. The authors believe that there’s an overlap of causes as well as distinctive ones. As an example of a cause distinctive to deep RL, temporal difference learning poses a very peculiar optimization problem that might be one of the culprits of plasticity loss: inherently it learns a value function from its own bootstrapped predictions. Such a setting is not common for supervised problems with fixed prediction targets that deep learning typically deals with.\n- We are not completely sure whether we have understood the last question correctly so we would like to kindly point out that identifying the loss of plasticity is one of the intended uses of the proposed plasticity injection approach. However, if we misunderstood the question, we are happy to engage in further conversation during the author-reviewer discussion period.\n\nAgain, we thank you for the review and for all of the comments.\n\n[1] Dohare, Shibhansh, J. Fernando Hernandez-Garcia, Parash Rahman, Richard S. Sutton, and A. Rupam Mahmood. \"Maintaining Plasticity in Deep Continual Learning.\" arXiv preprint arXiv:2306.13812 (2023).\n\n[2] Lyle, Clare, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. \"Understanding plasticity in neural networks.\" In International conference on machine learning. PMLR, 2023.\n\n[3] Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg et al. \"Human-timescale adaptation in an open-ended task space.\" In International conference on machine learning. PMLR, 2023.\n"}]

Author rebuttal:
We thank all reviewers for their efforts to provide the detailed feedback. We particularly appreciate that reviewers find the paper novel (Reviewers 227D, Foz7), well-written (x17L), comprehensive (227D), thorough (Foz7) as well as highlight that the method is simple (227D), well thought out (P5kx), and with an in-depth analysis of its elements (227D).
We provide responses to reviewers’ questions and concerns in individual replies and attach a supplementary PDF with an experiment aiming to address a concern raised by reviewer 227D.


Meta-review: Authors propose "plasticity injection" for neural networks to maintain their ability to learn from new data, given that previous training has shown to "lose plasticity" after previous training causing performance plateaus when used in reinforcement learning tasks. The authors have been thorough in responding and updating their paper w.r.t. reviewer requests and comments, which the reviewers on the whole have accepted and appreciated. After reading the paper and all the reviews and rebuttals, I appreciate this simple-yet-effective idea to introduce plasticity into the learning process without causing any discontinuities in the model's outputs when injected, including uses this method as a diagnostic tool to determine when networks lose their plasticity.
Decision: Accept (spotlight)