SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: Percentage improvement of the average performance after adding plasticity injection across all 57 Atari games. We take the maximum score among the agents with plasticity injection after 25M, 50M, and 100M steps to roughly estimate the improvement as if plasticity injection was applied at a proper timestep and to demonstrate what the performance could have been if plasticity loss was mitigated. Table 1 later presents a categorization of environments into buckets where the agent does and does not benefit from injection. Learning curves corresponding to each environment are available in Appendix A.
works and suggest a recipe for controlling the degree of plasticity loss by decreasing the learning rate or the replay ratio, increasing the network size, or employing normalizations<sup>2</sup>.
In addition to gaining scientific insight, we now discuss how the intervention can be useful in large-scale RL.
5.3 Plasticity Injection for Computational Efficiency
Over the recent years, RL agents have been trained at increasingly larger scales. For example, mastering particularly challenging environments required an equivalent of hundreds of years of human gameplay [Vinyals et al., 2019], or obtaining a diverse set of skills required a  $1\mathrm{B}+$  parameter networks [Reed et al., 2022]. Given the trend, computational considerations become increasingly relevant.
Figure 5: Percentage improvements of the IQM scores from plasticity injection in varying regimes controlling the degree of plasticity loss. The intervention effect size monotonically increases with the replay ratio (RR) and the learning rate (LR), monotonically decreases with the size of the neural network, and is smaller yet positive for an agent employing spectral normalization (SN). These observations can be seen as recommendations about how to address loss of plasticity.
2We follow the recommendation of Gogianu et al. [2021] and apply SN to the penultimate layer; since we apply injection to the last two layers, issues with their plasticity might be partially alleviated by SN. Given that Gogianu et al. [2021] notice that the spectral norm of other layers starts growing more and D'Oro et al. [2023] observe that the first layers benefit from partial resets, we conjecture that first layers' plasticity is still declining even with SN.
Figure 6: Left: Comparison of plasticity injection to other methods that can be applied to dynamically address loss of plasticity. The difference in performance between all methods and the baseline is insignificant except for injection; an agent with injection is capable of improving without having to re-train from scratch. Right: Performance over wallclock time for an agent with plasticity injection and an agents that uses a network with increased width from the beginning. Because Injection switches from a small network to a larger network through plasticity injection at 50M frames and uses less trainable parameters than Larger Net, it achieves similar IQM while saving an equivalent of 20 hours of computational resources.
Plasticity injection can be used to improve the computational efficiency of RL in the following ways.
Reincarnating with Plasticity Injection. Recently, Agarwal et al. [2022] proposed a workflow called "Reincarnating RL" which reuses computations from previously trained agents during the iterative process of agent design. For example, if we trained an agent for several days or weeks and then decided to change its design (such as the network size), the workflow suggests to leverage the spent computations instead of training again from scratch. Plasticity injection can be useful from this perspective when the agent is unable to improve due to loss of plasticity and re-training from scratch is expensive. To see the effectiveness of plasticity injection in such setting, we compared plasticity injection with several alternatives that address loss of plasticity dynamically during training, including Shrink-and-Perturb (SnP) [Ash and Adams, 2020], resets [Nikishin et al., 2022], and naive width scaling (we describe the methods in detail in Appendix C). Figure 6 (left) shows that plasticity injection achieves a higher aggregate score across 57 Atari games compared to the alternatives. The results suggest that plasticity injection can be used to "reincarnate" agents more efficiently compared to the alternatives, without re-training from scratch.
Minimizing Computations via Dynamic Growth. Although larger networks tend to maintain plasticity longer, they require more computations or can be more challenging to train [Team et al., 2023]. We hypothesize that the full capacity of a large network may not always be necessary early in training, even if it is useful to maintain plasticity later. If this hypothesis is true, we can save computations by starting from a smaller network and injecting plasticity during training, without compromising the final performance compared to using the large network from the beginning. To verify this hypothesis, we took a network with larger layer width, matching the total number of parameters in  $\phi(\cdot), h_{\theta}(\cdot)$ , and  $h_{\theta_1}(\cdot)$  combined. The results in Figure 6 (right) show that an agent with plasticity injection during training performs comparably to the alternative that uses a larger network from the start. At the same time, it saves about 20 hours of wallclock time on an A100 GPU since it uses a smaller network up to 50M frames and has fewer parameters that are updated after plasticity injection. These results confirm the hypothesis and suggest that plasticity injection can be used as a tool for minimizing computations when training RL systems at a large scale.
The first and foremost limitation of plasticity injection is an increase in memory and training time compared to the baseline agent with a standard-sized network. When using plasticity injection as a diagnostic tool, we believe the overhead is largely justified since preserving the network output makes it easier to isolate confounding factors like exploration. From the deployment viewpoint, a practitioner should decide whether the increase in compute and memory is justified based on how much plasticity injection improves performance. While the effect of the intervention was positive on the aggregate performance on Atari, it varied considerably across individual games: in some cases, it did not change scores much, in other cases, it had a significant positive effect.
Preserving the network outputs and keeping weights can be undesirable in case of parameter divergence that often occurs in the deep RL experimentation [Van Hasselt et al., 2018]. Such a scenario also qualifies as loss of plasticity; addressing it without drastic tools can be challenging. Lastly, while we propose a diagnostic and mitigation tool, we do not identify causal factors driving plasticity loss in deep RL. More research is needed here: understanding these causes could lead to avoiding plasticity loss in the first place.
Results in this paper can serve as a clear study of the plasticity loss phenomenon in deep RL and evidence that the optimization aspect in RL still leaves room for improvement. The version of plasticity injection we propose may yet not be optimal: we strived for simplicity rather than performance and view the intervention as a blueprint for future methods.
The experiments in this paper adopted the convolutional architecture from Van Hasselt et al. [2016] but modern deep RL practice not rarely involves ResNets [He et al., 2016, Espeholt et al., 2018] and Transformers [Vaswani et al., 2017, Chen et al., 2021, Reed et al., 2022]; we did not investigate settings with these advanced architectures. However, the idea of plasticity injection is agnostic to the choice of the architecture. For example, it can be applied for residual blocks in ResNets or decoder blocks in Transformers.
An exciting avenue for future research is understanding trade-offs between architectural design decisions: RL agents typically employ networks that were originally proposed for stationary problems, but perhaps dynamically growing networks would suit the non-stationary nature of RL better.
Applications of plasticity injection focus on diagnosing RL systems and their efficiency. We compliment a recent opinion paper from Mannor and Tamar [2023] by arguing that if deep RL is to become a technology that a non-expert can use, more research is needed on the process of iterating on the agent design and computational efficiency.
Although this paper attempted to understand and mitigate loss of plasticity in RL, there are still remaining open questions. Can we solve the problem of plasticity loss completely? Which properties of newly initialized networks enable high plasticity? Answering these questions is a key challenge for training truly intelligent agents.
EN thanks Tom Schaul, Greg Farquhar, John Quan, Dan Horgan, Mihaela Rosca, Angelos Filos, Diana Borsa, Luisa Zintgraf, Yash Chandak, Robert Lange, Chris Lu, David Parkes, Blanca Huergo, Rich Sutton, David Silver, Hado van Hasselt, Alexander Novikov, Julia Novikova, and especially Iurii Kemaev for their help and valuable discussions. EN also thanks many other interns and the broader DeepMind team for the great internship experience. This work was funded by DeepMind.
We acknowledge the Python community [Van Rossum and Drake Jr, 1995, Oliphant, 2007] for developing the core tools that enabled this work, including JAX [Bradbury et al., 2018, Babuschkin et al., 2020], Jupyter [Kluyver et al., 2016], NumPy [Oliphant, 2006, Van Der Walt et al., 2011], SciPy [Jones et al., 2014], matplotlib [Hunter, 2007], and pandas [McKinney, 2012].
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. arXiv preprint arXiv:2303.07507, 2023.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
Robert E Schapire. The strength of weak learnability. Machine learning, 5(2):197-227, 1990.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. arXiv preprint arXiv:2302.12902, 2023.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139-1147. PMLR, 2013.
Adrien Ali Taiga, William Fedus, Marlos C Machado, Aaron Courville, and Marc G Bellemare. Benchmarking bonus-based exploration methods on the arcade learning environment. arXiv preprint arXiv:1908.02388, 2019.
Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, et al. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608, 2023.
Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. The numpy array: a structure for efficient numerical computation. Computing in science & engineering, 13(2):22-30, 2011.
Hado Van Hasselt. Double q-learning. Advances in neural information processing systems, 23, 2010.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02648, 2018.
Guido Van Rossum and Fred L Drake Jr. Python tutorial, volume 620. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michael Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
Figure 7: Performance of the Double DQN with and without plasticity injection after 25M, 50M, and 100M frames on the full Atari 57 benchmark. The potential discontinuities in the plots such as in Road runner are caused by the evaluation each 1M frames, i.e. the first moment the agent with injection contributes to the plot is after learning for 1M frames.
Injection Effect	Environments
Consistent Improvement	Alien, Asteroids, Breakout, Chopper command, Enduro, Frostbite, Gopher, Phoenix, Space invaders, Surround, Wizard of wor, Yars revenge (12 total)
Minor Improvement	Amidar, Asterix, Atlantis, Bank heist, Beam rider, Berzerk, Boxing, Defender, Fishing derby, Jamesbond, Krull, Ms pacman, Road runner, Sequest, Time pilot, Up n down, Video pinball, Zaxxon (18 total)
Negligible	Battle zone, Bowling, Centipede, Crazy climber, Double dunk, Freeway, Gravitar, Hero, Ice hockey, Kangaroo, Kung fu master, Montezuma revenge, Name this game, Pitfall, Pong, Private eye, Qbert, Riverraid, Skiing, Solaris, Star gunner, Tennis, Tutankham, Venture (24 total)
Negative	Assault, Demon attack, Robotank (3 total)
Table 1: Summary of effects from applying plasticity injection to Double DQN on 57 Atari games.
A Complete Learning Curves
Figure 7 presents the return plots over the course of Double DQN training for 200M frames on the whole set of 57 Atari games. We informally categorized environments into four buckets upon visual inspection of effects from plasticity injection in Table 1. The most notable negative example is Demon attack, while on Assault and Robotank the effect is negative but minor. In the rest of the 54 games, plasticity injection either improves performance or has a negligible effect, possibly depending on the injection timestep.
B Ablations
This appendix presents an ablation analysis of the various design choices made during the study of plasticity injection. The purpose of such ablations is to build intuition on the behavior of plasticity injection under different conditions so that an RL practitioner can use it in their application.
Injection Variants. The proposed modification of the network architecture is not the only one possible. In Section 4, we initially described a version of plasticity injection without encoder sharing, that is, when the intervention is applied to the entire network (referred to as Injection, Whole Net in Figure 8). Another alternative is to create a whole new set of parameters and copy the encoder parameters of the old network without sharing it (denoted as Injection, Whole Net, Copy Enc). Lastly, for all three versions, there is the possibility of not freezing the old set of parameters (weights corresponding to the third, output correction term are always going to be frozen).
Figure 8 (left) summarizes the findings:
1. Creating a completely new encoder-head pair is the alternative with the lowest IQM scores;
2. Variants with encoder sharing or copying have comparable performance; the Injection, Whole Net, Copy Enc version has a slightly lower performance than the rest. We conjecture that it might be due to the larger number of frozen parameters;
3. Unfrozen variants generally perform not worse than their frozen counterparts. The unfrozen variants introduce more trainable parameters compared to the baseline, which require more computations during learning and increase the network expressivity. Since we were interested in a careful diagnosis of plasticity loss and extra expressivity may be a confounding factor, we decided to stick to the frozen version by default.
Multiple Injections. Given the improved performance from plasticity injection in the previous experiments, a natural question is whether applying plasticity injection multiple times would improve performance even further. To investigate this question, we applied plasticity injection at 100M and 150M frames, in addition to 50M frames, and plotted the IQM improvements with respect to a single injection at 50M frames. As shown in Figure 8 (right), additional injections do not improve the performance over a single injection in a setup with a standard network. We hypothesize that in our particular experimental setting, loss of plasticity can be largely mitigated with a single plasticity
Figure 8: Left: Comparison between variations of plasticity injection. Whole Net denotes injection of both the encoder  $\phi(\cdot)$  and the head  $h_\theta(x)$ ; Copy Enc denotes copying the  $\phi(\cdot)$  at the moment of injection without further sharing; Unfrozen denotes keeping parameters of the first term unfrozen. Relying on a new encoder leads to a lower performance; the rest of the alternatives have comparable scores. Right: Percentage improvements of the IQM score from multiple injections over a single injection for varying network sizes. Multiple injections are beneficial for smaller networks. Note that previous plots in Figure 5 show improvements when comparing one injection over no injections while this plot compares multiple injections over one.
injection. To verify this hypothesis, we applied multiple injections while varying the network size (similarly to Section 5.2, to make the network  $2\mathrm{x}$  smaller, we divide the width of the hidden layers by  $\sqrt{2}$ ). Figure 8 (right) confirms that the level of improvement grows monotonically as the agent uses smaller networks. Since the results in Figure 5 suggests that the degree of plasticity loss increases with smaller networks, this result indicates that multiple rounds of plasticity injection can be beneficial in situations where the agent network is too small to maintain plasticity.
No Output Correction. In the majority of the games, subtracting the initial copy of the newly introduced head  $h_{\theta_2}(\cdot)$  resulted in mostly similar learning curves as without the subtraction, although not always. In particular, the impact of the injection on Yars Revenge is smaller without compensating for the bias. Also, we observed a significant difference in high variance games (such as Berzerk and Hero). Note that removing effects on the predictions from introducing the new head would be possible by modifying the initialization [Brohan et al., 2022]. From the analysis viewpoint, we strove to have as clean experimental design as possible and wanted to remove initialization-specific confounders since initialization would affect network plasticity as well [Sutskever et al., 2013]. From the saving memory and computations viewpoint, it might be preferable to do plasticity injection without introducing the third network.
Optimizer. One might hypothesize that benefits from injection can be attributed to manipulations with the optimizer state. To test this hypothesis, we perform two ablations: the first resets statistics of the RMSProp optimizer [Tieleman et al., 2012] used by Double DQN after 50M steps, the second copies the optimizer state of the original head to the newly initialized head after the injection. Figure 9 (left) demonstrates that most of the effects from injection come from having additional weights rather than from interventions on the optimizer.
Injection Timestep. In Section 5.2, we presented the results for a selection of environments while varying injection timestep. Figure 9 (right) suggests that across all games, changing the timestep by a
Figure 9: Left: Comparison of an agent with injection, an agent with injection but copied optimizer state for the newly initialized head (Injection + Copy Opt), and an agent that resets the optimizer statistics of the last two layers (Reset Opt). The results suggest that effects from interventions on the optimizer state are marginal compared to having new weights. Right: Aggregate performance for agents with varying injection timesteps. Whilst Figures 7 and 10 suggest that loss of plasticity might be happening at different paces across environments, the final IQM score is relatively robust with respect to the injection moment.
Figure 10: Per-game ratios of weight magnitude after learning for 200M frames and before experiencing any data. The ratios can vary up to 10 times between games.
factor of two yields comparable aggregate performance. Note though that we measure the IQM score after 200M frames, so the transient performance would differ depending on the timestep.
Adaptive Criterion for Injection. As a step towards getting rid of the need to specify the injection timestep, we also explored the option of having a criterion for triggering the intervention. If the agent has the initial weight magnitude  $\| w_0\|$  ( $w$  denotes here both encoder and head weights), we inject plasticity after the weight norm surpasses the  $3\| w_0\|$  threshold. The IQM scores of the agent with injection after 50M steps and with this heuristic coincide, although the frame when the agent reaches the threshold differs per game significantly: for some environments, it can be as small as 20M (such as Enduro), for other environments, it can be beyond 200M (such as Robotank) implying that the agent will learn without injection. Figure 10 gives an overview of how much the weight norm grows over the course of training (suggesting how fast the agent reaches the  $3\| w_0\|$  threshold on each game). We view devising an even more powerful criterion as a promising avenue for future work.
L2 Regularization. The observations about the norm increase made us try adding L2 regularization to the Double DQN agent. A grid search over  $[10^{-7}, 3 \cdot 10^{-7}, 10^{-6}, 3 \cdot 10^{-6}, 10^{-5}, 3 \cdot 10^{-5}]$  coefficients resulted in the best coefficient of  $3 \cdot 10^{-6}$  but leaving the aggregate score mostly the same; higher values resulted in significant performance deterioration. The result gives evidence that controlling the weight norm itself does not address plasticity loss but allows multiple interpretations. We speculate that L2 might be prematurely encouraging weights to have zero magnitude before obtaining high rewards (the effect would be especially profound in sparse reward settings) or that L2 might have undesirable side effects of smoothing approximate value functions while the true value functions might be non-smooth [Dong et al., 2020]. We are puzzled about the inefficacy of L2 in our experiments and mixed results from applying it in RL in past works: the majority of deep RL algorithms do not use it [Mnih et al., 2015, Schulman et al., 2015, Lillicrap et al., 2015, Mnih et al., 2016, Bellemare et al., 2017], although not without exceptions [Schrittwieser et al., 2021]. Some works have explicitly reported negative effects from controlling the weight norm in deep RL [Nikishin et al., 2022], while others highlighted its benefits [Farahmand et al., 2008, Li et al., 2023]; more research is needed to understand its effect in RL.
C Details about the Baselines