Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages
Authors: Guozheng Ma
Lu Li
Sen Zhang
Zixuan Liu
Zhen Wang
Yixin Chen
Li Shen
Xueqian Wang
Dacheng Tao
Venue: ICLR 2024 poster (2024)
Keywords: Plasticity
Visual Reinforcement Learning
Deep Reinforcement Learning
Sample Efficiency
Primary area: reinforcement learning

Abstract: Plasticity, the ability of a neural network to evolve with new data, is crucial for high-performance and sample-efficient visual reinforcement learning (VRL). Although methods like resetting and regularization can potentially mitigate plasticity loss, the influences of various components within the VRL framework on the agent's plasticity are still poorly understood. In this work, we conduct a systematic empirical exploration focusing on three primary underexplored facets and derive the following insightful conclusions: (1) data augmentation is essential in maintaining plasticity; (2) the critic's plasticity loss serves as the principal bottleneck impeding efficient training; and (3) without timely intervention to recover critic's plasticity in the early stages, its loss becomes catastrophic. These insights suggest a novel strategy to address the high replay ratio (RR) dilemma, where exacerbated plasticity loss hinders the potential improvements of sample efficiency brought by increased reuse frequency. Rather than setting a static RR for the entire training process, we propose Adaptive RR, which dynamically adjusts the RR based on the critic’s plasticity level. Extensive evaluations indicate that Adaptive RR not only avoids catastrophic plasticity loss in the early stages but also benefits from more frequent reuse in later phases, resulting in superior sample efficiency.

Published as a conference paper at ICLR 2024
REVISITING PLASTICITY IN VISUAL REINFORCEMENT LEARNING: DATA, MODULES AND TRAINING STAGES
Guozheng Ma²
Lu Li
Sen Zhang²
Zixuan Liu'
Zhen Wang²
Yixin Chen Li Shen Xueqian Wang"! 11
Dacheng Tao
" 'Tsinghua University
"The University of Sydney
3 #Washington University in St. Louis
IJ Explore Academy "Nayyang Technological University, Singapore
www.sssssnnuaa.... www.ggaaa.ed..... cnm sen. zen.zanngeyyy eddu (mqz21, lilu21, 1, xx-1iu21 htt.taaaeeealasiinneeee.e
>wan4121 @uni sydneyy- wyydney.edu.ae adu chengcse. wust1.edo; mathshen1 8gmai c.com
wang. xq85x. inghnnggaaaa odu dacheng taw@nt...do.......
ABSTRACT
Plasticity, the ability neural network to evolve with new data, is crucial for high-performance and sample-efficient visual reinforcement learning (VRL). Al- though methods like resetting and regularization can potentially mitigate plastic- ity loss, the influences of various components within the VRL framework the agent's plasticity are still poorly understood. In this work, we conduct a systematic empirical exploration focusing three primary underexplored facets and derive the following insightful conclusions: (1) data augmentation is essential in main- taining plasticity; (2) the critic's plasticity loss serves as the principal bottleneck impeding efficient training; and (3) without timely intervention to recover critic's plasticity in the early stages, its loss becomes catastrophic. These insights suggest a novel strategy to address the high replay ratio (RR) dilemma, where exacerbated plasticity loss hinders the potential improvements of sample efficiency brought by increased reuse frequency. Rather than setting a static RR for the entire training process, we propose Adaptive RR, which dynamically adjusts the RR based on the critic's plasticity level. Extensive evaluations indicate that Adaptive RR not only avoids catastrophic plasticity loss in the early stages but also benefits from more frequent reuse in later phases, resulting in superior sample efficiency.
INTRODUCTION
The potent capabilities of deep neural networks have driven the brilliant triumph of deep reinforce- ment learning (DRL.) across diverse domains (Silver et al., 2017; Jumper et al., 2021; OpenAI, 2023). Nevertheless, recent studies highlight pronounced limitation of neural networks: they struggle to maintain adaptability and learning from new data after training on a non-stationary objective (Lyle al., 2021; Sokar et al., 2023), challenge known as plasticity loss (Lyle al., 2023; Abbas al., 2023). Since RL agents must continuously adapt their policies through interacting with environ- ment, non-stationary data streams and optimization objectives are inherently embedded within the DRL paradigm (Kumar et al., 2023). Consequently, plasticity loss presents fundamental challenge for achieving sample-efficient DRL. applications (Nikishin al., 2022; D' Oro et al., 2022).
Although several strategies have been proposed to address this concern, previous studies primar- ily focused on mitigating plasticity loss through methods such as resetting the parameters of neu- rons (Nikishin et al., 2022; D' D'Oro et al., 2022; Sokar et al., 2023), incorporating regularization tech- niques (Kumar et al., 2023; Lyle et al., 2023; 2021) and adjusting network architecture (Schwarzer et al., 2023; Dohare al., 2021; Abbas et al., 2023). The nuanced impacts of various dimensions within the DRL framework on plasticity remain underexplored. This knowledge gap hinders more precise interventions to better preserve plasticity. To this end, this paper delves into the nuanced mechanisms underlying DRL's plasticity loss from three primary yet underexplored perspectives: data, agent modules, and training stages. Our investigations focus on visual RL (VRL) tasks that enable decision decision-making directly from high-dimensional observations. As a representative paradigm
"Equal Contribution, 1 CCorrsponning authors.

---

Published as a conference paper at ICLR 2024
of end-to-end DRL, VRL is inherently more challenging than learning from handerafted state inputs, leading to its notorious sample inefficiency (Ma et al., 2022; Yarats et al., 202 Tomar et al., 2022).
We begin by revealing the indispensable role of data augmentation (DA) in mitigating plasticity loss for off-policy VRL algorithms. Although DA is extensively employed to enhance VRL's sample efficiency (Yarats et al., 2020; 2021a), its foundational mechanism is still largely elusive. Our inves- tigation employs a factorial experiment with DA and Reset. The latter refers to the re-initialization of subsets of neurons and has been shown to be a direct and effective method for mitigating plas- ticity loss (Nikishin et al., 2022). However, our investigation has surprisingly revealed two notable findings: (1) Reset can significantly enhance performance in the absence of DA, but show limited or even negative effects when DA is applied. This suggests significant plasticity loss when DA is not employed, contrasted with minimal or no plasticity loss when DA is utilized. (2) Performance with DA alone surpasses that of reset or other interventions without employing DA, highlighting the pivotal role of DA mitigating plasticity loss. Furthermore, the pronounced difference in plastic- ity due to DA's presence or absence provides compelling cases for comparison, allowing a deeper investigation into the differences and developments of plasticity across different modules and stages.
We then dissect VRL agents into three core modules: the encoder, actor, and critic, aiming to identify which components suffer most from plasticity loss and contribute to the sample inefficiency of VRL training. Previous studies commonly attribute the inefficiiecy of VRL training to the challenges of constructing a compact representation from high-dimensional observations (Tomar et al., 2022; Laskin et al., 2020a; Wang et al., 2022; Stooke et al., 2021; Shah & Kumar, 2021). A natural corollary to this would be the rapid loss of plasticity in the encoder when learning from scratch solely based on reward signals, leading to sample inefficiency. However, our comprehensive experiments reveal that it is, in fact, the plasticity loss of the critic module that presents the critical bottleneck for training. This insight aligns with recent empirical studies showing that efforts to enhance the representation of VRL agents, such as meticulously crafting self-supervised learning tasks and pre- training encoders with extra data, fail to achieve higher sample efficiency than simply applying DA alone (Li et al., 2022b; Hansen et al., 2023). Tailored interventions to maintain the plasticity of critic module provide a promising path for achieving samplc-efficient VRL in future studies
Given the strong correlation between the critic's plasticity and training efficiency, we note that the primary contribution of DA lies in facilitating the early-stage recovery of plasticity within the critic module. Subsequently, we conduct comparative experiment by turning on or turning off DA at certain training steps and obtain two insightful findings: (1) Once the critic's 's plasticity has been recovered to adequate level in the early stage, there's 's no need for specific interventions maintain it. (2) Without timely intervention in the early stage, the critic's 's plasticity loss becomes catastrophic and irrecoverable. These findings underscore the importance of preserving critic plasticity during the initial phases of training. Conversely, plasticity loss in the later stages is not a critical concern. To conclude, the main takeaways from our revisiting can be summarized as follows:
DA is indispensable for preserving the plasticity of VRL agents. (Section 3)
Critic's plasticity loss is critical bottleneck affecting the training efficiency. (Section 4) Maintaining plasticity in the early stages is crucial to prevent irrecoverable loss. (Section 5)
We conclude by addressing a longstanding question in VRL: how to determine the appropriate re- play ratio (RR), defined as the number of gradient updates per environment step, to achieve optimal sample efficiency (Fedus et al., 2020). Prior research set a static RR for the entire training process, facing a dilemma: while increasing the RR of off- off-policy algorithms should enhance sample effi- ciency, this improvement is offset by the exacerbated plasticity loss (Nikishin et al., 2022; Sokar et al., 2023; Schwarzer et al., 2023). However, aforementioned analysis indicates that the impact of plasticity loss varies throughout training stages, advocating for adaptive adjustment of RR based on the stage, rather than setting a static value. Concurrently, the critic's plasticity has been identified as the primary factor affecting sample efficiency, suggesting its level as a criterion for RR adjust- ment. Drawing upon these insights, we introduce simple and effective method termed Adaptive RR that dynamically adjusts the RR according to the critic's 's plasticity level. Specifically, Adaptive RR commences with a lower RR during the initial training phases and elevates it upon observing sig- nificant recovery in the critic's plasticity. Through this approach, we effectively harness the sample efficiency benefits of a high RR. while skillfully circumventing the detrimental effects of escalated plasticity loss. Our comprehensive evaluations on the DeepMind Control suite (Tassa et al., 2018) demonstrate that Adaptive RR attains superior sample efficiency compared to static RR baselines.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
In this section, we briefly review prior research works on identifying and mitigating the issue of plasticity loss, as well on the high RR dilemma that persistently plagues off-policy RL algorithms for more efficient applications. Further discussions on related studies can be found in Appendix A.
Plasticity Loss. Recent studies have increasingly highlighted a major limitation in neural networks where their learning capabilities suffer catastrophic degradation after training on non -tatiiono ob- jectives (Sokar et al., 2023; Nikishin et al., 2024). Different from supervised learning, the non- stationarity of data streams and optimization objectives is inherent in the RL. paradigm, necessi- tating the confrontation of this issues, which has been recognized by several terms, including pri- macy bias (Nikishin et al., 2022), dormant neuron phenomenon (Sokar et al., 2023), implicit under- parameterization (Kumar et al., 2020), capacity loss (Lyle et al., 2021), and more broadly, plasticity loss (Lyle et al., 2023; Kumar et al., 2023). Agents lacking plasticity struggle to learn from new experiences, leading to extreme sample inefficiency or even entirely ineffective training.
The most straightforward strategy to tackle this problem is re-initialize part of the network to regain rejuvenated plasticity (Nikishin et al., 2022; D'Oro et al., 2022; Schwarzer et al., 2023). However, periodic Reset (Nikishin et al., 2022) may cause sudden performance drops, impacting exploration and requiring extensive gradient updates to recover. To circumvent this drawback, ReDo (Sokar et al., 2023) selectively resets the dormant neurons, while Plasticity Injection (Nikishin et al., 2024) introduces a new initialized network for learning and freezes the current one as resid- ual blocks. Another line of research emphasizes incorporating explicit regularization or altering the network architecture to mitigate plasticity loss. For example, Kumar et al. (2023) introduces Init to regularize the network's weights back to their initial parameters, while Abbas et al. (2023) em- ploys Concatenated Rel.U (Shang et al., 2016) to guarantee a non-zero gradient. A Although existing methods have made progress in mitigating plasticity loss, the intricate effects of various dimensions in the DRL framework plasticity are poorly understood. In this paper, we aim to further explore the roles of data, modules, and training stages to provide a comprehensive insight into plasticity.
High RR Dilemma. Experience replay, central to off- policy DRL algorithms, greatly improves the sample efficiency by allowing multiple reuses of data for training rather than immediate discarding after collection (Fedus et al., 2020). Given the trial-and-error nature of DRI., agents alternate be- tween interacting with the environment to collect new experiences and updating parameters based on transitions sampled from the replay buffer. The number of agent updates per environment step is usually called replay ratio (RR) (Fedus et al., 2020; D'Oro et al., 2022) or update-to-data (UTD) ratio (Chen al., 2020; Smith et al., 2022). While it's intuitive increase the RR as strategy to improve sample efficiency, doing so naively can lead to adverse effects (Li et al., 2022a; Lyu et al., 2024). Recent studies have increasingly recognized plasticity loss as the primary culprit be- hind the high RR dilemma (Sokar et al., 2023; Nikishin et al., 2022; 2024). Within non-stationary objective, an increased update frequency results in more severe plasticity loss. Currently, the most effective method to tackle this dilemma is to continually reset the agent's parameters when setting a high RR value (D' Oro et al., 2022). Our investigation offers a novel perspective on addressing this ang-standing issue. Firstly, we identify that the impact of plasticity loss varies across training stages, implying a need for dynamic RR adjustment. Concurrently, we determine the critic's plastic- ity as crucial for model capability, proposing its level as a basis for RR adjustment. Drawing from these insights, we introduce Adaptive RR, a universal method that both mitigates early catastrophic plasticity loss and harnesses the potential of high RR in improving sample efficiency.
3 DATA: DATA AUGMENTATION IS ESSENTIAL IN MAINTAINING PLASTICITY
In this section, we conduct a factorial analysis of DA and Reset, illustrating that DA effectively maintains plasticity. Furthermore, in comparison with other architectural and optimization interven- tions, we highlight DA's pivotal role as a data-centric method in addressing VRL's plasticity loss.
A Factorial Examination of DA and Reset. DA has become an indispensable component in achieving sample-efficient VRL applications (Yarats et al., 2020; Laskin et al., 2020b; Yarats et al., 202 Ma et al., 2022). As illustrated by the blue and orange dashed lines in Figure 1, employing simple DA approach to the input observations can lead to significant performance improvements in previously unsuccessful algorithms. However, the mechanisms driving DA's notable effectiveness
3

---

Published as a conference paper at ICLR 2024
remain largely unclear (Ma al., 2024). On the other hand, recent studies have increasingly rec- ognized that plasticity loss during training significantly hampers sample efficiency (Nikishin et al., 2022: Sokar et al., 2023; Lyle et al., 2023). This naturally raises the question: does the remarkable efficacy of DA stem from its capacity to maintain plasticity? To address this query, we undertake factorial examination of DA and Reset. Given that Reset is well-recognized for its capability to mitigate the detrimental effects of plasticity loss on training, it can not only act as a diagnostic tool assess the extent of plasticity loss in the presence or absence of DA, but also provide benchmark to determine the DA's effectiveness in preserving plasticity.
Cheetah Rum
Walker Run
Quadruped Run
200
060
710
720
Eplarde Betaes
500
300
200
200
0.5
1.8
2.0
0.2
0.5
3.5
6.0
1.0
1.5 2.0
2.5
3.5
Enstrorment Steps
Eariinveeeett Staps
Drvtrmettnnt Steps
Whb Beset, Wo DA
wT Reset, wob DA
wib Reset, NJ DA
W0 Reset, Not DA
incorporating excluding Reset and DA. We adopt DrQ-v2 (Yarats et al., 2021a) as our baseline algorithm and follow the Reset settings from Nikishin et al. (2022). Mean and std are estimated over runs. Note that re-initializing 10 times in the Quadruped Run task resulted in poor performance, prompting us to adjust the reset times 5. For ablation studies on reset times and results in other tasks, please refer Appendix B.1.
Figure 1: Training curves across four combinations:
The results presented in Figure highlight three distinct phenomena: e In the absence of DA, the implementation of Reset consistently yields marked enhancements. This underscores the evident plasticity loss when training is conducted devoid of DA. e With the integration DA, the intro- duction of Reset leads to only slight improvements, that applying DA alone can sufficiently preserve the agent's 's plasticity, leaving little to no room for significant improvement. Comparativelly, the performance of Reset without DA lags behind that achieved employing DA alone, underscoring the potent effectiveness of DA in preserving plasticity.
occasionally, decrease. This indicates
Comparing DA with Other Interventions. We
sess the influence of various architectural and optimiza- DMC using the DrQ-v2 frame- work. Specifically, we implement the following tech-
tion interventions
Weggt Decay
niques: Weight Decay, where we set the L2 coefficient to 10 L2 Init (Kumar et al., 2023): This technique integrates L2 regularization aimed the initial param- eters into the loss function. Specifically, we apply it to the critie loss with a coefficient set to 10-2 Layer (Brmalizationn(a et al., 2016) after each convolutional and linear layer. Spectral Normalization (ariuiiiiaaioMMiiaotoo et al., 2018) after the initial linear layer for both the actor and critic networks. o Shrink and Perturb(Ash & Adams, 2020): This involves multiplying the critic network weights by small scalar and adding a perturbation equivalent to the weights of randomly initialized network. e Adoption of CReL. (Shang et al., 2016) as an alternative to ReLU in the critic network. We present the final performance of different interventions in Figure 2, which indicates that DA is the most effective method. For further comparison of interventions, see Appendix B.3.
Layer
Spectral
200
420
600
200
Egisode Retura Aftec 3M Stupo
Figure 2: Performance of various inter-
ventions in Cheetah Run across 5 seeds.
4 MODULES: THE PLASTICITY LOSS OF CRITIC NETWORK IS PREDOMINANT
In this section, we aim to investigate which module(s) of VRL agents suffer the most severe plas- ticity loss, and thus, are detrimental to efficient training. Initially, by observing the differential trends in plasticity levels across modules with and without DA, we preliminarily pinpoint the critic's plasticity loss as the pivotal factor influencing training. Subsequently, the decisive role of DA when using a frozen pre- pre-trained encoder attests that encoder's plasticity loss isn't 't the primary bottleneck for sample inefficiency. Finally, contrastive experiments with plasticity injection on actor and critic further corroborate that the critic's plasticity loss is the main culprit.
4

---

Published as a conference paper at ICLR 2024
Fraction of Active Units (FAU). Although the complete mechanisms underlying plasticity loss remain unclear, a reduction in the number of active units within the network has been identified as principal factor contributing to this deterioration (Lyle et al., 2023; Sokar et al., 2023; Lee et al., 2024). Hence, the Fraction of Active Units (FAU) is widely used as a metric for measuring plasticity. Specifically, the FAU for neurons located in module M, denoted as &.M, is formally defined as:
PM EnEm 1(an (z) > 0)
(I)
N
where an (x) represent the activation of neuron n given the input I. and N is the total number of neurons within module M. More discussion on plasticity measurement can be found in Appendix A.
Different FAU trends across modules reveal critic's plasticity loss as a hurdle for VRL training.
Within FAU as metric, we proceed to assess the plasticity disparities in the encoder, actor, and critic modules with and without DA. We adopt the experimental setup from Yarats et al. (2021a), where the encoder is updated only based on the critic loss. As shown in Figure 3 (left), the integration of DA leads to substantial leap in training performance. Consistent with this uptrend in performance, DA elevates the critic's FAU to a level almost equivalent to an initialized network. In contrast, both the encoder and actor's FAU exhibit similar trends regardless of DA's presence absence. This finding tentatively suggests that critic's plasticity loss is the bottleneck constraining training efficiency.
Quadruped Run
Encoder
Actor
Critic
100
Prrettin of Acttoe Units (1WU)
700
0.3
Egoeeda Returr
600
500
0.4
400
0.3
300
200
100
2.3
9.5
3.0
2.5 3.
Enstromment Steps
3el
Environment Steps
Environment Stops
Envirommen Steps
Iee
Figure 3: Different FAU trends across modules throughout training. The plasticity of encoder and actor displays similar trends whether DA is employed or not. Conversely, integrating DA leads to a marked improvement in the critic's plasticity. Further comparative results are in Appendix B.6.
Is the sample inefficiency in VRL truly blamed on poor representation? Since VRL handles high-dimensional image observations rather than well-structured states, prior studies commonly at- tribute VRL's sample inefficiency to its inherent challenge of learning a compact representation.
We contest this assumption by conducting a simple experiment. Instead of training the encoder from scratch, we employ an ImageNet pre-trained ResNet model as the agent's encoder and retain its parameters frozen throughout the training process. The specific implementation adheres Yuan et al.
Walker Walk
Walker Run
wio DA
700
800
500
Episode Ruturn
300
600
(2022), but employs the DA operation as in DrQ-v2. Building on this setup, we com- pare the effects of employing DA against not using it on sample efficiency, thereby isolating and negating the potential influ- ences from disparities in the encoder's 's rep- resentation capability on training. As de- picted in Figure 4, the results illustrate that employing DA consistently surpasses sce- narios without DA by a notable margin. This significant gap sample inefficiency in VRL cannot be predominantly attributed to poor representation. This pronounced disparity underscores two criti- cal insights: first, the pivotal effectiveness of DA is not centered on enhancing representation; and second, sample inefficiency in VRI. cannot be primarily ascribed to the insufficient representation.
300
200
200
200
0.0
0.2
0.4
0.5
0.8
1.0
0
Enviromment Steps
10s
Esviconmet Steps
1os
Figure 4: Learning curves of DrQ-v2 using frozen ImageNet pre-trained encoder, with and without DA.
Plasticity Injection on Actor and Critic as Diagnostic Tool. Having ruled out the encoder's influence, we next explore how the plasticity loss of actor and critic impact VRL's training efficiency.
5

---

Published as a conference paper at ICLR 2024
To achieve this, introduce plasticity injec- tion as a diagnostic tool (Nikishin et al., 2024). Unlike Reset, which leads to a periodic momen- tary decrease in performance and induces an exploration effect (Nikishin al., 2022), plas- ticity injection restores the module's plasticity its initial level without altering other char- acteristics or compromising previously learned knowledge. Therefore, plasticity injection al- lows us to investigate in isolation the impact training after reintroducing sufficient plasticity to certain module. Should the training perfor- mance exhibit a marked enhancement relative to the baseline following plasticity injection into a particular module, it would suggest that this module had previously undergone catas- trophic plasticity loss, thereby compromising the training efficacy. We apply plasticity injec- tion separately to the actor and critic when us- ing and not using DA. The results illustrated in Figure and Appendix B.4 reveal the subsequent findings and insights: When employing DA, the application of plasticity injection to both the actor and critic does not modify the training performance. This suggests that DA alone is sufficient to maintain plasticity within the Walker Run task. Without using DA in the initial 1M steps, adminis- tering plasticity injection to the critic resulted in a significant performance improvement. This fully demonstrates that the critic's 's plasticity loss is the primary culprit behind VRL's sample inefficiency.
Walker Run
820
wib DA bedor
Wio PI
22 Actor
P1 Critic
Episode Return
230
2.0
Environment Steps
Figure 5: Training curves of employing plasticity injection (PI) on actor or critic. For all cases, DA applied after the given environment steps.
5 STAGES: EARLY-STAGE PLASTICITY LOSS BECOMES IRRECOVERABLE
In this section, elucidate the differential attributes of plasticity loss throughout various training stages. Upon confirming that the critic's plasticity loss is central to hampering training efficiency, closer review of the results Figure underscores that DA's predominant contribution to effec- tively recover the critic's 's plasticity during initial phases. This naturally raises two questions: After recovering the critic's plasticity to an adequate level the early stage, will ceasing interventions to maintain plasticity detrimentally affect training? If interventions aren't 't applied early to recover the critic's plasticity, is it still feasible to enhance training performance later through such measures? To address these two questions, at certain training steps and obtain the following findings: Turning off DA after the critic's plas- ticity has been recovered does not affect training efficiency. This suggests that it is not necessary to employ specific interventions to maintain plasticity in the later stages of training. Turning on DA when plasticity has already been significantly lost and without timely intervention in the early stages cannot revive the agent's 's training performance. This observation underscores the vital importance of maintaining plasticity in the early stages; otherwise, the loss becomes irrecoverable.
conduct a comparative experiment by turning
or turning off DA
Cheetah Run
Walker Run
Quadruped Run
Episcel ftaturm
030
335
450
400
300
200
266
100
Environment Stops
3xe
Eow.ronment
1.05
Enveronment
Bvery Step
Every Step
Stepe
Stepe
Figure 6: Training curves for various DA application modes. The red dashed line shows when DA turned on turned off. Additional comparative results can be found in Appendix B.5.
We attribute these differences across stages the nature of online RL to learn from scratch in boot- strapped fashion. During the initial phases of training. bootstrapped target derived from low-quality and limited-quantity experiences exhibits high non-stationarity and deviates significantly from the

---

Published as a conference paper at ICLR 2024
actual state- actein values (Cetin et al., 2022). The severe non-stationarity of targets induces rapid decline in the critic's ' plasticity (Lee et al., 2024; Li et al., 2022a), consistent with the findings in Figure 3. Having lost the ability to learn from newly collected data, the critic will perpetually fail to capture the dynamics of the environment, preventing the agent from acquiring an effective policy. This leads to catastrophic plasticity loss in the early stages. Conversely; although the critic's plas- ticity experiences a gradual decline after recovery, this can be viewed as process of progressively approximating the optimal value function for the current task. For single-task VRL that doesn't require the agent to retain continuous learning capabilities, this is termed as benign plasticity loss. Differences across stages offer new perspective address VRL's plasticity loss challenges.
METHODS: ADAPTIVE RR FOR ADDRESSING THE HIGH RR DILEMMA
Drawing upon the refined understanding of plasticity loss, this section introduces Adaptive RR to tackle the high RR dilemma in VRL. Extensive evaluations demonstrate that Adaptive RR strikes a superior trade-off between reuse frequency and plasticity loss, thereby improving sample efficiency.
High RR Dilemma. Increasing the replay ratio (RR), which denotes the number of updates per en- vironment interaction, is an intuitive strategy to further harness the strengths of off off-policy algorithms to improve sample efficiency. However, recent studies O'Oro et al., 2022; Li al., 2022a) and the results in Figure 7 consistently reveal that adopting higher static RR impairs training efficiency.
Cheetah Run
Walker Run
Quadruped Run
800
830
660
100
500
800
400
430
0.5
310
RR-1
200
210
100
330
0.5
1.8
2.0
6-5
0.5
1.5
35
3.0
60
0.5
1.0
1.5
2.4
2.5
Enstrorment Stepe
Eovirrmenn
Staps
Dnserement Steps
Figure 7: Training curves across varying RR values. Despite its intent to enhance sample efficiency through more frequent updates, an increasing RR value actually undermines training.
The fundamental mechanism behind this counterintuitive failure of high RR is widely recognized as the intensified plasticity loss (Nikishin et al., 2022; Sokar al., 2023). As illustrated in Figure 8, increasing RR results in a progressively exacerbated plasticity loss during the early
Quadruped Run
66
RR-0.5
RR-2
BR-1
RR-1
BAU of Cattle
stages of training. Increasing RR from 0.5 to 1 notably di-
minishes early-stage plasticity, but the heightened reuse frequency compensates, resulting in marginal boost in
sample efficiency. However, as RR continues to rise, the detrimental effects of plasticity loss become predom- inant, leading to a consistent decline in sample efficiency. When RR increases to 4, even with the intervention of DA, there's no discernible recovery of the critic's plastic- ity in the early stages, culminating in a catastrophic loss. An evident high RR dilemma quandary arises: while higher reuse frequency holds potential for improving sample efficiency, the exacerbated plasticity loss hinders this improvement.
Envirenment Steps
Figure The FAU of critic across vary- ing RR values. A larger RR value leads to more severe plasticity loss.
Can we adapt RR instead of setting a static value? Previous studies addressing the high RR dilemma typically implement interventions to mitigate plasticity loss while maintaining consis- tently high RR value throughout training (D'Oro et al., 2022; Sokar et al., 2023; Nikishin et al., 2024). Drawing inspiration from the insights in Section 5, which highlight the varying plasticity loss characteristics across different training stages, an orthogonal approach emerges: why not dy- namically adjust the RR value based on the current stage? ? Initially, a low RR is adopted to prevent catastrophic plasticity loss. In later training stages, RR can be raised to boost reuse frequency, as the plasticity dynamics become benign. This balance allows us to sidestep early high RR draw- backs and later hamess the enhanced sample efficiency from greater reuse frequency. Furthermore, the observations in Section 4 have confirmed that the critic's plasticity, as measured by its FAU, is the primary factor influencing sample efficiency. This implies that the FAU of critic module can
7

---

Published as a conference paper at ICLR 2024
be employed adaptively identify the current training stage. Once the critic's FAU has recovered to a satisfactory level. it indicates the agent has moved beyond the early training phase prone to catastrophic plasticity loss, allowing for an increase in the RR value. Based on these findings and considerations, we propose our method, Adaptive RR.
Adaptive Replay Ratio
Adaptive RR adjusts the ratio according to the current plasticity level of critie, utilizing low RR in the early stage and transitioning it to high value after the plasticity recovery stages.
Evaluation on DeepMind Control Suite. We then evaluate the effectiveness of Adaptive RR O in improving the sample efficiency of VRL algorithms. Our experiments conducted on six chal- lenging continuous DMC tasks, which are widely perceived to significantly suffer from plasticity loss (Lee et al., 2024). We select two static RR values baselines: Low RR=0.5, which exhibits no severe plasticity loss but has room for improvement in its reuse frequency. e High RR=2, which shows evident plasticity loss, thereby damaging sample efficiency. Our method, Adaptive RR, starts with RR=0.5 during the initial phase of training, aligning with the default setting of DrQ-v2. Sub- sequently, we monitor the FAU of the critic module every 50 episodes, i.e., 10 environment steps. When the FAU difference between consecutive checkpoints drops below minimal threshold (set at 0.001 in our experiments), marking the end of the early stage, we adjust the RR to 2. Figure illustrates the comparative performances. Adaptive RR consistently demonstrates superior sample efficiency compared to a static RR throughout training.
Chestab Run
Walker Run
Quadruped Run
Pplocde Battur
100
800
230
RR-0.5
RR-2 (w/ DAj
Adptire RB (w/ DA
1.5
Environment Steps
Enstronttant Stapa
Earivonment Steps
Hopper Hop
Reacher Hard
Finger Turn Hard
1006
Tipoost Rekmm
3.0
0.00
Envinomment
160
Prstronntent
Envennement Stags
Figure 9: Training curves of various RR settings across 6 challenging DMC tasks. Adaptive RR
demonstrates superior sample efficiency compared both static low RR and high RR value.
Quadruped Run
Encoder
Actor
Critic
Frantoon of Astive Units (1NK3
Epioude Retum
700
044
400
RR-0 pwy DA)
Rit-2
100
Adagetie 3R (w) DAI
Enstrocment Stegs
Sed
Environment Staps
1
Environment
305
Envtronment Stegs
140
Figure 10: Evolution of FAU across the three modules in the Quadruped Run task under different RR configurations. The critic's plasticity is most influenced by different RR settings. Under high RR, the critic's 's plasticity struggles to recover in early training. In contrast, Adaptive RR successfully mitigates catastrophic plasticity loss in the early phases, yielding the optimal sample efficiency.
Through a case study on Quadruped Run, we delve into the underlying mechanism of Adaptive RR, as illustrated in Figure 10. Due to the initial RR set 0.5, the critic's 's plasticity recovers promptly to
Our code is available at: https: /ttpt/ttttttt//g///////gg com/ /uw.//uuzzeggggeeeeeee Ma Adapt i ve- re-Replay HypllayRaae Ratio
8

---

Published as a conference paper at ICLR 2024
considerable level in the early stages of training, preventing catastrophic plasticity loss seen with RR=2. After the recovery phases, Adaptive RR detects a slow change in the critic's FAU, indicating it's nearing peak, then switch the RR to higher value. Within our experiments, the switch times for the five seeds occurred at: 0.9M, 1 1.2M, 1.1M, 0.95M, and 0.55M. After switching to RR=2, the increased update frequency results in higher sample efficiency and faster convergence rates. Even though the critic's FAU experiences a more rapid decline at this stage, this benign loss doesn't t damage training. Hence, Adaptive RR can effectively exploits the sample efficiency advantages of high RR, while adeptly avoiding the severe consequences of increased plasticity loss.
Table 1:
Comparison of Adaptive RR versus static RR with Reset and ReDo implementations. The average episode returns are averaged over 5 seeds after training for 2M environment steps.
Average Episode Return
RR=0.5
RR=2
Adaptive RR
(After 2M Eov Stops)
defaul:
Reset
Relo
defsult
Reset
Rello
RR: 0.So2
Cheetuh Run
828 59
799 :: 26
788 5
793 9
885 + 20
873 173:100 19
880 :: 45
Walker Run
648 107
618
709 7
749 10
734
758
Quadruped Run
379 = 120
593 = 129
371 158
417 110
511 =
47
608 53
784 + 53
We further compare the performance of Adaptive RR with that of employing Reset (Nikishin et al., 2022) and ReDo (Sokar et al., 2023) under static RR conditions. Although Reset and ReDo both effectively mitigate plasticity loss in high RR scenarios, our method significantly outperforms these two approaches, as demonstrated in Table This not only showcases that Adaptive RR can secure a superior balance between reuse frequency and plasticity loss but also illuminates the promise of dynamically modulating RR in accordance with the critic's overall plasticity level as an effective strategy, alongside neuron-leved network parameter resetting, to mitigate plasticity loss.
Evaluation on Atari-100k. To demonstrate the applicability of Adaptive RR in discrete- action tasks we move our evaluation to the Atari-100K benchmark (Kaiser et al., 2019), as- sessing Adaptive RR against three distinct static RR strategies across 17 games. In static RR settings, as shown in Table 2, algorithm per- formance significantly declines when RR in- creases to 2, indicating that the negative impact of plasticity loss gradually become dominant.
Table 2: Summary of Atari- results. Com- prehensive scores are available in Appendix D.
DrQ(c)
ReDo Adaptive RR
Metries
RK=0.5
5 RR-1
RR-2
RR-1
KR:0.5to2
Mean HNS (%)
42.3
41.3
35.1
42.3
55.8
Median HNS (%)
22.6
30.3
26.0
41.6
48.7
# Saperhuman
1
2
# Best
1
3
11
However, Adaptive RR, by appropriately increasing RR from 0.5 to 2 the right moment, can effectively avoid catastrophic plasticity loss, thus outperforming other configurations in most tasks.
7 CONCLUSION, LIMITATIONS, AND FUTURE WORK
In this work, we delve deeper into the plasticity of VRL, focusing on three previously underexplored aspects, deriving pivotal and enlightening insights: + DA emerges as a potent strategy to mitigate plasticity loss. Critic's plasticity loss stands as the primary hurdle to the sample-efficient VRL. Ensuring plasticity recovery during the early stages is pivotal for efficient training. Armed with these insights, we propose Adaptive RR to address the high RR dilemma that has perplexed the VRI. community for a long time. By striking a judicious balance between sample reuse frequency and plasticity loss management, Adaptive RR markedly improves the VRL's sample efficiency.
Limitations. Firstly, our experiments focus on DMC and Atari environments, without evaluation in more complex settings. As task complexity escalates, the significance and difficulty of maintaining plasticity concurrently correspondingly rise. Secondly, we only demonstrate the effectiveness of Adaptive RR under basic configurations. A more nuanced design could further unlock its potential.
Future Work. Although neural networks have enabled scaling RL to complex decision-making scenarios, they also introduce numerous difficulties unique to DRL, which are absent in traditional RL contexts. Plasticity loss stands as a prime example of these challenges, fundamentally stemming from the contradiction between the rial-and-error nature of RL and the inability of neural networks to continuously learn non-stationary targets. To advance the real-world deployment of DRL, it is imperative to address and understand its distinct challenges. Given RL's unique learning dynamics, exploration of DRL-specific network architectures and optimization techniques is essential.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work is supported by STI 2030-Major Projects (No. 20212D0201105). We thank Zilin Wang and Haoyu Wang for their valuable suggestions and collaboration. We also extend our thanks to Evgenii Nikishin for his support in the implementation of plasticity injection. Furthermore, we sincerely appreciate the time and effort invested by the anonymous reviewers in evaluating our work, and are grateful for their valuable and insightful feedback.
REFERENCES
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos Machado. Loss of plasticity in continual deep reinforcement learning. In Conference Lifelong Learning Agents, pp. 620- 636. PMLR, 2023.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in neural information processing systems, 33:3884-3894, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprini arXiv: 160.064400 2016.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http: / //arxiv.orgg//oo/.. abs/ 11881.206111
Edoardo Cetin, Philip J Ball, Stephen Roberts, and Oya Celiktutan. Stabilizing off-policy deep reinforcement learning from pixels. In International Conference on Machine Learning, pp. 2784- 2810. PMLR, 2022.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W Ross. Randomized ensembled double q- learning: Learning fast without model. In International Conference Learning Represen- tations, 2020.
Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.0800 2021.
Pierluca D'Oro, Max Schwarzer. Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Kleventh International Conference on Learning Representations, 2022.
Jaameng Fan and Wenchao Li. Dribo: Robust deep reinforcement learning via multi-vies infor- mation bottleneck. In International Conference on Machine Learning, pp. 6074-6102. PMLR, 2022.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International Con- ference Machine Learning, pp. 3061-3071. PMLR, 2020.
Caglar Gulcehre, Srivatsan Srinivasan, Jakub Sygnowski, Georg Ostrovski, Mehrdad Farajtabar, Matthew Hoffman, Razvan Pascanu, and Arnaud Doucet. An empirical study of implicit regular- ization in deep offline rl. Transactions on Machine Learning Research, 2022.
Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a learning-frog scratch baseline. In International Conference on Machine Learning. pp. 12511-12526. PMLR, 2023.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873)
2021.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Mitos, Blažej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2019.
10

---

Published as a conference paper at ICLR 2024
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under- paadterrramttiizttionnn inhibits data-efficient deep reinforcement learning. In International Conference on Learning Rep- resentations, 2020.
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprini arXiv:2308. 1958, 2023.
Michael Laskin, Aravind Srinivas, and Pieter Abheel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, pp. 5639- 5650. PMLR, 2020a.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances neural information processing systems, 33: 19895, 2020b.
Hojoon Lee, Hanseul Cho, Hyunseung Kim, Daehoon Gwak, Joonkee Kim, Jaegul Choo, Se- Young Yun, and Chulhee Yun. Plastic: Improving input and label plasticity for sample efficient rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. Advances in Neural Information Processing Systems, 33:1 11111199-111911 2020.
Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting. In The Eleventh International Conference on Learning Represen- fations, 2022a.
Xiang Li. Jinghuan Shang, Srijan Das, and Michael Ryoo. Does self-supervised learning really im- prove reinforcement learning from pixels? Advances in Neural Information Processing Systems, 35:30865-300 30881. 2022b.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in rein- forcement learning. In International Conference on Learning Representations, 2021.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity neural networks. In International Conference on Machine Learning. pp. 13190-23211. PMLR, 2023.
Jiafei Lyu, Le Wan, Zongqing Lu, and Xiu Li. Off-policy rl algorithms can be sample-efficient for continuous control via sample multiple reuse. Information Sciences, pp. 120371, 2024.
Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang. Bo Yuan, and Dacheng Tao. A comprehensive survey of data augmentation in visual reinforcement learning. arXiv preprint arXiv:2210.04561, 2022.
Guozheng Ma, Linrui Zhang, Haoyu Wang, Lu Li, Zilin Wang, Zhen Wang, Li Shen, Xueqian Wang, and Dacheng Tao. Learning better with less: Effective augmentation for sample-efficient visual reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. In Advances in Neural Information Processing Systems, 2020.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A univer- sal visual representation for robot manipulation. In Conference on Robot Learning, pp. 892-909. PMLR, 2023.
Thanh Nguyen, Tung M Luu, Thang Vu, and Chang D Yoo. Sample-efficient reinforcement learning representation learning with curiosity contrastive forward dynamics model. In 2021 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 3471-3477. IEEE, 2021.

---

Published as a conference paper at ICLR 2024
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias deep reinforcement learning. In International conference on machine learning, pp. 16828-16847. PMLR, 2022.
Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. In International Conference on Machine Learning, pp. 17359-17371. PMLR, 2022.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictins representations. In International Conference on Learning Representations, 2020.
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agar- wal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level effi- ciency. In International Conference on Machine Learning, pp. 30365-30380. PMLR, 2023.
Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning. In Inter- national Conference Machine Learning. PMLR, 2021.
Wenling Shang. Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In international conference on machine learning, pp. 2217-2225, PMLR, 2016.
David Silver, Julian Schritwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.
Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv: 078660 2022.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference on Machine Learning, pp- 32145-32168. PMLR. 2023.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pp. 0870-9879 PMLR, 2021.
Chenyu Sun, Hangwei Qian, and Chunyan Miao. Celf: A contrastiveccurristty-driee learning framework for sample-efficient reinforcement learning. arXiv preprint arXiv:2205.009.0 2022.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 1801 00690, 2018.
Manan Tomar, Utkarsh Aashu Mishra, Amy Zhang, and Matthew E Taylor. Learning representations for pixel-based control: What matters and why? Transactions on Machine Learning Research, 2022.
Che Wang, Xufang Luo, Keith Ross, and Dongsheng Li, Vrl3: A data-driven framework for visual deep reinforcement learning. Advances in Neural Information Processing Systems, 35:32974- 32988, 2022.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173 2022.
12

---

Published as a conference paper at ICLR 2024
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference on learning representations, 2020.
Denis Yarats, Rob Fergus, Alessandro Lazarie, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2021a.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improv- ing sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 10674-10681, 2021b.
Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, Zhizheng Zhang, and Zhibo Chen. Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34:5276-5289, 2021.
Tao Yu, Zhizheng Zhang, Cuiling Lan, Yan Lu, and Zhibo Chen. Mask-based latent reconstruc- tion for reinforcement learning. Advances in Neural Information Processing Systems, 35:25117-
25131, 2022.
Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. Advances in Neural Information Processing Systems, 35:13022 13002211337, 2022.
Jinhua Zhu, Yingce Xia, Lijun Wu, Jiajun Deng, Wengang Zhou, Tao Qin, Yan-Yan Liu, and Houqiang Li. Masked contrastive representation learning for reinforcement learning, IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2022.
13

---

Published as a conference paper at ICLR 2024
A EXTENDED RELATED WORK
In this section, we provide an extended related work to supplement the related work presented in the main body.
Sample-Efficient VRL. Prohibitive sample complexity has been identified as the primary obsta- cle hindering the real-world applications of VRL (Yarats et al., 2021b). Previous studies ascribe this inefficiency to VRL's requirements to concurrently optimize task-specific policies and learn compact state representations from nigh-dimensional observations. As a result, significant efforts have been directed towards improving sample efficiency through the training of a more potent en- coder. The most representative approaches design auxiliary representation tasks to complement the RL objective, including pixel or latent reconstruction (Yarats et al., 2021b; Yu et al., 2022), future prediction (Lee et al., 2020; Schwarzer et al., 2020; Yu et al., 2021), and contrastive learning for instance (Laskin et al., 2020a; Sun et al., 2022; Fan & Li, 2022) or temporal discrimination (Zhu et al., 2022; Oord et al., 2018; Nguyen et al., 2021; Mazoure et al., 2020). Another approach is to pre-train visual encoder that enables efficient adaptation to downstream tasks (Shah & Kumar, 2021; Xiao et al., 2022; Parisi et al., 2022; Nair et al., 2023). However, recent empirical studies suggest that these methods do not consistently improve training efficiency (Li et al., 2022h; Hansen et al., 2023; Ma et al., 2024). indicating that insufficient representation may not be the primary bot- tleneck hindering the sample efficiency of current algorithms. Our findings in Section provide a compelling explanation for the limited impact of enhanced representation: the plasticity loss within the critic module is the primary constraint on VRL's sample efficiency.
Plasticity Loss in Continual Learning vs. in Reinforcement Learning. Continual Learning (CL) aims to continuously acquire new tasks, referred to as plasticity, without forgetting previously learned tasks, termed stability. A primary challenge in CL is managing the maaility-plastiicn off. Although online reinforcement learning (RL) exhibits characteristics of plasticity due to its non- stationary learning targets, there are fundamental differences between CL and RL. Firstly, online RL typically begins its learning process from scratch, which can lead to limited training data in the early stages. This scarcity of data can subsequently result in loss of plasticity early on. Secondly, RL usually doesn't 't require an agent to learn multiple policies. Therefore, any decline in plasticity during the later stages won't 't significantly impact its overall performance.
trade-
Measurement Metries of Plasticity. Several metrics are available to assess plasticity, including weight norm, feature rank, visualization of loss landscape, and the fraction of active units (FAU). The weight norm (commonly of both encoder and head) serves a dual purpose: it not only acts as a criterion to determine when maintain plasticity but also offers direct method to regulate plas- ticity through L2 regularization (Sokar et al., 2023; Nikishin et al., 2024). However, Nikishin et al. (2024) show that the weight norm sensitive to environments and cannot address the plasticity by controlling itself. The feature rank can be also regarded as proxy metric for plasticity loss (Kumar et al., 2020; Lyle et al., 2021). Although the feature matrices used by these two works are slightly different, they correlate the feature rank with performance collapse. Nevertheless, Gulcehre et al. (2022) observe that the correlation appears in restricted settings. Furthermore, the loss landscape has been drawing increasing attention for its ability to directly reflect the gradients in backpropaga- tion. Still, computing the network's 's Hessian concerning a loss function and the gradient covariance can be computationally demanding (Lyle et al., 2023). Our proposed method aims obtain reli- able criterion without too much additional computation cost, and leverage it to guide the plasticity maintenance. We thus settled on the vicoonzreeogniized and potent metric. FAU, for assessing plas- ticity (Sokar et al., 2023; Abbas et al., 2023). This metric provides an upper limit on the count of inactive units. As shown in Figure 9, the experimental results validate that A-RR based on FAU significantly outperforms static RR baselines. Although FAU's efficacy is evident in various stud- ies, including ours, its limitations in convolutional networks are highlighted by (Lyle et al., 2023). Therefore, we advocate for future work to introduce a comprehensive and resilient plasticity metric.
14

---

Published as a conference paper at ICLR 2024
B EXTENDED EXPERIMENT RESULTS
B.1 RESET
To enhance the plasticity of the agent's network, the Reset method periodically re-initializes the parameters of its last few layers, while preserving the replay buffer. In Figure 11, we present ad- ditional experiments on six DMC tasks, exploring four scenarios: with and without the inclusion of both Reset and DA. Although reset is widely acknowledged for its efficacy in counteracting the adverse impacts of plasticity loss, our findings suggest its effectiveness largely hinges on the hyper- parameters determining the reset interval, as depicted in Figure 12.
Quadruped Walk
Hopper Hop
Reacher Hard
320
100
300
Epoieds Ittur
600
630
230
1.5
2.0
35
3.0
1.0
25
Bastronzzent Stepoi
EariENatAA Stupe
Etrersaaiett Stopa
Cartpole Swingup
Walker Stand
Finger Spin
3000
210
030
100
6.0
0.3
0.2
0.4
0.6
0.3
6.2
0:48
0.5
1.0
6.0
6.2
6.4
6.6
6.3
3.3
Eistrorment Staps
Eaviinomment Staps
Erotememmnnt Steps
who Baree, wio
w/ Reset, wole
wel Roseel
my Ranel,
Figure 11: Training curves across four combinations: incorporating excluding Reset and DA.
Quadruped Run (1 (10x Reset)
Quadruped Run (5x Resel)
100
750
200
Splodd
3m0
2.5
2.0
Eavizzatient Stapa
Desivzzaan: Stopa
Hopper Stand (1 Ox Reset)
Hopper Stand (5x Reset)
0.13
300
200
0.3
Eavteraaeest Staps
Restromman Steps
Wil Smant VSD DA
W/ Dnnort, MA5 DA
Wil Smaoet sf BA
Smeet, Ny)
Figure 12: Learning curves for various reset intervals demonstrate that the effect the reset strongly depends the hyper-parameter that determines the reset interval.
15

---

Published as a conference paper at ICLR 2024
B.2 HEAVY PRIMING PHENOMENON
Heavy priming (Nikishin et al., 2022) refers to updating the agent 10 times using the replay buffer, which collects 2000 transitions after the start the training process. Heavy priming can induce the agent to overfit to its early experiences. We conducted experiments to assess the effects of using heavy priming and DA, both individually and in combination. The training curves can be found in Figure 13. The findings indicate that, while heavy priming can markedly impair sample efficiency without DA, its detrimental impact is largely mitigated when DA is employed. Additionally, we ex- amine the effects of employing DA both during heavy priming and subsequent training. illustrated in Figure 14. The results indicate that DA not only mitigates plasticity loss during heavy priming but also facilitates its recovery afterward.
Cartpole Swingup
Walker Stand
Hopper Stand
Trplode Beaurn
600
8.00
2.0
Stapps
Finger Spin
Chestah Run
Walker Run
800
Tpiortds Betur
800
2.0
204
100
6.3
9.2
3.6
06
1.5
0.6
9.5
3.0
1.5
24
2.5
6.5
1.5
3.0
2.5
3.5
Envinoment Steps
Enstronment Steps
Teviimnment Stags
Haavy Priming (ode DAj
Normal Training (w/n DA)
Heney Priming (w) DA
Normal Tmining (wi DAI
Figure 13: Heavy priming can severely damage the training efficiency when not employing DA.
Cartpola Swingup
Walker Stand
Happer Stand
Tpioots Deaurn
500
283
4.3
0.2
26
1.6
86
6.3
03
12
69
02
944
0.6
1.6
Envinomentt Stepe
Bnstronment Stepe
Envimement Staps
Finger Spin
Cheetah Run
Walker Run
Tgitodd 7447mm
830
283
Environment Steps
Enstronment Steps
Envinoament Stepo
Wol Priming DA Training DA
Priming
Training
Training
Petming w/ Training DA
Figure 14: DA not only can prevent the plasticity loss but also can recover the plasticity of agent after heavy priming phase.
16

---

Published as a conference paper at ICLR 2024
B.3
FURTHER COMPARISONS DA WITH OTHER INTERVENTIONS
Walker Stand
103
base
CReLU
Episoda Reward
0.8
12-Init
w/ LN
0.4
0.2
0.0
0.5
1.0
1.5
2.0
Environment Steps
1e5
Encoder
Actor
Critic
500
500
500
base
w/ CReLU
100
L2-Init
100
w/ LN
w/ DA
Environment Steps
Environment Steps
Environment Steps
Encoder
Actor
Critic
400
300
Woight
200
60
40
0.0
0.5
1.0
1.5
2.0
0.0
1.5
2,0
0.0
0.5
1.0
1.5
2.0
Environment Steps
le5
Environment Steps
165
Environment Steps
les
Encoder
Actor
Critic
0.6
Fraction Aclive
0.5
0.3
0.3
0.2
2,0
0.0
0.0
0.5
Environment Steps
Environment Steps
Environment Steps
Figure 15: We use three metrics Feature Rank, Weight Norm, and Fraction of Active Units (FAU) assess the impact of various interventions on training dynamics.
17

---

Published as a conference paper at ICLR 2024
B.4 PLASTICITY INJECTION
Plasticity injection is an intervention to increase the plasticity of a neural network. The network schematically separated into an encoder Φ(-) and head ha(-). After plasticity injection, the parameters of head 0 are frozen. Subsequently, two randomly initialized parameters, 0f and 02 are created. Here, 8f are trainable and 02 are frozen. The output from the head is computed using the formula h2(2) + hog (z) hog (z), where 2 = 6(x).
We conducted additional plasticity injection experiments on Cheetah Run and Quadruped Run within DMC, as depicted in Figure 16. The results further bolster the conclusion made in Section 4: critic's plasticity loss the primary culprit behind VRL's sample inefficiency.
Cheetah Run
Quadruped Run
Training
Normal Training
Plasticity Injectios to
Plasticity Injection Actor
502
Plasticity Injection to Critsc
Plasticity Injection Crttic
BA
wio DA
w/
win DA
Episode Return
Episode Return
500
C03
800
400
200
Environment Steps
1sc
Environment Steps
Figure 16: Training curves showcasing the effects of Plasticity Injection on either the actor or critic, evaluated on Cheetah Run and Quadruped Run.
B.5 TURNING ON OR TURNING OFF DA AT EARLY STAGES
In Figure 17, we present additional results across six DMC tasks for various DA application modes. As emphasized in Section 5, it is necessary to maintain plasticity during early stages.
Walker Stand
Finger Spin
Hopper Stand
1006
200
Fpioods Retern
500
932
600
600
992
260
1.6
0.0
0.3
Environatent Stage
Faviromment Stagus
Environment Staps
Cheetah Run
Walker Run
Quadruped Run
1200
700
Fpiisnde Teetorn
600
600
300
235
135
0.6
4.5
1.5
2.5
0.6
1.0
2.05
6.5
1.6
2.0
2.5
3.3
Envtronment
Stops
168
Eovtronment Stops
Environment Steps
Stop
Aftor
Sefore Given
Figure 17: Training curves across different DA application modes, illustrating the critical role of plasticity in the early stage.
18

---

Published as a conference paper at ICLR 2024
B.6 FAU TRENDS ACROSS DIFFERENT TASKS
In Figure 18 and Figure 19, we showcase trends for various FAU modules across an additional nine DMC tasks as a complement the main text.
Walker Stand
Encoder
Actor
Critic
1000
Prectoon Active Unite (PAL)
Egiccde Return
200
600
0.3
600
200
0.0
0.44
Envtroamentt Steps
Invironment Stops
Unvtronment Stops
Invironment Steps
Walker Walk
Encoder
Actor
Crilic
1000
Prsctton of active Unite (MAL)
0.6
Tpioods Nettrm
0.3
200
0.0
6.2
Enviromment Staps
Environment Stepe
Environmtent Staps
Environment Steps
Walker Run
Encoder
Actor
Critic
000
prretton of Netive Units (1AA)
0.5
Eqiiodda Return
200
2.0
0.5
1.3
15
2.3
255
3.3
3.5
3.3
1.5
2.0
25
30
03
0.5
3.3
155
2.0
255
3.0
3.0 0.0
0.5
1.0
1.5
3.0
2.5
Enstronment Steps
Environment Steps
Enviromment Steps
Envtronme Stops
Reacher Hard
Encoder
Actor
Critic
1000
W0-DA
Proccton Artive Units (FAU)
Fgoknds Retarm
400
0.2
400
0.
0.5
3.0
2.
2.5
30
6.0
05
2.0
1.3
2.8
2.5
3.0
0.0
2.5
1.0
3.5
200
25
3.
3.0 00
055
2.0
1.5
2.0
2.5
Environment Steps
Environment Staps
Envircoment Steps
Environment Staps
Figure 18: FAU trends for various modules within the VRL agent across DMC tasks (Walker Stand, Walker Walk, Walker Run and Reacher Hard) throughout the training process.
19

---

Published as a conference paper at ICLR 2024
Hopper Stand
Encoder
Actor
Critic
Fraction Actooo Uiits (UAU)
04
0.3
Epiecee Neeturm
660
62
0.4
0.6
1.0
0.2
0.4
0.4
0.9
0.8
1.0
Enstronment Stage
Environment Steps
Environment Stops
Covtronment mt Steps
Hopper Hop
Encoder
Actor
Critic
Pracctonn of Antere Unitts (1AG)
Egoonda Beturn
0:2
84
15
2.3
20
0.5
2.0
Enstrommest Stepa
3ed
Environment Stops
Envizonment Stups
Environme nt Steps
Finger Spin
Encoder
Actor
Critic
1006
Praction Action UnatedPall (PAl)
201 DA
0.5
Egilonda Ntatrm
600
200
0.0
0.2
8.4
8.3
1.
0.2
0.2
0.4
0.6
1.0
Inviroament Stops
Environment Stops
Unntronment Stops
190
Invtronment Steps
Quadruped Walk
Encoder
Actor
Critic
Practoon of Actave Units (13U)
0.4
Epioode Tatumm
600
400
0.3
Cnntronment
3ed
Invironment
145
Environment
305
Dovtronmec
146
Quadruped Run
Encoder
Actor
Critic
Prestion of Antore Unets (INU
Egierde Elaturrs
0.4
0.3
Staps
1m9
Staps
Stegs
Figure 19: FAU trends for various modules within the VRL agent across DMC tasks (Hopper Stand, Hopper Hop, Finger Spin, Quadruped Walk and Quadruped Run) throughout the training process.
20

---

Published as a conference paper at ICLR 2024
Figure 20 provides the FAU for different random seeds, demonstrating that the trend is consistent across all random seeds.
Cheetah Run (w DA) Cheetah Run (w/o DA)
Walker Run (w/ DA)
Walker Run (w/o DA)
Epxide Rature
1
0.45
FA0 C.l00
0.40
2astrommet Stage
Enstennment Stape
Dnmevoment Steps
Dovinvmmen Strps
Quadruped Walk (w/ DA)
Quadruped Walk (wif DA)
Quadruped Quadruped
Epp00c Darturm
Rptaada Dotuzs
1.5
32
Dnstronment Steps
Domiivmment Stape
Envirnmment Steps
Encommment
Hopper Hop Ow/ DA)
Hopper Hop (wil DA)
Reacher Hard (w/ DA
Reacher Hard (wio DA)
350
Ejtaad
0.66
6.40
EMM Cotte
210
3.0
Figure 20: FAU trends for various modules within the VRL agent, evaluated across six DMC tasks and observed for each random seed throughout the training process.
21

---

Published as a conference paper at ICLR 2024
B.7
ADDITIONAL METRICS TO QUANTIFY THE PLASTICITY
Walker Stand
Encoder
Actor
Critic
16.0
Weight Norma
65
75
60
53
40
35
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.
0.0
0.5
1.0
1.5
2.0
Environment Steps
166
Environment Steps
In5
Environment Steps
Encoder
Actor
Critic
5.0
Feature Rank
3000
350
500
200
200
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.5
2.0
0.0
0.5
1.5
2.0
Environment Steps
Environment Steps
1e0
Environment Steps
Quadruped Run
Encoder
Actor
Critic
17.75
1.00
Weight Norta
17:25
100
80
16.75
16:35
16.00
w/ DA
Environment Steps
105
Envirenment Steps
165
Environment Steps
Encoder
Actor
Critic
Feature Raak
130
130
150
100
130
3
Environment Steps
Environment Steps
166
Enviroument Steps
Figure 21: Measuring the plasticity of different modules via feature rank and weight norm.
22

---

Published as a conference paper at ICLR 2024
C
EXPERIMENTAL DETAILS
In this section, we provide our detailed setting in experiments.
C.I ALGORITHM
Algorithm 1 Adaptive RR
Require: Check interval I, threshold total steps T
1: Initialize RL training with a low RR
2: while <
do
3:
if +%I= and | 3 < 7 then
4:
Switch high RR 5: end if
6: Continue RI training with the current RR
7: Increment step &
8: end while
C.2
DMC SETUP
We conducted experiments on robot control tasks within Deep.Mind Control using image input as the observation. All experiments based on previously superior DrQ-v2 algorithms and maintain all hyper-parameters from DrQ-v2 unchanged. The only modification made was to the replay ratio, adjusted according the specific setting. The hyper-parameters presented in Table 3.
Table 3: A default set of
used in DMControl evaluation.
Algorithms Styper-parameters
Replay buffer capacity
106
Action repeat
2
Seed frames
4000
Exploration steps
2000
n-step returns
Mini-batch size
256
Discount ?
0.99
Optimizer
Adam
Learning rate
10 4
Critic Q-function soft-update rate
0.01
Features dim.
50
Repr. dim.
32 x 35 35
Hidden dim.
1024
Exploration stddev. clip
0.3
Exploration stddev. schedule
linear finear(1.0, 0.1, 01. 500000)
23

---

Published as a conference paper at ICLR 2024
D EVALUATION ON ATARI
Implement details. Our Atari experiments and implementation were based on the Dopamine frame- work (Castro al., 2018). For ReDo (Sokar al., 2023) and DrQ(e), We used the same setting as Dopamine, shown in Table 4. we use 5 independent random seeds for each Atari game, The detailed results are shown in Table 5.
Table 4: Hyper-parameters for 100K.
Common Parameter-Dr((6)
Value
Optimizer
Adam
Optimizer: Learning rate
10
Optimizer:
1.5 10 10-4
Training e
0.01
Evaluation e
0.001
Discount factor
0.99
Replay buffer size
10
Minibatch size
32
Q network: channels
32, 64, 64
netwwork: filter size
8 8, 4
4, 3
Q-network: stride
4, 2,
Q-network: hidden units
512
Initial collect steps
1600
n-step Training iterations
10
40
Training environment steps per iteration
10K
ReDo Parameter
Value
Recycling period
1000
7-Dormant
0.025
Minibatch size for estimating neurons score
64
Adaptive RR Parameter
Value
check interval
2000
threshold
0.001
low Replay Ratio
0.5
high Replay Ratio
2
24

---

Published as a conference paper at ICLR 2024
Table 5: Evaluation of Sample Efficiency on Atari-100k. We report the scores and the mean and median HNSs achieved by different methods Atari-100k.
DrQ(c)
DrQ(c)
DrQ(c)
ReDo
Adaptive RR
Game
Human
Random
(RR=0.5)
(RR=1)
(RR=2)
(RR=1)
(RRO.5to2)
Alien
7127.7
227.8
815
865
917
794
935
Amidar
1719.5
5.8
114
138
133
163
200
Assault
742.0
222.4
755
580
579
675
823
Asterix
8503.3
210.0
470
764
442
684
Bank Heist
753.1
14.2
451
232
91
61
553
Boxing
12.1
0.1
16
6
18
Brcakout
30.5
17
Chopper Command
7387.8
1037
1129
1650
Crazy Climber
35829.4
10780.5
18108
21539
17193
24492
22986
Demon Attack
1971.0
152.1
1993
1321
1125
2091
2098
Enduro
861
128
223
138
224
200
Freeway
29.6
Kung FU Master
22736.3
258.5
5342
11467
8423
11642
12195
Pong
14.6
-20.7
-16
-10
3
-6
-9
Road Runner
7845.0
11.5
6478
11211
9430
8606
12424
Seaquest
42054.7
352
SpaceInvaders
1669
148
388
402
408
379
493
Mcan HNS (%)
100
42.3
41.3
35.1
42.3
55.8
Median HNS (%)
22.6
26.0
# Superhuman
N/A
3
Best
N/A
25


Reviews:
[{"review_id": "OzenJ56N6N", "reviewer": "Reviewer_7W3D", "summary": "This paper scrutinizes plasticity loss in visual reinforcement learning (VRL) by exploring data augmentation (DA), agent modules, and training stages. It reveals that DA is pivotal in mitigating plasticity loss, particularly within the critic module of VRL agents, which is identified as the main contributor to the loss of plasticity. To this end, the paper proposes an Adaptive Replay Ratio (RR) strategy that adjusts the RR based on the critic module's plasticity, significantly enhancing sample efficiency over static RR methods, as validated on the DeepMind Control suite.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper offers a novel perspective on the loss of plasticity in a visual reinforcement learning setup which includes:\n\n-  **Analysis of Data Augmentation:**  A significant strength of this paper is its thorough analysis of why data augmentation is so effective in reinforcement learning. By conducting extensive experiments and analysis, the paper sheds light on the underlying mechanisms through which data augmentation enhances plasticity, particularly in visual reinforcement learning settings. This nuanced understanding helps bridge the gap between empirical success and theoretical underpinnings, providing valuable insights for future algorithm design.\n-  **Module-Specific Analysis of Plasticity Loss:** Another strength is the paper's detailed dissection of plasticity loss across different agent modules, namely the encoder, actor, and critic. The authors' methodical comparison uncovers that the critic module is the primary culprit behind plasticity loss, challenging previous assumptions about the encoder's role. \n- **A simple method to tackle the plasticity loss:** By proposing a simple, adaptive method to modulate the update ratio based on plasticity metrics, the paper presents a novel approach that improves upon the static replay ratio. This innovation has the potential to enhance the sample efficiency of diverse reinforcement learning models by aligning training intensity with the model's current capacity to learn.", "weaknesses": "This paper offers interesting insights and experimental results within the domain of visual reinforcement learning (VRL). Despite this, there are aspects that call into question the robustness of the findings:\n\n- **Metrics-Related Concerns:** The primary metric used to assess plasticity loss, the fraction of active units, might be inadequate. The reliance on this metric is questionable since different activation functions such as CReLU and LeakyReLU can inherently bias the number of active units. In addition, as noted by Lyle et al. [1], fraction of active units does not fully encapsulate plasticity dynamics. I recommend that the authors broaden their analysis with additional metrics—like feature rank, weight norms, or even the curvature of the loss surface (e.g., the Hessian's rank)—to substantiate their claims more convincingly. While no single metric may definitively quantify plasticity loss, a consistency of findings across multiple measures would strengthen the empirical foundation of their arguments.\n\n- **Analytical Concerns:** The paper claims the indispensability of data augmentation (DA) in mitigating plasticity loss, but it lacks a comprehensive analysis of why DA is superior. Other methods like layer normalization may have similar effects; what happens when these are combined, as done by Lee et al.? The paper would benefit from deeper exploration into why DA outperforms other methods. Providing a theoretical framework or empirical evidence for DA's efficacy in reducing plasticity loss would be a valuable addition to the field, potentially guiding future development of more robust VRL systems.\n\n- **Applicability Concerns:**  I recognize that calling for additional experiments might seem an unfair request, as it can be levied against any paper. However, I do believe that extending the empirical validation to include at least one more analysis would greatly benefit the paper's credibility. It would be enlightening to see how the findings translate to non-actor-critic methods or across diverse benchmarks. For instance, assessing the proposed adaptive replay ratio method on a well-established sample-efficiency benchmark such as Atari-100k, with Rainbow algorithm, would improve the practical significance of the adaptive RR method. Without such additional validation, the scope and applicability of the conclusions remain in question.\n\nI'm willing to increase the score if the authors address the outlined limitations.\n\n[1] Understanding plasticity in neural networks., Lyle et al., ICML 2023.\n\n[2] PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning., Lee et al., NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NZm0clqzQh", "reviewer": "Reviewer_Kom1", "summary": "This paper addresses the issue of plasticity loss where RL agents struggle to learn from new experiences after being trained on non-stationary data. The authors suggest three key findings. Firstly, they discovered that data augmentation (DA) holds greater significance in alleviating plasticity loss compared to reset. Secondly, their extensive experiments demonstrate that the bottleneck for training lies in the plasticity loss of the critic module. Lastly, restoring the critic's plasticity to an adequate level in the early stages is an important factor. They analyzed plasticity of the neural network for the static high Replay Ratio (RR) during RL agent training and based on their findings, they introduced an adaptive replay ratio which shows superior performance compared to its static counterpart and effectively addresses the high RR dilemma.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe paper is well-written and easy to follow.\n\n-\tTheir suggested method, adaptive RR, is straightforward and intuitive. The results demonstrate promising potential for the proposed approach.\n\n-\tWell-established experiments demonstrated the effectiveness of DA and the importance of Critic's plasticity.", "weaknesses": "-\tTheir proposed methodology, adaptive RR, is based on the analysis of the correlation between replay ratio and plasticity, which is relatively unrelated to the DA. This makes main message of this paper unclear. Are the authors emphasizing the importance of DA or are the authors emphasizing the importance of adaptive RR? The manuscript would benefit if it were to make their main message clearer by aligning their findings and their proposed method. It seems that although the findings regarding the relationship between DA and reset is intriguing, the main method is unrelated with their explanation regarding the effectiveness of DA.\n\n-\tTheir experiments on data augmentation w/ and w/o reset contradict existing research. According to Nikishin et. al. [1], they conducted a comparison between DrQ w/ reset and DrQ w/o reset, finding that the DrQ w/ reset demonstrated superiority over DrQ w/o reset (Check Figure 18 in [1]). Perhaps the results driven by the author are driven by the high variance of data augmentation w/o reset as in [1] or due to the difference in the DMC environments selected for evaluation. I have intentions to raise my score if the authors are able to show experimental results that their findings regarding the relationship between DA & reset also holds for other DMC environments. \n\n-\tIn addition, the current replay ratio values utilized for adaptive RR seems to be limited to a small range (e.g., [0.5, 1, 2]). In order to properly evaluate the efficacy of the proposed adaptive RR method, the authors should conduct further experiments for RR=4 and RR=8. Although higher RR does require longer training time and can be computationally expensive, it is essential for the authors to provide the following results in order to verify whether the proposed adaptive RR is indeed superior compared to static RR. It will also be best if the authors are able to provide results for high RR & reset & DA to further validate their findings that DA is indeed better off along and not paired with reset.\n\n-\tAs the authors note as a limitation, their experimental environment is confined to DMC, and they only demonstrate the effectiveness of Adaptive RR under basic configurations. However, it appears that the paper can provide a promising potential for a new methodology to deal with plasticity loss.\n\n[1] The Primacy Bias in Deep Reinforcement Learning, Nikshin et. al., ICML 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZSIU3HmIh", "reviewer": "Reviewer_m8KG", "summary": "This paper studies the interaction between data augmentation, replay ratios, and dormant neurons in deep reinforcement learning. In particular, it demonstrates a striking similarity in the efficacy of data augmentation and resets in preserving plasticity in deep RL agents, and shows that data augmentation mitigates a decline in the fraction of active units in the network. It further shows that the fraction of active units can also be used to design an adaptive replay ratio scheme which tailors the replay ratio to the phase of learning in the network.", "questions": "- The definition of FAU is a bit unclear: is it measuring average activity of neurons over a large batch, i.e. the sparsity of the network's representation, or is the expression inside the indicator function nonzero if the unit is active for *any* input, i.e. FAU =  1 - (fraction of dormant neurons)?\n- The replay ratio adaptation method is not described in sufficient detail. How specifically is the \"recovery of plasticity\" measured? How robust is the method to different variations on this measurement?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is an enjoyable read. It is very well-written and presented; claims and evidence are clear and easy to follow, and the figures are clearly explained and easy to interpret\n  - The paper offers a fresh take on plasticity -- I wouldn't have predicted that data augmentation would be such an effective regularizer, and this phenomenon is not an obvious conclusion from past literature.\n  - The study of manipulation tasks presents a nice counterpoint to prior work on plasticity which has focused largely on Atari games.\n  - I appreciated the comparisons against a variety of interventions in Figure 2, highlighting the effect size of data augmentation in contrast to other regularizers that had been studied in the literature.\n  - The work is able to ground its evaluation of plasticity not only in the performance of the agents on the RL tasks, but also using the Fraction of Active Units metric, which gives some insight into what is going on inside the network. This is useful because performance-only metrics are unable to disentangle plasticity from other factors in the RL training process such as exploration and generalization.\n  - The idea of adapting the replay ratio based on the stability of the network's training process makes a lot of sense and, despite the potential issues I highlight below with this particular instantiation, could be a generally useful quantity to keep track of.\n  - Section 5, which identifies \"critical periods\" in the training process, is particularly insightful as it suggests that regularization can be modulated throughout training", "weaknesses": "- The paper highlights the FAU as a major indicator of plasticity in the network, but glaringly omits evaluations of ReDO as a baseline. \n    - The domains studied in this paper are all from similar environments, and I'm not sure how much this paper is telling us about general properties of data augmentation, replay ratios, and plasticity, vs how much it is telling us about the interaction between these things in a specific class of robotic manipulation tasks. \n    - The causality around FAU and plasticity is quite vague in this paper. I wasn't sure whether it was claiming that the FAU is a *symptom* of plasticity loss, and that DA and ARR are affecting some hidden causal factor which reduces both plasticity loss and increases FAU, or whether the claim was that *by maintaining the FAU* these interventions are able to avoid plasticity loss.\n    - Data augmentation has the opposite on FAU in the actor and the critic, but it is not explained why a higher FAU would be beneficial in the actor but not the critic. In particular, data augmentation *dramatically* reduces the FAU in the actor, without seeming to have any ill effects. This seems to suggest that FAU might not be a particularly useful measure of plasticity for all learning problems, or at least that the story isn't as simple as that described in the paper.\n    - The specific mechanism by which data augmentation is helping to prevent plasticity loss is unclear. While it is observed to correlate with a reduction in the number of dormant neurons, it isn't obvious whether this is a causal mechanism I would like to see an evaluation of a few data augmentation classes on at least one environment to see how the choice of data augmentation influences its effect on plasticity.\n    - Minor nit: the claim that using a fixed encoder network means that the effect of plasticity on the representation is completely eliminated is too strong, as there is still some representation learning happening in the actor and critic MLPs. This should be noted in the paper text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "89rMrft0Hi", "reviewer": "Reviewer_c6tM", "summary": "This paper studies the plasticity loss problem in visual reinforcement learning. By using Fraction of Active Units (FAU) as a metric to measure the plasticity loss, several observations are concluded:\n- Data augmentation is critical.\n- The critic part in the actor-critic framework suffers from the plasticity loss most.\n- Plasticity loss in early states is irrevocable.\n\nThe authors also propose a technique to mitigate the plasticity loss issue, by using adaptive replay ratio.  Experiments are mainly conducted across 6 DMC tasks, showing the improvements of the proposed technique.", "questions": "See *weakness* above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Nice paper writing**. The motivation, experiments, and conclusions from observations are well stated, making this paper interesting to read.\n- **Good ablations**. The careful experiments on different modules and stages are interesting, which could give some insights into Visual RL.\n- **Interesting measurement**. The select measurement (FAU) is interesting. Though similar things have been well studied in pure deep RL [1,2], this work might be an early work to study the activation units in visual-based deep RL.\n\n[1] Nikishin, Evgenii, et al. \"The primacy bias in deep reinforcement learning.\" ICML 2022.\n\n[2] Sokar, Ghada, et al. \"The dormant neuron phenomenon in deep reinforcement learning.\" ICML 2023.", "weaknesses": "- **Limited insights revealed**. As the analysis in this work shows, the most critical factor for visual RL is **data augmentation (DA)**. However, this has been a very well-known factor for the community, and a quite large amount of recent works have extensively studied DA in visual RL. I think this work mainly gives a new perspective to explain the effectiveness of DA. This is certainly interesting, but considering the metric is directly from the previous works and similar measurements have been also extensively studied, applying FAU simply to a well-known critical factor does not give enough insights to the community.\n\n- **Limited technical contributions**. The only technical contribution of this work is to apply a low replay ratio and a high replay ratio in different stages. This is simple yet not effective enough to support its simplicity and also is not very related to the entire story that this work tells.\n\n- **Limited evaluations**. The experiments are only conducted on 6 DMC tasks, and therefore the diversity in domains and tasks are both limited. It could be good to see the observation and the technique are more universal, considering their simplicity.\n\nThough I like the analysis given in this work, I think this paper is not qualified for the acceptance of ICLR, with the reasons above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)