DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization
Authors: Guowei Xu
Ruijie Zheng
Yongyuan Liang
Xiyao Wang
Zhecheng Yuan
Tianying Ji
Yu Luo
Xiaoyu Liu
Jiaxin Yuan
Pu Hua
Shuzhen Li
Yanjie Ze
Hal Daumé III
Furong Huang
Huazhe Xu
Venue: ICLR 2024 spotlight (2024)
Keywords: Visual RL; Dormant Ratio
Primary area: reinforcement learning
TL;DR: DrM, a visual RL algorithm, minimizes the dormant ratio to guide exploration-exploitation trade-offs, achieving significant improvements in sample efficiency and asymptotic performance across diverse domains.

Abstract: Visual reinforcement learning (RL) has shown promise in continuous control tasks.
Despite its progress, current algorithms are still unsatisfactory in virtually every aspect of the performance such as sample efficiency, asymptotic performance, and their robustness to the choice of random seeds.
In this paper, we identify a major shortcoming in existing visual RL methods that is the agents often exhibit sustained inactivity during early training, thereby limiting their ability to explore effectively. 
Expanding upon this crucial observation, we additionally unveil a significant correlation between the agents' inclination towards motorically inactive exploration and the absence of neuronal activity within their policy networks.
To quantify this inactivity, we adopt dormant ratio as a metric to measure inactivity in the RL agent's network.
Empirically, we also recognize that the dormant ratio can act as a standalone indicator of an agent's activity level, regardless of the received reward signals.
Leveraging the aforementioned insights, we introduce DrM, a method that uses three core mechanisms to guide agents' exploration-exploitation trade-offs by actively minimizing the dormant ratio. 
Experiments demonstrate that  DrM achieves significant improvements in sample efficiency and asymptotic performance with no broken seeds (76 seeds in total) across three continuous control benchmark environments, including DeepMind Control Suite, MetaWorld, and Adroit.
Most importantly, DrM is the first model-free algorithm that consistently solves tasks in both the Dog and Manipulator domains from the DeepMind Control Suite as well as three dexterous hand manipulation tasks without demonstrations in Adroit, all based on pixel observations.

Published as a conference paper at ICLR 2024
DRM: MASTERING VISUAL REINFORCEMENT LEARN- ING THROUGH DORMANT RATIO MINIMIZATION
Guowei Xu-+f
Ruijie Zheng
Yongyuan Liang** Xiyao Wang
Zhecheng Yuan
Tianying Ji
Yu Luo Hal Daumé III Furong Huang² Huazhe 14
Xiaoyu Liu?
Jiaxin Yuan²
Pu Hua
Shuzhen Li
Yanjie Ze
Tsinghua University
University of Maryland, College Park
3 Shanghai Jiao Tong University
Shanghai Qi Zhi Institute
5 "Shanghai AI Lab
ABSTRACT
Visual reinforcement learning (RL. has shown promise in continuous control tasks. Despite its progress, current algorithms are still unsatisfactory in virtually every aspect of the performance such as sample efficiency, asymptotic performance, and their robustness to the choice of random seeds. In this paper, we identify major shortcoming existing visual RL methods that is the agents often exhibit sustained inactivity during early training, thereby limiting their ability to explore effectively. Expanding upon this crucial observation, we additionally unveil a significant corre- lation between the agents' inclination towards motorically inactive exploration and the absence of neuronal activity within their policy networks. To quantify this inac- tivity, we adopt dormant ratio (Sokar et al., 2023) as metric measure inactivity in the RI. agent's 's network. Empirically, we also recognize that the dormant ratio can act standalone indicator of agent's 's activity level, regardless of the ceived reward signals. Leveraging the aforementioned insights, we introduce DrM, method that uses three core mechanisms to guide agents' exploration-exploitatioo trade-offs by actively minimizing the dormant ratio. Experiments demonstrate that DrM achieves significant improvements sample efficiency and asymptotic per- formance with no broken seeds (76 seeds in total) across three continuous control benchmark environments, including DeepMind Control Suite, Meta World, and Adroit. Most importantly, DrM is the first model-free algorithm that consistently solves tasks in both the Dog and Manipulator domains from the DeepMind Control Suite well as three dexterous hand manipulation tasks without demonstrations in Adroit, all based on pixel observations.
Deepmind Control Suite (8 Hard Tasks)
MetalWordd (8 Tasks))
Adroit (3 Tasks)
100%
1025
Telede Rewarr
5:0
8.%%
600
Suucosss Ralte
50%
Succass Ratt
6:0%
40%
200
130
20%
205
098
090
20%
65% Training
10%
100%
20%
100%
Training Progress
Training Progriss
DrM
DrQ
ALIX
TACO
Figure 1: Success rate and episode reward function of training progress for cach the three domains that we consider (Deepmind Control Suite, Meta World, Adroit). All results are averaged over 4 random seeds, and the shaded region stands for standard deviation across different random seeds.
INTRODUCTION
Visual deep reinforcement learning (RL) agents that tackle complex continuous control tasks using high-dimensional pixels are crucial. Recent progress has been made through the incorporation of
"Equal contribution.
1 'Corresponding author. 1xgw230mail 1s, huazhe xuêmai 1 }. ts inghua edu. en Please refer to https: / htt///dmrrr/ .iggtthub. o for experiment videos and benchmark results.

---

Published as a conference paper at ICLR 2024
data augmentation (Yarats et al., 2022; 2021; Laskin et al., 2020a), self-supervised representation learning (Zheng et al., 2023; Laskin et al., 2020b; Stooke et al., 2021; Schwarzer et al., 2021; D Orr et al., 2023), regularization of the temporal difference update (Cetin et al., 2022), and high update- to-data (UTD) ratio (Hiraoka et al., 2022). Nonetheless, the sample efficiency exhibited by these RI. agents remains unsatisfactory. To be more specific, visual RL's inability first appears in the face of complex kinematics and a high number of degrees of freedom (DoFs), such as the Dog and Humanoid tasks in the DeepMind Control Suite (Tassa et al., 2018) or dexterous hand manipulation tasks in Adroit (Rajeswaran et al., 2018) without demonstrations. Second, the current leading visual RL agents might get stuck in the local optimum during the learning process under different initial random seeds. The inability to deal with complex systems and the presence of broken random seeds combined pose significant challenges to deploying visual RL agents in real-world applications.
In this paper, we examine the behaviors of visual RI. agents at different stages of training. Intriguingly, a recurrent issue we identify by observing the learning agents' behavior is that the agents frequently become motorically inactive during the initial phases of training, hindering the effective exploration of useful behaviors. When the agent is experiencing motor inactivity, we find that the policy neural network also possesses high rate of inactive neurons, which is defined as dormant neurons (Sokar et al., 2023) in the literature. As the training progresses, the agents' acquisition of new skills is usually accompanied by decline in the portion of dormant neurons i.e., dormant ratio. Hence, we hypothesize and empirically verify that the dormant ratio acts as an inherent gauge of an agent's activity level, irrespective of the external rewards it receives. Such a connection opens up new path for balancing between exploration and exploitation in RL agents. Remarkably, this pattern of inactivity in motor skills and neurons mirrors the arousal theory (Harrison & W, 2015; Güzel et al., 2020) in neuroscience, which states that an optimal neural network activity level is essential for enhancing attention, memory, and learning efficiency.
Based on this observation and insight. we propose to train visual RL agents with Dormant ratio Minimization (DrM). DrM introduces three simple mechanisms to effectively balance between exploration and exploitation while lowering the dormant ratio: periodical neural network weight perturbation mechanism, a coomannt-ratiobee exploitation mechanism extended from Chen et al. (202la). Consequently, the agent could emphasize exploration when the dormant ratio is high and shift its focus to exploitation when the dormant ratio is low. DrM is easy to implement, computationally efficient, and empirically sample efficient.
exploration scheduler, and a dormant-ratio-biat
DrM is evaluated across three different domains, Deepmind Control Suite (Tassa et al., 2018), MetaWorld (Yu et al., 2019), and Adroit (Rajeswaran et al., 2018), including 19 tasks within the realm of locomotion control, tabletop manipulation, and dexterous hand manipulation. Most notably, DrM is the first documented model-free algorithm that reliably solves complex dog and manipulator tasks, as well as demonstration free Adroit dexterous hand manipulation tasks from pixels. Furthermore, compared with previous state-of-the-ar model-free algorithms, DrM is significantly more sample efficient, especially on tasks with sparse rewards. To be precise, our technique requires 70%, 45%, and 60% fewer samples to match the peak asymptotic performance seen in the three baseline methods on the Deepmind Control suite, MetaWorld, and Adroit, respectively. Moreover, in terms of asymptotic performance, our method exhibits improvements of 65%, 35%, and 75% over the berf-performing baseline on the Deepmind Control suite, Meta and Adroit, respectively.
Below, we summarize our key contributions:
1. Through systematic examinations of the dormant ratio within agents performing continuous control tasks, we establish a crucial insight that a decline in the dormant ratio is an early indicator of successful skill acquisition, even before the increase of reward.
2. We introduce a mechanism that periodically perturbs the model weights of the agent, effectively reducing the dormant ratio and hence accelerating skill acquisition.
3. We additionally design a formant-ratio-based self-adaptive exploration-exploitttio sched- uler that ensures the agent explores when the dormant ratio is high and exploits its past success when the dormant ratio is low.
4. Extensive experiments Deepmind Control Suite, MetaWorld and Adroit show that DrM is particularly adept at handling tasks with sparse rewards or complex dynamics, achieving state-of-the-ar the-art performance against current leading visual RL baselines. DrM is the first.
2

---

Published as a conference paper at ICLR 2024
model-free freel-ffee RL algorithm that can reliably solve complex tasks such Dog, and Manipulator, as well as demonstration- Adroit dexterous hand manipulation tasks directly from pixels.
PRELIMINARY
Visual reinforcement learning. In visual RL (Kaelbling et al., 1998), the landscape is characterized by the inherent challenge of partial observability when dealing with image imputs, which prompts us to approach the problem as Partially Observable Markov Decision Process (POMDP) (Bellman, 1957), encapsulated within the tuple (S,O,A, A,7 P. R. 7). Here, is the state space, O is the observation A + A(S) defines the state transition kernel,
space and A stands for the action space. P where ((S) is distribution over the state space. R : S x A + R denotes the reward function and 7 E [0, 1) represents the discount factor. Starting from an initial state 80 E S, the overarching objective within this framework is to discover an optimal policy x² S - (AA) that maximizes the expected cumulative return, formulated as Ex 12r= rrr=].
Dormant Ratio of Neural Network The notion of dormant neurons, as originally introduced in Sokar et al. (2023), identifies neurons that have become nearly inactive, displaying minimal activation levels. This concept plays an important role in analyzing neural network behavior since networks used in online RL tend to lose their expressive ability.
Definition 2.1. (Sokar et al.. 2023) Consider fully connected layer with N neurons in total. Given an imput distribution D, let (x) denote the output of neuron i in layer under imput x E D. The score of a neuron i is:
ETED (x))
(1)
I N:. Ekei REED (
Then we define a neuron i in layer to be r-dormant if S T.
Definition 2.2. For a fully connected layer I, we denote the number of T -orormnnt neurons as H The r-dormant ratio of a neural network & can be formally defined as follows:
3r = > // N
LEG
IE
METHOD
In this section, begin by discussing key empirical observation: there is connection between the sharp reduction of an agent's dormant ratio and the agent's 's skill acquisition in visual continuous control tasks. This is detailed in Section 3.1. Building on top of this erucial insight, in Section 3.2, we introduce our proposed algorithm DrM In particular, we come with three simple yet effective mechanisms DrM such that they aim to not only reduce the agent's dormant ratio but also utilize the calculated dormant ratio to strike a balance between exploration and exploitation.
3.1 KEY INSIGHT: DORMANT RATIO AND BEHAVIORAL VARIETY
While previous works Lyle al. (2022); Sokar al. (2023) have highlighted that the actor/critic network of RL agents tends to lose expressivity during training, our empirical study offers a unique perspective on visual reinforcement learning for continuous control tasks: the dormant ratio and the agent's 's behavioral variety are correlated.
To illustrate this, we choose DrQ-v2, leading model-free RL algorithm that learns directly from pixel observations. In Figure 2, we display the dormant ratio of an agent's 's policy network, alongside the behaviors learned by the agent during its training on the Hopper Hop task from DeepMind Control depicted in this figure, we notice that a sharp decline in Suite as example. Interestingly, the dormant ratio of an agent's 's policy network serves as an intrinsic indicator of the agent executing meaningful actions for exploration. Namely, when the dormant ratio is high, the agent becomes immobilized and struggles to make meaningful movements. However, as this ratio decreases, observe clear progression in the agent's mobility, demonstrated in the figure: starting with crawling, advancing to standing, and ultimately, hopping. We refer the readers to Appendix A and project webpage for more visualizations of the dormant ratio.
Based on these empirical observations, we conclude that the decline in the dormant ratio is closely linked to the agent's 's initiation of meaningful actions, marking departure from its prior monotonous
3

---

Published as a conference paper at ICLR 2024
Hopper Hop
(Higgh dornant ratio) stuck and immobile
0.3
02
(Mediun dormortt
ratio) crawling slowly
Number Environmant Staps
Dormant Ratin
Epieade Reaaadd
(Low dormant
ratio)
Standing and noving
(Dormant ratio of DrQ-v2 agent trained on Hopper Hop task during the first 1M frames); Interestingly, we find that with declining dormant ratio, the agent incrementally acquires action capabilities. Even though the reward stays minimal during this phase, dormant ratio provides more insightful gauge of the agent's initial learning progress than the reward does.
Figure 2:
or random behaviors. Interestingly, this shift can happen without a corresponding rise in the agent's s rewards. This suggests that the dormant ratio acts as an intrinsic metric, influenced more by the diversity and relevance of the agent's behaviors than by its received rewards, which underscores the value of the dormant ratio as a meaningful metric for understanding the behaviors of visual RL. agents.
Motivated by this insight, we aim utilize dormant ratio as pivotol tool for balancing exploration and exploitation. Many existing strategies adjust exploration noise based on static factors such as task complexity and training stage. Nonetheless, an agent's 's performance can fluctuate across tasks and with different initializations, making adjustments based solely on these static factors less efficient and often mandating exntensive, task-specific fine-tuning hyperparameters. In contrast, customizing exploration noise according to the agent's s current performance offers a more flexible and effective approach. While an intuitive approach would be to rely on reward signals, this strategy brings up the following challenges: 1) Reward values definitions vary across different tasks and domains, necessitat- ing domain-spedne knowledge for interpretation and hyperparameter tuning. 2) Even within specific task, rewards might not indicate the agent's 's underlying learning phase. As depicted in Figure 2, an agent can attain similar rewards regardless of whether has mastered motion or remains stagnant.
In light of this, the dormant ratio emerges more effective metric for adjusting the exploration and exploitation tradeoff, as it faithfully reflects the dynamic changes in the agent's behavior. Our design of DrM follows this simple intuition: a higher dormant ratio suggests the need for increased exploration, whereas a lower ratio calls for exploitation. As the dormant ratio captures the intrinsic characteristics of agent's 's policy and behaviors, DrM is demonstrated to be effective across diverse tasks and domains with minimal hyperparameter tuning required.
3.2 DRM: VISUAL REINFORCEMENT LEARNING THROUGH DORMANT RATIO MINIMIZATION
As shown in the previous subsection, given fixed network capacity, it is essential for visual RL agent to actively reduce its dormant ratio, thereby enabling it to explore the environment through purposeful actions. Driven by this insight, introduce the three mechanisms of our proposed DrM algorithm detail.
perturbation. The goal of this mechanism is perturb the model weights when the RL agent's 's network displays a high dormant ratio, losing its expressivity. Here, we utilize the perturbation reset method (D' Oro et al., 2023; Ash & Adams, 2020) that employs soft resets, process that interpolates all the agent's parameters between their prior values and randomly initialized values. This can be expressed with the following equation:
Dormant-ratio-guilee
0, = ab,-1 + (1 - a)Φ, ~~ initializer
Here, a is referred to as the perturb factor, 0.-1 indicates the network weights before the reset, 0, is the network weight after the reset, and á is randomly initialized weights. Note that this is fundamentally different from the approach of NoisyNet (Fortunato et al., 2018b), which is designed to encourage exploration by injecting noise into the model weights at every timestep. Our goal here is to refresh the dormant weights only after a relatively long time interval (every 20+5 frames). The value of is controlled by the dormant ratio 3: clip(1 kBB Ominn Owar), where is the perturb rate.
4

---

Published as a conference paper at ICLR 2024
Awaken exploration scheduler. We aim to emphasize exploration with large exploration noise when the dormant ratio is high, and reduce the exploration noise when the dormant ratio is low. Thus, rather than utilizing the linear decay of exploration noise variance in the original DrQ-v2, we introduce a dormant-ratio-basee awaken exploration scheduler. Specifically, let / denote a low dormant ratio threshold. We define the agent as "awakened" when its dormant ra- tio is below B. Let to be the number of timesteps until the agent becomes "awakened" from the start of training. The standard deviation of the exploration noise, (t), is then defined as:
Awoken Exploration Scheduler
of Exploration Noise
1+exp (-B8-)/T)) Ainear (1 - 100 if awakened
max
a(t)
0.6
otherwise
(rexp(-(0-)/T)
(4)
0.
Here, T is the exploration temperature hyperparameter. Tinear (-)
0.2
24
05
0.5
is the linear schedule of exploration noise defined in DrQ-v2.
Ratio
We visualize the awaken exploration scheduler in Figure as
Figure 3: Visualization of the awaken exploration scheduler as a function of the dormant ratio 3
a function of the dormant ratio. Initially, when the dormant ratio is high, ration noise to encourage effective exploration of the environment. As training progresses and the dormant ratio decreases to relatively low level (below the threshold 9). this indicates that the agent should transition from exploration to exploitation.
would like to give the agent big explo-
Explot Lation Parareter
Dormant-ratio-uuide exploitation. Furthermore, we introduce an an- other mechanism that skillfully prioritizes exploitation when the dormant ratio low. For continuous control tasks using actor-critic algorithms, the critic aims to approximate r(s, a) + +*Q(s', (ss)). In Ji et al. (2023), it demonstrates that value underestimation often occurs in the early stages of training, when the replay buffer could contain scarce high-quality
Exploitotion Porureter
episodes that the agent has encountered through exploration. In this train- ing stage, is suboptimal, and the Q-value is often underestimated due to insufficient exploitation of high-quality samples in the replay buffer. To address this, proposes to approximate high expectile of Q values with V function using expectile regression, making the new target value
Dommant Rotio
Figure 4: Visualization of exploitation hyperparameter as a function of the dormant ratio S
r(s, a) + (s' + (1 - A)Q(s' ( X E [0.
(5)
As V converges more rapidly than Q-values, this mechanism allows the RL agent to quickly exploit its historically successful trajectories without introducing additional overestimation. Here, À serves the exploitation hyperparameter. Higher values of X focus more on exploiting past successes through the fitted V function, the value of the best actions in that state. This emphasis exploitation in context refers to utilizing the V function to extract more value from historical experiences, aligning with its traditional usage of maximizing rewards based on known information. We introduce dormant- matio- exploitation technique A, which is now defined as a function of the dormant ratio 33:
X
x(3) 1 + exp((3 B)/T")
Here, Ā is the maximum exploitation hyperparameter, and T' is the exploitation temperature hyperparameter. 3 and 9 represent the dormant ratio and its threshold, as previously defined. In Figure 4, we plot the exploitation hyperparameter À as a function of the dormant ratio B. When the agent's dormant ratio exceeds the threshold B. lower À is selected to emphasize exploration. Conversely, when the dormant ratio is low, indicating the agent can perform meaningful actions, higher À chosen prioritize exploitation.
4 EXPERIMENT
In this section, we evaluate DrM on three visual continuous control benchmarks for both locomotion and robotic manipulation: DeepMind Control Suite (Tassa et al., 2018), Meta World (Yu et al.,
5

---

Published as a conference paper at ICLR 2024
(a) Deep.Mind Control Suite
(b) MetaWorld
(c) Adroit
Figure Three visual continuous control benchmarks evaluate our proposed algorithm: Deep.Miid Control Suite, MetaWorld, and Adroit.
2019), and Adroit (Rajeswaran et al., 2018). These environments feature rich visual elements such as textures and shading, necessitate fine-gra one-grained control due to complex geometry, and introduce additional challenges such as sparse rewards and high- gigh-dimensioall action spaces that previous visual RL algorithms such as DrQv2 (Yarats et al., 2022) have been unable to solve.
Baselines. We compare our algorithm with the three strongest existing model-free visual RL algorithms: DrQ-v2 (Yarats al., 2022), ALIX (Cetin al., 2022), and TACO (Zheng al., 2023). ALIX and TACO build upon DrQ-v2. ALIX adds adaptive regularization to the encoder's gradients to stabilize temporal difference learning from encoders. TACO incorporates an auxiliary temporal action-driven contrastive learning objective to learn state and action representations.
Deep.Mind control suite. For Deepmind Control Suite, we evaluate DrM on eight hardest tasks from the Humanoid, Dog, and Manipulator domain, well as Acrobot Swingup Sparse The Manipulator domain is particularly challenging due to its sparse reward structure and the long horizon required for skill acquisition, while Humanoid and Dog tasks feature intricate kinematics, skinning weights, collision geometry, as well as muscle and tendon attachment points. This complexity makes these domains extremely difficult for algorithms to learn control effectively. Following the experimental procedure described by Yarats et al. (2022), we evaluate DrM and all baseline algorithms over 30 million frames of online interaction, while Acrobot Swingup Sparse was run for million frames. Intriguingly, in four dog tasks, we observe that existing baselines encounter a sudden performance decline for some random seeds. We have confirmed this not due to the checkpoint loading mechanisms, and in contrast, DrM does not exhibit this issue in any of the four tasks. As shown Figure 6, we note that DrM the first documented model-free visual RI. algorithm that is capable of solving both Dog and Manipulator domains in the DeepMind Control Suite using pixel observations. Additionally, we notice that the variation across different random seeds, as indicated by the shaded areas in results, is considerably smaller for DrM compared to baseline algorithms. This reduced variation implies that DrM is more robust to different random initializations. In contrast, baseline algorithms frequently experience issues with broken seeds, where the agent fails to acquire any meaningful behaviors and receives consistently low rewards throughout the training process.
Dog Run
Dog Trot
Dog Walk
Dog Stand
Eplocee fowweedd
150
100
200
200
3R7 10t 15T Zhn 25m 33m
177
370
2007
25T
20
25th 3017
37 5m 10
1ST 2 25th 2077
371
35t
13t
201
25 367
Manipulator Bring Ball
Humancid Run
Humanoid Walk
Acrebot Swingup Sparse
800
600
Episade Reqwerd
600
90
330
100
200
3t1 1tth 15th
10m 35
33
30
370 10m 155
25th 25mh 30m
3m
5m
15th 15m
2im
31 23m 30m
27 30
3m
ST.
DrM
DrQ
ALIX
TACO
Figure 6: Performance of DrM against baseline algorithms DrQ-v2. ALIX, and TACO on Deepmind Control suite. All results are averaged over 4 random seeds, and the shaded region stands for standard deviation across different random seeds.
Meta World. As shown in Figure/ 7, we evaluate DrM and baselines on eight challenging tasks including 4 very hard tasks with dense rewards following prior works and 4 medium tasks with sparse

---

Published as a conference paper at ICLR 2024
success signals. Consistently across the spectrum of tasks within MetaWorld, our method outperforms other visual RL baselines, which demonstrates the significantly improved sample efficiency of DrM Especially in more challenging scenarios featuring only sparse task completion rewards, existing visual RI. baselines struggle to find a good policy, while DrM shines by achieving success rates on par with those using dense reward signals. This underscores the remarkable advantages brought by formant-ratio-based exploration when dealing with tasks with sparse rewards.
Asscmbly (dense)
Stick Pull (dense)
Pick Placee Wall (dense)
Disassomble (densc)
1034
130%
1004
10%
50%
40%
409
20%
274
0%
050
014
Coffee (sparse)
(sparse)
Sweep (sparse)
Hammer (sparse)
100%
100%
100%
50%
62%
60%
60%
M
40%
10%
20%
29
0%
0%
Number of Framess
Number of Framnes
Number of framms
Number of Framis
DrQ-VZ
Figure 7: Success rate for DrM and baselines and 4 medium tasks with spare rewards. All results aggregated 4 random seeds, with shaded areas representing the standard deviation across seeds. Notably, our method demonstrates significantly higher sample efficiency, especially in tasks with sparse rewards.
World including very hard tasks with dense rewards
Hammer
Door
Pen
100%
100%
80%
80%
80%
Success Rate
50%
40%0
40%
40%
0%
0%
0%
17A
200
4m
0m
271
210
31t
4m11
5m
mm
1m
2m
20
6m
Frames
DrM
DrQv2
ALIX
TACO
Figure Performance DrM against baseline algorithms DrQ-v2. ALIX, and TACO Adroit. results are averaged over random seeds, and the shaded region stands for standard deviation across different random seeds.
Adroit. In Figure 8, we also evaluate DrM on the Adroit domain, focusing on three dexterous hand manipulation tasks: Hammer, Door, and Pen, which requires controlling robotic hand with 24 degrees of freedom. For additional task details, we refer readers Rajeswaran et al. (2018). Given the task's high-dimensional action space and intricate physics, previous reinforcement learning algorithms have faced significant challenges, especially when learning from pixel observations. Notably, DrM is the first documented model-free visual RL algorithm that is capable of reliably solving tasks in the Adroit domain without expert demonstrations.
Acrabot Swingup Sparse
Humanaid Run
Manipulator Bring Ball
0.8
0.8
Dormant Reti
2..6
Dormant Ratio
0.66
Dormant Ratio
0.4
04
64
0.2
0.2
6m
1.2m
2.4m
3 6m
4.8m
6m
om
2m
4m
6'm
8m
10mm
om
2m
4m
6m
8m
10m
Number of Environment Steps
Number of Environment Steps
Number of Envirorment: Steps
DrQ+2
TACO
Figure 9; Dormant ratio our method DrM and other baseline algorithms on three DMC tasks.
7

---

Published as a conference paper at ICLR 2024
Dormant Ratio Analysis In this section, conduct detailed analysis and comparison of the dormant ratio changes during the training process of DrM and three baseline algorithms. We carry out experiments in three visual DMC tasks, and the experimental results are shown in Figure 9. From this figure, we observe that as training progresses, the dormant ratio of DrM rapidly decreases, indicating that our method effectively minimizes the dormant ratio. In comparison, other exisiting baselines all fail to effectively reduce the dormant ratio. This also explains why our approach exhibits high sample efficiency and performance.
Ablation Study We conduct ablation studies on the Adroit environment to evaluate the contribution of each component to our method. i.e.. dormant-ratio-guinn -guiddd perturbation, awaken exploration, and dormant-ratio-guinn exploitation. Additionally, to show that the dormant ratio plays crucial role in integrating these three components, we also compare with a baseline where we use fixed parameters for the three mechanisms without being guided by the dormant ratio. (i.e., Drg perturbation with perturb factor a fixed, fixed linear exploration schedule, and Drg exploitation with exploitation parameter À fixed.)
The experiment results are shown in Figure 10, From the results, we find that all three components are necessary to achieve the best results.
Adroit Tasks)
We observe that after removing the dormant-ratio-guidee ex- ploitation (DrM w/o Drg Exploitation), the final success rate decreased by nearly 20%, while eliminating either the dormant- ratio-guides guided perturbation (DrM w/o Drg Perturbation) or the awaken exploration (DrM w/o Awaken Exploration) lead to decline of close 40%, highlighting the importance of each component. In our ablated version without dormant-ratio- guided perturbation, the agent only converges suboptimal policy, reaching a success rate of just about 40%. This is likely due to the fact that without the awaken exploration, the agent lacks sufficient exploration, making it easy to get stuck in a sub-optimal policy. Additionally, when removing the dormant- antio-guided exploitation component, the agent lacks the ability to exploit its past success, and there fore exhibits significantly slower learning curve.
DrM
DrM w/o Drg Perturaation
80%
wask Exploration
DrM wes Dig Explsiittion
Siccess Ratee
60%
DrQ V2 Fied Mecbarionr
40%
20%
0%
0m
1.5m
3m Environment Steps
Figure 10: (Ablation) Aggregated re- sults over Adroit domain. Drg repre- sents Dormant-ratio-guu guided.
RELATED WORK
Visual reinforcement learning. Visual reinforcement learning (RL) faces substantial challenges when training agents to make decisions based on pixel observations. Within this domain, two primary categories of approaches have emerged: model-based and model-free methods. Model-based methods (Hansen et al., 2022; Hafner et al., 2020; 2021; ;; 2019; Lee et al., 2020; Hafner et al., 2023) accelerate visual RL by learning world models of the environment. On the other hand, model-free methods have made significant strides in improving data efficiency. These advancements include auxiliary losses, such as the contrastive objective in CURL (Laskin et al., 2020b), ATC (Stooke et al., 2021) for state representations, TACO (Zheng et al., 2023) for learning state and action representations through mutual information, and self-prrdiction representations in SPR (Schwarzer et al., 2021) and SR-SPR (D'Oro et al., 2023). Data augmentation techniques, exemplified by RAD (Laskin et al., 2020a), DrQ (Yarats et al., 2021), and its enhanced version DrQv2 (Yarats et al., 2022), have been instrumental in enabling robust learning directly from pixel data, effectively bridging the gap between state-based and image-based RL. Additionally, regularization methods such as A-LIX (Cetin et al., 2022) have been introduced mitigate catastrophic self-ovvrfitting by providing adaptive regularization to convolutional features. Furthermore. strategies such as scaling network sizes (Schwarzer et al., 2023), high update-to-data (UTD) ratios (D'Oro et al., 2023) and ensemble Q (Chen et al., 2021b; Hiraoka et al., 2022) have been explored to enhance sample efficiency in visual RL. TD-MPC (Hansen et al., 2022) merges the advantages of model-based and model-free methods through temporal difference learning. V-MPO (Song et al., 2020b), an on-policy adaptation of MPO (Song et al., 2020a), exhibits high asymptotic performance on challenging pixel-control tasks (Tassa al., 2018). These various techniques collectively represent the oftte-of-the-art in visual RL, addressing the multifaceted challenges associated with decision maainng from raw visual input. However, our proposed framework differs in that we address the sample efficiency challenge from
8

---

Published as a conference paper at ICLR 2024
the perspective of dormant ratio. We propose more effective DrM that achieves superior performance than prior model-free baselines.
Loss of expressivity of deep RL. In deep RL, there is growing body of evidence suggesting that neural networks tend to lose their capacity and expressiveness for fitting new targets over time and ultimately harm their final performance. To alleviate this issue, Lyle et al. (2022) and Kumar et al. (2021) primarily focus on adjusting the learned feature values. Nikishin et al. (2022) shed light on the primacy bias when training on early data, which can impede further learning progress. Their proposal involves periodic parameter reinitialization for the last few layers while keeping the replay buffer unchanged. Lyle et al. (2023) aims to identify that the loss of plasticity fundamentally influenced by the curvature of the loss landscape. Additionally, the dormant neuron phenomenon, as demonstrated by Sokar et al. (2023) prompts the development of ReDo, a method aimed at reducing dormant neurons and preserving network expressivity during training. Nikishin et al. (2023) introduces plasticity injection, a minimalistic intervention that temporarily freezes the current network and leverages newly initialized weights to facilitate continuous learning. These diverse approaches collectively address the issue of expressivity loss in deep RL, offering insights and methods to enhance computational efficiency and continual learning capabilities in deep RL algorithms. In our paper, we leverage the dormant ratio to gain valuable insights and interpretability into agent behavior in visual RL. We introduce novel perturbation technique and exploration strategy based on the dormant ratio for addressing visual continuous control tasks.
Exploration in RL. Efficient exploration remains substantial challenge in online RL, particularly in high-dimensional environments with sparse rewards. Based on different key ideas and principles, exploration strategies can be classified into two major categories. The first category is uncertainty- oriented exploration (Jin et al., 2020; Ménard et al., 2021 aa;;; Kaufmann et al., 2021; Wang et al., 2023), which often employs techniques such as the upper confidence bound (UCB) (Auer, 2002) to capture value estimate uncertainty to guide exploration. Another category is intrinsic motivation- oriented exploration, which encourages agents to explore by maximizing intrinsic rewards. These rewards are often based on prediction errors (Houthooft et al., 2016; Pathak et al., 2017; Burda et al., 2019; Sckar et al., 2020; Badia et al., 2020) or count-based state novelty (Bellemare et al., 2016; Tang et al., 2017; Ostrovski et al., 2017), motivating the agent to visit states with high prediction errors or the unexplored states. A close idea is exploration by maximizing state entropy as an intrinsic reward (Lee et al., 2019; Hazan et al., 2019; Mutti et al., 2022; Yang & Spaan, 2023). Exploration methods have proven effective in enhancing sample efficiency in vision-based RL. RE3(Seo et al., 2021) utilizes a fixed random encoder to obtain a stable state entropy estimate, along with a value-conditional extension proposed in Kim et al. (2023). MADE (Zhang et al., 2021) introduces an adaptive regularization that maximizes deviation from explored regions, while BEE (Chen et al., 2021a) leverages past successes to capitalize on fortuitous circumstances. Closely relevant techniques involve injecting noise into action (Wawrzynski, 2015; Lillicrap et al., 2016) or parameter spaces (Rückstieß et al., 2010; Sehnke et al., 2010; Fortunato et al., 201 8a; Plappert et al., 2018). Furthermore, strategies that dynamically adjust exploration noise based on factors like agent performance, environmental complexity, and training stage have shown promise in Amos et al. (2021); Yarats et al. (2022). Our method distinguishes itself by directly perturbing the model weights of the agent to reduce the dormant ratio and design dormant-ration -atmann-ratio-uudee -giide exploration technique to improve exploration efficiency.
6 CONCLUSION
In this paper, we introduce a highly efficient online RL algorithm, DrM, which resolves the most complex visual control tasks that previous models failed to tackle, setting a new benchmark in both sample and time efficiency. Looking ahead, we perceive two main avenues for future RL. exploration research. Firstly, the dormant ratio's 's interpretability is a captivating aspect, and subsequent research could delve into why it has a significant correlation with the diversity and significance of an agent's 's action, from theoretical standpoint. Secondly, the dormant ratio delivers more precise depiction of an agent's 's early learning outcomes compared to rewards, it could be used in unsupervised RL. Additionally, although this work primarily concentrates mechanisms we propose for DrM could well be adapted for discrete action tasks on DQN/Efficient Rainbow algorithms with some minor adjustments. We are confident that the dormant ratio's 's value
continuous control, the three key
9

---

Published as a conference paper at ICLR 2024
extends beyond our current understanding, and that its strategic application could greatly enhance the performance of future visual reinforcement algorithms.
REFERENCES
Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. In LADC, volume 144 of Proceed- ings of Machine Learning Research, pp. 6-20. PMLR, 2021. 9
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in neural
information processing systems, 33:3884-3894, 2020. 4
Peter Auer. Using confidence bounds for exploiatiion-exploraiion trade-offs. Mach. Learn, Res., 397-422, 2002. 9
Adrià Puigdomenech Badia, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. In ICLR. OpenRe- view.net, 2020. 9
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In NIPS, pp. 1471-1479, 2016.
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pp- 679-684, 1957. 3
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In ICLR (Poster). OpenReview.net, 2019.
Edoardo Cetin, Philip J. Ball, Stephen J. Roberts, and Oya Çeliktutan. Stabilizing off-policy deep reinforcement learning from pixels. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 2784-2810. PMLR, 2022. 2, 6,
Annie S. Chen, HyunJi Nam, Suraj Nair, and Chelsea Finn. Batch exploration with examples for scalable robotic reinforcement learning. IEEE Robotics Autom. Letr., 643):400144408, 2021a. 2,
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q leaarnnn Learning fast without a model. In ICLR. OpenReview.net, 2021b. 8
Pierluca D' D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G. Bellemare, and Aaron C. Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In ICLR. OpenReview.net, 2023. 2, 8
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Rémi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In ICLR (Poster). OpenReview.net, 2018a. Sa. 9
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference on Learning Representations, 2018b. URL https: http://ppnnneeeeeeee n http://ntt/totur/////// hwt.iiirrryyaaara. KAW. 4 wttpet//oorrmiirrrrrctkkaaaaaaa
Özlem Güzel, IIker Sahin, and Chris Ryan. Push-motivatioo--aseed emotional arousal: A research study in a coastal destination. Journal of Destination Marketing & Management, 100428, 2020.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML,, volume 97 of Proceedings of Machine Learning Research, pp. 2555-2565. PMLR, 2019.
Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR. OpenReview.net, 2020. 8
10

---

Published as a conference paper at ICLR 2024
Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In ICLR. OpenReview.net, 2021. 8
Danijar Hafner. Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.0410 2023. 8
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmen- tation. In International Conference on Robotics and Automation, 2021. 24
Nicklas Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 8387-8406. PMLR, 2022. 8
Harrison and David W. Arousal theory. Brain Asymmetry and Neural Systems: Foundations in Clinical Neuroscience and Neuropsychology, pp. 427-427-444 435, 2015. 2
Elad Hazan, Sham M. Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In ICML, volume 97 of Proceedings of Machine Learning Research, pp. 2681-2691. PMLR, 2019.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. ICLR. OpenReview.net, 2022.
2
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: variational information maximizing exploration. In NIPS, pp. 1109-1117, 2016. 9
Tianying Ji, Yu Luo, Fuchun Sun, Xianyuan Zhan, Jianwei Zhang, and Huazhe Xu. Seizing serendip- ity: Exploiting the value of past success in off off-policy actor-critic. arXiv preprint :xXi:23306002855, 2023. 5
Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 4870-4879. PMLR, 2020. 9
Leslie Pack Kaelbling, Michael Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134 1998. 3
Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Edouard Leurent, and Michal Valko. Adaptive reward-free exploration. In ALT, volume 132 of Proceedings of Machine Learning Research, pp. 865-891. PMLR, 2021. 9
Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Accelerating reinforcement learning with conditionn state entropy exploration. arXiv preprint arXiv:2305. 19476, 2023.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit mnder-parameterizzttonn inhibits data-efficient deep reinforcement learning. ICLR. OpenReview.net, 2021. 9
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In NewrIPS, 2020a. 2,
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: contrastive unsupervised representations for reinforcement learning. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 5639-5650. PMLR, 2020b. 2,8
Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In NeurlPS, 2020.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. 052744 2019. 9
Efficient exploration via state marginal matching. arXiv preprint
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
11

---

Published as a conference paper at ICLR 2024
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In ICLR. OpenReview.net, 2022. 3, 9
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Ávila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In ICML., volume 202 of Proceedings of Machine Learning Research, pp. 23190-23211 13211. PMLR, 2023. 9
Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Emilic Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 7599-7608. PMLR, 2021a. 9
Pierre Ménard, Omar Darwiche Domingues, Xuedong Shang, and Michal Valko. UCB momentum q-learning: Correcting the bias without forgetting. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 7609-761 PMLR, 2021b. 9
Mirco Mutti, Riccardo De Santi, and Marcello Restelli. The importance of non-markovianity in maximum state entropy exploration. In ICML, volume 162 of Proceedings of Machine Learning Research, pp.
PMLR, 2022. 9
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron C. Courville. The primacy bias in deep reinforcement learning. In ICML,, volume 162 of Proceedings of Machine Learning Research, pp. 16828-16847. PMLR, 2022.
Eygenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. arXiv preprint arXiv:2305. 15555, 2023. 9
Georg Ostrovski, Mare G. Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 2721-2730. PMLR, 2017. 9
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-drive exploration by self-supervised prediction. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 2778-2787. PMLR, 2017. 9
Andrew Patterson, Samuel Neumann, Martha White, and Adam White, Empirical design in reinforce- ment learning, 2023. 23
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In ICLR (Poster). OpenReview.net, 2018. 9
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems, 2018. 6.
Thomas Rückstieß, Frank Sehnke, Tom Schaul, Daan Wierstra, Yi Sun, and Jürgen Schmidhuber. Exploring parameter space in reinforcement learning. Paladyn J. Behav. Robotics, 1(1):14-24, 2010. 9
Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In ICLR. OpenReview.net, 2021. 2,
Max Schwarzer, Johan Samir Obando-Ceron, Aaron C. Courville, Marc G. Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 30365-30380. PMLR, 2023. 8
Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmid- huber. Parameter-exploringg policy gradients. Neural Networks, 23(4):551-555 2010. 9
12

---

Published as a conference paper at ICLR 2024
Ramanan Sekar, Oleh Rybkin, Kostas Danillidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In ICML, volume 119 Proceedings of Machine Learning Research, pp. 3883-592. PMLR, 2020. 9
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In ICML, volume 139 of Proceedings of Machine Learning Rexearch, pp. 9443-9454. PMLR, 2021.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 32145-32168. PMLR, 2023. 1, 3,9,
H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg. Aidan Clark, Hubert Soyer, Jack W. Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, Nicolas Heess, Dan Belov, Martin A. Riedmiller, and Matthew M. Botvinick. V-MPO: on-policy maximum posteriori policy optimiza- tion for discrete and continuous control. ICLR. OpenReview.net, 2020a. 8
H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W. Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, Nicolas Heess, Dan Belov, Martin A. Riedmiller, and Matthew M. Botvinick. V-MPO: on-policy maximum a posteriori policy optimiza- tion for discrete and continuous control. In ICLR. OpenReview.net, 2020b. 8
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 9870 9879. PMLR, 2021. 2,
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In NIPS, pp. 2753-2762, 2017. 9
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00699, 2018. 2, 8
Xiyao Wang, Ruijie Zheng, Yanchao Sun, Ruonan Jia, Wichayaporn Wongkamjan, Huazhe Xu, and Furong Huang. Coplanner: Plan to roll out conservatively but to explore optimistically for model-basee rl, 2023.
Pawel Wawrzynski. Control policy with autocorrelated noise in reinforcement learning for robotics. International Journal of Machine Learning and Computing, 5(2):91, 2015. 9
Qisong Yang and Matthijs T. J. Spaan. CEM: constrained entropy maximization for task-agnostic
safe exploration. In AAAI, pp. 10798-10806. AAAI Press, 2023. 9
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In ICLR. OpenReview.net, 2021. 2,
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforeement learning. In ICLR. OpenReview.net, 2022. 2, 6, 8,
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, volume 100 of Proceedings of Machine Learning Research, pp. 1094-1100. PMLR, 2019. 2,
Zhecheng Yuan, Sizhe Yang, Pu Hua, Can Chang, Kaizhe Hu, and Huazhe Xu. RL-vigen: A reinforcement learning benchmark for visual generalization. In shirth- Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: //oeerreview weteeeewwweeeeeeeeeeeeeeeeee.eee www.:enettoootteeeeeeeeeeee forum? d=RwNI qaNogd. 24
Tianjun Zhang, Paria Rashidinejad, Jiantao Jiao, Yuandong Tian. Joseph E. Gonzalez, and Stuart Russell. MADE: exploration via maximizing deviation from explored regions. In NeurIPS, pp. 9663-9680, 2021. 9
13

---

Published as a conference paper at ICLR 2024
Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang. TACO: Temporal latent action-driven contrastive loss for visual reinforcement learning. In seventh Conference Neural Information Processing Systems, 2023. URL www. net / forum? d=ezCsMoy1 w9. 2, 6, 8
14

---

Published as a conference paper at ICLR 2024
A
MORE VISUALIZATION RESULTS OF DORMANT RATIO
Acrobot Swingup Sparse (Success)
150
Gartpole Swingup Sparse (Success
Y000
Hopper Hop (Success)
Dermant Rabig
120
Episode Rewara
Domant Ratie
800
Episade Reward
Domant Ratio
Episade Reward
0.6
0.6
0.6
0.4
0.4
200
Number Environment
Number Enviranment Steps
Number Environment Steps
Acrabot Swingup Sparse (Falled)
Cartpale Swingup Sparse (Failed),
Hopper Hop (Failed)
1.0
100
1.0
100
1.0
100
Dormant Ratio
0.8
eo
Epicode Reward
Domant Ratio
0.8
Episode Reward
Dommntt Ratio
0.0
80
Eplsodee Rewardd
0.0
0.4
0.4
40
0.4
02
0.2
20
20
0m 0.6m
1.0m
Om 0.2m
1m
0.6m 0.0m
Number of Environment Steps
Number of Environment Steps
Number of Envirooment Steps
Dormant Ratio
Episode Reward
Figure 11: Analysis of the dormant ratio in successful vs. broken seeds reveals distinct behavior patterns. In successful seed, decreasing dormant ratio allows the agent to effectively explore environment learn skills. Conversely, a broken seed, the agent becomes immobile and fails discover meaningful motions.
B TIME EFFICIENCY DRM
To assess the algorithms' speed, measure their frames per second (FPS) on the same DeepMind Control Suite task, Dog Walk, using identical Nvidia RTX A5000 GPU. As Figure 12 shows, while achieving significant sample efficiency and asymptotic performance, DrM only slightly compromises wall-clock time compared to DrQ- v2. Compared with two other baselines, DrM is roughly as time- efficient as ALIX and about three times faster than TACO, which needs a batch size four times larger than that of DrQ-v2 to compute its temporal contrastive loss.
105-
Time Efficincyy
Frames Pour Second
DrM
DrOv2
AUX
TACO
Figure 12: Comparison time- efficiency
15

---

Published as a conference paper at ICLR 2024
C IMPLEMENTATION DETAILS
In this section, we describe the implementation details of DrM. We have built DrM upon the publicly available source code of DrQ-v2. Subsequently, we present the pseudo-code outlining our approach.
C. DORMANT RATIO CALCULATION
In this subsection, we demonstrate how the dormant ratio is calculated in Algorithm 1.
Algorithm Dormant Ratio Calculation
1: procedure CAL.DORMANT_R RATIOoodooo inputs, r-dormant threshold)
2: Initialize counters: neurons, dormant neurons
3: Operate forward propagation: model(inputs)
4: for each module in model do
5:
module is Fully Connected Layer then
6:
output average over the batch (Joutput of the forward propagation)) average_output = average over the neurons (output)
7:
8:
dormant_neurone
count output < geraageoouipur 7-dormant threshold
9:
teurons + = neurons in this layer
end if
11:
end for
return dormant ratio dormant_neurons / total total_ _neurons
13:
end procedure
C.2 DERD PERTURBATION
In this subsection, we demonstrate how the perturbation is performed based on the dormant ratio in Algorithm
Algorithm 2 Dormant-ratio-guided Perturbation
1: procedure PERTURB(network, optimizer, perturb factor)
2: Create a new network new net which has the same shape as the original network
3: Initialize weights of new_net
4: for each layer and parameter in the network do if layer Fully Connected Layer then
5:
6:
Compute noise as: new net (1 perturb.factor) Update parameter with: net perturb factor noise
7:
8:
end end for
9:
10: Reset the state of the optimizer
11: return updated network, optimizer
12:
end procedure
16

---

Published as a conference paper at ICLR 2024
C.3 AWAKEN EXPLORATION SCHEDULER
In this subsection, we demonstrate how the awaken exploration scheduler is performed in Algorithm 3.
Algorithm 3 Awaken Exploration Scheduler
1: Initialize awaken step None
2:
function (stee)
3:
if' awaken_step None then
4:
return dormant.stddev
5: else
linear-stddev linear scheduueeeeet awaken step) 7: return max (ooooooottsdddeeeeeoeeo linear_stddey stddev)
6:
8: end if
9: end function
10: function UPDATE_AWAKEN SWW.EEAWWWAE.STPPSSSPPP
11: if d'awakenssstep None and dormant.ratio < dormant thressold then
12:
awaken step +- step
13: end
14: end function
C.4 DORMAN--RATO-GUIDDE EXPLOITATION
In this subsection, demonstrate how the dormant-ratio-guided exploitation performed in Algorithm 4.
Algorithm Dormant-ratio-guicee exploitation
function UPDATE VALUE (obs, action)
2:
Q1, Q2 critic(obs, action)
3:
Q min(Q1,
4:
V Vone((bs) (obs)
5: error
Q
sign { error >
6:
{0 otherwise
7:
weight = (1 sign)expectile + sign(1 - expectile)
8:
value_loss mean(weight x error?)
9:
Update value network using value loss
end function
11:
function CAI_TAARGET_QneeX
reward, discount) action distribution actor(nextoobs, awaken exploration scheduler)
12:
13:
Sample nextaction from the distribution with clipping
14:
target_Q). Q1, target_ = critic_ crrtiictarget(nnett next_obs, next_action)
15:
Varget.V-epplore min min(target_QU target.Q2)
16:
target exploit = Vinec (next obs)
target V À target V exploit (1 A) target V explore
18:
target_Q = reward + (discount x target_
return target.
20: end function
17

---

Published as a conference paper at ICLR 2024
D HYPERPARAMETERS
D.1 HYPERPARAMETERS IN DRM
We summarize all the hyperparameters of DrM in Table While are trying keep the settings identical for each of the task, there are few specific deviations of DrM hyperparameters for some tasks. Additionally, D.2, we demonstrate the performance of DrM with a single set of hyperparameters across these tasks.
Hammer, Pen, Door of Adroit: Exploitation expectile 0.7 Dog [Stand, Walk, Run], Humanoid Run, Coffee Push & Soccer: Maximum perturb factor amar 0.6
Parameter
Setting
Replay buffer capacity Action repeat
10"
Seed frames
4000
Exploration steps
2000
n2-teep returns
Mini-batch size
256
Discount ?
0.99
Optimizer
Adam
Learning rate
10 (DeepMind Control Suite)
10-4 (MetaWorld & Adroit)
Agent update frequency Soft update rate Features dimension
2
0.01
100 (Humanoid & Dog)
50 (Others)
Hidden dimension
1024
T-Dormant ratio
0.025
Dormant ratio threshold /
0.2
Minimum perturb factor Omin
0.2
Maximum perturb factor mma
0.9
Perturb rate k
Perturb frames
200000
Linear exploration stddev. clip
0.3
Linear exploration stddev. schedule
linear( 1.0, 0.1. 200000)) (DeepMind Control Suite) linear(1 1.0,0.1,300000) (MetaWorld & Adroit)
Awaken exploration temperature T
0.1
Target exploitation parameter À
0.6
Exploitation temperature Exploitation expectile
0.02
0.9
Table 1: A default set of hyper-paramen parameters used our experiments.
Note: For learning rate, feature dimension, and linear exploration schedule, simply follow the standard of DrQ-v2, which has a separate setting for hard DMC tasks (lower learning rate, larger feature dimensionality, longer exploration schedule). These three hyperparameters not introduced by DrM, and we do not do any tuning on these three.
18

---

Published as a conference paper at ICLR 2024
D.2 PERFORMANCE WITH ONE SET OF DRM HYPERPARAMETER.
Here we show the performance of Dr on tasks where we use a single set of hyperparameter's instead of the domain-specific ones.
DrM
DrM w. Fixed Hyperparameters
Dog
Dog Walk
Dog
1000
500
Episode Reeward
Coisade Reward
2:30
402
Episode
400
500
200
100
13th
15t
20m
25m
Xim
15mh 21n Number frames
137
30m
16m Number of Frames
10m
Number Fromes
Humanold
Coffee Push
200
0014
87%
Egisode Rewwrrd
250
Success Rate
0000
50%
200
40%
0%
5m
5m
13m
15m
20m Fremes
25m
30m
2m
0.4m
3.5m
1i5
1.5m
2m
0m
7.4m
2.5m
2,3m
3.5m
2m
Hammer
Door
Pen
100%
80%
80%
80%
Success Htte
60%
60%
40%
40%
20%
20%
20%
0%
0%
0m
1m
2m
3m
4m Fremes
5m
6m
0m
1m
271
3m
4m
5m
6m
0m
1m
2m
3m
4m
5m
6m
Number
Number
Frames
Number Frames
Figure 13: Comparison of DrM against DrM with fixed hyperparameters nine tasks where used the non- default hyperparameter settings.
In general, we find that applying DrM with the default hyperparameter setting, using a unified set of hyperparameter off-the-shelf has decently great performance across all tasks. But for some tasks such as Humanoid Run in DMC and Hammer in Adroit, some additional hyperparameter tuning would be beneficial get further performance gain.
19

---

Published as a conference paper at ICLR 2024
E DRM VERSUS INTRINSIC REWARD BASED EXPLORATION TECHNIQUES
While the primary advantage of our method (DrM) lies employing the dormant ratio to guide the agent's exploration-exi -expoiitoonellliiaatiinn tradeoff, encouraging more active exploration of the environment, DrM demonstrates remarkable performance on the most challenging tasks, particularly those involving complex environmental dynamics and sparse rewards. Here, we compare DrM with an intrinsic reward- based exploration approach, which also aims for encouraging the exploration of RL agent. For this comparison, we select Random Network Distillation (RND), a popular and widely used technique in intrinsic reward based exploration.
For RND, we implement RND on top of DrQ-v2 and compare its performance against DrM on three Adroit tasks. We introduce random encoder f whose architecture is same as the encoder in DrQ-v2, and we introduce a predictor network 9 with the same architecture. The predictor network is then trained to predict representations from random encoder given the same observations, ie., minimizing < = [lf(s;: - (( We use prediction error as an intrinsic reward and learn policy that maximizes ploaa r + pr¹. We perform hyperparameter search over the weight E {0.1, 1.0, 10.0) on the Pen task and and then use the best hyperparameter (p = 1.0) for the three
Adroit tasks.
Hammer
Door
Pen
100%
100%
100%
80%
80%
80%
Success Rate
60%
60%
60%
40%
405
20%
20%
20%
056
0%
0%-
07/
1m
2th
30
4m
5m
6m
0m
Im
2m
3m
4m
5m
6m
0m
Im
7m
30
40%
5m
6m
Number Frames
Number of Frames
Number of Frames
DrM
DrQ-v2 w. RND
Figure 14: Comparison of DrM against an intrinsic exploration baseline (DrQ-v2 with RND) on three Adroit tasks.
As shown in Figure 14, additional intrinsic exploration mechanism such RND could indeed improve DrQ-v2's performance. (DrQ-v2 achieves 0% success rate across all tasks.) However, adding only an intrinsic exploration mechanism still insufficient for the agents discover the optimal policies. The performance gap again demonstrates the significance of DrM, which uses the dormant ratio guide the agent's ppplooaaiio-exploiaaiiii intrinsic exploration technique could also be combined with DrM to further boost exploration. This integration could be implemented as a separate mechanism or as replacement for the awaken exploration schedule in DrM. We could also use similar strategy in DrM, using the dormant ratio to control the magnitude of the intrinsic noise While we encourage future research this direction, such exploration falls outside the scope of the current work.
tradeoff. Furthermore, principle, an RND-Iike
20

---

Published as a conference paper at ICLR 2024
F COMPARISON WITH REDO
To compare approach with ReDo (Sokar al., 2023), which only resets the weights of dormant neurons, we conducted experiments three different environments on Meta Meral In Figure 15, can be observed that our method significantly outperforms the approach of resetting only dormant neurons and DrM with ReDo. We speculate that this improvement is due to the positive impact of resetting non con-dormant neurons on exploration. Additionally, our use of the dormant ratio to guide exploration strategy distinguishes our approach from previous works.
Soccer (sparse)
(sparse)
Coffee (sparse)
100%
100%.
0074
80%
80%
Success Rate
Success Rate
6.0%
60%%
Success
400%
45%
20%
20%
25%
0%
0%
50
Environment Stops
15m
300
US
Environment Stops
Lom
13m
0m
Number Environment Stops
15m
2m
DrQ-22
DrQ+ReDo
DrM(ReDo)
Figure 15: Comparison of DrM against ReDo (Sokar et al., 2023)
G DETAILED ABLATION RESULTS
In this section, we present the results of additional ablation studies for DrM in addition to Figure 10. In particular, we conduct additional ablation studies two MetWorld tasks, and we show the ablation study in three Adroit tasks separately instead of an aggregated plot.
Stick Pull
Pick Place Wall
100%
80%
20%
Success Rate
40%
40%
20%
0m
3.4m
0 8m
1.2m
1.6m
Steps
5m
Number of Environment Steps
1.6m
2m
Pen
Door
Hammer
100%
100%
80%
Success Raki
60%
60%
60%
40%
40%
20%
20%
20%
0%
0%
0%
0m
1m
2m
4m
Sm
6m
0m
1m
2m
3m
4m
3m
6m
0m
1m
2m
3m
4m
6m
Number of Framess
Number of Frames
Number Frames
DrM
DrM w/o Awaken Exploration
DrM w/o Drg Exploitation
DrM w/o Drg Perturbation
Figure 16: Additional ablation studies: Drg represents Dormant-ratio guided.
21

---

Published as a conference paper at ICLR 2024
H AN ADDITIONAL ABLATIONON STUDY ON DORMAN--RATIO-GUIDED EXPLORATION
To justify design choice of dormant-ratio-guidee exploration, here have conducted ad- ditional ablation study on the Adroit domain, where we compare DrM with a baseline such that sets maximal exploration noise (i.e., std 1) before awakening and then using the linear schedule afterward.
Hammer
Door
Pen
100%
100%
100%
60%
60%
40%
40%
Number Frames
Number of Framnes
Number Frames
DrM
DrM w. Alternative Exploration Schedule
Figure 17: Comparison of DrM with an alternative exploration strategy
As shown from Figure 17, we find that using the maximal exploration noise instead of our adaptive adjustment of exploration noise based the dormant ratio results in significant performance degrada- tion across all three Adroit tasks. This further justifies our design choice of the Dormant-ratio-guide exploration mechanism.
22

---

Published as a conference paper at ICLR 2024
I RESULTS WITH MORE RUNS
Considering the variability in results introduced by random seeds and for statistical rigor, we con- ducted 10 runs of experiments for both DrQ-v2 and DrM across multiple MetaWorld environments, following the suggestion by Patterson et al. (2023). This was done to compare the performance of the algorithms across a broader range of seeds. It is evident that our algorithm is sensitive to the randomness of seeds, and it consistently maintains a significant performance lead over baseline algorithms.
Disassemble
Pick Place Wall
Sweep Into (sparse)
100%
100%
80%-
60%
60%
Success
Success
M
Number of Environment Steps
Number of Environment Steps
Number of Environment Steps
DrQ-v2 (4 runs)
DrQ+v2 (10 runs)
DIM(4 runs)
DrM//1 runs)
Figure 18: 10 runs of DrM vs. DrQ-v2 on three Metaworld Tasks.
J EFFECT OF SHRINK-AND AND--K-URDD-ERTUR
Regarding the potential complementary effects of "dithering" exploration introduced by the awaken scheduler and deeper exploration induced by network resets, we perform experiments by replacing reinitialized perturbations with the original initialization parameters. The results, depicted in the following figures, demonstrate that 1-shrink perturbations caused only 10% decrease perfor- mance in the Sweep-Into while maintaining comparable performance in Stick-Pull. This suggests that the combination of these exploration strategies may be complementary, providing a nuanced understanding of their interplay.
Sweep
100%
1003
50%
Success Rate
80%
Success
40%
40%
0%
0%
10m
Number Environment Steps
3:4m
B.lm
122m
1.
27
5m
Enviranment Steps
1.5m
1m
7m
DrM
DrM
(1-shrink)
23

---

Published as a conference paper at ICLR 2024
K FURTHER DISCUSSION OF DORMANT RATIO
In this section, aim evaluate an alternative hypothesis regarding the dormant ratio and DrM. The hypothesis is that dormant ratio measures the change from frame to frame and by minimizing it, DrM promotes the agent to maximize the velocity of change between frames, thereby encouraging exploration and yielding good empirical results. Through three experiments, we demonstrate that this not the case. Instead, as argued in the main paper, the dormant ratio should be an intrinsic indicator of the agent's 's activity level, rather than merely reflecting the speed of change in observations from external environments.
First, we performed an additional experiment using the DMC-Generalization Benchmark (Hansen Wang, 2021; Yuan et al., 2023). Unlike standard DMC tasks where the agents have static introduces dynamic element by inserting video clip into the background. If low dormant ratio truly corresponded to significant frame-to-frame, changes, then we would expect the dormant ratio in DMC-Gen to be low throughout the training process. However, findings contradict this assumption. As demonstrated in Figure 19, the pattern of the dormant ratio in DMC-Gen mirrors that of ordinary DMC tasks. Initially, the agent's dormant ratio remains high, instead of being consistently low throughout the training, challenging the hypothesis that dormant ratio is merely reflection of rapid frame-to-frame changes. Hopper (Gen)
background,
(Gen)
Dormant Ratio
0.45
Dormant Ratio
0.45
0.3
0.3
0.15
0.15
120k
80cc
120k
Number of Environment Steps
Number Environment Steps
Figure 19: The dormant ratio DrM OMCC-eenrallzzttion Benchmark.
Next, to further investigate whether using the difference between frames as an intrinsic reward for optimization is effective, we conducted an additional experiment on three Adroit tasks. The hypothesis suggests that by using the difference between consecutive observation frames as an intrinsic reward, an agent could achieve performance comparable to DrM by maximizing this reward. However, our empirical tests on three Adroit tasks indicate otherwise. For these experiments, we defined the intrinsic reward rf as the difference between the first and third frame of the agent's observations at timestep I. (Here, same as DrM, we follow the standard practice to use stack of three consecutive image frames as the agent's 's observation at each timestep.) For our experiments, we calculated the LI difference between the first and third frames. We then normalized this intrinsic reward using the running mean and standard deviation and trained a policy to maximize the total reward ,otal r
Hammer
Door
Pen
100%
100%
100%
80%
805
80%
Success Rale
60%
60%
60%
40%
405
20%
20%
20%
056
0%
0%-
07m
17
2th
301
4m
3m
6m
5m
Im
2m
3m
4m
5mm
5m
0mm
Im
2m
3m
4m
5m
Bm
Number of Frames
Number of Frames
Number of Frames
DrM
DrQ-v2 w. frame LI distance
Figure 20: DrM vs. DrQ-v2 with intrinsic reward defined as LI difference between consecutive frames
In Figure 20, we show the performance of D rM against such baseline. As shown from the plot, simply maximizing the difference between frames clearly cannot solve the three tasks.
24

---

Published as a conference paper at ICLR 2024
Finally, it's important to note that if DrM solely focused imizing the difference between frames, it would likely strug- gle with tasks that require minimal motion, such Acrobot Swingup, Humanoid Stand, and Dog Stand. In these tasks, ex- cessive motion could lead low reward. Here conducted an additional experiment on Cartpole Balance Sparse, task that also necessitates reduced motion for maintaining balance. As illustrated in Figure 21, despite its simplicity, DrM continues perform well when compared baseline algorithms. This further indicates that DrM 's effectiveness is not merely result of maximizing frame-to-frame differences.
Caripola Balance
1008
Epicade
TACO
Cdeed 13015 30005
Number Enviromment
Figure 21: DrM against baseline algo- rithms Cartpole Balance Sparse
25


Reviews:
[{"review_id": "48c8U9VuWL", "reviewer": "Reviewer_jJ9Z", "summary": "This paper identifies a pervasive issue in pixel-based RL, where the agent's\npolicy has few activations and limited activity in the early phases of learning.\nRelated to this phenomenon is neuron dormancy, and the authors propose an\nalgorithm that uses this insight in three mechanisms: weight perturbation,\nexploration and exploitation. The combination of these three improvements is\nreferred to as Dormant ratio Minimization (DrM), and the authors claim that this\nalgorithm is the first (documented and model-free) algorithm to solve dog and\nmanipulator, demonstration-free adroit, and is generally sample efficient in\nvarious environments.", "questions": "- Section 3 (Language surrounding Dormant Ratio, specifically beginning of 3.2): While you demonstrate the empirial obervation that low neuron dormancy is correlated with meanginful high-level behavior, it is overclaiming to say that it is essential. I can imagine an approach to construct a neural network with an arbitrarily high dormancy ratio by encapsulating any policy within a much larger but inactive network. Thus, it is not necessarily the case that high dormancy ratio translates to less meaningful behavior, but that it can be correlated with it during training.\n- Section 3 (Expectile Regression and Exploitation): This section seems to build off recent work, and I am not familiar with \"blended exploitation and exploration\". However, I do not see how the proposed method has anything to do with exploitation. Exploitation and exploration are fundamentally about control and action, but this section is primarily about the target of a temporal difference method. But even putting that aside, I do not see why placing higher weight on the state value function should empahsize exploitation, or why placing higher weight on the action value function should emphasize exploration.\n- Section 3 (Hyperparameters): The method involves three components, each with at least one hyperparameter. This gives your proposed algorithm a number of additional degrees of freedom over the base algorithm and the other baselines. Looking at the experiments, there is an ablation study for each individual component but no results on the sensitivity to the various hyperparameters. It would have been good to explore this, at least in one of the simpler environments.\n- Section 4 (Dormant ratio analysis): Validating the fact that the dormancy ratio is indeed minimized vs the base algorithm is interesting. One thing that would strengthen these results further is to show that the other baselines (A-LIX, TACO) are similar to DrQ-v2 in that they do not minimize the dormancy ratio on at least some problem. It would also be interesting to investigate why there is some periodicity in the dormancy ratio such as in humanoid run and manipulator.\n- Section 4 (Ablation study): I appreciate the effort to ablate your algorithm in Adroit, a relatively complex environment. One thing that would help is to show whether each component of DrM is faithful to its motivation. For example, the exploration mechanism could be shown to have a large influence on performance in environments that require exploration and less influence on environments that do not require as much exploration. This would also serve as an opportunity to further elaborate on what the exploitation mechanism accomplishes.\n\n# Minor\n- Section 2 (visual RL): Is it necessarily the case that your setting is POMDP and that the observations provided to the agent lack some information? While this can be true in some settings, the environment is usually constructed so that the observatiosn do include all necessary information (e.g. concatenation of frames in Atari).\n- Section 2 (Dormant Ratio): You use the term \"Linear layer\" in both definitions, but I am not sure whether this is required in the definition or what it is that you mean. Does it mean the last layer that usually maps the penultimate activations to the output?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The insight is derived from seemingly unrelated but recent work on Neuron\n  Dormancy, which explains how the phenomenon can impact learning in neural\n  networks through a potential plasticity mechanism. This paper seems to\n  provides a novel lens for the phenomenon, showing that neural activity can be\n  correlated with exploration, and that explicitly minimizing dormancy,\n  reinforcement algorithms can be improved in their ability to both explore and\n  'exploit'.\n- The paper is overall well written. The empirical analysis is good and,\n  although the statistical power is low due to a small sample size, the\n  improvements seem statistically significant. I particularly like that the\n  dormancy ratio is in-fact minimized, because the algorithm seems to be doing\n  this implicitly. The algorithm does seem to achieve SOTA on a few challenging\n  continuous control problems with relevant baselines. Overall, the\n  contributions are convincing.", "weaknesses": "- The foundational motivation for dormancy ratio minimization is not entirely\n  convincing. It is not clear why high neuron dormancy should necessarily lead\n  to a decrease in exploration, even if this was empirically observed. It is\n  also not that clear why a lower neuron dormancy may lead to more exploration.\n  I think the paper would benefit from a more careful treatment of this finding,\n  either with further experiment, a toy model/study, theory or merely some text\n  describing why this should be expected.\n- There are a few aspects of the proposed algorithm that are mildly worrying.\n  First is the exploitation mechanism. I do not understand how exploitation is\n  determined by a hyperparameter controlling a value update towards a\n  state-value or an action-value. \n- The second thing is that the algorithm\n  proposes three mechansism, each with a hyperparmeter that governs a schedule.\n  This provides several degrees of freedom to improve upon the base algorithm,\n  and no hyperparameter study is conducted. This is not good empirical science,\n  but the empirical demonstration of success on hard problems does at least hold\n  promise for further work. \n\nOverall, I think the first and second weakness are both more pressing and addressable within a rebuttal period.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AB6DTHrW8v", "reviewer": "Reviewer_2E41", "summary": "The paper provides insight that the dormant ratio of the network acts can indicate if the agent is effectively exploring or not. Based on this insight, several mechanisms are proposed to use the dormant ratio to handle the exploration-exploitation trade-off. The results show that the proposed mechanism significantly improves performance and sample efficiency over existing methods.", "questions": "What value of $\\hat{\\beta}$ is used in Figures 3 and 4? This should be specified in the figure itself. \n\nWhat exactly is the linear schedule, $\\sigma_{linear}$, of exploration? The paper says that it's the same as defined in DrQ-v2. However, I believe the paper should be as self-contained as possible, so details like this should be provided in the appendix.\n\n---------------\n\nI've increased my score as new results largely overcome my initial concerns.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The central insight provided in the paper is very interesting. The main idea is that when the dormant ratio is very high, the agent is not getting diverse experience, so it would be good to increase exploration at this point. The insight is verified by looking at the dormant ratio and the agent's behaviour for some seeds.\nThis insight is then used to control the exploration-exploitation trade-off. Using an internal indicator like the dormant ratio is a promising direction to tackle the exploration-exploitation dilemma as it does not depend on any external signals like reward. \n\nThe results show that the proposed method, DrM, is effective in many visual RM domains and significantly outperforms existing methods.", "weaknesses": "Although the paper has interesting ideas and promising results, some of the weaknesses stop me from recommending a full acceptance of the paper.\n-**Weak empirical evaluation** All of the experiments are performed with just four random seeds, which raises questions about the statistical significance of the results. I refer the authors to Patterson et al. (2023) on how to perform good empirical experiments. I realize that it might not be feasible to do 30 runs for all environment-algorithm pairs, but there should be at least one experiment in the paper that will stand the test of time. I suggest that the authors perform at least ten runs for all algorithms in some environments, maybe on the four dense tasks in MetaWorld (from Figure 7).\n-**Too strong claims** For example, the last line of Section 3.1 says, \" ... all the hyperparameters in this approach are robust to tasks and domains.\". However, no evidence is provided for this claim. Their method introduces 4 or 5 new hyper-parameters, and different values are used for different environments. This means that the same hyperparameter is not optimal across domains.\n\nPatterson, A., Neumann, S., White, M., & White, A. (2023). Empirical Design in Reinforcement Learning. arXiv preprint arXiv:2304.01315.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yRyujD18D9", "reviewer": "Reviewer_R5aS", "summary": "This paper presents an algorithm which leverages the fraction of dormant units in the network to guide exploratory behaviour. The resulting method demonstrates significant improvements over baseline algorithms that use naive exploration strategies on a variety of challenging robotic manipulation domains.", "questions": "- There is a lack of clarity in the causality of the phenomenon studied by the paper: is the claim that interesting behaviour leads to greater data variety and thus less overfitting and inactive units, or is the idea that reduced representation capacity limits the range of behaviours that a network can express?\n- How sensitive is the method to hyperparameters? \n- Could the authors provide additional justification for the expectile prediction component of their method, and illustrate what types of environments this might prove detrimental for?\n- Can the authors comment on the design choices of their algorithm highlighted in the \"weaknesses\" section and provide justification for why the variant used in the paper is employed, rather than a simpler and more direct version of achieving the goal claimed for each component? For example, why the method perturbs all units rather than resetting dead ones, why exploration noise is potentially greater later in training than earlier due to the max operator, and why the expectile method which is described as *exploiting* past success should have the effect of prioritizing high-variance states and thus correspond to greater *exploratory* behaviour?\n- How robust is the algorithm to the trajectory of dead units in networks trained on a particular environment? For example, if we considered a different architecture or set of tasks (perhaps ProcGen or Atari), would the strategies employed by the proposed method still provide the same benefit, or would they need to be re-tuned?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper evaluates on more than one domain\n- The proposed method significantly outperforms the baselines compared against.\n- The paper performs an ablation on a single environment of the different components of the proposed method\n- Nice contrast with other adaptive exploration methods which use e.g. bandit methods to select epsilon-greedy exploration parameter. \n- Pseudocode is provided for each component of the proposed method\n- The proposed method depends on a single easy to measure property of the network state\n- The correlation between the dormant ratio and the qualitative properties of the network's exploration is intriguing.\n- The idea of strategically exploring based on properties of an agent's representation is a new and exciting approach which I think could be applied more broadly in RL and could be really useful to the community.", "weaknesses": "- The design choices going into the main method proposed by the paper are not given sufficient motivation, and I am concerned that although the algorith does indeed improve performance, it does not do so for the reasons claimed. In particular:\n        - Why is the method using shrink and perturb rather than directly resetting inactive units? Is it possible that S&P is introducing additional exploration due to the noisy updates to the parameters, similarly to the \"noisy nets\" strategy of Fortunato et al.?\n        - The exploration strategy appears to actually introduce *more* noise after the network is \"activated\" than it used before activation due to the max operator. However, the motivation for this noise adaptation is to introduce more noise early in training when the network's behaviours are less interesting. I agree that this should have the effect of increasing exploratory behaviour throughout training compared to using a schedule, but not in the way claimed by the paper.\n        - What motivates the modelling assumption that the network only becomes \"activated\" once in its training trajectory? Why is it assumed that the number of inactive units will not drop repeatedly?\n        - The use of expectiles introduces a large benefit in training based on the ablation study, but the optimal expectile seems to depend on the environment. Is there additional justification for using expectiles beyond the empirical benefit and the additional knob to tune in the training algorithm?\n- I have some concerns about the fairness of the comparisons employed in this work:\n        - Each of the components of the method seem to increase exploratory behaviour. However, the baselines compared against do not incorporate any particularly sophisticated exploration strategies (DrQv2 uses a linear scheduler and it doesn't look like A-LIX or TACO improve upon this). \n        - The proposed algorithm has a number of hyperparameters which have different optimal values for different environments. As a result, I'm not sure how much of the performance gain observed in the paper is due to finetuning of these additional hyperparameters vs the fundamental benefits of the method.\n        - Given the different optimal values of the expectile and perturbation scale for different environments, the robustness of the method to its hyperparameters is a major concern to me. How much worse is performance if the optimal hyperparameters for one domain are used in another?\n- While I appreciate the pseudocode provided in the appendix, there are some issues with clarity. For example, algorithm 2 says that one creates a deep copy of the network and stores this in the variable new_net, but then the next line randomly initializes the weights of new_net. What did the copy step do? Algorithm 1 lines 6-7 are also extremely vague and should be clarified with an equation.\n- Related to my concerns about design choices, there are many aspects of the algorithm that seem unnecessarily complicated -- for example, one could simply use an exploration noise value equal to the sigmoid of the dormant unit fraction discussed, or use a maximal exploration noise until the awakened threshold is met and then start the noise decay. I would appreciate ablations which illustrate the importance of these design choices.\n- The use of the term \"exploitation\" seems inaccurate to me. Exploitation is usually used to refer to greedy behaviour with respect to the network's predictions. In this case, exploitation refers more to optimistic Q-target updates which bias towards higher expectiles, encouraging the network to visit states which have occasionally yielded high rewards in the past. This approach would presumably lead to overestimation of Q-values in noisy environments and potentially result in suboptimal policies where the agent is attracted to high-variance, low-expectation states. I think the paper would benefit from a clearer discussion of what precisely this component of the method is doing, and potentially also a rebrand to characterize it more as a \"risk-seeking\" parameter rather than an \"exploitation\" parameter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FTVegchYBB", "reviewer": "Reviewer_Jxw3", "summary": "This paper is inspired by recent work on the \"dormant ratio\", a measure of how many neurons are inactive in a deep RL agent's neural net. Previous work (Sokar et al., 2023) showed that a high dormant ratio is detrimental to learning, but this paper flips things around by considering the converse, interpreting a declining dormant ratio as a proxy for learning progress. The authors propose three adaptive hyperparameter scheduling mechanisms based on this idea. They evaluate the resultant algorithm, named DrM, in three visual continuous control domains, where it performs more stably and achieves better asymptotic performance than three other recent methods.", "questions": "- In the \"dormant-ratio-guided exploitation\" section, it's mentioned that \"value underestimation occurs as the agent starts to acquire skills, given that $\\pi$ is sub-optimal\". I don't quite follow this. As the agent starts to improve, wouldn't we expect $\\pi$ to become less sub-optimal?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The greatest strength of the paper is the significance of the results. DrM is the clear winner on *all* of the tasks reported on in the paper, and in some domains (particularly Adroit) it achieves a step change improvement. I like that the authors have considered three separate domains (DeepMind Control Suite, MetaWorld and Adroit), with multiple tasks from each.\n\nThe main idea of treating the dormant ratio as a progress proxy, and hence using it as the basis for adaptive hyperparameter scheduling, makes intuitive sense. While I have some minor qualms about the writing (see below), the paper was clear and easy-to-read overall. Lastly, the coverage of the huge amount of related work from the last 2-3 years is about as thorough as possible.", "weaknesses": "My biggests concerns with the paper are its limited novelty a lack of proper ablations.\n\nNovelty:\n\nNone of the three mechanisms introduced by DrM are novel in-and-of themselves.\n- As noted in the paper, \"dormant-ratio-guided perturbation\" follows the previous work of (D'Oro et al., 2023; Ash & Adams, 2020).\n- Similarly, \"dormant-ratio-guided exploitation\" is based on the work of Ji et al. (2023).\n- The idea of scheduling the exploration rate goes back a very long way, pre-dating deep RL. Some recent approaches, e.g., Agent57 (Badia et al.), also use a form of adaptive exploration.\n\nWhat *is* new is the idea of using the dormant ratio to schedule the hyperparameters of these methods. I don't disagree that this is novel; however, it's rather limited novelty in my opinion.\n\nAblations:\n\nWhile Figure 10 contains some ablations, the three mechanisms are each ablated in their entirety. I'd prefer it if only the adaptive scheduling component were ablated for each method, since this is the novel part. In other words, for a fair comparison, DrM ought to be compared against DrQ + perturbation resets (with fixed $\\alpha$) + \"Blended Exploration and Exploitation operator\" (with fixed $\\lambda$).\n\nMoreover, the ablations only consider one domain (Adroit) and the results are aggregated across all Adroit tasks. I'd prefer to see more ablations in greater granularity. For me, they're the most important part of the experiments, since without them it's very hard to understand *why* the improvements work. (The argument that DrM lowers the dormant ratio and hence improves performance seems very \"chicken-egg\" to me; it's unclear which is the cause and which is the effect. Only the dormant-ratio-guided perturbations seem to directly target the dormant ratio.)\n\nMinor things:\n- Section 3.1 is too unscientific, e.g., the claim \"This suggests that the dormant ratio acts as an intrinsic metric, influenced more by the diversity and relevance of the agent's behaviors than by its received rewards\". This is all based on one example, and the claim that RL agents learn relevant behaviours before their returns improve sounds dubious to me. Another too-bold claim is: \"As the dormant ratio captures the intrinsic characteristics of an agent's policy network, all the hyperparameters in this approach are robust to tasks and domains.\" This is never established, and in fact Table 1 in the Appendix suggests that some parameters of DrM require domain-specific tuning.\n- The claim in the abstract that DrM has \"no broken seeds (76 seeds in total)\" is a little vague (what constitutes \"broken\"?) and it's never really returned to in the main body of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)