SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 5: Learning curves on six control tasks from DeepMind Control Suite (Tassa et al., 2020) as measured on the episode return. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
Results Figure 2 shows that A2C+VCSE consistently outperforms A2C+SE on diverse types of tasks, including simple navigation without obstacles (Empty-16x16), navigation with obstacles (LavaGapS7 and SimpleCrossingS9N1), and long-horizon navigation (DoorKey-6x6, DoorKey-8x8, and Unlock). For instance, on LavaGapS7, A2C+VCSE achieves an average success rate of  $88.8\%$ , while A2C+SE achieves  $13.9\%$  after 100K steps. This shows that VCSE encourages the agent to effectively explore high-value states beyond a crossing point. On the other hand, SE excessively encourages the agent to visit states located before the crossing point for increasing the state entropy.
Comparison to SE with varying  $\beta$  One might think that adjusting the scale of intrinsic rewards could address the issue of state entropy exploration in the supervised setup, by further encouraging exploration in high-value states or discouraging exploration in low-value states. However, as shown in Figure 4, simply adjusting the scale cannot address the issue. Specifically, A2C+SE fails to solve the task with a large  $\beta = 0.05$ , because large intrinsic rewards could make the agent ignore the task reward and encourages redundant exploration. On the other hand, small  $\beta = 0.0005$  also degrades sample-efficiency when compared to the performance with  $\beta = 0.005$  as it
Figure 4: Learning curves on SimpleCrossingS9N1 as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across eight runs.
discourages exploration. In contrast, A2C+VCSE learns to solve the task in a sample-efficient manner even with different  $\beta$  values (see Figure 16 for supporting experiments).
5.2 DeepMind Control Suite Experiments
Setup We consider a widely-used DeepMind Control Suite (DMC; Tassa et al. 2020) benchmark mainly consisting of locomotion tasks. For evaluation on a more challenging setup, we conduct experiments on pixel-based DMC environments by using a state-of-the-art model-free RL method DrQv2 (Yarats et al., 2021a) as our underlying RL algorithm. For computing the intrinsic bonus, we follow Laskin et al. (2021) by using the features from an intrinsic curiosity module (ICM; Pathak et al. 2017) trained upon the frozen DrQv2 encoder. For SE, we find that normalizing the intrinsic reward with its running mean improves performance, which is also done in Laskin et al. (2021). We do not normalize the intrinsic reward for our VCSE exploration. We also disable the noise scheduling
<sup>5</sup>We provide additional results on state-based DMC tasks in Appendix C, where trends are similar.
Figure 6: Learning curves on six visual manipulation tasks from Meta-World (Yu et al., 2020) as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
in DrQv2 for SE and VCSE as we find it conflicts with introducing the intrinsic reward. We instead use the fixed noise of 0.2. We use  $k = 12$  for both SE and VCSE. See Appendix A for more details.
Results Figure 5 shows that VCSE exploration consistently improves the sample-efficiency of DrQv2 on both sparse reward and dense reward tasks, outperforming all other baselines. In particular, our method successfully accelerates training on dense reward Walker Walk task, while SE significantly degrades the performance. In Appendix B, we further show that the performance of DrQv2+SE improves with smaller  $\beta$ , but it still struggles to outperform DrQv2 on Walker Walk. This implies that SE might degrade the sample-efficiency of RL algorithms on dense reward tasks by encouraging the agent to explore states that might not be helpful for solving the task. Moreover, We show that introducing a decaying  $\beta$  schedule for SE struggles to improve performance in Appendix B.
5.3 Meta-World Experiments
Setup We further evaluate our method on visual manipulation tasks from Meta-World benchmark (Yu et al., 2020) that pose challenges for exploration techniques due to its large state space with small objects. For instance, moving a robot arm toward any direction enables the agent to visit novel states, but it would not help solve the target task. As an underlying RL algorithm, we use DrQv2 (Yarats et al., 2021a). We follow the setup of Seo et al. (2022a) for our experiments by using the same camera configuration and normalizing the extrinsic reward with its running mean to make its scale be 1 throughout training. For computing the intrinsic reward, we use the same scheme as in DMC experiments (see Section 5.2) by using ICM features for computing the intrinsic reward and only normalizing the SE intrinsic reward. We disable the noise scheduling for all methods and use  $k = 12$  for SE and VCSE. See Appendix A for more details.
Results Figure 6 shows that VCSE consistently improves the sample-efficiency of DrQv2 on visual manipulation tasks, while SE struggles due to the large state spaces. Notably, DrQv2+VCSE allows for the agent to solve the tasks where DrQv2 struggles to achieve meaningful success rates, e.g.,  $\mathrm{DrQv2 + VCSE}$  achieves  $85\%$  success rate on Door Open after 100K environment steps while DrQv2 achieves zero success rate. On the other hand, SE struggles to improve the sample-efficiency of DrQv2, even being harmful in several tasks. This shows the effectiveness of VCSE for accelerating training in manipulation tasks where state space is very large. We also report the performance of SE and VCSE applied to model-based algorithm (Seo et al., 2022a) on Meta-World in Appendix B, where the trend is similar in that SE often degrades the performance due to the large state space.
(a) Effect of value approximation
Figure 7: (a) Learning curves on SimpleCrossingS9N1 that compares VCSE using neural networks for value function approximation against VCSE that employs policy evaluation (Sutton & Barto, 2018) using the privileged simulator information. Learning curves aggregated on two control tasks from DeepMind Control Suite that investigate the effect of (b) value conditioning and (c) batch size. We provide results on individual task in Appendix B. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
(b) Effect of value conditioning
(c) Effect of batch size
5.4 Ablation Studies and Analysis
Effect of value approximation Since we use neural networks for encoding states and approximating the value functions, our objective  $\mathcal{H}(S|V_{\pi}^{\mathrm{vs}})$  could change throughout training. To investigate the effect of using the approximated value function for our objective, we provide experimental results where we (i) use ground-truth states for computing the intrinsic bonus and (ii) computes the value estimates via policy evaluation (Sutton & Barto, 2018) using the MiniGrid simulator. Figure 7a shows that VCSE based on policy evaluation using the privileged knowledge of ground-truth simulator performs the best, but VCSE based on value function approximation with neural networks can match the performance and significantly outperforms SE. This supports that our method works as we intended with the approximated value functions. We expect that our method can be further improved by leveraging advanced techniques for learning value functions (Bellemare et al., 2017; Espeholt et al., 2018; Hafner et al., 2023), reaching the performance based on the privileged knowledge.
Effect of value conditioning We compare VCSE with a baseline that maximizes reward-conditional state entropy (RCSE) in Figure 7b to investigate our design choice of conditioning the state entropy on values instead of one-step rewards. The result shows that VCSE largely outperforms RCSE because the value captures more long-term information about the state when compared to the reward. We note that RCSE cannot be applied to navigation tasks in MiniGrid, where the reward is only given to goal-reaching states, which also demonstrates the benefit of VCSE in terms of applicability.
Effect of batch size We investigate how estimating the value-conditional state entropy using samples from minibatches by reporting the results with increased batch sizes. Specifically, we use the batch size of 1024 for computing the intrinsic bonus but use the same batch size of 256 or 512 for training the actor and critic of RL agent. As shown in Figure 7c, we find that results are stable with both batch sizes, which shows that our value-conditional state entropy estimation can be stable without using extremely large batch sizes or all the samples within the replay buffer.
Heatmap visualization To help understand how VCSE improves sample-efficiency, we provide the visualization of visited state distributions obtained during the training of A2C agents with SE and VCSE. For this analysis, we modified the original SimpleCrossingS9N1 task to have a fixed map configuration (see Appendix A for more details). As shown in Figure 8a, we find that the agent trained with SE keeps exploring the region before a narrow crossing point instead of reaching the goal until 100K steps, even if it has experienced several successful episodes at the initial phase of training. This is because the crossing point initially gets to have a high value and thus exploration is biased towards the wide low-value state region, making it difficult to solve the task by reaching the green goal. On the other hand, Figure 8b shows that VCSE enables the agent to successfully explore high-value states beyond the crossing point.
(a) State entropy exploration
Figure 8: Visualization of visited state distribution during the training of A2C agent with (a) maximum state entropy (SE) exploration and (b) maximum value-conditional state entropy (VCSE) exploration to solve SimpleCrossing task from MiniGrid (Chevalier-Boisvert et al., 2018).
(b) Value-conditional state entropy exploration
Limitation and future directions One limitation of our work is that we lack a theoretical understanding of how value-conditional state entropy exploration works. Considering the strong empirical performance evidenced in our experiments, we hope our work facilitates future research on a theoretical investigation. Our work only considers a non-parametric state entropy estimator due to its benefit that does not learn a separate estimator, but considering learning-based estimator can be also an interesting direction. Moreover, applying our idea to methods that maximizes the state entropy of policy-induced state distributions (Lee et al., 2019; Mutti & Restelli, 2020) can be also an interesting future work. Our work might be limited in that it works exactly the same as the state entropy exploration until the agent encounters a supervised reward from an environment. Developing a method that utilizes an intrinsic signal to partition the state space into meaningful subspaces without the need for task reward is a future direction we are keen to explore in future work.
Conclusion We present a new exploration method that maximizes value-conditional state entropy which addresses the imbalanced distribution problem of state entropy exploration in a supervised setup by taking into account the value estimates of visited states for computing the intrinsic bonus. Our extensive experiments show that our method can consistently improve the sample-efficiency of RL algorithms in a variety of tasks from MiniGrid, DeepMind Control Suite, and Meta-World benchmarks. We hope our simple, intuitive, and stable exploration technique can serve as a standard tool for encouraging exploration in deep reinforcement learning.
Acknowledgements and Disclosure of Funding
We want to thank anonymous reviewers and colleagues at KAIST ALIN-LAB and UC Berkeley RLL for providing helpful feedback and suggestions for improving our paper. We also thank Shi Zhuoran for helpful comments on the draft. This research is supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2022-0-00953, Self-directed AI Agents with Problem-solving Capability; No.20190-00075, Artificial Intelligence Graduate School Program (KAIST)).
Badia, Adrià Puigdomènech, Sprechmann, Pablo, Vitvitskyi, Alex, Guo, Daniel, Piot, Bilal, Kapturowski, Steven, Tieleman, Olivier, Arjovsky, Martín, Pritzel, Alexander, Bolt, Andew, et al. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020.
Bellemare, Marc, Srinivasan, Sriram, Ostrovski, Georg, Schaul, Tom, Saxton, David, and Munos, Remi. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, 2016.
Bellemare, Marc G., Dabney, Will, and Munos, Rémi. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, 2017.
Bingham, Ella and Mannila, Heikki. Random projection in dimensionality reduction: applications to image and text data. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, 2001.
Zhang, Chuheng, Cai, Yuanying, Huang, Longbo, and Li, Jian. Exploration by maximizing rényi entropy for reward-free rl framework. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Ziebart, Brian D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy, 2010.
Societal Impact
We do not anticipate immediate negative consequences from conducting this work because our experiments are based on simulation environments designed to conceptually evaluate the capabilities of reinforcement learning (RL) algorithms. Recent studies, however, demonstrate that large-scale RL when integrated with robotics can effectively work on real-world environments (Kalashnikov et al., 2021; Herzog et al., 2023). This makes it crucial to be aware of potential societal harms that RL agents could inadvertently cause. While this work does not aim to address these safety issues, our method might mitigate unintentional harm during the training process. For instance, it might prevent the agent from exhibiting potentially dangerous novelty-seeking behaviors, such as moving robot arms towards low-value, empty regions where a human researcher is likely to be situated.
A Experimental Details
Compute For MiniGrid experiments, we use a single NVIDIA Titan Xp GPU and 8 CPU cores for each training run. It takes 15 minutes for training the agent for 1M environment steps. For DeepMind Control Suite and Meta-World experiments, we use a single NVIDIA 2080Ti GPU and 8 CPU cores for each training run. It takes 36 minutes and 90 minutes for training the agent for 100K environment steps on DeepMind Control Suite and Meta-World benchmarks, respectively.
A.1 Implementation Details
Value normalization For normalizing value estimates to stabilize value-conditional state entropy estimation, we compute the mean and standard deviation using the samples within the mini-batch. We empirically find no significant difference to using the running estimate.
Extrinsic critic function For training the extrinsic critic function described in Section 4.2, we introduce another set of critic and target critic functions based on the same hyperparameters used for the main critic the policy aims to maximize. Then we use the target critic for obtaining value estimates. We apply the stop gradient operation to inputs to disable the gradients from updating the extrinsic critic to update other components. For the policy, we use the same policy for training both main and extrinsic critic functions. We empirically find no need for training another policy solely for the extrinsic critic.
A2C implementation details We use the official implementation of RE3 (Seo et al., 2021) and use the same set of hyperparameters unless otherwise specified. Following the setup of RE3, we use a fixed, randomly initialized encoder to extract state representations and use them for computing the intrinsic reward. We use the same hyperparameter of fixed intrinsic scale  $\beta = 0.005$  and  $k = 5$  for both SE and VCSE following the original implementation. For RE3, we normalize the intrinsic reward with its standard deviation computed using the samples within the mini-batch, following the original implementation. But we do not normalize our VCSE intrinsic reward.
DrQv2 implementation details We use the official implementation<sup>7</sup> of DrQv2 (Yarats et al., 2021a) and use the same set of hyperparameters unless otherwise specified. For both SE and VCSE exploration, we find that using  $\beta = 0.1$  achieves the overall best performance. We also use  $k = 12$  for both SE and VCSE. For computing the intrinsic reward, we follow the scheme of Laskin et al. (2021) that trains Intrinsic Curiosity Module (ICM; Pathak et al. 2017) upon the representations from a visual encoder and uses ICM features for measuring the distance between states. We note that we detach visual representations used for training ICM to isolate the effect of training additional modules on the evaluation. For both SE and VCSE exploration, we disable the noise scheduling scheme of DrQv2 that decays  $\sigma$  from 1 by following a pre-defined schedule provided by the authors. This is because we find that such a noise scheduling conflicts with the approaches that introduce additional intrinsic rewards. Thus we use the fixed noise of 0.2 for SE and VCSE exploration. For Meta-World experiments, we also disable the scheduling for the DrQv2 baseline as we find it performs better. But we use the original scheduling for the DrQv2 baseline following the official implementation.
$^{6}$ https://github.com/younggyoseo/RE3
<sup>7</sup>https://github.com/facebookresearch/drqv2
Figure 9: Examples of tasks in our MiniGrid experiments: (a) LavaGapS7, (b) Empty-  $16\times 16$ , (c) DoorKey-  $6\times 6$ , (d) DoorKey-  $8\times 8$ , (e) Unlock, and (f) SimpleCrossingS9N1.
Figure 10: Examples of tasks we used in our Meta-World experiments: (a) Door Open, (b) Printer Open, (c) Fingerprint Open, (d) Window Open, (e) Button Press, and (f) Fingerprint Close.
Figure 11: Examples of tasks we used in our DeepMind Control Suite experiments: (a) Hopper, (b) Walker, (c) Cheetah, (d) Cartpole, (e) Pendulum
Heatmap analysis details For experiments in Figure 8, we use the easy version of SimpleCrossingS9N1 task from MiniGrid benchmark (Chevalier-Boisvert et al., 2018). Specifically, we disable the randomization of map configurations to make it possible to investigate the heatmap over a fixed map. For visualizing the heatmaps, we record  $x$ ,  $y$  position of agents during the initial 100K steps. We train A2C agent with both SE and VCSE exploration as we specified in Section 5.1 and Appendix A.1 without any specific modification for this experiment.
A.2 Environment Details
MiniGrid We conduct our experiments on six navigation tasks from MiniGrid benchmark (Chevalier-Boisvert et al., 2018): LavaGapS7, Empty-  $16\times 16$ , DoorKey-  $6\times 6$ , DoorKey-  $8\times 8$ , Unlock, and SimpleCrossingS9N1. We provide the visualization of the tasks in Figure 9. We use the original tasks without any modification for our experiments in Section 5.1.
Meta-World We conduct our experiments on six manipulation tasks from Meta-World benchmark (Yu et al., 2020): Door Open, Printer Open, Window Open, Button Press, and Camera Close. We provide the visualization of the tasks in Figure 10. We follow the setup of Seo et al. (2022a) that uses a fixed camera location for all tasks.
DeepMind Control Suite We conduct our experiments on six locomotion tasks from DeepMind Control Suite benchmark (Tassa et al., 2020): Hopper Stand, Walker Walk Sparse, Walker Walk, Cheetah Run Sparse, Cartpole Swingup Sparse, and Pendulum Swingup. We use the sparse reward tasks introduced in Seyde et al. (2021), by following RE3. We provide the visualization of the tasks in Figure 10.
B Additional Experiments
B.1 Experiments with Model-Based RL
Setup As a model-based underlying RL algorithm, we consider Masked World Models (MWM; Seo et al. 2022a) that has shown to be able to solve more challenging, long-horizon tasks compared to DrQv2. We consider four tasks of various difficulties: Box Close, Handle Pull Side, Lever Pull, and Painter Open. We use the official implementation<sup>8</sup> of MWM (Seo et al., 2022a) and use the same set of hyperparameters unless otherwise specified. For both SE and VCSE exploration, we find that using  $\beta = 1$  performs best. Following the idea of Seo et al. (2022b) that introduces an additional reward predictor for intrinsic reward in the world model of DreamerV2 (Hafner et al., 2021), we introduce the reward network that predicts our intrinsic reward  $r_t^{\mathrm{VCSE}}$ . For computing the intrinsic reward, we also follow the idea of Seo et al. (2022b) that uses a random projection (Bingham & Mannila, 2001) to reduce the compute cost of measuring distances between states. Specifically, we project 2048-dimensional model states into 256-dimensional vectors with random projection. Because the original MWM implementation normalizes the extrinsic reward by its running estimate of mean to make its scale 1 throughout training, we find that also normalizing intrinsic rewards with their running estimates of mean stabilizes training. We use  $k = 12$  for both SE and VCSE.
Results Figure 12 shows that VCSE consistently accelerates and stabilizes the training of MWM agents on four visual manipulation tasks of different horizons and difficulties, which shows that the effectiveness of our method is consistent across diverse types of RL algorithms. On the other hand, we observe that SE could degrade the performance, similar to our observation from experiments with DrQv2 on Meta-World tasks (see Figure 6). This supports our claim that SE often encourages exploration to be biased towards low-value states especially when high-value states are narrowly-distributed, considering that manipulation tasks have a very narrow task-relevant state space.
Figure 12: Learning curves on six visual manipulation tasks from Meta-World (Yu et al., 2020) as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
B.2 Ablation Study
In Figure 13, we provide the results on individual task used for reporting the aggregate performance that investigate the effect of value conditioning and batch size (see Figure 7b and Figure 7c).
(a) Effect of value conditioning
Figure 13: Learning curves on two visual locomotion control tasks from DeepMind Control Suite that investigate the effect of (a) value conditioning and (b) batch size. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across eight runs.
(b) Effect of batch size
<sup>8</sup>https://github.com/younggyoseo/MWM
B.3 Experiments with Varying Intrinsic Reward Hyperparameters
A2C+VCSE with varying  $k$  We provide additional experimental results with varying  $k \in \{3,5,7\}$  on SimpleCrossingS9N1 task in Figure 14. We find that the performance of A2C+VCSE tends to degrade as  $k$  increases. We hypothesize this is because higher  $k$  leads to finding less similar states and this often leads to increased intrinsic reward scale.
Figure 14: Learning curves as measured on the episode return. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across eight runs.