Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration
Authors: Dongyoung Kim
Jinwoo Shin
Pieter Abbeel
Younggyo Seo
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
State Entropy
Exploration
TL;DR: We present a novel exploration technique that maximizes the value-conditional state entropy, which takes account into the value estimates of visited states for computing intrinsic bonus.

Abstract: A promising technique for exploration is to maximize the entropy of visited state distribution, i.e., state entropy, by encouraging uniform coverage of visited state space. While it has been effective for an unsupervised setup, it tends to struggle in a supervised setup with a task reward, where an agent prefers to visit high-value states to exploit the task reward. Such a preference can cause an imbalance between the distributions of high-value states and low-value states, which biases exploration towards low-value state regions as a result of the state entropy increasing when the distribution becomes more uniform. This issue is exacerbated when high-value states are narrowly distributed within the state space, making it difficult for the agent to complete the tasks. In this paper, we present a novel exploration technique that maximizes the value-conditional state entropy, which separately estimates the state entropies that are conditioned on the value estimates of each state, then maximizes their average. By only considering the visited states with similar value estimates for computing the intrinsic bonus, our method prevents the distribution of low-value states from affecting exploration around high-value states, and vice versa. We demonstrate that the proposed alternative to the state entropy baseline significantly accelerates various reinforcement learning algorithms across a variety of tasks within MiniGrid, DeepMind Control Suite, and Meta-World benchmarks. Source code is available at https://sites.google.com/view/rl-vcse.

Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration
Dongyoung Kim KAIST
Jinwoo Shin KAIST
Pieter Abbeel UC Berkeley
Younggyo Seo" KAIST
Abstract
A promising technique for exploration to maximize the entropy of visited state distribution, i.e. state entropy, by encouraging uniform coverage of visited state space. While it has been effective for an unsupervised setup, it tends to struggle supervised setup with task reward, where agent prefers visit high-value states to exploit the task reward. Such a preference can cause an imbalance between the distributions of high-value states and low-value states, which biases exploration towards low-value state regions as a result of the state entropy increasing when the distribution becomes more uniform. This issue is exacerbated when high-value states are narrowly distributed within the state space, making it difficult for the agent to complete the tasks. In this paper, we present novel exploration technique that maximizes the value-conditions the state entropies that are conditioned on the value estimates of each state, then maximizes their average. By only considering the visited states with similar value estimates for computing the intrinsic bonus, our method prevents the distribution of low-value states from affecting exploration around high-value states, and vice versa. We demonstrate that the proposed alternative the state entropy baseline significantly accelerates various reinforcement learning algorithms across variety of tasks within MiniGrid, DeepMind Control Suite, and World benchmarks. Source code is available at https: /sittss..././//////////////e lssiteeggoggle............eo.eee www.gle.coomeeeooooo www.wwww/wcc./eee.oe.........eee wwwe/rllcse.......
state entropy, which separately estimates
Introduction
Recent advances in exploration techniques have enabled us to train strong reinforcement learning (RL) agents with fewer environment interactions. Notable techniques include injecting noise into action or parameter spaces (Sehnke et al., 2010; Rückstiess et al., 2010; Wawrzynski, 2015; Lillicrap et al., 2016; Fortunato al., 2018) and using visitation counts (Thrun, 1992b; Bellemare et al., 2016; Sutton & Barto, 2018; Burda et al., 2019) or emrrrs from predictive models (Stadic et al., 2015; Pathak et al., 2017, 2019; Sekar et al., 2020) as an intrinsic reward. particular, recently developed approach that maximizes the entropy of visited state distribution has emerged as promising exploration technique (Hazan al., 2019; Lee al., 2019; Mutti et al., 2021) for unsupervised RI,, where the agent aims to leamn useful behaviors without any task reward (Liu & Abbeel, la,;; Yarats et al., 202
The idea to maximize the state entropy has also been utilized in supervised setup where the task reward is available from environments to improve the sample-efficiency of RL algorithms (Tao et al., 2020; Seo et al., 2021; Nedergaard Cook, 2022; Yuan al., 2022). Notably, Seo al. (2021) have shown that maximizing the sum of task reward and intrinsic reward based on state entropy estimate can accelerate RL training. However, this supervised setup, we point out that this approach often suffers from an imbalance between the distributions of high-value and low-value states, which occurs as an agent prefers to visit high-value states for exploiting the task reward. Because state entropy increases when the distribution becomes more uniform, low-value states get to receive higher
Now at Dyson Robot Learning Lab. Correspondence to younggyo.. seo@dyson.com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

State
kNN (k-3)
Intrinsic Reward VCSB
Value
State Norm Value Norm 2y
Sorting with Maximum Norm max s,
||s:
sjlla
1|
2ylla
1l. l|e, 2yll))
Minibatch
Extrinsic Reward
Critic
Objective
Actor
r? + 8 yesE
Maximize
Reinforcement Learning Agent
Figure 1: Illustration of our method. We randomly sample states from replay buffer and compute the Euclidean norm in state and value spaces using pairs of samples within a minibatch. We then sort the samples based on their maximum norms. We find the k-th nearest neighbor among samples (e.g., = 3 in the figure) and use the distance to it as an intrinsic reward. Namely, our method excludes the samples whose values significantly differ for computing the intrinsic reward. Then train our RL agent to maximize the sum of the intrinsic reward and the extrinsic reward.
intrinsic bonus than high-value states, which biases exploration towards low-value states. This makes difficult for the agent to explore the region around high-value states crucial for solving target tasks, which exacerbates when high-value states are narrowly distributed within the state space.
In this paper, we present novel exploration technique that maximizes value-conditional state entropy which separately estimates the state entropies that are conditioned on the value estimates of each state, then maximizes their average. This intuitively can be seen as partitioning the state space with value estimates and maximizing the average of state entropies of partitioned spaces. Namely, our method avoids the problem of state entropy maximization by preventing the distribution of low-value states from affecting exploration around high-value states, and vice versa, by only considering the states with similar value estimates for computing the intrinsic bonus. For value-conditional state entropy estimation,
utilize the Traskov-SSööbauce ----trbbuee-rrrasbbergg estimator (Kraskov et al., 2004)
along with a value normalization scheme that makes value distribution consistent throughout training. We define intrinsic reward proportional to the value-conditional state entropy estimate and train RL agents to maximize the sum of task reward and intrinsic reward.
We summarize the main contributions of this paper:
We present a novel exploration technique that maximizes value-conditional state entropy which addresses the issue of state entropy exploration in supervised setup by taking into account the value estimates visited states for computing the intrinsic bonus.
We show that maximum value-conditional state entropy (VCSE) exploration successfully accelerates the training of RI. algorithms (Mnih et al., 2016; Yarats et al., 2021a) on a variety of domains, such as MiniGrid (Chevalier-Boiise et al., 2018), DeepMind Control Suite (Tassa et al., 2020), and Meta-We World (Yu et al., 2020) benchmarks.
2 Related Work
Exploration in RL Exploration has been actively studied to solve sparse reward tasks or avoid being stuck at local optima. One of the classical approaches to encourage exploration is e-greedy algorithm (Sutton& & Barto, 2018). This idea has been extended approaches that inject noise into the action space (Wawrzynski, 2015; Lillicrap et al., 2016) or parameter space (Sehnke et al., 2010; Rückstiess al., 2010; Fortunato et al., 2018; Plappert et al., 2018) and approaches that maximize the action entropy (Ziebart, 2010; Haarnoja et al., 2018). Similarly to our work that introduces an intrinsic reward, there have been approaches that use the emoor from predictive models as an intrinsic reward (Schmidhuber, 1991; Oudeyer et al., 2007: Stadie et al., 2015; Pathak et al., 2017, 2019; Sekar al., 2020; Badia et al., 2020). Instead of using the knowledge captured in the model, we use metric that can be guantified from data. The idea of using the state visitation count an intrinsic

---

reward (Thrun, 1992b; Bellemare et al.. 2016; Tang et al.. 2017; Ostrovski et al., 2017; Burda et al., 2019) is also related. However, our method differs that we directly maximize the diversity of data. Our work also differs from line of work that balances exploration and exploitation (Thrun, 1992a; Brafman Tennenholtz, 2002; Tokic, 2010; Whitney et al., 2021; Schäfer et al., 2022) as we instead consider a situation where exploitation biases exploration towards specific state space regions.
Maximum state entropy exploration The approach most related to our work is to utilize state entropy as intrinsic reward. A popular application of maximizing state entropy is unsupervised RI. (Lee et al., 2019; Hazan et al., 2019; Mutti & Restelli, 2020; Mutti et al., 2021; Liu & Abbeel, 2021b; Yarats et al., 2021b; Zhang al., 2021; Guo al., 2021; Mutti et al., 2022a.b; Yang & Spaan, 2023), where the agent learns useful behaviors without task reward. We instead consider setup where task reward is available. A potentially related approach of Yang & Spaan (2023) which introduces safety constraint for exploration in that one can also consider the approach of introducing a value constraint for exploration. However, our work differs in that our motivation is not to discourage exploration on specific state regions. The work closest to ours is approaches that make agents maximize state entropy along with task rewards (Tao et al.. 2020; Seo al., 2021; Nedergaard & Cook, 2022; Yuan et al., 2022). We point out they often suffer from an imbalance between distributions of high-value and low-value states and propose to take into account the value estimates to address this issue.
3 Preliminaries
Definition Let Z (X, Y) be a random variable whose joint entropy is defined as H(X, Y)
((x.,))>p((.g) [logp(x y)] and marginal entropy is defined as H(X) = - -Ez-py (x) tejllog loooox px (z)] The conditional entropy is defined as H(Y X) = E(m.)))((z.a) [log p(v r)), which quantifies the amount of information required for describing the outcome of Y given that the value of known.
Reinforcement learning We formulate problem Markov decision process (MDP; Sutton & Barto 2018), which is defined as a tuple (S, p, r": 7). Here, S denotes the state space, A denotes the action space, p(s++1 |s, a,) is the transition dynamics, r is the extrinsic reward function rf = r (8t, oz), and y E [o, 1). Then we train an agent to maximize the expected return.
3.1 Maximum State Entropy Exploration
k-nearest neighbor entropy estimator When we aim to estimate the entropy of X but the density Px is not available, non-parametric entropy estimators based on k-nearest neighbor distance can be used. Specificallyy, given N i.i.d. samples { the Sorachenko-Leonemn
(KL) entropy estimator (Kozachenko & Leonenko, 1987; Singh et al., 2003; Kraskov et al., 2004) is defined as:
dx N
[g((x) = (kb) (NN) + log Cdx
log Dz),),
where vi is the digamma function, (1) is twice the distance from I; to its k-th nearest neighbor, dx is the dimensionality of X, and Cax is the volume of the dx-dimmesionnal unit ball.2
State entropy as an intrinsic reward Let B be a replay buffer and S be random variable of dimension d, with probability density (s(s)) Exer 1s3x/BB]. The main idea of maximum state entropy exploration is encourage an agent maximize the state entropy H(S) (Liu & Abbeel, 2021b; Seo et al., 2021). The KL entropy estimator in Equation is used to estimate the state entropy (S). and the intrinsic reward ,f is defined by ignoring constant terms and putting logarithm along with an additive constant for numerical stability as below:
ds N
N
AgL(S) = (k)) + (NN) + log Cas +
Σ 1=1
log (i
BE - (O) +1)
(2)
where D2(1) is twice the distance from 8; to its k-nearest neighbor.
For instance, cax = #xxx///1(11 dx/2) for Euclidean where @ as 3.14159 is constant.
3

---

3.2 Conditional Entropy Estimation
Naive conditional entropy estimator Given N ii.d. samples { where 2; (24, ye), the conditional entropy can be estimated by using the chain rule of conditional entropy H(Y (X) Y) ( Specifically, one can compute Hy (X, Y) and (X) and use these estimates for computing (Y X). But a major drawback of this naive approach is that it does not consider length scales in spaces spanned by X and Y can be very different, i.e., the scale of distances between samples and their n-nearest neighbors in each space could very different.
KSG conditional entropy estimator To address the issue of estimating entropy in different spaces spanned by X and Y, we employ Kraskov-Stögbant Sttrbbaurr-rrsssbrrggee (KSG) estimator (Kraskov et al., 2004), which is originally designed for estimating mutual information. The main idea of KSG estimator is to use different k-values in the joint and marginal spaces for adjusting the different length scales across spaces. Given sample in the joint space and the maximum norm || 2' =l max z'11, llv 3'Il,² we find a k-th nearest neighbor ANNN = (a.NN, y, NNN To find a value that makes .:NNN be nz (i)-th nearest neighbor of r; in the space spanned by X, we count the number (1) of points 2j whose distances Di less than ear (1)/2, where ez (x) is twice the distance from 24 .r. .,.NN (see Appendix D for an illustrative example). We note that (y(() and (y can be similarly defined by replacing by y. Now let <ii) be twice the distance between 2; and 2, 2,NN i.e. e(i) 2- ||2--
2,
AANN
naxx-
Then KSG estimators for the joint and marginal entropies given as:
Ags (X,Y) (k) + (NN log(cax cav) + dx dy N
log (i)
dx
+ +1 HKso = N Σ 11) (NN) + log Cdx
+
log (1)
N
Then the chain rule of conditional entropy (X. Hgso (X) and estimators from Equation 3 and Equation 4 to obtain conditional entropy estimator and its lower bound as below:
N
AKsO (Y (x) =
Σ
(n, (i) + + dx (log E(i) - log (i)) + dy log e(i) +
N
(5)
(nx 1) dy log E(i) - (bk) + log Cay
N
where C' denotes (k) + log cay and lower bounds holds because e(i) 2 (z) always holds.
4 Method
We present a new exploration technique that maximizes the coliie-onniiionall state entropy (VCSE), which addresses the issue of state entropy exploration in supervised setup by taking into account the value estimates of visited states for computing the intrinsic bonus. Our main idea is prevent the distribution of high-value states from affecting exploration around low-value states, and vice versa, by filtering out states whose value estimates significantly differ from each other for computing the intrinsic bonus. In this section, we first describe how to define the value-conditional state entropy and use as an intrinsic reward (see Section 4.... We then describe how we train RL agents with the intrinsic reward (see Section 4.2). We provide the pseudocode of our method in Algorithm 1.
4.1 Maximum Value-Conditionss State Entropy
Value-conditional state entropy Let be stochastic policy, f be an extrinsic critic function, and V be a random variable with a probability density py (v) 2 1 (s)=m/ |B] Our key idea is to maximize the colde--coniiional state entropy H(S V = Ev-pye (o) [H(S) V = u)}, which
22 and z' are arbitrary samples from a random variable Z. We also note that this work uses Euclidean norm for ||c x'1 and ly I/1 while any norm can be used for them.

---

end for Algorithm Maximum Value-Conditional State Entropy Exploration
1: Initialize policy म, extrinsic critic J critic Jū. replay buffer B
2: for each environment step & do
3: Collect transition Ti (s, at s(+1,77) and update B + B (T))
4: Sample mini-batch {Tt, 18 j=1 B
5: for j = to do
6:
Find (st, 1/ ( from (ri, J using max(Ils s'11, 11 r'll)
7:
Compute distances ea(t)) f 2- -1s,,,, - s
and (t,) + 2- 1 - 11
ANN
8:
Compute joint state-value space distance e(4,) t- max((s (()))
9:
Compute bias ny (4,) by counting {v, 1/ E (1) (((1))/2 Fz, (c(l)/2)))
Compute intrinsic reward rt; + (ny 1)/dd + log (tt)
11:
Compute total reward rī, + rī, + B r, end for
VOSE
12:
13:
Update policy и, critic für with {(t,, de, +1 r )
Update extrinsic critic f with {(Ry, Bt #1, rt, }]
15:
corresponds to separately estimating the state entropies that are conditioned on the value estimates of each state and then maximizing their average. This intuitively be seen partitioning the visited state space with value estimates and averaging the state entropy of each partitioned space.
Estimation and intrinsic reward To estimate H(S) V=*),, we employ the KSG conditional entropy estimator in Equation 5. Specifically, we estimate the aalue-conditional state entropy as below:
1 Σ N (nn 1) + dv (log c(1) log c. (i)) ds log c()) +C
Agsa (S (
(6)
where denotes (k)) + log Cds. In practice, we maximize the lower bound of value-conditional state entropy because of its simplicity and ease of implementation as below:
Axsa (S (SIV)
(nn((t) + + ds log (oge(() (k+) log Cds
We then define the intrinsic reward r VosE similarly to Equation 2 by ignoring constant terms as below:
1 (n. + 1) log loge(i) where e(i) max ds
VCSE
s; ANN ||0, - (NNN
(8)
To provide intuitive explanation of how reward encourages exploration, note that (sj, Uz) pair whose value , largely differs from v is not likely to be chosen as k-th nearest neighbor because maximum-now will be large due to large value norm ||u uyll- This means that k-th nearest neighbor will be selected among the states with similar values, which corresponds to partitioning the state space with values. Then maximizing <(i) can be seen maximizing the state entropy within each partitioned space. We provide an illustration of our method in Figure 1.
4.2 Training
Objective We train RI. agents to solve the target tasks with the reward r. = rf VCSE where 3 > 0 is scale hyperparameter that adjusts the balance between exploration and exploitation. We do utilize decay schedule for unlike prior work (Seo al., 2021) because our method is designed to avoid redundant exploration. We provide the pseudocode of our method in Algorithm 1.
To compute the intrinsic reward based on value estimates of expected return from environments, we train an extrinsic critic l to regress the expected return computed only with the extrinsic reward rg. Then we use the value estimates from l to compute the intrinsic reward in Equation 8. To train the agent to maximize rT. we train a critic function f that regresses
Implementation detail
5

---

LavaGap87
Enepty- 18x16
DacfKay-Ed6
100
100
A2C
AACx-EE
Success Rate (16)
00
Success Rate (16)
(16)
A2C-VE2E
20
AFC
20
ARCidE ARCMVCE
6410
2-10
4+10
5-10
5-12"
6x10 55koll tetd 1501 2-10
2.5-10 3ond
6-100
1412"
2x10
4000
Ensfronment Stup
Envfronment
Environmmnt Stop
DoorKay-BB
Unlork
SimpleCroosingSi0
ABC+BE
A2D+SE AIC-V
Succses Flate (8)
ARCHVOSE
Shccoss Ratts (14)
82
20-
2
15-10" 20-10" 23-10" soord'
4x10
8x10"
Eriiroement Slep
Envirorment Step
Environment
Stap
Figure 2: Learning curves on six navigation tasks from MiniGrid Sheealler-Boisver et al., 2018) as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
the expected return computed with ri. which the policy aims to maximize. To ensure that the distribution value estimates be consistent throughout training, we normalize value estimates with their mean and standard deviation computed with samples within a mini-batch When an environment partially observable as a eigh-dimensiona observation o, O is only available instead of a fully observable state, we follow a common practice (Mnih et al., 2015) that reformulates the problem as MDP by stacking observations (Ot, 0r-1, Ot- ...) to construct a state 8t. We also note that we use a replay buffer B for entropy estimation following Liu & Abbeel (202 1b); Seo et al. (2021) in that it explicitly encourages the policy to visit unseen, high-reward states which are not the buffer.
Experiments
We design our experiments to evaluate the general- ity of our maximum value-condittonal state entropy (VCSE) exploration sample-efficinny of various RL algorithms (Mnih al., 2016; Yarats et al., 2021a). We conduct exten- sive experiments on a range of challenging and high- dimensional domains, including obbbeally-bbsevvable navigation tasks from MiniGrid Schevalier-Boisvedr al., 2018), pixel- pixel-based locomotion tasks from DeepMind Control Suite (DMC; Tassa et al. (2020)), and pixel-based manipulation tasks from World (Yu et al., 2020).
technique for improving the
Figure 3: Examples of tasks from MiniGrid, DeepMind Control Suite, and MerldWorld.
5.1 MiniGrid Experiments
Setup We evaluate our method navigation tasks from MiniGrid benchmark (Chevalier-Boisver) et al., 2018) consisting of sparse reward goal-reaching tasks. This environment is partially observable the agent has access to a 7 x 3 encoding of the 7 x 7 grid in front of it instead of the full grid.4 4 As a baseline, we first consider RE3 (Seo et al., 2021) which trains Advantage Actor- (A2C; Mnih et al. 2016) agent with state entropy (SE) exploration where the intrinsic reward is obtained using the representations from a random encoder. We build our method upon the official implementation of RE3 by modifying the SE reward with our VCSE reward. Unlike RE3, we do not normalize our intrinsic reward with standard deviation and also do not utilize a separate buffer for computing the intrinsic reward using the on-policy batch. We use l 5 for both SE and VCSE by following the original implementation. See Appendix A for more details.
WWe provide additional results observbbee MiniGrid tasks in Appendix C, where trends similar.

---

Hopper
Walker Sperse
Walker Welk
1,00
1,000
1,000
030
Eploads Retum Retumm
Eptkada
400
630
200
200
DrQu2i5E
230
Droydid
Droigivits
22+10'
5-10
20+10"
30-1h'
42x10"
50-th"
10+10"
22+10"
30-10"
40-10"
10+10"
Enviromment
Cheetah Run Sparss
Caripole Swingup Sparss
Pendulum Swingup
1,000
1,000
1,000
820
800
82D
Epozce Rotu
Episocc Retum
820
820
Episace
220
DOG
250
D10-2-52
D-G-2-95
D:012-92
0x16 10x10"
20w10"
50old
0<16"
10w10"
20-11"
aovid
20-10"
Environnent Step
Environmnt Stop
Environment Step
Figure 5: Learning curves six control tasks from DeepMind Control Suite (Tassa et al., 2020) as measured on the episode return. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
Results Figure shows that A2C+VCSE consistently outperforms A2C+SE on diverse types of tasks, including simple navigation without obstacles (Empty-16x16) navigation with obstacles (LavaGapS7 and SimpleCrossingSNNI) and long-hon oong-horizon navigation (DoorKey-6xx DoorKey-8k. and Unlock). For instance, on LavaGapS7, A2C+VCSE achieves an average success rate of 88.8%, while A2C+SE achieves 13.9% after 100K steps. This shows that VCSE encourages the agent to effectively explore high-value states beyond crossing point. On the other hand, SE excessively encourages the agent visit states located before the crossing point for increasing the state entropy.
Comparison to SE with varying 33 One might think that adjusting the scale of intrinsic rewards could address the issue of state entropy exploration in the supervised setup, by further encouraging explo- ration in high-value states or discouraging exploration in low-value states. However, shown in Figure 4, simply adjusting the scale cannot address the issue. Specifically, A2C+SE fails to solve the task with large = 0.05, because large intrinsic rewards could make the agent ignore the task reward and encour- ages redundant exploration. On the other hand, small 0.0005 also degrades sample-efficiency when compared to the performance with 3 0.005 as it discourages exploration. contrast, A2C+VCSE learns to solve the task in sample-efficiee manner even with different 3 values (see Figure 16 for supporting experiments).
SimpiesrrosingSaN
100
Succoss (96)
80
A2C+YCBE
40
A2C+SE (2-0.06)
20
(000.000)
A2C+SE (p-0.0005)
Ox10
2x10
4x10"
6x10
8x10
10x10"
Environment
Figure Learning curves on SimpleCross- ingS9N1 as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, re- spectively, across eight runs.
5.2 DeepMind Control Suite Experiments
Setup We consider widely-used DeepMind Control Suite (DMC; Tassa et al. 2020) benchmark mainly consisting of locomotion tasks. For evaluation on a more challenging setup, we conduct experiments on pixel-based DMC environments by using state-of-the-ar model-free RL method DrQv2 (Yarats et al., 2021a) as our underlying RL algorithm. 3 For computing the intrinsic bonus, we follow Laskin et al. (2021) by using the features from intrinsic curiosity module (ICM; Pathak et al. 2017) trained upon the frozen DrQv2 encoder. For SE, we find that normalizing the intrinsic reward with its running mean improves performance, which is also done in Laskin et al. (2021). We do not normalize the intrinsic reward for our VCSE exploration. We also disable the noise scheduling
We provide additional results state- based DMC tasks in Appendix C, where trends similar.
7

---

Door Open
Open
Open
100
100
100
60-
60-
Success R.ae
Success Rate
40-
40-
20-
20-
DrQy2idE
20-
proiavvcs
5H10
15x10"
5+10
38-1h
5+10
0+12"
Erifronment Stup
Envfronmmnt Step
Enviromment
Window Open
Button Press
Faurel Close
Paare15)
Rare:%)
Successs Farer%)
30
40
D
DGG D.G---82
DO2
D10-2-5E
D-012-92
0-10"
10"10"
20xld
10w10" 10x10"
Ax10"
Enviroement Step
Environment Stap
Enstronment
Stop
Figure 6: Learning curves six visual manipulation tasks from Meta- Woold (Yu et al., 2020) as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
in DrQv2 for SE and VCSE as we find it conflicts with introducing the intrinsic reward. We instead use the fixed noise of 0.2. We use k 12 for both SE and VCSE. See Appendix A for more details.
Results Figure 5 shows that VCSE exploration consistently improves the sample-efficinny of DrQv2 on both sparse reward and dense reward tasks, outperforming all other baselines. In particular, our method successfully accelerates training on dense reward Walker Walk task, while SE significantly degrades the performance. In Appendix B, we further show that the performance of DrQv2+SE improves with smaller 3, but it still struggles to outperform DrQv2 on Walker Walk. This implies that SE might degrade the sample-efficennyy of RL algorithms on dense reward tasks by encouraging the agent explore states that might not be helpful for solving the task. Moreover, We show that introducing a decaying 3 schedule for SE struggles to improve performance in Appendix B.
5.3
World Experiments
Setup We further evaluate our method on visual manipulation tasks from Meta- Woold benchmark (Yu et al., 2020) that pose challenges for exploration techniques due its large state space with small objects. For instance, moving robot arm toward any direction enables the agent to visit novel states, but would not help solve the target task. As an underlying RI. algorithm, we use DrQv2 (Yarats et al., 2021a). We follow the setup of Seo et al. (2022a) for our experiments by using the same camera configuration and normalizing the extrinsic reward with its running mean make its scale be throughout training. For computing the intrinsic reward, we use the same scheme in DMC experiments (see Section 5.2) by using ICM features for computing the intrinsic reward and only normalizing the SE intrinsic reward. We disable the noise scheduling for all methods and use 12 for SE and VCSE. See Appendix A for more details.
Results Figure 6 shows that VCSE consistently improves the sample-efficiency of DrQv2 on visual manipulation tasks, while SE struggles due to the large state spaces. Notably, DrQv2+ allows for the agent to solve the tasks where DrQv2 struggles to achieve meaningful success rates, e.g., DrQv2+VCSE achieves 85% success rate on Door Open after 100K environment steps while DrQv2 achieves zero success rate. On the other hand, SE struggles to improve the sample-effice eeficleeifccnccy of DrQv2, even being harmful in several tasks. This shows the effectiveness of VCSE for accelerating training in manipulation tasks where state space is very large. We also report the performance of SE and VCSE applied to model- algorithm (Seo et al., 2022a) on Meta-World in Appendix B, where the trend is similar in that SE often degrades the performance due the large state space.
8

---

9C
1,000
802
Eposod Raturn
Epieode Return
Swctees Fari
62-
002
40-
402
20
200
201
4+10
20x10" $6HT0
SExT0" 40o00"
Envirommnn Stap
Envirommnn Step
Ervironmant Stop
(a) Effect of value approximation
(b) Effect of value conditioning
(c) Effect of batch size
Figure 7: (a) Learning curves on SinpleCrossingS91i that compares VCSE using neural networks for value function approximation against VCSE that employs policy evaluation (Sutton Barto, 2018) using the privileged simulator information. Learning curves aggregated on two control tasks from DeepMind Control Suite that investigate the effect of (b) value conditioning and (c) batch size. We provide results on individual task in Appendix B. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
5.4 Ablation Studies and Analysis
Effect value approximation Since we use neural networks for encoding states and approximating the value functions, our objective H(S)V?*s could change throughout training. To investigate the effect of using the approximated value function for our objective, we provide experimental result where we (i) use ground-truth states for computing the intrinsic bonus and (ii) computes the value estimates via policy evaluation (Sutton & Barto, 2018) using the MiniGrid simulator. Figure 7a shows policy evaluation using the privileged knowledge of ground-truth simulator performs the best, but VCSE based on value function approximation with neural networks can match the performance and significantly outperforms SE. This supports that method works intended with the approximated value functions. We expect that our method can be further improved by leveraging advanced techniques for learning value functions (Bellemare al., 2017; Espeholt et al., 2018; Hafner et al., 2023), reaching the performance based on the privileged knowledge.
that VCSE based
Effect of value conditioning We compare VCSE with baseline that maximizes reward-conditional state entropy (RCSE) in Figure 7b to investigate our design choice of conditioning the state entropy on values instead of one-step rewards. The result shows that VCSE largely outperforms RCSE because the value captures more long-term meng-term information about the state when compared to the reward. We note that RCSE cannot be applied to navigation tasks in MiniGrid, where the reward is only given to goal-reaching states, which also demonstrates the benefit of VCSE in terms of applicability.
Effect of batch size We investigate how estimating the value-conditional state entropy using samples from minibatches by reporting the results with increased batch sizes. Specifically, the batch size of 1024 for computing the intrinsic bonus but use the same batch size of 256 or 512 for training the actor and critic of RL agent. As shown in Figure 7c, we find that results are stable with both batch sizes, which shows that our value-conditional state entropy estimation can be stable without using extremely large batch sizes or all the samples within the replay buffer.
use
Heatmap visualization To help understand how VCSE improves sample-efficiency, we provide the visualization of visited state distributions obtained during the training of A2C agents with SE and VCSE. For this analysis, we modified the original SimpleCrossingS9N task to have a fixed map configuration (see Appendix A for more details). As shown in Figure 8a, we find that the agent trained with SE keeps exploring the region before narrow crossing point instead of reaching the goal until 100K steps. even if it has experienced several successful episodes at the initial phase of training. This is because the crossing point initially gets to have a high value and thus exploration biased towards the wide low-value state region, making it difficult to solve the task by reaching the green goal. On the other hand, Figure 8b shows that VCSE enables the agent to successfully explore high-value states beyond the crossing point.

---

SinpleCoosing
25k Sogs
SKC, Sugs
15% Sups
LOOK Sape
25K Siggs
S3K Skaps
2RR Stege
1000 Skg
(a) State entropy exploration
(b) conddiionll state entropy exploration
Figure 8: Visualization of visited state distribution during the training of A2C agent with (a) maximum state entropy (SE) exploration and (b) maximum value-conditional state entropy (VCSE) exploration to solve SimpleCrossing task from MiniGrid Schevalier-Boisverr et al., 2018).
6 Discussion
Limitation and future directions One limitation our work is that we lack theoretical under- standing of how calde-condiiional state entropy exploration works. Considering the strong empirical performance evidenced in our experiments, we hope our work facilitates future research on a the- oretical investigation. Our work only considers non-parametric state entropy estimator due its benefit that does not learn a separate estimator, but considering learning-based estimator can be also an interesting direction. Moreover, applying our idea methods that maximizes the state entropy of policy-inducee state distributions (Lee et al., 2019; Mutti & Restelli, 2020) can be also an interesting future work. Our work might be limited in that works exactly the same the state entropy exploration until the agent encounters a supervised reward from an environment. Developing a method that utilizes intrinsic signal partition the state space into meaningful subspaces without the need for task reward is a future direction we are keen to explore in future work.
Conclusion We present new exploration method that maximizes value-conditional state entropy which addresses the imbalanced distribution problem of state entropy exploration in supervised setup by taking into account the value estimates of visited states for computing the intrinsic bonus. Our extensive experiments show that our method can consistently improve the sample-cfficiency of RL algorithms benchmarks. We hope our simple, intuitive, and stable exploration technique can serve as a standard tool for encouraging exploration in deep reinforcement learning.
variety of tasks from MiniGrid, DeepMind Control Suite, and Meta- World
Acknowledgements and Disclosure of Funding
We want to thank anonymous reviewers and colleagues at KAIST ALIN-LAB and UC Berkeley RLL for providing helpful feedback and suggestions for improving our paper. We also thank Shi Zhuoran for helpful comments on the draft. This research is supported by Institute of Information & Commu- nications Technology Planning Evaluation (ITPP) grant funded by the Korea government(MSIT) (No.2022-0-00953. Self-directed AI Agents with Problem-solving Capability; No.20190-00075. Artificial Intelligence Graduate School Program (KAIST)).
References
Badia, Adrià Puigdomènech, Sprechmann, Pablo, Vitvitskyi, Alex, Guo, Daniel, Piot, Bilal, Kap- turowski, Steven, Tieleman, Olivier, Arjovsky, Martín, Pritzel, Alexander, Bolt, Andew, et al. Never give up: Learning directed exploration strategies. In International Conference Learning Representations, 2020.
Bellemare, Marc, Srinivasan, Sriram, Ostrovski, Georg, Schaul, Tom, Saxton, David, and Munos, Remi. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Informa- tion Processing Systems, 2016.
Bellemare, Mare G, Dabney, Will, and Munos, Rémi. A distributional perspective on reinforcement learning. In International Conference Machine Learning, 2017.
Bingham, Ella and Mannila, Heikki. Random projection in dimensionality reduction: applications to image and text data. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, 2001.
10

---

Brafman, Ronen and Tennenholtz, Moshe. R-max-a general polynomial time algorithm for near- optimal reinforcement learning. Journal of Machine Learning Research, 2002.
Burda, Yuri, Edwards, Harrison, Storkey, Amos, and Klimov, Oleg. Exploration by random network
distillation. In International Conference on Learning Representations, 2019.
Boiiser Maxime, Willems, Lucas, and Pal, Suman. Minimalistic gridworld environment for openai gym. https://githee com wommmaxximcb////oeoooooooeoooeem gengggggnnmniiin 2018. 2018.
Espeholt, Lasse, Soyer, Hubert, Munos, Remi, Simonyan, Karen, Mnih, Vlad, Ward, Tom, Doron, Yotam, Firoiu, Vlad, Harley, Tim, Dunning, lain, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. International Conference on Machine Learning, 2018.
Fortunato,
Meire, Azar, Mohammad Gheshlaghi, Piot, Bilal, Menick, Jacob, Osband, Ian, Graves, Alex, Mnih, Vlad, Munos, Remi, Hassabis, Demis, Pietquin, Olivier, et al. Noisy networks for exploration. International Conference on Learning Representations, 2018.
Guo, Zhaohan Daniel, Azar, Mohammad Gheshlaghi, Saade, Alaa, Thakoor, Shantanu, Piot, Bilal, Pires, Bernardo Avila, Valko, Michal, Mesnard, Thomas, Lattimore, Tor, and Munos, Rémi. Geometric entropic exploration. arXiv preprint arXiv:2101.0000 2021.
Haarnoja, Tuomas, Zhou, Aurick, Abbeel, Pieter, and Levine, Sergey. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, 2018.
Hafner, Danijar, Lillicrap, Timothy, Norouzi, Mohammad, and Ba, Jimmy. Mastering atari with discrete world models. In International Conference Leurning Representations, 2021. 2021.
Hafner, Danijar, Pasukonis, Jurgis, Ba, Jimmy, and Lillicrap, Timothy, Mastering diverse domains through world models. arXiv preprint S.KV:22301.01000 2023.
Hazan, Elad, Kakade, Sham, Singh, Karan, and Van Soest, Abby. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, 2019.
Herzog, Alexander, Rao, Kanishka, Hausman, Karol, Lu, Yao, Wohlhart, Paul, Yan, Mengyuan, Lin, Jessica, Arenas, Montserrat Gonzalez, Xiao, Ted, Kappler, Daniel, et al. Deep rl scale: Sorting waste in office buildings with fleet of mobile manipulators. arXiv preprint arXiv:2305.03270 2023.
Kalashnikov, Dmitry, Varley, Jacob, Chebotar, Yevgen, Swanson, Benjamin, Jonschkowski, Rico, Finn, Chelsea, Levine, Sergey, and Hausman, Karol. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXi::2 2021.
Kozachenko, Lyudmyla F and Leonenko, Nikolai N. Sample estimate of the entropy of random vector. Problemy Peredachi Informatsii, 23(2):9-16, 1987.
Kraskov, Alexander, Stögbauer, Harald, and Grassberger, Peter. Estimating mutual information. Physical review E, 69(6):066133, 2004.
Laskin, Michael, Yarats, Denis, Liu, Hao, Lee, Kimin, Zhan, Albert, Lu, Kevin, Cang, Catherine, Pinto, Lerrel, and Abbeel, Pieter. Urlb: Unsupervised reinforcement learning benchmark. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Lee, Lisa, Eysenbach, Benjamin, Parisotto, Emilio, Xing, Eric, Levine, Sergey, and Salakhutdinov, Ruslan. Efficient exploration via state marginal matching, arXiv preprint D00.:vv9900.05277.. 2019.
Lillicrap, Timothy P. Hunt, Jonathan J, Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, and Wierstra, Daan. Continuous control with deep reinforcement learning. In International Conference Learning Representations, 2016.
Liu, Hao and Abbeel, Pieter. Aps: Active pretraining with successor features. International Conference Machine Learning, 2021a.

---

Liu, Hao and Abbeel, Pieter. Behavior from the void: Unsupervised active pre-training, In Advances in Neural Information Processing Systems, 2021b.
Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, al. Human- level control through deep reinforcement learning. nature, 2015.
Mnih, Volodymyr. Badia, Adria Puigdomenech, Mirza, Mehdi, Graves, Alex, Lillicrap, Timothy, Harley, Tim, Silver, David, and Kavukcuoglu, Koray. Asynchronous methods for deep reinforce- ment learning. In International Conference on Machine Learning, 2016.
Mutti, Mirco and Restelli, Marcello. An intrinsically -nttiivvalyy approach for learning highly exploring and fast mixing policies. In Proceedings of the AAAI Conference Artificial Intelligence, 2020.
Mutti, Mirco, Pratissoli, Lorenzo, and Restelli, Marcello. A policy gradient method for task-agnostic exploration. In Proceedings of the AAAI Conference Artificial Intelligence, 2021.
Mutti, Mirco, De Santi, Riccardo, and Restelli, Marcello. The importance of non markoviaaitty in maximum state entropy exploration. In International Conference on Machine Learning, 2022a.
Mutti, Mirco, Mancassola, Mattia, and Restelli, Marcello. Unsupervised reinforcement learning in multiple environments. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022b.
Nedergaard, Alexander and Cook, Matthew. k-means maximum entropy exploration. arXiv preprint arXiv:2205.15663, 2022.
Ostrovski, Georg, Bellemare, Mare G. Oord, Aaron van den, and Munos, Rémi. Count-based exploration with neural density models. In International Conference on Machine Learning, 2017.
Oudeyer, Pierre Yves, Kaplan, Frdric, and Hafner, Verena V. Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation, 2007.
Pathak, Deepak, Agrawal, Pulkit, Efros, Alexei A, and Darrell, Trevor. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, 2017.
Pathak, Deepak, Gandhi, Dhiraj, and Gupta, Abhinav. Self-supervised exploration via disagreement. In International Conference on Machine Learning, 2019.
Plappert, Matthias, Houthooft, Rein, Dhariwal, Prafulla, Sidor, Szymon, Chen, Richard Y, Chen, Xi, Asfour, Tamim, Abbeel, Pieter, and Andrychowicz, Marcin. Parameter space noise for exploration. In International Conference on Learning Representations, 2018.
Rückstiess, Thomas, Sehnke, Frank, Schaul, Tom, Wierstra, Daan, Sun, Yi, and Schmidhuber, Jürgen. Exploring parameter space in reinforcement learning. Paladyn, 1:14-24, 2010.
Schäfer, Lukas, Christianos, Filippos, Hanna, Josiah P. and Albrecht, Stefano V. Decoupled rein- forcement learning to stabilise maaasiically-moivattee exploration. In International Conference on Autonomous Agents and Multiagent Systems, 2022.
Schmidhuber, Jürgen. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991.
Sehnke, Frank, Osendorfer, Christian, Rückstieß, Thomas, Graves, Alex, Peters, Jan, and Schmidhu- ber, Jürgen. Parameter-exploring policy gradients. Neural Networks, 23(4)5551-559, 2010.
Sekar, Ramanan, Rybkin, Oleh, Danilidis, Kostas, Abbeel, Pieter, Hafner, Danijar, and Pathak, Deepak. Planning explore via sepe-supe world models. In International Conference Machine Learning, 2020.
Seo, Younggyo, Chen, Lili, Shin, Jinwoo, Lee, Honglak, Abbeel, Pieter, and Lee,, Kimin. State entropy maximization with random encoders for efficient exploration. International Conference on Machine Learning, 1021.
12

---

Seo, Younggyo, Hafner, Danijar, Liu, Hao, Liu, Fangchen, James, Stephen, Lee, Kimin, and Abbeel, Pieter. Masked world models for visual control. In Conference on Robot Learning, 2022a.
Seo, Younggyo, Lee, Kimin, James, Stephen, and Abbeel, Pieter. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning, 2022b.
Seyde, Tim, Schwarting, Wilko, Karaman, Sertac, and Rus, Daniela. Learning to plan optimistically: eneertainty-uuidee deep exploration via latent model ensembles. In Conference on Robot Learning, 2021.
Singh, Harshinder, Misra, Neeraj, Hnizdo, Vladimir, Fedorowicz, Adam, and Demchuk, Eugene. Nearest neighbor estimates of entropy. American journal of mathematical and management sciences, 2003.
Stadie, Bradly C, Levine, Sergey, and Abbeel, Pieter. Incentivizing exploration reinforcement learning with deep predictive models. arXiv preprint 1xXiv:507.0088 2015.
Sutton, Richard S and Barto, Andrew G. Reinforcement learning: An introduction. MIT Press, 2018.
Tang, Haoran, Houthooft, Rein, Foote, Davis, Stooke, Adam, Chen, OpenAl Xi, Duan, Yan, Schulman, John, DeTurck, Filip, and Abbeel, Pieter. # exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
Tao, Ruo Yu, François-Lavet, Vincent, and Pineau, Joelle. Novelty search in representational space for sample efficient exploration. In Advances in Neural Information Processing Systems, 2020.
Tassa, Yuval, Tunyasuvunakool, Saran, Muldal, Alistair, Doron, Yotam, Liu, Siqi, Bohez, Steven, Merel, Josh, Erez, Tom, Lillicrap, Timothy, and Heess, Nicolas. dm_control: Software and tasks for continuous control. arXiv preprint arXiv:2006. 12983, 2020.
Thrun, Sebastian. Efficient exploration reinforcement learning. Technical Report. Carnegie Mellon University, 1992a.
Thrun, Sebastian The role of exploration in learning control with neural networks. Handbook of intelligent control: Neural, fuzzy and adaptive approaches, 1992b.
Tokic, Michel. Adaptive e-greedy exploration in reinforcement learning based value differences. In KI 2010: Advances in Artificial Intelligence: 33rd Annual German Conference on A1, Karlsruhe, Germany, September 21-24, 2010. Proceedings 33. Springer, 2010.
Wawrzynski, Pawel. Control policy with autocorrelated noise in reinforcement learning for robotics. International Journal of Machine Learning and Computing, 2015.
Whitney, William F, Bloesch, Michael, Springenberg, Jost Tobias, Abdolmaleki, Abbas, Cho, Kyunghyun, and Riedmiller, Martin. Decoupled exploration and exploitation policies for sample- efficient reinforcement learning. arXiv preprint arXiv:2101.09456 2021.
Yang, Qisong and Spaan, Matthijs TJ. Cem: Constrained entropy maximization for task-agnostic safe exploration. The Thirty-Sevntth AAAI Conference on Artificial Intelligence, 2023.
Yarats, Denis, Fergus, Rob, Lazaric, Alessandro, and Pinto, Lerrel. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint 207.v:2107.09645 2021a.
Yarats, Denis, Fergus, Rob, Lazaric, Alessandro, and Pinto, Lerrel. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, 2021
Yu, Tianhe, Quillen, Deirdre, He, Zhanpeng, Julian, Ryan, Hausman, Karol, Finn, Chelsea, and Levine, Sergey. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, 2020.
Yuan, Mingqi, Pun, Man-On, and Wang, Dong. Rényi state entropy maximization for exploration acceleration in reinforcement learning, IEEE Transactions Artificial Intelligence, 2022.
13

---

Zhang, Chuheng, Cai, Yuanying, Huang, Longbo, and Li. Jian. Exploration by maximizing rényi entropy for reward-free rl framework. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Ziebart, Brian D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy, 2010.
14

---

Societal Impact
We do not anticipate immediate negative consequences from conducting this work because our experiments are based on simulation environments designed to conceptually evaluate the capabilities of reinforcement learning (RL) algorithms. Recent studies, however, demonstrate that large-scale RL when integrated with robotics can effectively work on real-world environments (Kalashnikov et al., 2021; Herzog et al., 2023). This makes it crucial to be aware of potential societal harms that RL agents could inadvertently cause. While this work does not aim to address these safety issues, method might mitigate unintentional harm during the training process. For instance, it might prevent the agent from exhibiting potentially dangerous novelty-seeking behaviors, such moving robot arms towards lawuvalue, empty regions where a human researcher is likely to be situated.
A Experimental Details
Compute For MiniGrid experiments, we use a single NVIDIA TITAN Xp GPU and 8 CPU cores for each training run. It takes 15 minutes for training the agent for 1M environment steps. For DeepMind Control Suite and Meta- World experiments, we use a single NVIDIA 2080Ti GPU and CPU cores for each training run. It takes 36 minutes and 90 minutes for training the agent for 100K environment steps on DeepMind Control Suite and Woold benchmarks, respectively.
A.1 Implementation Details
Value normalization For normalizing value estimates to stabilize value-conditionaa state entropy estimation, we compute the mean and standard deviation using the samples within the We empirically find no significant difference to using the running estimate.
Extrinsic critic function For training the extrinsic critic function described in Section 4.2, introduce another set of critic and target critic functions based on the same hyperparameters used for the main critic the policy aims maximize. Then estimates. We apply the stop gradient operation to inputs to disable the gradients from updating the extrinsic critic to update other components. For the policy, use the same policy for training both main and extrinsic critic functions. We empirically find no need for training another policy solely for the extrinsic critic.
the target critic for obtaining value
A2C implementation details We use the official implementation' of RE3 (Seo et al., 2021) and use the same set of hyperparameters unless otherwise specified. Following the setup of RE3, we use fixed, randomly initialized encoder to extract state representations and use them for computing the intrinsic reward. We use the same hyperparameter of fixed intrinsic scale 3 = 0.005 and k for both SE and VCSE following the original implementation. For RE3, normalize the intrinsic reward with its standard deviation computed using the samples within the mini-batch, following the original implementation. But we do not normalize our VCSE intrinsic reward.
DrQy2 implementation details We use the official implementation" of DrQv2 (Yarats et al., 2021a) and use the same set of hyperparameters unless otherwise specified. For both SE and VCSE exploration, we find that using 3 = 0.1 achieves the overall best performance. We also use k = 12 for both SE and VCSE. For computing the intrinsic reward, we follow the scheme of Laskin et al. (2021) that trains Intrinsic Curiosity Module (ICM; Pathak et al. 2017) upon the representations from a visual encoder and uses ICM features for measuring the distance between states. We note that we detach visual representations used for training ICM to isolate the effect of training additional modules on the evaluation. For both SE and VCSE exploration, we disable the noise scheduling scheme of DrQv2 that decays from 1 by following a pre-defined schedule provided by the authors. This is because we find that such a noise scheduling conflicts with the approaches that introduce additional intrinsic rewards. Thus we use the fixed noise of 0.2 for SE and VCSE exploration. For Meta-World experiments, we also disable the scheduling for the DrQv2 bascline as we find it performs better. But we use the original scheduling for the DrQv2 bascline following the official implementation.
15

---

Figure 9: Examples of tasks in our MiniGrid experiments: (a) LavaGapS7, (b) Empty-16x 16, (c)
DoorKey-6x6, (d) DoorKey-8x x88, (e) Unlock, and (f) SimpleCrossingSNN.
Figure 10:
Examples of tasks we used in our Meta- World experiments: (a) Door Open, (b) Drawer
Open, (c) Faucet Open, (d) Window Open, (c) Button Press, and (f) Faucet Close.
Figure 11:
Examples of tasks used in our DeepMind Control Suite experiments: (a) Hopper, (b)
Walker, (c) Cheetah, (d) Cartpole, (e) Pendulum
Heatmap analysis details For experiments in Figure 8, we use the easy version of SimpleCross- ingS9N1 I task from MiniGrid benchmark (Chevalier-Boisee et al., 2018). Specifically, we disable the randomization of map configurations to make possible investigate the heatmap over fixed map. For visualizing the heatmaps, we record position of agents during the initial 100K steps. We train A2C agent with both SE and VCSE exploration as we specified Section 5.1 and Appendix A. without any specific modification for this experiment.
A.2 Environment Details
MiniGrid We conduct our experiments on six navigation tasks from MiniGrid benchmark "Chevalier-Boisver et al.. 2018): LavaGapS7, Empty-16 16, DoorKey-6x DoorKey-8xx 888, Unlock, and SimpleCrossingS9N.. We provide the visualization of the tasks in Figure We use the original tasks without any modification for our experiments in Section 5.1.
Meta-World We conduct our experiments on six manipulation tasks from Meta-Wor World benchmark (Yu et al., 2020): Door Open, Drawer Open, Faucet Open, Window Open, Button Press, and Faucet Close. We provide the visualization of the tasks in Figure 10. We follow the setup of Seo al. (2022a) that uses fixed camera location for all tasks.
DeepMind Control Suite We conduct our experiments on six locomotion tasks from DeepMind Control Suite benchmark (Tassa et al., 2020): Hopper Stand, Walker Walk Sparse, Walker Walk, Cheetah Run Sparse, Cartpole Swingup Sparse, and Pendulum Swingup. We use the sparse reward tasks introduced in Seyde et al. (2021), by following RE3. We provide the visualization of the tasks Figure 10.
16

---

B Additional Experiments
B.1 Experiments with Model-Based RL
Setup As a model-based underlying RL algorithm, we consider Masked World Models (MWM; Sco al. 2022a) that has shown to be able solve more challenging, long-horizon tasks compared to DrQv2. We consider four tasks of various difficulties: Box Close, Handle Pull Side, Lever Pull, and Drawer Open. We use the official implementation" of MWM (Seo et al., 2022a) and use the same set of hyperparameters unless otherwise specified. For both SE and VCSE exploration, we find that using 3 = performs best. Following the idea of Seo et al. (2022b) that introduces an additional reward predictor for intrinsic reward in the world model of DreamerV2 (Hafner et al., 2021), we introduce the reward network that predicts our intrinsic reward r. VCSE, For computing the intrinsic reward, we also follow the idea of Seo et al. (2022b) that uses random projection (Bingham & Mannila, 2001) to reduce the compute cost of measuring distances between states. Specifically, project 2048- 1948-dimensional model states into 256-dimensional vectors with random projection. Because the original MWM implementation normalizes the extrinsic reward by its running estimate of mean to make its scale throughout training, find that also normalizing intrinsic rewards with their running estimates of mean stabilizes training. We use k = 12 for both SE and VCSE.
Results Figure 12 shows that VCSE consistently accelerates and stabilizes the training of MWM agents on four visual manipulation tasks of different horizons and difficulties, which shows that the effectiveness of our method is consistent across diverse types of RL algorithms. On the other hand, we observe that SE could degrade the performance, similar to our observation from experiments with DrQv2 Meta- World tasks (see Figure 6). This supports our claim that SE often encourages exploration to be biased towards low-value states especially when high-value states are narrowly- distributed, considering that manipulation tasks have very narrow task-relevant state space.
Soxe
Landsa Side
Lover
Dowar Open
SLCOOD HER
VCSE
20-1"" 1501t
15012" Erattormatt Stop
19612"
511"
6:12"
Ervinimment Stop
Eraitonmett Stop
Enwitorant
Figure 12: Learning curves six visual manipulation tasks from Meta-World (Yu et al., 2020) as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
B.2 Ablation Study
In Figure 13, we provide the results on individual task used for reporting the aggregate performance that investigate the effect of value conditioning and batch size (see Figure 7b and Figure 7c).
Cartoole Sevtnger Sparne
Hopper Stard
Cergole Svirgue Sporss
Hopper Stand
200
VIDE
aorid" 2010
0'12" 10'10 20n18
2000
2001? sprit
20417 10117"
(a) Effect of value conditioning
(b) Effect of batch size
Figure 13: Learning curves on two visual locomotion control tasks from DeepMind Control Suite that investigate the effect of (a) value conditioning and (b) batch size. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across eight runs.
17

---

B.3 Experiments with Varying Intrinsic Reward Hyperparameters
A2C+VCSE with varying k We provide additional experimental results with varying f3.5. 7] on SimpleCrosingS9NI task in Figure 14. We find that the performance of A2C+VCSE tends to degrade as & increases. We hypothesize this because higher & leads finding less similar states and this often leads to increased intrinsic reward scale.
SimplaCCrossnggSii
Stutcaoo Rain (76
60
60
20-
Alceettle jong jark
8-10"
50H10"
Step
Learning curves as measured on the episode return. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across eight runs.
Figure 14:
DrQv2+SE on walker walk with varying 3 We provide additional experimental results with varying B (0.1,0.01. 0.001} on Walker Walk task where DrQv2+SE significantly struggles to improve the sample-efficinnc of DrQv2. In Figure 15, we find that the performance of DrQv2+SE consistently worse than the vanilla DrQv2 with different values. This implies that adding SE intrinsic reward can be sometimes harmful for performance by making difficult for the agent to exploit the task reward.
Walker Walk
903
Epiooda Rotumn
000
400
200
50-10"
20-10"
20-15"
S0-10"
Step
Learning curves as measured on the episode return. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across eight runs.
Figure 15:
A2C+VCSE with varying 3 We also report the experimental results of A2C+VCSE with varying in Figure 16. We find that A2C+VCSE consistently improves A2C, in contrast to A2C+SE which fails to significantly outperform A2C even with different 3 values (see Figure 4).
Succes Ran (h
8
8
Aecorreer (oin mis (w0
0+10
2-10
4-10"
6x10"
9+10"
10+10"
Enveenmentt
Learning curves measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across eight runs.
Figure 16:
DrQv2+SE with decaying 3 the state entropy baseline that uses decaying schedule for 3, similarly to Seo al. (2021) that uses schedule for the intrinsic reward. In Figure 17, we find that such a schedule cannot significantly
We conduct additional experiments that compare DrQv2+CCSE with
18

---

improve the performance of SE, except Hopper Stand where the performance is stabilized. Moreover, we find that the decaying schedule sometimes could degrade the performance, i.e., Walker Walk Sparse. We also note that designing such decaying schedule is a tedious process that requires researchers to tune the performance, making less desirable even it works reasonably well, We indeed find that the performance becomes very sensitive to the magnitude of decaying schedule." On the other hand, DrQv2+VCSEB exhibits consistent performance without the decaying schedule, which highlights the benefit of our approach that maximizes the value-conditional state entropy.
Hopper
Sporse
Walker Walk
1,000
1,000
0-0x2
1,000
D-GI2YSE (Pleed
D'GUONSE (Decay B1
Eploads Retum
Fplcoda Ruturn
6:00
(Deaay
430
OXGvEW/CE
Drov
200
Dr (Foed
Dx (Decay
200
200
0400
22-10' 30-10' 42-10"
Spoop
6x10"
20+10"
30<10'
42x10"
5pn0
0400
12+10"
20-10"
30-1s"
42-10"
50-10"
Environment
Cheetah Run Sparaa
Caripole Swing.p Sparsa
Pendulum Swingup
1,000
1,000
1,000
030
OrOvc+SE
000
000
Epasoe Retum
Epicots Ratum.
600
Epiaatia
230
28x10" Environment Step
bvin
10x10"
20x10"
43x10" Sext0"
bx10
30x10"
28x10"
30w10"
4tx10"
50x10"
Environment Step
Environment Step
Figure 17: Learning curves on six visual locomotion control tasks from DeepMind Control Suite (Tassa et al., 2020) as measured on the episode return. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across eight runs.
C Experiments with Ground-Trute States
C.1 MiniGrid Experiments
To demonstrate that our method also works fully-observable MiniGrid where we do not use state encoder for computing the intrinsic bonus, we provide additional experiments that use fully observable states instead of partially observable grid encoding (see Section 5. Specifically, we use set of one-hot vectors that represents a current map as inputs to the agent, and use the location of agent as inputs for computing the intrinsic bonus. Figure 18 shows that VCSE consistently accelerates the training, which highlights the applicability of VCSE to diverse domains with different input types.
traate Ihe'd
LineSladt
Doatfoy
Neooen 2h11
6m1 2-12
o012
m010 Enattorant Stop
e:12
2012
4012
10-20
2012
Ieatoormnt Stop
Enatorrert Stop
Enstrrrett
Figure 18: Learning curves on four navigation tasks from fully-observable MiniGrid (Chevalier- Boisvert et al., 2018) as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
'Due to the unstable performance from introducing the decay schedule, we run experiments with multiple decaying schedules and report the best performance for each task.
19

---

C.2 Deep Control Suite Experiments
We further report experimental results state-based DeepMind Control Suite experiments where we do not use state encoder for computing the intrinsic bonus. To make the scale of state norms be consistent across diverse tasks with different state dimensions, we divide the state input with its state dimension before computing the intrinsic bonus. As our underlying RI. algorithm, we used Soft Actor- Critic (SAC; Haarnoja et al. 2018). For SE and VCSE, we disabled automatic tuning hyperparameter a and used lower value of 0.001 in SAC, because we find that such an automatic tuning conflicts with introducing the intrinsic reward, similar to noise scheduling in DrQv2. Figure 19 shows that VCSE consistently accelerates the training, which highlights the applicability of VCSE to diverse domains with different input types.
Ep0000 Maam
802
Secrsica
200
19912
Enerserent
Figure 19: Learning curves on four locomotion tasks from state-based DeepMind Control Suite (Tassa et al., 2020) as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
D Additional Illustrations
We provide the additional illustration that helps understanding the procedure of estimating the conditional entropy with KSG estimator, which is explained in Section 3.2.
ANN
(#s, vt)
cx(i)
(x)
Figure 20: Illustration of procedure for computing <z. (1) and (1) when using the KSG estimator with k = 2. Given a centered point 2,, we first find k-nearest neighbor state ANN Then eg (i) is twice the distance from 2; to x ANN and ny can be computed as by counting all the points located within (z; (2) ex (1) We note that ey (1) and (i) can be also similarly computed.
20


Reviews:
[{"review_id": "FhdcakfX9e", "reviewer": "Reviewer_CsYB", "summary": "The paper investigates an exploration technique that maximizes the entropy of visited states while taking into account the expected return of the states. The goal is to explore the part of the state space that is both less visited while avoiding too much exploration for the low-value states.", "questions": "Can you clarify my main concern (see above). If so, I would happily increase my score.", "limitations": "The authors adequately address limitations.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is overall well written and presents an interesting approach to taking into account an estimate of the expected return together with state entropy exploration. ", "weaknesses": "My main concern is that it is a bit unclear how the presented algorithm actually enforces more exploration on the high expected return states as compared to previous approaches. In Figure 8, I'm unsure how VCSE will have a different behaviour than SE given that the agent needs to at least visit once the reward to be able to see that there is a lower bonus by keeping exploring the low expected return states. In addition, previous approaches that use a combination of intrinsic and extrinsic rewards should also tend to visit more the high expected return parts of the state space. \n\nAdditional minor comments:\n- I'm unsure why \"supervised setup\" is mentioned in line 4 (abstract) and line 29.\n- line 91: $\\gamma$ can't be 1.\n- Typos: line 105", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer CsYB,\n\nWe sincerely appreciate your efforts and insightful comments to improve the draft. We respond to each of your comments one-by-one in what follows.\n\n--------\n\n**Q1. Unclear how the presented algorithm actually enforce more exploration on the high expected return state as compared to previous approach.**\n\n**A1.** We would like to clarify that VCSE does not aim to encourage more exploration only on high-value state space, but aims to prevent a scenario where the exploration is biased towards specific regions within the state space. Specifically, VCSE prevents the states with drastically different value estimates from having an effect on computing the intrinsic reward. This prevents the distribution of low- (or high-) value states from affecting exploration around high- (or low-) value states. Thus VCSE can encourage uniform exploration within each value-conditional state space, which is the main difference to the previous approaches that uses a combination of intrinsic and extrinsic reward for exploration. We will try to further clarify this in the final draft.\n\nFor instance, in Figure 8 of the original draft, one can expect that VCSE and SE would behave exactly the same before it encounters a reward from an environment. We observed that both methods indeed experience successful episodes and the agent learns to visit more states within the optimal path towards the goal at similar steps for both methods. This starts to make a visible difference between SE and VCSE. Specifically, this high visitation count within the optimal path leads to higher SE intrinsic rewards for non-optimal states outside the path, which biases exploration towards non-optimal states. On the other hand, VCSE encourages exploration on both state regions within and outside the optimal path, allowing for quickly learning to solve the target tasks.\n\n--------\n\n**Q2. I'm unsure why \"supervised setup\" is mentioned in line 4 (abstract) and line 29.**\n\n**A2.** We use the “supervised setup” term to denote that we consider a setup where the external task reward is available from the environment, following the terminology used in [Laskin et al., 2021].\n\n[Laskin et al., 2021] Laskin, Michael, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. \"URLB: Unsupervised reinforcement learning benchmark.\" arXiv preprint arXiv:2110.15191 (2021).\n\n--------\n\n**Q3. $\\gamma \\in [0, 1]$**\n\n**A3.** We followed the notation of [Sutton & Barto, 2018] where the discount factor $\\gamma$ is defined to be located within $[0, 1]$. We also note that $\\gamma$ can be set to 1 if the task is episodic so that there is a terminating state [Pitis, 2019]\n\n[Sutton & Barto, 2018] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.\n\n[Pitis, 2019] Pitis, Silviu. \"Rethinking the discount factor in reinforcement learning: A decision theoretic approach.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 7949-7956. 2019.\n\n--------\n\n**Q4. Typos error in line 105**\n\n**A4** Thank you for pointing this out. We’ll fix this in the final draft.\n"}, {"review_id": "Y1oHSNmpCG", "reviewer": "Reviewer_MwTR", "summary": "The paper proposes an improvement over a popular intrinsic reward based on state entropy. Instead of encouraging large state entropy uniformly over all states which may deviate the policy toward failure states, the proposed intrinsic reward motivates the agent to maximize the value-conditioned state entropy. The value-conditioned state entropy is estimated with a classical kNN method.", "questions": "Besides the effect of k, I wonder how if authors have tried other entropy/mutual information estimator (for example, the one based on contrastive learning) and why ending up selecting the kNN one.", "limitations": "kNN method for entropy estimation usually suffers in high-dimensional space, but the proposed method works well in environments with visual observations. So I don't find any significantly limitation.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The intrinsic reward is an important direction of improving RL exploration and sample efficiency, and how to balance between instrinsic and extrinsic reward is a challenge for intrinsic reward design. This paper proposes a novel method to achieve this goal --- conditioning the state entropy on the state value to partition the state space and focus on state entropy over states with similar value. The paper is well written and the proposed method is strongly supported by a large range of experiments.", "weaknesses": "There is no significant weakness that I notice except for an ablation study on the effect of k for the kNN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer MwTR,\n\nWe sincerely appreciate your efforts and insightful comments to improve the draft. We respond to each of your comments one-by-one in what follows.\n\n---\n\n**Q1. Analysis on the effect of $k$**\n\n**A1.** Following your suggestion, we conducted additional experiments with $k \\in \\{3, 5, 7\\}$ and observed that the performance of our method tends to degrade as $k$ increases As shown in Figure 3 available from the attached PDF in the general response, We hypothesize this is because higher $k$ leads to finding less similar states and this often leads to increased intrinsic reward scale. We will include the relevant results in the final draft.\n\n---\n\n**Q2. Other entropy/mutual information estimator?**\n\n**A2.** It’s an interesting question. We employed a $k$-NN estimator following prior work, mainly because it’s non-parametric so that it does not incur additional training cost of training another neural network based estimator. But investigating the possibility of leveraging other estimators for online RL is definitely an interesting future direction and we will include the relevant discussion in the final draft. Thank you!"}, {"review_id": "n7BR3bhQBt", "reviewer": "Reviewer_pL2M", "summary": "This paper present a exploration technique that maximizes the value-conditional state entropy, which separately estimates the state entropies that are conditioned on the value estimates of each state, then maximizes their average. By only considering the visited states with similar value estimates for computing the intrinsic bonus, it prevents the distribution of low-value states from affecting exploration around high-value states, and vice versa. The experiments demonstrate that the proposed alternative to the state entropy baseline significantly accelerates various reinforcement learning algorithms across a variety of tasks within MiniGrid, DeepMind Control Suite, and Meta-World benchmarks.\n", "questions": "(1) In line 117, what is the meaning of  z and z’? \n\n(2) In Equ.(1)，what is the meaning of dx ? It seems unexplained.\n\n(3) In section 5, the selected experimental environment is relatively simple with a low dimension. Can we select some complex environments with a high-dimension state and action spaces for verification?\n\n(4) In Fig1, how to do “State Norm” and “Value Norm”?\n\n(5) In Fig1, why do you only choose the third one(kNN (k=3))instead of the others？\n", "limitations": "In section 6, the authors talk about some limitation and future directions about this paper. At present, the potential negative social impact of this work has not been identified.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The exploration method proposed in this paper show that maximum value-conditional state entropy (VCSE) exploration successfully  accelerates the training of RL algorithms. It can be used for reference in other RL algorithms.\n\n(2) For section 3, the development process of the entropy estimator is very clearly described.\n\n(3) For section 5, extensive experiments have been conducted to demonstrate the effectiveness of the algorithm proposed in this paper.\n", "weaknesses": "(1) The writing of this article seems difficult to understand in some aspects，especially for certain formulas.\n\n(2) The motivation of this article is not fully explained.\n\n(3) To be honest，it lacks a certain degree of novelty. As is well known, the methods used in this paper like “k-nearest neighbor entropy estimator” and “KSG conditional entropy estimator” are all previous work. We just utilize some of them.\n\n(4) The main body of this paper is clear to understand, here is space for improvement. I defer some of my issues in the appendix to \"Questions\".\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer pL2M,\n\nWe sincerely appreciate your efforts and insightful comments to improve the draft. We respond to each of your comments one-by-one in what follows.\n\n---\n\n**Q1. Motivation and Novelty**\n\n**A1.** In this work, we aim to improve the sample-efficiency of deep RL algorithms by introducing a novel exploration technique that can address the problem of balancing exploration and exploitation, which is an important and challenging problem for deep RL as highlighted by Reviewer MwTR and FYPf. Specifically, we propose a new exploration technique that builds upon a widely-used state entropy (SE) exploration by pointing out that SE suffers from an imbalance between the distributions of high-value and low-value states and introducing a new objective that takes into account the value estimates of states. \n\nWe would also like to emphasize that our key novelty lies in introducing a new objective that maximizes the value-conditional state entropy $H(S|V)$ instead of the state entropy $H(S)$. To the best of our knowledge, our work is first to propose this objective and demonstrate its effectiveness. Moreover, we would like to note that which estimator to use is one of our design choices and not the main focus of this work. As Reviewer MwTR mentioned in their review, any entropy/mutual information estimator is compatible with our method. We will try to further clarify our motivation and novelty in the final draft.\n\n--------\n\n\n**Q2. What is the meaning of $z$ and $z’$ in Line 117?**\n\n**A2.** $z$ and $z’$ are arbitrary two samples from a random variable Z = (X, Y), which is a notation used for introducing the concept of the maximum norm. Thanks for pointing this out and we will clarify this in the final draft.\n\n--------\n\n**Q3. The meaning of $d_{X}$ in Equation 1.**\n\n**A3.** $d_{X}$ is the dimensionality of a random variable $X$. Similarly, $d_{S}$ and $d_{Y}$ are the dimensionalities of random variables $S$ and $Y$, respectively. Thanks for pointing this out and we will fix this in the final draft.\n\n--------\n\n**Q4. Additional experiments on more complex environments**\n\n**A4.** We would like to emphasize that our experiments are conducted in environments where the observation and action spaces are high-dimensional. For instance, we reported main experimental results on pixel-based environments where the inputs to the agents are high-dimensional multi-channel inputs. Moreover, we note that our method is effective on Meta-World environments where the robot arm can freely move inside the wide robot workspace so that state-action space becomes very high-dimensional.\n\nNonetheless, to further address your concern, we provide additional experimental results on the Quadruped Walk task which has more high-dimensional state/action spaces compared to other considered tasks like Hopper. As shown in Figure 1 available from the attached PDF in the general response, we find that our VCSE clearly outperforms SE especially in the initial phase of training, which implies that the effectiveness of our method is consistent on even more complex tasks. We will include relevant results in the final draft.\n\n--------\n\n**Q5.  How to compute state and value norms in Figure 1?**\n\n**A5.** To compute the state norm in Figure 1, we randomly sample states from a replay buffer and compute the Euclidean norm between states. For computing the value norm, we pass these states through the critic function to get value estimates, and compute the Euclidean norm between value estimates. We will add a more detailed explanation on the caption for Figure 1 in the final draft.\n\n--------\n\n**Q6.  In Figure 1, why do you only choose the third one(kNN (k=3))instead of the others?**\n\n**A6.** We note that $k$ is a hyperparameter in our method and basically we can use any $k$ for our method. We choose $k=3$ in Figure 1 for illustrative purposes. We will clarify this in the final draft."}, {"review_id": "09NiBLnKdF", "reviewer": "Reviewer_FYPf", "summary": "This paper addresses the problem of sample-efficient exploration in sparse-rewards deep reinforcement learning. It builds on previous literature on the maximization of the state entropy as an exploration objective, which was mostly used in reward-free settings, proposing a value-conditioned state entropy objective along with the task reward. The latter pushes the agent to maximize the state entropy while penalizing the entropy over the estimated values of the visited states. This value-conditioned state entropy is implemented through an off-the-shelf non-parametric conditional entropy estimator. Finally, the paper compares the performance of value-conditioned entropy maximization against standard entropy maximization in MiniGrid, DeepMind Control, and Meta-World.", "questions": "This paper advocates for using state entropy bonuses to accelerate RL in sparse-rewards tasks, instead of training an entropy maximizing policy in reward-free settings, which is the most common mode of previous works. \nTo adapt state entropy bonuses to the demand of the sparse-reward task, which requires to deal with the exploration-exploitation trade-off, the paper proposes to condition the entropy of the visited states with the value of those states, effectively driving exploration towards rewarding regions of the state space.\nThis interesting notion leads to a methodology that can be easily incorporated in previous algorithms while benefitting the resulting sample efficiency in a variety of domains.\nHowever, the paper does not provide a formal theoretical understanding of the proposed approach, which leaves some doubts on the generality of their results. \nAnyway, I am currently providing a slightly positive score while encouraging the authors to improve the theoretical ground for value-conditioned entropy bonuses, and especially how the estimation error of the value function impacts the learning process.\n\n**(Clarification on the objective)**\n\nFrom my understanding of Algorithm 1, the proposed method maximizes the value-conditional state entropy over the samples taken from the replay buffer, and not just the samples drawn with the latter policy. While this is common in previous works as well (e.g., Liu and Abbeel, 2021), I am wondering whether looking at the value-conditional entropy of the latest samples is more appropriate in a setting where external reward is also present. Can the authors discuss their implementation choice and comment pros and cons of the two alternatives?\n\n**(Sensitivity to $\\beta$ also for VCSE)**\n\nThe experiments report an interesting analysis on how the value of $\\beta$ affects the performance of A2C+SE. It would be interesting to inspect also the sensitivity of VCSE to the corresponding $\\beta$ value.\n\n**(Comparison with CEM)**\n\nA recent work (Yang and Spaan, CEM: Constrained entropy maximization for task-agnostic safe exploration, 2023) considers state entropy maximization under cost constraints. I am wondering whether their approach could be also used to address the problem of reducing task-irrelevant exploration, e.g., by introducing appropriate value constraints. Can the authors discuss the comparison between their solution and CEM, possibly highlighting pros and cons of the two?", "limitations": "Some limitations of the paper and the proposed methodology are reported in the first paragraph of Section 6. I believe that a formal discussion on how the value estimation error affects their method would be extremely valuable.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes an intrinsic reward can be easily incorporated in any previous method;\n- The experiments show that the value-conditional objective brings improved/matching performance in a variety of domains;\n- The paper clearly presents ideas and contribution, and also include some compelling visualization of the conditional entropy estimation.", "weaknesses": "- The intrinsic reward computation requires robust estimates of the value function, which is a challenge per se, especially in sparse-rewards domains;\n- As mentioned by the authors in their conclusion, the theoretical ground for value-conditional state entropy is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer FYPf,\n\nWe sincerely appreciate your efforts and insightful comments to improve the draft. We respond to each of your comments one-by-one in what follows.\n\n---\n\n**Q1. Importance of value estimation**\n\n**A1.** In Figure 7(a) of the original draft, we reported that the performance further improves when using the ground-truth value estimated with privilege information from the simulator. As you mentioned, this implies that the performance of our method can indeed depend on the quality of learned value functions. Further quantitative evaluation on the relation between the quality of value functions and the performance of our method is indeed an interesting question, but we leave it to future work as measuring the quality of value functions is an open problem. It would be also very appreciated if the reviewers might perhaps be able to suggest experimental designs for this analysis experiment.\n\n--------\n\n\n**Q2. Discussion on the pros and cons of maximizing value-conditional state entropy using the samples from a replay buffer**\n\n**A2.** This is an interesting question. *The benefit of maximizing the entropy using the samples from a replay buffer* is that it explicitly encourages the policy to visit unseen states which are not in a replay buffer yet. This is more aligned with a supervised RL setup where the goal of exploration is supporting the policy to find a novel state with a high extrinsic reward. We adopted this buffer-based strategy in our work because our method is designed for the supervised RL setup. On the other hand, *maximizing the entropy of a policy-induced state distribution* is beneficial in case we want to learn a policy that can visit diverse states. For instance, this can be particularly useful in an unsupervised RL setup where the goal is to train a policy that can be quickly adapted to solve the downstream task when the reward becomes available. We will include the relevant discussion in the final draft.\n\n--------\n\n**Q3. Analysis on the effect of $\\beta$.**\n\n**A3.** Following your suggestion, we conducted additional experiments with varying $\\beta$ for VCSE. As shown in Figure 2 available from the attached PDF in the general response, we find that A2C+VCSE consistently improves A2C, in contrast to A2C+SE which fails to significantly outperform A2C even with different $\\beta$ values.\n\n--------\n\n**Q4. comparison with CEM**\n\n**A4.** Thanks for your suggestion to include additional discussion on relevant work [Yang and Spaan, 2023]. As you mentioned, one can think of an idea to discourage the exploration in task-irrelevant state space by introducing a value constraint similar to CEM that introduces a safety constraint. This idea can be also useful in our setup in that it can introduce a scheduling for value constraint so that it can first encourage the agent to visit all the states and thus focus on exploration around high-value regions.\n\nHowever, we would like to note that VCSE is based on a different motivation in that VCSE aims to prevent the states with different high- (or low-) values from affecting exploration around low- (or high-) value states, not to discourage exploration on specific state regions.  Moreover, we note that a mechanism to similar value constraint scheduling can be observed when the environment is sparsely-rewarded because VCSE acts the same as SE until it observes a sparse reward. We will include the relevant discussion in the final draft. We will include the relevant discussion in the final draft.\n\n[Yang and Spaan., 2023] Yang, Qisong, and Matthijs TJ Spaan. \"CEM: Constrained Entropy Maximization for Task-Agnostic Safe Exploration.\" The Thirty-Seventh AAAI Conference on Artificial Intelligence. 2023.\n\n"}]

Author rebuttal:
Dear reviewers,

We sincerely appreciate your efforts and insightful comments. Following your suggestions, we provide a one-page pdf that contains (i) the analysis on the effect of $\beta$ and $k$ and (ii) additional results on a high-dimensional Quadruped environment. We will incorporate them into the final draft.

Sincerely,

Authors

Meta-review: This paper has been positively evaluated by the majority of reviewers. Only one reviewer expressed concerns about the significance of the proposed work compared to other approaches/methods in the literature. These concerns have been extensively addressed by the authors in the rebuttal. Even if the reviewer has not been fully convinced and kept the grade of borderline reject. Considering the outcome of the discussion phase, it is clear that the average opinion of the reviewers is positive and that the remaining concerns are not critical.

I encourage the authors to address all the comments and to incorporate the recommended improvements in the final version.
Decision: Accept (poster)