Skew-Explore: Learn faster in continuous spaces with sparse rewards
Authors: Xi Chen
Yuan Gao
Ali Ghadirzadeh
Marten Bjorkman
Ginevra Castellano
Patric Jensfelt
Venue:  (2020)
Keywords: reinforcement learning
exploration
sparse reward

Abstract: In many reinforcement learning settings, rewards which are extrinsically available to the learning agent are too sparse to train a suitable policy. Beside reward shaping which requires human expertise, utilizing better exploration strategies helps to circumvent the problem of policy training with sparse rewards. In this work, we introduce an exploration approach based on maximizing the entropy of the visited states while learning a goal-conditioned policy. The main contribution of this work is to introduce a novel reward function which combined with a goal proposing scheme, increases the entropy of the visited states faster compared to the prior work. This improves the exploration capability of the agent, and therefore enhances the agent's chance to solve sparse reward problems more efficiently. Our empirical studies demonstrate the superiority of the proposed method to solve different sparse reward problems in comparison to the prior work. 

Under review as a conference paper at ICLR 2020
SKEW-EXPLORE: LEARN FASTER IN CONTINUOUS SPACES WITH SPARSE REWARDS
Anonymous authors Paper under double-blind review
ABSTRACT
In many reinforcement learning settings, rewards which are extrinsically available to the learning agent are too sparse to train suitable policy. Beside reward shap- ing which requires human expertise, utilizing better exploration strategies helps circumvent the problem of policy training with sparse rewards. In this work, we introduce exploration approach based maximizing the entropy of the visited states while learning a goal-conditioned policy. The main contribution this work is to introduce novel reward function which combined with goal proposing scheme, increases the entropy of the visited states faster compared to the prior work. This improves the exploration capability of the agent, and there- fore enhances the agent's 's chance to solve sparse reward problems more efficiently. Our empirical studies demonstrate the superiority of the proposed method to solve different sparse reward problems in comparison to the prior work.
INTRODUCTION
Reinforcement Learning (RL) is based on performing ex- ploratory actions in trial-and-error manner and reinforc- ing those actions that result in superior reward outcomes. Exploration plays important role in solving a given se- quential decision maaking problem. A RL agent cannot improve its behaviour without receiving rewards exceed- ing the expectation of the agent, and this happens only the consequence of properly exploring the environment.
with sparse reward or
Polioy Leaming
In this paper, propose method to train a policy which efficiently explores continuous state space. Our method is particularly well-suited to solve sequential decision- making tasks with sparse terminal rewards, i.e., rewards received at the end of successful interaction with the environment. We propose to directly maximize the en- tropy of the history states by exploiting the mutual infor- mation between the history states and a number of refer- ence states. To achieve this, we introduce novel reward function which, given the references, shapes the distribu- tion of the history states. This reward function, combined with goal proposing learning frameworks, maximizes the entropy of the history states. We demonstrate that this way of directly maximizing the state entropy, compared to indirectly maximizing the mutual information (Warde-
Figure 1: A sparse-reward task trained by a YuMi robot simulation and de- ployed to the real hardware. The task consists (1) opening door, (2) press- ing button, and (3) closing the door. The reward is given only at the end of successful trial.
Farley et al. 2018 Pong et al. 2019) improves the ex- ploration the state space as well as the convergence speed at solving tasks with sparse terminal rewards.
Maximizing the mutual information between the visited states and the goal states, I(S; G), re- sults in a natural exploration of the environment while learning to reach to different goal states Warde-Farley Farley et al. 2018; Pong et al. 2019). The mutual information can written as ((; G) = (G) hGG) therefore maximizing the mutual information is equivalent maximizing the en-

---

Under review as a conference paper at ICLR 2020
tropy of the goal state while reducing the conditional entropy (conditioned on the goal state). The first term, encourages the agent to choose its own goal states as diverse as possible, therefore im- proving the exploration, and the second term forces the agent to reach the different goals it has specified for itself, i.e., training goal-conditioned policy, π (-1s,g). g). Instead of maximizing the mu- tual information, we propose to maximize the entropy of the visited states directly, i.e., maximizing h(S) = (Z) / (S) h (Z]S), where Z is a random variable that represents the reference points of promising areas for exploration. Therefore, in our formulation, we have an extra term, ((SIZ), which encourages maximizing the entropy of the state conditioned on the reference points. This extra term, implemented by the proposed reward function, helps the agent to explore better at the vicinity of the references. We call our method Skew-Explors since similar to Skew-Fit introduced by Pong et al. (2019), it skews the distribution of the references toward the less visited states, but instead of directly reaching the goals, it explores the surrounding areas of them.
We experimentally demonstrate that the new reward function enables an agent to explore the state space more efficiently in terms of covering larger areas in less time compared to the earlier meth- ods. Furthermore, we demonstrate that our RL agent is capable of solving long-term sequential decision-making problems with sparse rewards faster. We apply the method to three simulated tasks, including a problem to find a trajectory of a YuMi robot end-effector, to open a door of a box, pressing a button inside the box and closing the door. In this case, the sparse reward is given only when the button is pressed and the door is closed, i.e., at the end of about one minute of continuous interaction with the environment. To validate appropriateness of the trajectory found in simulation, we deployed it on a real YuMi robot, as shown in Figure | The main contributions of this paper can be summarized as (I) introducing a novel reward function which increases the entropy of the history states much faster compared to the prior work, and (2) experimentally demonstrating the superi- ority of the proposed algorithm to solve three different sparse reward sequential decision-makinn problems.
2 RELATED WORK
Prior works have studied different algorithms for addressing the exploration problem. In this section, we summarize related works in the domain where rewards from the enviromment are sparse or absent.
Intrinsic Reward: One way to encourage exploration is to define an aetiiniicallymmoivvtttt reward,
including methods that assimilate the definition of curiosity in psychology (Oudeyer et al.] [007]] Pathak et al.
2017). These methods have found success in domains like video games (Ostrovski et al.) 2017 7 Burda et al. 2018). In these approaches, the "novelty", "curiosity" or "surprise" of a state is computed as an intrinsic reward using mechanisms such as visiiivsiiing count and prediction error (Schmidhuber) [991, Stadie et al. 2015: Achiam & Sastry] 2017 Pathak et al. 2017). By considering this information, the agent is encouraged to search for areas that are less visited or have complex dynamics. However, as pointed out by Ecoffet et al. (2019), an agent driven by intrinsic reward may suffer from the problem of detaching from the frontiers of high intrinsic reward area. Due to catastrophic forgetting, it may not be able to go back to previous areas that have not yet been fully explored (Kirkpatrick al. 2017: Bllefsen al. 2015). Our method is able to keep tracking the novelty frontier and train policy to explore different areas in the frontier.
Diverse Skill/Option Discovery: Methods that aim to learn set of behaviours which are distinct from each other, allow the agent to interact with the environment without rewards for a particular task. Gregor et al. (2016) introduced an option discovery technique based on maximizing the mutual information between the options and the final states of the trajectories. Eysenbach et al. (2018); Florensa al. 201 Savinov et al. (2018) proposed to learn a fixed set of skills by maximizing the mutual information through an internal objective computed using a discriminator. [Achiam et al. (2118) extended the prior works by considering the whole trajectories and introduced a curriculum learning approach that gradually increases the number of skills to be learned. In these works, the exploration is encouraged implicitly through learning diverse skills. However, it is difficult to control the direction of exploration. In method, maintain proposing module which tracks the global information of the states we have visited so far, and keep proposing reference points that guide the agent to the more promising areas for exploration.
Self-Goal Proposing: Self-goal proposing methods are often combined with a goal-conditioned policy (Kaelbling. 1993. Andrychowicz et al., 2017) where goal (or task) generation model is
2

---

Under review as a conference paper at ICLR 2020
trained jointly with goal reaching policy. The agent receives rewards in terms of completing the internal tasks which makes it possible to explore the state space without any supervision from the environment. [Sukhbaatar et al. (2017) described a scheme with two agents. The first one proposes tasks by performing a sequence of actions and the other repeats the actions in reverse order. Held et al. (2018) introduced method that automatically label and propose goals at the appropriate level of difficulty using adversarial training. Similar works are proposed by [Colas et al. (2018): Veeriah
al. (2018): Florensa et al. (0177)), where goals are selected based on the learning progress. Warde-
Farley et al. 2018) trained goal-conditioned policy by maximizing the mutual information between the goal states and the achieved states. The goals are selected from the agent's 's recent experience
with strategies. Later. [Pong et al. (019) applied similar idea of using mutual information. They maximize the entropy of a goal sampling distribution. The focus of these methods is on learning a policy that can reach diverse goals. Although gradually increasing the scale of the goal proposing network, the agent may eventually cover the entire state space, exploration itself is not efficient. In our work, we adopt the same idea of maximizing the entropy of the goal sampling distribution by Pong et al. (2019). However, instead of using the goal-conditioned policy, we introduce a reference point-conditioned policy which greatly increases the efficiency of exploration.
3 SKEW-EXPLOORE: SEARCHING FOR THE SPARSE REWARD
We discuss the policy learning problem in continuous state and action spaces, which we model as an infinite-horizon Markov decision process (MDP). The MDP is fully characterized by a tuple (8, A. Pa (s, Ss,s'), s'), Rig (s. s')), where S, the state space, and A, the action space, are subsets of IR", the unknown transition probability p: : S x A x S -> [0, inf) indicates the probability density function of the next state s' given the current state E S and the action associated environment E emits an extrinsic reward according to function R' :S x A + R. The objective of the agent is to maximize the discounted return, i.e. return R -t.=07 where is a discounted factor and rt, is the reward received at each step 1..- In this study, we consider an agent interacting in an environment E with sparse reward. The sparse reward r is modelled as a truncated Gaussian function with a narrow range. From previous interactions, the agent holds an interaction set I, in which transaction triples as S, $,+1), Vj- E fl.... T - 1} are contained. We also extract the states s, from I₁ to form a history state set S,, which contains all visited states by the agent until iteration continuous state space and converge to a policy that maximizes the R as fast as possible. This involves two processes 1) Find the external reward through efficient exploration. 2) Converge to a policy that maximizes R once the external reward is found.
A. For each transition, the space
The objective of our method is to find an arbitrary external goal in
We can use the entropy of the history state set as neutral objective to encourage exploration, since an agent that maximizes this objective should have visited all valid states uniformly. To describe it mathematically, we define a random variable S to represent the history states that the agent has visited. The distribution of S is estimated from the history state set S,. Our goal is to encourage exploration by maximizing the entropy I(S) of the history states. However, using the entropy as the intrinsic reward directly may suffer from problems similar to other intrinsic motivated methods [Schmidhuber [991; Stadie et al., [2015; Achiam & Sastry [017]] Pathak et al. (017). As the reward of the same state is changing, the agent has the risk of detaching from the frontiers of high intrinsic reward area.
We introduce concept called novelty frontier reference point, which can be sampled from a dis- tribution that represents the novelty frontier (Ecoffet et al. 2019). The novelty frontier defined in our work represents the areas near the states with lower density in distribution p(s). The frontier reference points are sampled after the distribution of the novelty frontier is updated. We define Z to represent all the history frontier reference points with probability density p(z) estimated from a set Z, that contains all novelty frontier reference points until iteration t.
The conditional probability p(s|z) defines the behaviour of the agent with respect to each refer- ence point. In this work, we model this behaviour using state distribution function (2(s z) parameterized by the displacement between the state and the reference point. The function K; needs to be chosen carefully as it should satisfy expectation of the policy behaviour and also, provides an informative reward signal to train the policy. Mathematically, we can rewrite p(s) as p(s) f ((slz)p(z)dd = K2(s z)p(z)dz. Generally, K₂ can be different for different z.
3

---

Under review as a conference paper at ICLR 2020
However, to reduce the complexity of learning, constrain K-(+) to be consistent for any z, mean- ing (s (2) (s (s- z). The defimition of (-) satisfies the definition of a kernel function. Using ((s z). p(s) can then be further represented as
p(s) = 118-200 K(s-2)p(zzdds
(K p)))).
By considering the law of convolution of probability distributions, obtain S Z N, where N is a random variable characterized by a density function (-). Now with this setup, we are able analyze our method's performance using information theory. By considering the entropy's relationship with mutual information h(S) = h(S Z) + I(S; Z), we receive the final decomposition of our objective under the novelty frontier reference point cant-condiiioned policy framework
hS) = (Z) + 4 (S (Z[s).
(1)
Eq. indicates that in order to maximize the h(S). we can individually maximize/minimize each term while making other terms fixed. In the following section, we will explain the optimization process in detail.
3.1 MAXIMIZING hZ)): OBTAINING AN EXPANDING SET OF NOVELTY FRONTIER REFERENCE POINTS
As introduced above, h (Z) is the entropy estimated from the novelty frontier reference points set 2,- To increase l(Z), we need to add a new reference point to 2, such that, the entropy estimated form Ze+1 is larger than the entropy estimated from Z,. In our method, the frontier reference points are sampled from the novelty frontier distribution which represents less history areas according to the current history states. Pong al. (2019) proposed method to skew the distribution of the history states using importance sampling, such that states with lower density can be proposed more often. In our work, we use similar way to estimate the novelty frontier distribution. There are three steps in our process. In the first step, we estimate the p(s) from S, using density estimator e.g. Kernel Density Estimation (KDE). In the second step, we sample Q states {sp. ... (8Q) from p(s), and compute the normalized weight for each state using Eq. 2
N a E [-if, 0). Ep(s= Cpss=s)ppf so) n=1
1 W/: = Y p(s:)p(s;)
(2)
where Y. normalizing constant. The state with lower p(s) has higher weight and vice versa. Fi- nally, we utilize a generative model training scheme (: ) (e.g. weighted KDE), together with sam- pled states and weights to get skewed distribution Pekewed Tg((so. snh (wo. wn)) to represent the novelty frontier distribution.
Exciiples sampee rovely
Ecerp weliraaedd cherthetton 1209 vodtt
6.30
Exaimple therth monelyy
0.06
0.00
Figure 2: This figure shows a comparison between the state distribution p(s) (dashed blue) and a corresponding novelty frontier distribution skewed from p(s) (red).
Q big enough, by choosing a appropriately, we are able to expand our frontiers after each iteration. As a consequence, the distribution estimated from Z, will become more and more uniform and its range will become larger and larger, just like annual ring of the tree. The entropy of continuous uniform function U(p,g q) is In(p g) and if the distribution has larger range, the entropy is larger as well. Fig illustrates the estimated frontier distribution skewed from p(s).
3.2 MAXIMIZING (S|Z) - hZ((S): INCREASING THE EXPLORATION RANGE AROUND
REFERENCE POINTS
The conditional entropy ZS and hZZ are highly correlated, maximizin//nnnimizingg them individually are difficult. Therefore, in this section, consider maximize h(SIZ) -i(S)))
4

---

Under review as a conference paper at ICLR 2020
whole. Using the relation S Z N, rewrite the expression h([|Z)
h(Zss)
h(Z + N - h h Z + N), which can be further simplified (see Appendix D as
h(Z[S) h(S|Z) h(N) - -b(Z).
This implics that there is a lower bound for the expression h(SlZ) - h(Z]s). For a fixed h(Z), we can maximize the lower bound ((N) /zZ) by increasing ([N). k(NN is related to the shape and variance of the exploration distribution near the reference point. In our method, we model N as a Gaussian distribution with zero mean. In ideal case, we would like to have large variance as possible. However, increasing the variance also results in learning difficulty, as we need a longer trajectory to evaluate the performance and more samples to update the network. Therefore, use the variance to control the trade-off between exploration efficiency and learning efficiency.
Scnglo Inme
Skow
FII
Sarple
Exploce
Our method, Skew- Neew-Explore, aims to obtain uniform state visitation distribution estimated from the history state set. We start by sampling from our history state set, and weighting the states such that less-visited states are assigned with higher weights. We then train skewed distribution Pskewed (s) using the weighted samples as the novelty frontier distribution. Next we sample reference points the novelty frontier distribution and run our policy to explore around the points.
Figure 3:
3.3 DESIGNING THE REWARD FUNCTION
Our algorithm requires the policy to move around a given reference point, and the distribution of the states in the trajectory should follow a Gaussian distribution centered at the reference point. In this section, we introduce an intrinsic reward function to train such policy by minimizing the the Kullback-Leibler (KL) divergence between the trajectory distribution and the desired Gaussian distribution. For each given reference point Z!, collect trajectory т; by running the policy with the given reference point, indicated as क (2,). for M steps. Then, we estimate the probability density of each state in T;, referred as Pr, (s), using density estimator. Finally, we check the probability density of s in the Gaussian distribution centered at Z; referred to as Pz. between the trajectory distribution Pr. and the desired distribution Pz, is formulated as follows
Pz,(s). (s). The KL-divergence
Pr. (s)
DKL (+) px, = (2, (2))))))))))))))))))))))))))) Pr. (s) log
(3)
Pz. (s)
To minimize the KL-divergence, the intrinsic reward of s with respect z, is computed as Vine (s, 2.) = log(pz, (s)) log(p>;(s)
(4)
The intrinsic reward function measures the difference between the desired density of in the trajec- tory and the actual density achieved. The reward is positive when the actual density is smaller than the desired one, when states in the trajectory are too far from the reference point, and the reward is negative when the actual density is larger than the desired one, when the agent stays too long at the reference point. An extrinsic reward reer(s) (s) is provided by the environment and the total reward of a time step is defined as the weighted sum of the intrinsic and the extrinsic reward. The extrinsic reward should be much greater than the intrinsic reward. The reward of each time step r(s, 2;) defined as
r(s, 2.) Wint (s,e + Wext (s):
(5)
where, Wint and West are respective weights for internal and external rewards. The performance of the policy is closely related the set Z,, as records the reference points we used to train the policy until iteration t. As described in section 3.1 while we increase the entropy h(Z) by proposing new reference points form the novelty frontier to train the policy, the policy gradually gain skills to explore different areas. When state with large extrinsic reward is discovered, the policy eventually ignores all given reference points and converge to reach the state with the extrinsic reward. Algorithm]] shows the whole Skew- ekew-Explore algorithm using pseudo code and our implementation of the algorithm is available online
4open sci www.eeeeeeeeeeeee 44b--444b 4ace--44c6--200
5

---

Under review as a conference paper at ICLR 2020
Algorithm Skew-Explore
procedure SKEW-EXPLORE
2:
History state set S = 0
3: History novelty frontier reference points set 2g-
0
4: Randomly sample L novelty frontier points
5:
Zg Zn (
6:
for t= 3... do
7:
Collect set of states S, by running policy giving different frontier reference point 2,.
8:
Compute reward for each state using Eq.
9:
Update history state set S, = S U [s....., sQ} and estimate p(s) from S:
10:
Estimate the novelty frontier distribution Pskewed(s) by skewing p(s).
11:
Sample L new novelty frontier points 21 ~ Pskewed(s).
Update history novelty frontier reference points Z, Zz-1 (Zl, Zt}.
13:
Update policy according to the rewards
3.4
SCALING TO HIGHER DIMENSIONAL STATES
4
EXPERIMENT
In this section, we evaluate our algorithm from three perspectives. 1) How efficient our algorithm in terms of exploring the entire state space, and how different choice of variance affects the efficiency? 2) Is our algorithm able to converge to stable solution for tasks with sparse reward? 3) Is our algorithm able to solve complicated sparse reward task with long horizon? The implementation details of the ex- periments can be found in Appendix E Two metrics are considered to evaluate the performance. They are the state distribution entropy h(S) and the cov- erage, which are estimated from history state set S;- We describe how estimate the metrics and their difference in Appendix A and short video regarding the experiments can found online
Figure 4: The PointMaze environment and the DoorOpen environment.
4.1 POWER OF EXPLORATION
In the first experiment, we evaluate our algorithm in term of the efficiency of exploring the state space. We test our algorithm in two simulated environments, the PointMaze and the DoorOpen environments (Fig. 4. In the PointMaze environment, point agent is controlled to move inside a maze with narrow passages. In the DoorOpen environment, a YuMi robot can open a door by grabbing the door handle. The PointMaze environment 2017a); Eysenbach et al.] (2018): Pong et al. (2019), whilst environments similar to the DoorOpen environment used by KKlakrishhan al. (011); Chebotar et al. (017)); Pong al. (2019). The objective of the tasks is to explore the entire state space in a minimum amount time. order evaluate the performance, we measure the efficiency as the overall coverage and the entropy the density estimated from all history states. We compare our algorithm with two baseline algorithms: the random network distillation (RND) proposed by Burda et al. (2018) which is an approach using prediction error as the intrinsic reward, and Skew-Fin -ii proposed by Pong et al. (2019) which combines a goal proposing network with a goal-conditioned policy.
previously used by Florensa et al.
We consider two configurations. The first one is the proximal policy optimization (PPO) Schulman (2017) together with a Long Short-Term Memory network (LSTM) Hochreiter Schmidhuber (997). The second configuration is soft actor-critic (SAC) Haarnoja et al. (2018) and hindsight experience replay (HER) Andrychowicz et al. (2017). We note here that RND is only tested with PPO and LSTM it was not designed for off-policy methods. For each configuration, we run
https: httpps://www.r http::///wwwwwwppooooooo http://pww....coooo/////oaaaaaaa
deo_ subm 8si asionnmp on.mp4

---

The power of exploration is an important aspect, but also converge to stable policy that maximizes the extrinsic reward for different sparse re- ward tasks. To this end, we use the same environments (Fig. 4 as in the previous experiment. In each environment, we se- lect five uniformly distributed target points from the area of interest and assign extrin- sic rewards when the agent reaches these points. For each target point, we train individual policy to reach it. Hypotheti- cally, influenced by the extrinsic reward, the agent eventually ignores the internal Under review as a conference paper at ICLR 2020
ParthAon
DoorOpen
BACHHER-SUNN-F
Gowerage
Entory
RRO
PPOHLSTM-SSeNN'I
PPO Stow.exoo Exploro
Namberd
Narber
beraction
Figure 5: Results of how coverage and entropy changes iterations in PointMaze (left) and the DoorOpen (right) environments.
12 times and compare the mean and variance. Fig. 5 shows the results for 5 configurations. We that method, SAC+HER=SKew-ENw faster than the other methods. It also increases with relatively small variances. Fig 6 illustrates
makes both coverage and entropy increase
100
Skew-Explore:
21
100
Skew-Fit:
Figure Results of how coverage changes over iterations the PointMaze environment.
how coverage changes for both our method Skew-Explore and Skew-Fit. In this figure, we see how our method is able to cover the state space (area in this case) faster than Skew-Fit. To further analyze how different choices of the variance of affects the exploration efficiency, several values of variances 0², 1 22, 32, 42 and 62 are tested in the PointMaze environment. After 80 iterations, estimated entropy 1.22 49.7 1..8. 51.4 11.7 and 48.9 48.9433. are received. We observe that while the variance increases, the performance first increases and then decreases.
4.2 POWER OF SOLVING A SINGLE SPARSE REWARD TASK
our algorithm
Tagen
Point "Srget Poinss
Target Points
Torget Teins Peinss
PoinoMaze
(90x5%)
1975
2388
(90+3%)
Table 1: This shows the number of trajectories needed for the algorithm to converge to five uniformly sampled target states in each environment. A successful con- vergence is measured as 90% of the states receive the extrinsic reward with a standard deviation of less then 3%. The two other methods Skew-Fit and RND cannot solve the problem below upper limit 3000 trajectories.
goals generated by the goal proposing module and reaches the target points consistently. In or- der to evaluate the performance of our algorithm, we measure how reliably the agent is able to reach each target point. To this end, we collect the final states from the most recent trajectories and define criteria for convergence as the percentage of receiving the extrinsic rewards. If more than 90% the states receive the extrinsic reward with a standard deviation less then 3%, we say the agent solved the task successfully. In this experiment, we use the configuration SAC+HER+Skew: Explore which achieves the best performance in the previous experiment. Table shows how many trajectories the algorithm needs to reach the criteria of convergence. This experiment thus shows that our algorithm able to solve a sparse reward task, by obtaining policy with a limited number of trajectories. Additional results can be found in Appendix

---

Under review as a conference paper at ICLR 2020
4.3 TASK WITH LONG HORIZON AND REAL WORLD DEMONSTRATION
In the third experiment, we evaluate the ability of our algorithm in terms of solving a sparse reward task with long horizon and test the performance of the converged policy using real world YuMi robot. We increase the complexity of the environment by adding a box and a button to the DoorOpen environment used in the previous two experiments. We design task called OpenPressClose which needs a long sequence of procedures to be solved. The sequence includes 1) open the box, 2) press the button inside the box and 3) close the box. The extrinsic sparse reward is only given to the agent after all procedures in the sequence are done. This task is exceptionally challenging as each intermediate procedure requires set of continuous actions in correct order to be achieved and no intermediate reward is provided guide the search. Therefore this task requires the power of efficient exploration to discover the state that provides an extrinsic reward. the algorithm fails to explore efficiently, the rewarding state would never be found and policy will be learned. The results show that the algorithm is able to discover the extrinsic reward and converge to a stable solution. Fig 7 shows the change of average extrinsic reward per step iterations and Fig 7 shows the converged policy in sequential order. The resulting policy is deployed to a real world A demonstration of the real robot solving the task can be found in YuMi robot as shown Fig I the video.
Rein
poer Opened
250
300
500
000
Dee Cored
Figure 7: (a) Relation between average extrinsic reward per iterations. (b) From up to down, left to right, this figure shows sequence of the converged policy for the OpenPressClose task, which is long horizon task with a sparse reward given only at the end.
5 CONCLUSION
In this work, propose algorithm named Skew-Explore, general framework for continuous state exploration. Inspired by Skew-Fit [Pong et al. 2019), the main idea of Skew-Explore is to encourage exploration around the novelty frontier reference points proposed by latent variable proposing network. The algorithm is able to track the global information of entropy of density dis- tribution estimated by the states stored in history state set, which helps maximize corresponded metrics, namely entropy and coverage. Two experiments are conducted to test the power of Skew- Explore on the exploration problem and the single sparse reward problem. In the first experiment, we found that our algorithm Skew-Explors Explore, using SAC and HER together, has the fastest exploration rate. In the second experiment, we found that our algorithm is also able to converge to a stable policy when a single sparse reward is given. As demonstrator, we used an environment where robotic manipulator needs to 1) open door, 2) press a button inside and 3) close the door in a sequence but only with a sparse reward given at the end. We implemented the fully converged policy on real YuMi robot using policy transfer. Future work will include investigating we can improve the efficiency of policy convergence by adjusting the proposing network's 's distribution. Additionally, will examine whether clustering can increase the efficiency for exploration. Moreover, we will look for better reward function than KL divergence between Gaussian-basse goal distribution and the trajectory distribution.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint arXiv: 1033.07703.0132. 2017.
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint
2018.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience play. In Advances in Neural Information Processing Systems, pp. 5048-5058, 2017.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12880 2018.
Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine. Path integral guided policy search. In 2017 IEEE international conference on robotics and au- tomation (ICRA), pp. 3381-3388. IEEE, 2017.
Cédric Colas, Olivier Sigaud, and Pierre- Oudeyer. Curious: Intrinsically motivated multi-task multi-goal reinforcement learning. arXiv preprint wiXiv:1800.0600 2018.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint (rrXiv/900) 19900/110 10995, 2019.
Kai Olav Ellefsen, Jean- -an--Batistt Mouret, and Jeff Clune. Neural modularity helps organisms evolve to learn new skills without forgetting old skills. PLoS computational biology, 1140:e1004128, 2015.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06077 2018
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning, arXiv preprint
2017a
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. arXiv preprint 1exivv/705.0635 2017b.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint
2016.
Charles Miller Grinstead and James Laurie Snell. Introduction to probability. American Mathemat-
ical Soc., 2012.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. urXiv preprint 1rXiv:1801.01290, 2018.
Thomas C Hales. A proof of the kepler conjecture. Annals of mathematics, pp. 1065-1185, 2005.
David Held, Xinyang Geng, Carlos Florensa, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, pp. 1094-1099. Citeseer, 1993.
Mrinal Kalakrishnan, Ludovic Righetti, Peter Pastor, and Stefan Schaal. Learning force control policies for compliant manipulation. In 2011 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 4639 4644. IEEF, 2011.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-355 2017.

---

Under review as a conference paper at ICLR 2020
Georg Ostrovski, Mare G Bellemare, Aäron van den Oord, and Rémi Munos. Count-based ploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning-Volume Volume 70, pp. 2721-2730. JMLR. org, 2017.
Pierre-Yve Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary computation, 11(2):665-266, 2007.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp. 16-17, 2017.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. arXiv preprint 2019.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. arXiv preprint weXvv:8033000533 2018.
Jürgen Schmidhuber. possibility for implementing curiosity and boredom in model-building neu- ral controllers. In Proc. of the international conference simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint 15X.001507.008874 2015.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint 17035:1703.05407. 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Vivek Veeriah, Junhyuk Oh, and Satinder Singh. Many-goals reinforcement learning. arXiv preprint wrXiv:/806.09600 2018.
David Warde-Farlcy, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-pant -on-paramttric discriminative rewards. arXiv preprint XXI1:/8/1.11355 2018
10

---

Under review as a conference paper at ICLR 2020
Appendices
A DIFFERENCE BETWEEN COVERAGE AND h(S)
Since the reward is given to an arbitrary state in the whole space, two metrics could be considered evaluate the performance of the agent, namely the state distribution entropy h(S) and the cov- erage fc(S.). The distribution of S can be estimated using any density estimator trained using S
Daja points
The thind point
0.20
Daza po
The third point Essimated densty using three points
Overal
Estimated density
of
of three poines
0.30
using teo point
0.1
0.05
0.00
-2
Figure 8: The change of coverage (left) and estimation of probability density (right) when adding new point, in mie-dimesionnl state space. In the left image, the overall state coverage constructed by two black points is represented the area of dark grey. When third point (red dot) is added, the overall state coverage increases and the amount of increase is the area of light grey. In the right image, the red curve shows the density distribution constructed by black points. When new point (red dot) is added, the density distribution changes to the dashed curve. The state entropy decreases as the new point is in region with high density.
The coverage Se is defined as a mapping Se S R. K(S.) := Is (o(s)) ds, where 6: [0. inf) > R is a radial basis function with a Gaussian kernel centred at s; E Sy and n equals |S.). Intuitively, these two measures describe similar characteristics of the data distribution. How- ever, they also have significant differences. One major difference is that when new data point presented in the state space, the coverage can only increase whilst the entropy can decrease if the new data point is given in an often visited area. Figure |illustrates the difference between coverage and density estimation when two and three points are given. Coverage gives a better intuition about where the agent should search for the sparse reward.
To find the sparse reward, maximize the history states entropy to increase the coverage. A proof for one-dimensional state space is given in Appendix [ Proof of high dimensional cases is related to Kepler theorem Hales (2005) and is beyond the scope of this paper. We aim maximize the entropy of the state density in the S₁, which H(S).
B ESTIMATING STATE ENTROPY AND COVERAGE
In practice, we uniformly discretize the entire state space along each dimension and use the center of the discretized grids to estimate the entropy and the coverage. We do not check the validity of each grid. Therefore, there are grids that are not reachable from the initial state, and the maximum possible coverage is less than 1. The PointMaze environment is discretized to 50 50 grids. The DoorOpen environment is discretized to 10 10 10 2 10 grids. The entropy and coverage are summed over the discretized states.
C PROOF OF RELATION BETWEEN COVERAGE AND h(S)
In this section, we prove Lemma C.I
Lemma C.1. Given N movable points P = (P1. P2. ... PN distributed in R with boundary Bi S D4 S Br. Maximizing the coverage estimated from these points using Gaussian kernel will make the points to be uniformly distributed within the boundary, thus maximizing the entropy of the points' distribution.
Proof. According to the definition of coverage, the area under all Gaussian kernels could be written
as :
f((P) / max(o(pi)) dx
(6)
11

---

Under review as a conference paper at ICLR 2020
We rewrite the equation as subtraction of all the areas and the overlapping areas. Fig 2
illustrates
these concepts. The area from negative infinity to x under Gaussian distribution is defined as.
$(x)= { p(x)dr = 2
1+ erf
I
(7)
V2
N-1
ti
fc(P) = (20 - Σ
2
(8)
2
1=1
-4
1 Σ erf
(9)
2
(10)
As maximizing - -ー erf 2v?
is equivalent to minimizing - erf
We formulate
2/2
an optimization problem as follows:
N-1
-4
minimize faaa. d4
Σ orf
2
subject to
0, Vt
N-1
Σ
4 S Bn
B₁:
Solving this problem using the KKT conditions will give that when t; Ba-B Vt,, the pression is minimized. This result shows that when the coverage estimated from these points is maximized, the points need to be distributed evenly within the boundary. As a consequence, the entropy of the distribution of these points also maximized.
Datz points
B
B
0.3
Caverage
Overlspping drea
0.1
Figure 9: This figure demonstrates the relationship between overall coverage and overlapping area.
D PROOF OF ((SIZ) hZ(Ss(/)))))) ((N) - (Z)
In this section, we show our analysis and proof of the expression
M(SIZ) - M(Z[S) 2 ((N) - M(Z).
Let Z and be jointly distributed continuous random variables, where related Z through conditional PDF [x]== defined for all z. The conditional PDF f(s]2) follows a Gaussian distribution centred at different with standard deviation o >
Lemma D.1. S can be represented as the sum of Z and N, where N is r.v. with Gaussian PDF of mean and standard deviation a, independent from Z.
Z N
(11)
Proof
12

---

Under review as a conference paper at ICLR 2020
Using the law of total probability, the density of f() can be written
f(s)
f(s)z)
1 (e-s)) (0--0//2000 (()))(())))) f(=)dz
O 0/2
K(ss - 22)(z
(K f)(s),
(12)
where K is the density function of N. According to Theorem 7.1 in Grinstead & Snell (0012).
when the density function ((s) the convolution of the density function of G and N, S G holds.
Q.E.D.
Lemma D.2. When Z + N holds, the following expression
h(SIZ) hZZ[)) 2 ((N) h(Z)
(13)
holds.
Proof.
Remark. Here we note that we will use following equations later proof,
h Z) = (Z (Z + N(N)
(14)
(NN) hNN Z) h(Z + N/Z).
(15)
iZ) = ((Z and ((N) ((N Z) are true because Z and N are indepen- dent. Moreover, hZ NN h(Z N N) and ((N h + N are because there is no
In Eq. 14 and Eq. 15
gain of information by adding what is given already.
We expand the Eq. 13 by replacing the S with Z N and get
h(S)Z) - h(Z(S) lZ + 2 (
((N) h Z + N)
(16)
Now we proceed to prove that h( N) nN +
h iZ N) M(G + N| + +(Z) N
=N(N) + l(() - 44(2 + N)
+ -N(N)
1(Z+N)
M(N, Z+NN h(Z N)
M(N/Z + NN.
(17)
We replace hZZ/Z + N) with M(N N) Eq. 16 have
A(SIZ) - (Z[()) h(N) A(N +N)
[( + N;N)
I(S: N).
(18)
According to the property mutual information. 7(S; N) 2 O. with equality if and only if and N are independent. As S Z + N, N)S 0 is true only when the PDF of N is Dirac delta function. In our case, PDF NN is a Gaussian function and a > O, as consequence, the expression h(S)Z)) h (Z|S) > 0 holds. Additionally, since h hZ Z + N) is the same (NN + N), by law of conditioning reduces entropy, we has an inequality as follows
h(SlZ) (Z(S) h(N) h(N + +N)
z( - (Z[(S) = h(N) hZ + N)
ZS b(Z(S) 2 A(N) h(Z).
(19)
13

---

Under review as a conference paper at ICLR 2020
Q.E.D
If we link it back to the paper, Z is the novelty frontier reference points proposing distribu- tion and S represents the achieved state distribution by executing goals sampled form Z. N defines given reference point. When N has Dirac delta distri-
the behaviour the policy according bution, the policy is goal-conditioned policy (as in Skew-Fit). When N is Gaussian distribution, the policy moves around the reference point following a Gaussian distribution. Compared to the goal conditioned policy, the reference point-conditioned policy achieves higher state entropy. The extra amount entropy equals the mutual information I(Z + N)
E IMPLEMENTATION DETAILS
E. ENVIRONMENT DETAILS
The three environments used in the experiments are implemented using Mujoco Todorov et al. 2012).
PointMaze: this environment, an agent travels in a maze contains narrow passages. The observa- tion is the v, y position of the agent and the action the velocity along the J, y axis. The maximum velocity for each dimension is 0.12. In the sparse reward experiment, the agent receives an extrinsic reward when the distance the goal is less than 0.15.
DoorOpen: In this environment, we use single arm Yumi robot with 7 degrees of freedoms (DOFss. The robot is controlled in Cartesian space and the orientation of the end-effector is fixed. The robot can grab a door handle and open the door on a table. The maximum opening of the door is 90 degrees. The observation space is 5 dimensional, including the z, y, position of the end-effector, the open/close status of the gripper and the opening angle of the door. The action is the velocity along v, y, 2 axis. The valid action space of the robot is 10cm x 11em x 10cm. In the sparse reward experiment, the agent receives extrinsic reward when the angle difference to the goal is less than 2 degrees.
OpenPressClose This environment contains single arm Yumi robot, a box with a door and button inside the box. The robot configuration the same as the DoorOpen environment. The maximum opening of the door is 143 degrees. The observation space of the robot is 6 dimensional, including the x, y, & position of the end end-effector, the open/close status of the gripper, the opening angle of the door and the status of the button. The action is the velocity along x, y, axis. The valid action space of the robot is 2cm 28.5cm x 16em. In the sparse reward experiment, the agent needs to press the button down for more than lem and close the door completely.
E.2 HYPERPARAMETER
We use the same network structure for all experiments. The policy net- work contains fully-connected lay- with the number of units all lay- ers 32, 64, 128, 64, 32. We use ReLU as the activation function and there is no activation for the output layer and for all experiments, the length of the trajectory is 200.
50'
56'
Target door andles
The RND contains colly-connected layers with the number of units 32, 64, 64 the random target net- work and contains 4 fully-connected layers with the number units 32, 64, 64, 128 in the prediction network.
Figure 10: Five uniformly distributed points in PointMaze environment and the DoorOpen environment.
14

---

Under review as a conference paper at ICLR 2020
For PPO, the batch of 5, 000 and 15 epochs per iteration. We update the state distribution and the goal distribution at every 5, 000 step.
F ADDITIONAL RESULTS
The second experiment test whether the algorithm is able to converge to given sparse reward after exploration. As mentioned in the section 4.2 five points are selected to test the convergence of the algorithm given a sparse reward. Fig 10 shows the exact points in the PointMaze and DoorOpen environments. The points selected uniformly from the of interests. For each point, twelve experiments have been executed to receive information about mean and the variance of the algo- rithm's 's performance. Fig [ Fig [2 show the relationship between the convergence criteria and the number of iteration for PointMazz environment and DoorOpen environment respectively.
Figure 11: This figure show the relationship between convergence criteria and the number of itera- tions in PointMazz environment.
Ohe Conegens
36
54
72"
90'
Figure 12: This figure shows the relationship between convergence criteria and the number of itera- tions in DoorOpen environment.
15


Reviews:
[{"review_id": "SJl_0LyRtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new exploration algorithm by proposing a new way of generating intrinsic rewards. Specifically, the authors propose to maintain a \"novelty frontier\" which consists of states that have low-likelihood under some likelihood model trained on their replay buffer. The authors propose to sample from the novelty frontier using a scheme similar to a prior method called Skew-Fit, but replace the VAE with a kernel-based density model. To construct an exploration reward, the authors estimate the KL divergence between the resulting policy state distribution and the desired `state distribution, where the desire state distribution is a Gaussian centered around a point sampled from the novelty frontier.\n\nOverall, the paper tackles an important questions of exploration, and while the concept of a frontier set is not novel, the authors propose a concrete instantiation that has promising results on continuous state spaces. I'm skeptical that this exact algorithm would work on domains with complex state spaces (e.g. images), where adding Gaussian noise to your state won't produce reasonable nearby states. That said, the general idea of fitting a new model to the latest trajectory and using KL as reward seems like a promising principle that could on its own scale. However, the theory seems a bit off and there are a few experimental details that make me hesitant to increase my score.\n\nIn details:\n\nTheory:\nI found the proof surprisingly long given that it amounts to saying that if (1) S = Z + N and (2) Z and N are independent, then\n  H(S) >= H(N)\nand so\n  H(S | Z) - H(Z | S) = H(S) - H(Z) >= H(N) - H(Z)\nPerhaps more worrisome is the statement, \"we consider to maximize h(S|Z) - h(Z|S)\". Unless I misread the paper, the authors do not maximize this quantity. Instead, they *fix* this quantity by choosing a fixed entropy of N. Worse yet, this quantity is actually minimizes since, while h(N) is fixed for the duration of the experiment, h(Z) is maximized (\"To increase h(Z), we need to add...\"). It would be good for the authors to address this concern, given that the claim of the paper is that they are \"maximizing the entropy of the visited states.\" It seems like a simple answer is the following: given that S = Z + N, if N is fixed to some Gaussian distribution, then the authors simply need to maximize H(Z), which they are already doing. I'm not sure why the authors need to reason about H(S | Z) - H(Z | S).\n\nExperiments:\nCan Table 1 be replaced with the learning curves? The numbers 90% success and standard deviation of 3% seem like arbitrary numbers. It doesn't preclude the possibility that (e.g.) Skew-Fit or RND receives a 99% success rate with standard deviation of 3.1%. Figure 11 and 12 of the Appendix don't convince me that threshold at 90% and 3% is a particularly good choice.\n\nCan the authors summarize the difference between coverage and entropy in the main paper? It seems like an important distinction. Given that the authors did not use all 8 pages, it would be good to explain it there rather than in the Appendix.\nHow sensitive is the method to the hyperparameter alpha? How was it chosen? Is it the same alpha chosen for Skew-Fit?\nHow was N chosen for the door environment?\nIs Figure 7 (left) showing the performance on the simulated or real-world robot?\nIf it was done on the real-world robot, were there any important details in getting sim-to-real-world to work?\nIn Figure 5, why does there seem to be discrete jumps in the learning curves for \"DoorOpen Coverage\"?\n\nI would be inclined to raise my score if:\n1. The authors clarify why studying the quantity H(S | Z) - H(Z | S) is particularly important.\n2. Address the concerns raised over the experiments.\n3. Discuss more explicitly under what assumption they expect for this method (with a Gaussian KDE) to work well\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lXWHd45H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of exploration in reinforcement learning. The key idea is to learn a goal-conditioned agent and do exploration by selecting goals at the frontier of previously visited states.  This frontier is estimated using an extension of prior work (Pong 2019). The method is evaluated on two continuous control environments (2D navigation, manipulation), where it seems to outperform baselines.\n\nOverall, I like that the proposed method integrates some notion of novelty with the language of mutual information and density estimation. While this idea has been explored in prior work (as noted in the related work section), the proposed idea seems like a useful contribution to the literature. The use of convolutions to obtain a lower bound on mutual information seems neat. The experimental results are quite strong.\n\nMy main concern with the paper is a lack of clarity. I currently have enough reservations and questions (listed below) about the experimental protocol that I am learning towards rejecting this paper. However, if the paper clarified the concerns below, I'd be willing to increase by review.\n\nQuestions / Concerns:\n* \"[Prior work on mutual-information cannot] guarantee that the entire state space can be covered\" -- In theory, I think these prior methods should cover the entire state space. Take the DIAYN objective, I(s, z) = H[s] - H[s | z]. This objective is maximized when p(s) is uniform over the entire state space and p(s | z) is a Dirac.\n* \"It is time consuming to collect enough samples to estimate an accurate state entropy\" -- Can you provide a citation/proof for this? Why should we expect the proposed method to require fewer samples?\n* \"...entropy itself does not provide efficient information to adjust the action at each step.\" -- Can you provide a citation / proof for this? Also, what does \"efficient information\" mean?\n* It seems like, if S is a finite collection of states in a continuous space, then it has measure zero, so its entropy should be zero. Can you explain why this is not the case?\n* If I'm not mistaken, in equation 2, if we take (say) alpha = -0.5, then w_i is proportional to sqrt(p(s_i)), so w_i is an increasing function in p(s_i), not a decreasing function.\n* Can you discuss how you might scale a KDE to high dimensions?\n* \"If the distribution has a larger range, the entropy is larger as well.\" -- Technically, this is not correct. You can construct distributions with larger ranges but smaller entropies.\n* I think that Equation 3 should be the KL divergence between state marginal distributions, not between trajectories. If it were the KL between trajectories, it would include actions and policy terms.\n* How are w_int and w_ext chosen? It seems like the method depends critically on the balance between these hyperparameters. Is w_int decayed over time? If not, why does the policy stop exploring once it has found the goal?\n* What policy is used at convergence? It seems like the policy is conditioned on Z_t, so how is the Z_t chosen for evaluation?\n* Fig 5 -- How are entropy and coverage computed? What are the maximum possible values for both of these quantities? What precisely does the X axis correspond to?\n* Table 1 -- How did the baseline algorithms perform on this task?\n* Fig 7 -- How did the baseline algorithms perform on this task? If the reward were sparse, shouldn't the Y axis be in the interval [0, 1]?\n* \"using coverage only during training is not suitable\" -- Can you provide a citation/proof for this?\n* \"As a consequence, the entropy of the distribution of these points is also maximized\" -- I believe that a finite number of points in a discrete space have measure zero, so they have zero entropy, regardless of the position of the points.\n\n\n\nOther comments\n* \"What is the difference between *S* (in bold) and S_t?\n* I would recommend using some notation other than p(s) to denote the smoothed/convolved density. \n* \"history states\" -- I was confused about what this meant until it was introduced two sections later.\n* \"assimilate the definition of curiosity in psychology\" -- I think that others (e.g., Oudeyer 2007, Pathak 2017) have noted the similarities between curiosity in humans and RL agents.\n* Check for backwards quotes in the related work section.\n* \"Self-Goal Proposing\" -- Some more related works are [Florensa 2017, Savinov 2018]\n* \"space associated environment\" -- I don't know what this means.\n* \"disc rewards\" -- I'd recommend spelling out discounted\n* \"truncated Gaussian function with a narrow range\" -- Can you explain precisely what this is?\n* In equation 2, I think it'd be clearer to write p^(1+\\alpha).\n* For the experiment on the effect of variance, I'd recommend making a plot instead of just listing the values.\n* In Section 4.3, it's unclear whether the physical robot was successful at solving the task.\n* \"We rewrite the equation…\" -- This paragraph is repeated.\n* Double check that \\citet and \\citep are used properly\n\n--------------UPDATE AFTER AUTHOR RESPONSE------------------\nThanks for answering many of my questions. This was helpful for clarifying my understanding. However, since a large fraction of my concerns were not addressed, so I am inclined with stick with my original vote to reject the paper. Nonetheless, I should emphasize that I think this paper is on the right track and the empirical results seems strong. With a bit more work on writing, I think it would be a fantastic paper at the next conference.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gZ0XznYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an exploration objective for solving long-horizon tasks with sparse rewards. While the paper is heavily influenced by the recent Skewfit work (Pong et al, 2019),  it aims to solve a different problem (that of exploration for sparse reward RL), since Skewfit is interested in exploration and goal sampling for self-supervised (i.e. no external rewards) RL. The central idea in this paper is to encourage exploration by maintaining a distribution over the \"current frontier\", i.e. the set of points in the state space that are closed to the \"edge\" of what is explored by the policy , sampling points around this frontier, and encouraging the policy to reach these sampled points. The paper compares to RND (a method that is representative of state-of-the-art in exploration bonuses, I believe) and also to Skewfit, and outperforms these methods in two non-standard environments: door opening and point-mass navigation. \n\nWhile the I think the empirical contributions are solid, and authors provide code to reproduce the results, I found the paper a bit hard to follow and understand, and different subsections in the technical section (Section 3) did not seem to have a unified narrative running through them. I will wait to see if other reviewers agree or disagree with me on this front, and if they do agree, then I think the paper will need substantial edits to improve its clarity before it is ready for publication. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers generally appreciated the ideas presented in the paper and found the overall aims and motivation of the paper to be compelling, there were too many questions raised about the experiments and the soundness of the technical formulation to accept the paper at this time, and the reviewers did not feel that the authors had adequately addressed these issues in their responses. The main concerns were (1) with the correctness and rigor of the technical derivation, which the reviewers generally found to be somewhat questionable -- while the main idea seems reasonable, the details have a few too many question marks; (2) the experimental results have a number of shortcomings that make it difficult to fully understand whether the method really works, and how well.
Decision: Reject