Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
Authors: Vitchyr H. Pong
Murtaza Dalal
Steven Lin
Ashvin Nair
Shikhar Bahl
Sergey Levine
Venue:  (2020)
Keywords: deep reinforcement learning
goal space
goal conditioned reinforcement learning
self-supervised reinforcement learning
goal sampling
reinforcement learning

Abstract: Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manually-designed reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing the entropy of the goal distribution together with goal reaching performance, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. Skew-Fit enables self-supervised agents to autonomously choose and practice reaching diverse goals. We show that, under certain regularity conditions, our method converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that it can learn a variety of manipulation tasks from images, including opening a door with a real robot, entirely from scratch and without any manually-designed reward function.

Under review as a conference paper at ICLR 2020
SKEW-FIT: STATE-COVERING SELF-SUPERVISED REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with manually- designed reward function limits this repertoire and imposes a manual engineering burden. supervipe agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objec- tive for goal-reaching policies that maximizes state coverage. We show that this objective equivalent to maximizing the entropy of the goal distribution together with goal reaching performance, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning goal distributions. Skew-Fit enables self-supervised agents to autonomously choose and practice reaching diverse goals. We show that, un- der certain regularity conditions, method converges a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that can learn a variety of manipulation tasks from images, including opening a door with real robot, entirely from scratch and without any manually -eeualll- reward function.
INTRODUCTION
Reinforcement learning (RL) provides appealing formalism for automated learning of behavioral skills, but separately learning every potentially useful skill becomes prohibitively time consuming, both in terms of the experience required for the agent and the effort required for the user to design reward functions for each behavior. What if we could instead design an unsupervised RL algorithm that automatically explores the environment and iteratively distills this experience into general-purpose -puposse policies that can accomplish new user-specified tasks test time?
Slaw-Fit
(M) Unweighted
Figure I: Left: Robot learning to open a door with Skew-Fit, without any task reward. Right: Samples from goal distribution when using (a) Skew-Fit and (b) unweighted (ic. uniform) sam- pling. When used as goals, the diverse samples from Skew-Fit encourage the robot to practice opening the door more frequently.
For an agent to learn autonomously, it needs an explo- ration objective. In the absence of any prior knowledge about which states are more useful, an effective exploration scheme is one that visits as many states as possible, allowing policy to au- tonomously prepare for user-specified task that it might see at test time. This objective has been formalized as maximizing the entropy of the learned policy's visited state distribution H(S) (Hazan et al. 2018a), since policy that maximizes this objective should approach a uniform distribution over valid states. Unfortunately, directly optimizing H(S) requires accurate model of the policy and environment (Hazan et al. 2018a). Moreover, even if this optimization were tractable, another short-coming of this objective is that the resulting policy cannot be used solve new tasks: it only knows how to maximize state entropy. In other words, to develop principled unsupervised RI. enough. We need a mechanism that
algorithms that result useful policies, maximizing H(S) allows us to control the resulting policy to achieve new tasks at test-time.
We consider the distribution over terminal states in finite horizon task and believe this work can be extended infinite horizon stationary distributions.

---

Under review as a conference paper at ICLR 2020
We argue that this can be accomplished by performing goal-directed exploration. In addition to maximizing the state entropy, we should be able to control where the policy goes by giving it a goal G that corresponds to state that it must reach. Mathematically, a goal-conditioned policy should minimize the conditional entropy over the states given goal, H(S G). This objective provides us with principled way for training policy to explore all states, by maximizing H(S), such that the state that is reached can be controlled by commanding goals, which means minimizing Hi( G).
Directly using this objective is often intractable, since it requires optimizing the entropy of the marginal state distribution of the policy, H(S). However, we can sidestep this issue by noting that the objective is the mutual information between the state and the goal, (S; G), which can be written as:
H(S) - H(|G)) = L(S G) = H(G) H((())).
(I)
Equation thhus gives an equivalent objective for an unsupervised RL algorithm: the agent should set diverse goals, maximizing 7 (G), and learn how to reach them, minimizing H(G S).
While the second term the typical objective studied in goal-conditione RL (Kaelbling 1993 Andrychowicz et al. 2017), maximizing the diversity of goals is crucial for effectively learning to reach all possible states. In new environment, acquiring such a maximum-entropy goal distribution is challenging: how can agent set diverse goals when it does not even know what states exist?
In this paper, we address this question via a new algorithm, Skew-Fit, which learns to model the uniform distribution over states, given only access to data collected by autonomous goal- conditioned policy. Our paper makes the following contributions. First, we propose a principled objective for unsupervised RL, based on Equation [ While number of prior works ignore the H(G) term, we argue that jointly optimizing the entire quantity is needed to develop effective and useful exploration. Second, we propose method called Skew-Fit and prove that, under some regularity conditions, it learns a generative model that converges to a uniform distribution over the goal space, even when the set of valid states is unknown (e.g., as in the case of images). Third, we empirically demonstrate that, when combined with goal-conditioned RL, Skew-Fit allows us to autonomously train goal-conditioned policies that reach diverse states. We test this method on a variety of simulated dision-based robot tasks without any task sask-specific reward function. In these experiments, Skew-Fit reaches substantially better final performance than prior methods, and learns much more quickly. We also demonstrate that our approach solves real-world manipulation task, which requires a robot to learn to open door from scratch in about five hours, directly from images, and without any manually-designed reward function.
2 PROBLEM FORMULATION
To ensure that an unsupervised reinforcement learning agent learns to reach all possible states in a controllable way, we maximize the mutual information between the state and the goal G, I (S: G). as stated in Equation || This section discusses how to optimize Equation [yb splitting the optimization into two parts: minimizing H(G S) and maximizing (G).
2.1 MINIMIZING H(G |S) GOAL-CONDITIONED REINFORCEMEN LEARNING
Standard RL considers a Markov decision process (MDP), which has state space S, action space A. and unknown dynamics p( S;, a:): : S x S x A + [0, +o)). Goal-conditioned RL also includes a goal space G. For simplicity, we will assume in our derivation that the goal space matches the state space, such that G S, though the approach extends trivially to the case where G is hand-specified subset of S, such as the global x-y position of robot. A goal-conditioned policy π s, g) maps a state S and goal g E S to a distribution over actions E A, and its objective is to reach the goal, i.e., to make the current state equal to the goal.
Goal-reaching can be formulated as minimizing H(G S), and many practical goal-reaching algorithms (Kalbling 1993 Lillicrap et al. 2016 Schaul et al. 2015 Andrychowicz et al., 2017 Nair et al. 2018 Pong et al. 2018 Florensa et al. 2018a) can be viewed as approximations to this objective by observing that the optimal goal coal-conditioned policy will deterministically reach the goal, resulting in a conditional entropy of zero: H(G S) 0. See Appendix E for more details. Our method may thus be used in conjunction with any of these prior goal-condi RL methods in order to jointly minimize H(G S) and maximize H()).
2

---

Under review as a conference paper at ICLR 2020
2.2 MAXIMIZING H(G): SETTING DIVERSE GOALS
We now turn to the problem of setting diverse goals or, mathematically, maximizing the entropy of the goal distribution H(G).. Let Us be the uniform distribution over S, where we assume S has finite volume so that the uniform distribution is well-defined. Let Ps be the goal distribution from which goals G are sampled. Our goal is to maximize the entropy of Ps. which we write as H (G). Since the maximum entropy distribution over S is the uniform distribution Us. maximizing H(G) may seem as simple as choosing the uniform distribution to be our goal distribution: Pa Us. However, this requires knowing the uniform distribution over valid states, which may be difficult to obtain when S is subset of IR", for some For example, if the states correspond to images viewed through a robot's camera, S corresponds to the (unknown) set of valid images of the robot's environment, while RD corresponds to all possible arrays of pixel values of particular size. In such environments, sampling from the uniform distribution R" is unlikely to correspond to a valid image of the real world. Sampling uniformly from S would require knowing the set of all possible valid images, which we assume the agent does not know when starting to explore the environment.
While we cannot sample arbitrary states from S, we can sample states by performing goal-directed exploration. To derive and analyze our method, we introduce simple model of this process: goal G ~~ Pe is sampled from the goal distribution pa, and then the agent attempts to achieve this goal, which results in distribution of states S E S seen along the trajectory. We abstract this entire process by writing the resulting marginal distribution over S as p(S Ps). We assume that ((S | Pa) has full support, which can be accomplished with an apsilon-greede goal reaching policy in communicating MDP. We also assume that the entropy of the resulting state distribution H(p(S) Po)) is no less than the entropy of the goal distribution H(ps(S) the goal and stay in a single state, no matter how diverse and realistic the goals are. Note that this assumption does not require that the entropy of p(S P&) is strictly larger than the entropy of the goal distribution, ps- This simplified model allows us to analyze the behavior of our goal-setting scheme separately from any specific goal-reaching algorithm. We will however show in Section 6 that we can instantiate this approach into a practical algorithm that jointly learns the goal-reaching policy. In summary, our goal is acquire maximum-entropy goal distribution pa over valid states S, while only having access to state samples from p(S) P+).
Without this assumption, policy could ignore
SKEW-FIT: LEARNING A MAXIMUM ENTROPY GOAL DISTRIBUTION
Our method, Skew-Fit, learns a maximum entropy goal distribution ps using samples collected from a goal-conditioned policy. We analyze the algorithm and show that Skew-Fit maximizes the entropy of the goal distribution, and present a practical instantiation for unsupervised deep RL.
3.1 SKEW-FIT ALGORITHM
To learn uniform distribution over valid goal states, we present a method that iteratively increases the entropy of generative model pa- In particular, given a generative model Po, at iteration i, we would like to train a new generative model Po: such that p has higher entropy than Pa, over the set of valid states. While we do not know the set of valid states S, we can sample states from p(S Po.). resulting in an empirical distribution Pemp, over the states
N
Pemp, (s)
S, ~ p(S p(.)) S.).
N
and use this empirical distribution to train the next generative model parrr However, if we simply train P model this empirical distribution, it may not necessarily have higher entropy than Pa,-
The intuition behind our method is quite simple: rather than fitting a generative model to our empirical distribution, we skew the empirical distribution so that rarely visited states are given more weight. See Figure 2 for a visualization of this process. How should we skew the empirical distribution if we want to maximize the entropy of Po..,? If we had access the density of each state, Pemp, (S). then could simply weight each state by 1 1/Demp, (S). We could then perform maximum likelihood
3

---

Under review as a conference paper at ICLR 2020
Skew
Fit
With Skow-Fit
Fit
Without Skew-Fit
Replay Buffer
Training Data
Proposed Goals
Replay Buffer [+]
Pb
Figure 2: Our method, Skew-Fit, samples goals for goal-conditioned RL in order induce uniform state visitation distribution. We start by sampling from our replay buffer, and weighting the states such that rare states are given more weight. We then train generative model Por41 with the weighted samples. By sampling new states with goals proposed from this new generative model, we obtain higher entropy distribution of states in replay buffer at the next iteration.
estimation (MLE) for the uniform distribution by using the following loss train Φ1+1:
(s(S) Peepp (S)
(Φ)) = ESsUs [log Poggs(( (S) ES--Parr
log Po)(S) x ES
log Po Po
Pemp,
where we use the fact that the uniform distribution US( has constant density for all states in S. However, computing this density Pemp, (S) requíres marginalizing out the MDP dynamies, which requires accurate model of both the dynamics and the coal-conditioned policy.
We avoid needing to model the entire MDP process by approximating Pemp, (S) with our previous learned generative model: Pemp, (S) p(S P&.) = Pa. (S). We therefore weight each state by the following weight function
Wr,a (S) Pe. (S)):
0.
where is hyperparameter that controls how heavily weight each state. If our approximation Pa. was exact, we could choose - and recover the exact importance sampling procedure described above. If a O, then this skew step has no effect. By choosing intermediate values of a, we can trade off the reliability of our estimate Pd. (S) with the speed at which we want to increase the entropy of the goal distribution.
Variance Reduction As described, this procedure relies on importance sampling (IS), which have high variance, particularly if (S) &e O. We therefore choose a class of generative models where the probabilities are prevented from collapsing zero, as will describe in inSectionn To further reduce the variance, we train Po++, with sampling importance resampling (SIR) (Rubin 1988).
Rather than sampling from Pemp, and weighting the update from each sample by Wr,a. SIR explicilly defines a skewed distribution as
N
1 Z. Pomp, (s))),a(s), Za
Pakewod, (s) &
Pemp, (Sr
(4)
where Zι is the normalizing coefficient and Pemp, is given by Equation We note that computing Za adds little computational overhead, since all of the weights already need to be computed. We then fit the generative model at the next iteration Po.=i to Pakewed. using standard MLE. We found that using SIR resulted in significantly lower variance than IS. See Appendix B.3 for this comparision.
Goal Sampling Alternative Because P Pskewed, at iteration t + 1, one can sample goals from either Potja or Pakewed, Sampling goals from Pakewed, may be preferred sampling from the learned generative model P computationally otherwise challenging. In either case, still needs to train the generative model Po. to create Pskewed. In our experiments, we found that both methods perform well.
4

---

Under review as a conference paper at ICLR 2020
Summary Overall, Skew-Fit samples data from the environment and weights different samples by their density under the generative model Pœr. We prove in the next section conditions under which this weighting makes the generative model at the next iteration Pøt+1 have higher entropy. With higher entropy, the p is more likely to generate goals at the frontier of unseen states, which results in more uniform state coverage. Skew-Fit is shown in Figure 2 and summarized in Algorithm [
Algorithm Skew-Fit
1: for Iteration / 1, ... do
2: Collect .V states { by sampling goals from Ps. (or Pskowed,) ) and running goal- conditioned policy.
3: Construct skewed distribution Pskewed, (Equation and Equation
4: Fit Por+1 to skewed distribution Pikewed, using MLE.
5: end for
3.2 SKEW-FIT ANALYSIS
In this section, we provide conditions under which Pe, converges in distribution to the uniform distribution over the state space S. To make this analysis possible, consider the case where N + 30, which allows us to study the limit behavior of the goal distribution Pstewed, Our most general result is stated follows:
Lemma 3.1. Let S be a compact set. Define the set of distributions Q {p: support of p is S). Let F: : Q + Q be a continuous function and such that H(F(p)) 2 H(p) with equality if and only if is the uniform probability distribution on S, Us. Define the sequence of distributions (pi, P2. .... by starting with any P E Q and recursively defining p++1 F(p)).
The sequence P converges to Us-
Proof. See Appendix Section
We will apply Lemma 3.1 be the map from Pskewed, to Pskewedea1 to show that Pskewed, converges to Us. If we assume that the goal-conditioned policy and generative model learning procedure are well behaved ( i.e., the maps from Ps, (S) to Pemp, and from Pskewed, to Por+1 are continuous ), then apply Lemma 3.1 we only need show that (Pskewed,,) ) 2 (pemp, with equality if and only if Pemp, Us. For the simple case when Ps. = Pemp, identically at each iteration, we prove the convergence of Skew-Fit true for any value of a E [-1,0] 0) in (Appendix A.3) However, in practice, Ps: only approximates Pemp.- To address this more realistic situation, we prove the following result:
Lemma 3.2. Given two distribution Pemp, and po. where Pemp,
< Por
and
CovS-poon [log Pemp, 1s),logPo. Po. (S)] 0,
(5)
Let Ha(a) be the entropy of Pstewed, for a fixed a. Then there exists constant a < O such that for all E [a, 0),
define the distribution Pskewed, as in Equation 4
H(pstewed,) = Ha(ai) > H(Pemp,)-
Proof. See Appendix Section
Thus, our generative model Ps, does not need to exactly fit the empirical distribution. We merely need for the log densities of po, and Pemp, to be correlated, which we expect to happen frequently with an accurate goal-conditioned policy, since Pemp, the set of states seen when trying to reach goals from P&, In this case, if we choose negative values of a that are small enough, then the entropy of Pikewed, will be higher than that of Pemp,- Empirically, we found that a values as low as a -1 performed well.
In summary, we see that under certain assumptions, Pskewed, converges to Us- Since we train each generative model Por+1 by fitting it to Pskewed, we expect Po, to also converge to Us-
p & q means that p is absolutely continuous with respect to z, i.e. p(s) ((s)= 0.
5

---

Under review as a conference paper at ICLR 2020
4 TRAINING GOAL-CONDITIONES POLICIES WITH SKEW-FIT
Thus far, we have presented and derived Skew-Fit assuming that we have access to a goal-reaching policy, allowing us to separately analyze how we can maximize H(G)) However, in practice we do not have access to such a policy, and in this section we discuss how we concurrently train goal-reaching policy.
Maximizing T(S; G) can be done by simultaneously performing Skew-Fit and training a goal conditioned policy to minimize H(G S), or, equivalently, maximize (G S). Maximizing -HI(G | S) requires computing the density log p G S), which may be difficult to compute without strong modeling assumptions. However, for any distribution q. the following lower bound for
-H(G S) holds:
-H(G S) = (GS [G S)] + (KL(P) 1) 2 (.G. [lg (G [S]]
where DKL denotes Bealbackk-eibler divergence as discussed by Barber & Agakov (2004). Thus, to minimize H S), we train a policy to maximize the following reward:
rSS, G) = loog QG S).
For the RL algorithm, use reinforcement learning with imagined goals (RIG) (Nir et al. 2018 8), though in principle any goal-conditioned method could be used. RIG is an efficient off-policy goal- conditioned method that solves the vision- based RL problem in learned latent space. In particular, RIG fits a 3-VAE and uses it to encode all observations and goals into a latent space, which it uses as the state representation. RIG also uses the 3-VAE to compute rewards, log q( S). Unlike RIG, we use the goal distribution from Skew- Fit to sample goals, both for exploration and for relabeling goals during training (Andrychowicz al. (017). Since RIG already trains generative model over states, we reuse this 3-VAE for the generative model Pa of Skew-Fit. To make the most use of the data, Ps is trained on all visited state rather than only the terminal states, which we found work well in practice. In other words, our method uses the likelihood estimates from the 3-VAE to choose the probability of sampling each state in Equation 3] To prevent these probabilities from collapsing to zero, we model the posterior of the 5-VAE as multivariate Gaussian distribution with fixed variance and only learn the mean. We include detailed summary of RIG and description our how we combine Skew-Fit and RIG in Appendii C.I
RELATED WORK
Many prior methods for training goal-conditioned policies assume that a goal distribution is available
to sample from during exploration (Kaclbling 1993, Schaul et al. 2015 Andrychowicz et al. 2017]]
Pong et al., 2018). Other methods use data collected from a randomly initialized policy or heuristics based data collected online to design non non- -arrammmetr (Colas al. 2018b Warde-Farley et al.)
2018: Florensa et al., 201 8a, ZZha & Tresp 2019) or parametric (Péré et al., 2018: Nair et al.) (018) goal distribution. We remark that Warde-Farley
al. (2018) also motivate their work in terms of minimizing lower bound for Hc (G S). S). Our work is complementary to these goal-reaching methods: rather than focusing on how to train goal-reaching policies, we propose a principled method for maximizing the entropy of a goal sampling distribution, H()).
Our method learns without any task rewards, directly acquiring policy that can be reused reach user-specified goals. This stands in contrast to exploration methods that give bonus rewards based on state visitation frequency (Bellemare et al. 2016 Ostrovski et al., [017]] Tang et al.) 2017; Savinov
et al., 2018 Chentanez et al. 2005 Lopes er al. 2012; Stadie et al. 2016: Pathak et al. 2017]
Burda al. 2018| 2019 Mohamed
| & Rezendel 2015 Tang et al. 2017]] Fu al. 2017). While
these methods can also be used without a task reward, they provide
mechanism for distilling the
knowledge gained from visiting diverse states into flexible policies that can be applied to accomplish
new goals test-time: their policies visit novel states, and they quickly forget about them other states become more novel.
Other prior methods extract reusable skills in the form of latent- latent variables can be interpreted as options (Sutton et al. 1999) or abstract skills (Hausman et al. 2018. Gupta et al. 2018b: Eysenbach et al. 2019 Gupta et al. 2018a
policies, where
Florensa et al.) 2017). The

---

Under review as a conference paper at ICLR 2020
resulting skills may be diverse, but they have grounded interpretation, while method can be used immediately after unsupervised training to reach diverse euer-specified goals.
Some prior methods propose to choose goals based heuristics such as learning progress (Baranes & Oudeyer 2012: Veeriah et al. 2018; Colas et al.. Sa), how oollcc the goal is (Nachum et al. 2018), level of difficulty (Florensa et al. 20188) or likelihood ranking (Zhao & Trespp 2019). In contrast, our approach provides a principled framework for optimizing a concrete and well-motivated exploration objective, and can be shown maximize this objectíve under regularity assumptions. The work of Hazan et al. (2018b) also provably optimizes a well-motivated exploration objective, but limited to tabular MDPs, while Skew-Fit able to handle high dimensional settings such as vision-based continuous control.
EXPERIMENTS
Our experiments study the following questions: (1) Does Skew-Fit empirically result in a goal distribution with increasing entropy? (2) In image-based domains, how does Skew-Fit compare to prior work on choosing goals for coal-conditioned RL? (3) Can Skew-Fit be applied a real-world, vision-based robot task?
Does Skew-Fit Maximize Entropy? To see the effects of Skew-Fit on goal distribution entropy in isolation of learning goal-reaching policy, we begin by studying an idealized example where the policy is a near serr-perfect goal-reaching policy. The MDP is defined on 2-by-2 unit square-shaped corridor (see Figure 3). At the beginning of an episode, the agent begins the bottom-left corner and samples goal from the goal distribution Ps.- To simulate the stochasticity of the policy and environment, we add a Gaussian noise with standard deviation of 0.05 to this goal. The policy reaches the state that is closest to this noisy goal and inside the corridor, giving us state S to add to empirical distribution. We compare Skew-Fit to sampling uniformly from the replay buffer (labeled MLE). The 3-VAE hyperparameters used to train Pe. given in Appendix C.5 As seen in Figure 3
Fouch
Entropy by Epoch
MLE
Entropy
Foudh
Skew- Fit
MLF
Skew Fit
20
40
Epoch
Figure 3: (Left) The set of final states visited by agent and MLE over the course of training. In contrast MLE, our method quickly approaches a uniform distribution over the set of valid states. (Right) The entropy of the sample data distribution, which quickly reaches its maximum for Skew-Fit. The entropy calculated via discretization onto 60 by 60 grid.
goals results in policy that primarily sets goal near the initial state distribution and only relies on the stochasticity of the policy and environment to explore. In contrast, Skew-Fit results in quickly learning high entropy, near-uniform distribution over the state space.
naively using previous experience
Vision-Based Continuous Control Tasks We now evaluate Skew-Fit on a variety of continuous control tasks, where the policy must control robot arm using only image observations, without access to any ground truth reward signal. We test our method on three different simulated continuous control tasks released by the authors of RIG (Nair et al. 2018): Visual Door, Visual Pusher, and Visual Pickup. To our knowledge, these are the only conl-condiiioned, vision-based continuous control environments that are publicly available and used in experimental evaluations in prior work, making them good point of comparison. See Figure for visuals and Appendix Cfoor details of these environments. The policies are trained in a completely unsupervised manner, without access to any prior information about the state- space any pre-defined goal-sampling distribution. To evaluate their performance, we sample goal images from a uniform distribution over valid states and report the agent's final distance to the corresponding simulator states (e.g., distance of the object
7

---

Under review as a conference paper at ICLR 2020
Figure 4: We evaluate on these continuous control environments. From left right: Visual Pusher, simulated pushing task; Visual Door, door opening task; Visual Pickup, picking task; and Real World Visual Door, a real world door opening task. tasks are solved from images and without any task soec-speciiccc reward. See Appendix D for details.
Goal
RIG+Skew-Pis
RIG
Final Object Distance
Visual Object Pickup
8.00
am
Distance 0.02
Distance 0.04
Timestens Visual Puck Pushing
Final Puck Distange
EKG (Dur))
RIG Rosed Piieayy
ENG
0.10
DISCERN
0.08
RKG DISUIZING
..... RXI HII
-... RXT Assioal GAN
RXC Exploaaioo
0.04
IGOK
Timestops
200K
300K
Distance
Final Angle Difference
Visual Door Opening
Dislance 0,03 Distance 0,16 Figure 5: (Left) Learning curves for simulated continuous control experiments. Lower better. For each environment and method, we show the mean and standard deviation of 6 seeds and smooth temporally across 25 epochs within each seed. Skew-F Fit consistently outperforms RIG and various baselines. See the text for description of each method. (Right) The first column displays example test goal images for each environment. In the next two columns, we display final images reached by Skew-Fit and RIG respectively. Under each image the final distance state space provide notion of the behavior of each method in the plots.
Timesteps
to the target object location), but the agent never has access to this true uniform distribution nor the ground-truth state information during training. While this evaluation method and metric is only practical in simulation, it provides with quantitative measure of policy's 's ability to reach broad coverage of goals in a vision-basse setting.
We these domains to compare Skew-Fit to number of existing methods goal-sampling. We compare to Warde-Farley et al, 2018). a vision-based method which uses a non- parameeeiiiicc approach based clustering sample goals and image discriminator to compute rewards. We denote this method as DISCERN. The other methods that we compare to were developed in non-vision, state- based environments. To ensure a fair comparison across methods, combine these prior methods with policy trained using RIG. First, we compare to RIG without Skew-Fit, We also compared to RIG using the relabeling scheme described in the hindsight experience replay (labeled HER). We compare to curiosity- driven prioritization (Ranked- Based Priority) (Zhao & Tresp 2019), variant of HER that samples goals for relabeling based on their ranked likelihoods. FFloresaa al. (20T samples goals from GAN based on the difficulty of reaching the goal. We compare against this method by replacing Ps with the GAN and label AutoGoal GAN. We also separately compare to the goal proposal mechanism proposed by Warde-Farley et al. (2018) and otherwise train the policy with RIG, which we label DISCERN-g. Lastly, to demonstrate the difficulty of the exploration challenge in these domains, compare # Exploration (Tang al. 2017). exploration method that assigns bonus rewards based on the novelty of new states. Implementation details of the prior methods is given in Appendix C.3
8

---

Under review as a conference paper at ICLR 2020
We see in Figure 5 that Skew-Fit significantly outperforms prior methods both in terms of task performance and sample complexity. The most common failure mode for prior methods is that the goal distributions collapse, resulting in the agent learning to reach only fraction of the state space, as shown in Figure 1 For comparison, additional samples of Po when trained with and without Skew-Fit are shown in Appendix B.4] Those images show that without Skew Fit, pe produces small, non-diverse distribution for each environment: the object is in the same place for pickup, the puck often in the starting position for pushing, and the door is always closed. In contrast, Skew-Fit proposes goals where the object is in the air and on the ground, where the puck positions are varied, and the door angle changes.
The direct effect of these goal choices can be seen by visualizing more example rollouts for RIG and Skew-Fit. Due to space constraints, these visuals are in Figure 16 Appendix B.4 The figure shows that standard RIG only learns to reach states close to the initial position, while Skew-Fit learns to reach the entire state space. A quantitative comparison of the various methods on the pickup task can be seen in Figure 6 which gives the cumulative total exploration pickups for each method. From the graph, we can see that only Skew-Fit learns to pay attention to the object and therefore consistently increases the rate at which the policy picks up the object during exploration. In contrast, the other methods have near constant slopes past 40k steps, meaning that they do not continue to learning, and many methods have near-constant rate of object lifts throughout all of training.
Visual Object Pickup
RIG + Skew-Fit (Ours)
Exploration Pickups
RIG
60
RIG + Rank-Based Priority
DISCERN
40
RIG + DISCERN-g
20
RIG + HER
RIG + AutoGoal GAN
20K
40K
60K
80K
100K
120K
RIG + Exploration
Timesteps
Figure 6: Cumulative total pickups during exploration for each method. The prior methods fail to pay attention to the object and only pick it up at the same rate the initial policy. In contrast. after seeing the object picked up a few times, Skew-Fit practices picking up the object more often by sampling the appriopriate exploration goals.
Real-World Vision-Basee Robotic Manipulation We also demonstrate that Skew-Fit scales well to the real world with a door opening task, Real World Visual Door. See Figure a picture of this environment. While a number of prior works have studied RL-based learning of door Chebotar opening Kalakrishnan et al. [hebotar et al. (2017).
Real World Visual Door Opening
Chontaatee Stccesies
we demonstrate the first method for autonomous learning of door opening without a user-provided task-specific re- ward function. As in simulation, we do not provide any goals to the agent and simply let it interact with the door to solve the door opening task from scratch, without any human guidance reward signal. We train two agents using Skew-Fit with RIG and RIG alone. Unlike in sim- ulation, cannot measure the difference between the policy's ' achieved and desired door angle since we do not have access the true state of the world. Instead, we simply visually denote a binary success/failure for each goal based on whether the last state in the trajectory achieves the target angle. Every seven and a half minutes of interaction time we evaluate on 5 goals and plot the cumulative successes for each method. As Figure 7 shows, standard RIG only starts to open the door after five hours of training. In contrast, Skew-Fit learns to occasionally open the door after three hours of training and achieves near-perfect success rate after five and a half hours of interaction time, demonstrating that Skew-Fit is a promising technique for solving real world tasks without any human- Videos of Skew-Fit solving this task and the simulated tasks can be viewed on our website
20
Inceraction Time (lous))
Learning curve for Real World Visual Door environment. We visually label a success if the policy opens the door to the target angle by the last state of the trajec- tory. Skew-Fit results considerable sample efficiency gains over prior work this real- world task.
Figure 7:
reward function.
Anonymous while under review:

---

Under review as a conference paper at ICLR 2020
Additional Experiments To study the sensitivity of our method to the hyperparameter a, we sweep a across the values -01,5 -0.5, -0.25, 0] on the simulated image-based tasks. Due to space constraints, the sensitivity analysis over the hyperparameter is in Appendix BJ and the results demonstrate that Skew-Fit works across a large range of values for a, and a = -1 consistently O, where the empirical distribution is not skewed. Additionally, Appendix C provides a complete description our method hyper-parametens including network architecture and RL algorithm hyperparameters.
outperform
CONCLUSION
We presented a formal objective for self-supervised goal-directed exploration, allowing researchers to quantify progress and compare progress when designing algorithms that enable agents to au- tonomously learn. We also presented Skew-Fit, an algorithm for training a generative model to approximate uniform distribution forcement learning, and our theoretical analysis gives conditions under which Skew-Fit converges to the uniform distribution. When such model is used to choose goals for exploration and to relabeling goals for training, the resulting method results in much better coverage of the state space, enabling our method to explore effectively, Our experiments show that when we concurrently train goal-reaching policy using self- -eferaeedd goals, Skew-Fit produces quantifiable improvements on simulated robotic manipulation tasks, and can be used to learn a door opening skill reach a 95% success rate directly on real-world robot, without any human-provider reward supervision.
valid states, using data obtained via goal-conditioned rein-
REFERENCES
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., Megrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight Experience Replay. In Advances in Neural Information Processing Systems (NIPS), 2017.
Baranes, A. and Oudeyer. P.-Y. Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots. Robotics and Autonomous Systems, 61([):49-77) 2012. doi: 00.10166..obbo.2012.00 0.000.0000.......
Barber, D. and Agakov, F. V. Information maximization in noisy channels: variational approach. In Advances in Neural Information Processing Syxtems, pp. 201-208, 2004.
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems (NIPS), pp- 1471-1479, 2016.
Billingsley, P. Convergence of probubility measures. John Wiley & Sons, 2013.
Burda, Y., Edwards. H.. Storkey, A., and Klimov, O. Exploration random network distillation. arXiv preprint arXiv: 1810. 12894, 2018
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, and Efros, A. Large-scale study of curiosity-driven learning. In International Conference on Learning Representations (ICLR), 2019.
Chebotar, Y., Kalakrishnan, M., Yahya, A., Li, A., Schaal, S., and Levine, S. Path integral guided policy search. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 3381 3388. IEEE, 2017.
Chentanez, N.. Barto, A. G., and Singh, S. P. Intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pp. 1281-1288, 2005.
Colas, C., Fournier, Sigaud, O.,, and Oudeyer, P. CURIOUS: intrinsically motivated multi-task, multi-goal reinforcement learning. CoRR, abs/1810.06254 2018a.
Colas, C., Sigaud, O., and Oudeyer, P.-Y. Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms. International Conference on Machine Learning (ICML), 2018b.
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is All You Need: Learning Skills without a Reward
Function. In International Conference on Learning Representations (ICLR), 2019.
Florensa, C., Duan, Y., and Abbeel, P. Stochastic neural networks for hierarchical reinforcement learning. In International Conference on Learning Representations (ICI.),, 2017.
10

---

Under review as a conference paper at ICLR 2020
Florensa, C., Degrave, J., Heess, N., Springenberg, J. T., and Riedmiller, M. Self-supervised Learning of Image Embedding for Continuous Control. In Workshop on Inference to Control at NeurlPS, 2018a.
Florensa, C.,, Held, D., Geng, X., and Abheel, P. Automatic Goal Generation for Reinforcement Learning Agents. In International Conference on Machine Learning (ICML), 2018b.
Fu, J., Co-Reyes. J. D., and Levine, S. EX 2 Exploration with Exemplar Models for Deep Reinforcement Learning. In Advances in Neural Information Processing Systems (NIPS), 2017.
Fujimoto, S., van Hoof, H., and Meger, D. Addressing Function Approximation Error in Actor-Critic Methods. In International Conference on Machine Learning (ICML), 2018.
Gupta, A., Eysenbach. B., Finn, C., and Levine. S. Unsupervised meta-learning for reinforcement learning. CoRR, abs:1806.0466D0 201 8a.
Gupta, A., Mendonca, R., Liu, Y., Abbeel, P. and Levine, S. Meta-Reinforremmen Learning of Structured Exploration Strategies. In Advances in Neural Information Processing Syxtems (NIPS), 2018b.
Нaarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. Soft actor-critic algorithms and applications. CoRR, abs//812.000 181.0905,, 2018.
Hausman, K., Springenberg, J. T., Wang. Z., Heess, N., and Riedmiller, M. Learning an Embedding Space for Transferable Robot Skills. In International Conference on Learning Representations (ICLR), pp. 1-16, 2018.
Hazan, E., Kakade, M., Singh, K., and Soest, A. V. Provably efficient maximum entropy exploration. CoRR, abs/ 1os/1812.02690, 2018a.
Hazan, E., Kakade, M., Singh, K., and Soest, A. V. Provably efficient maximum entropy exploration. CoRR, abs/1812.02690, 2018b.
Kaelbling, Learning to achieve goals. In International Joint Conference on Artificial Intelligence (ICAI)) volume vol.2. pp. 1094 8, 1993.
Kalakrishnan, M., Righetti, L., Pastor, P., and Schaal, Learning foree control policies for compliant manipulation. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4639 4644.
IEEE, 2011.
Lillicrap. T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016. ISBN 0-7803-3213-X, doi: 10. V01...3.0.
Lopes, M., Lang. T.. Toussaint, M., and Oudeyer, P--.. Exploration in model-based reinforcement learning by empirically estimating learning progress. In Advances in Neural Information Processing Systems, pp. 206-214, 2012.
Mohamed, S. and Rezende, J. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pp. 2125-2133, 2015.
Nachum, O., Brain, G., Gu, S., Lee, H., and Levine, S. Data-Efficient Hierarchical Reinforcement Learning. In Advances in Neurul Information Processing Systems (NeurIPS), 2018.
Nair, A., Pong, V., Dalal, M., Bahl, S., Lin, S., and Levine, S. Visual Reinforcement Learning with Imagined Goals. In Advances in Neural Information Processing Sysiems (NeurIPS), 2018
Nielsen, F. and Nock, R. Entropies and cross-entie cooss-ent of exponential families. In Image Processing (ICIP), 2010 17th IEEE International Conference on, pp.
IEEE, 2010.
Ostrovski, G., Bellemare, M. G., Oord, A., and Munos, R. Count-based exploration with neural density models. In International Conference on Machine Learning, pp. 2721-2730, 2017.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity- -Driven Exploration by Self-Supervised Prediction. In International Conference on Machine Learning (ICML), pp. 488 489. IEEE, 2017.
Péré, A., Forestier, S., Sigaud, O., and Oudeyer, P.-Y. Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration. In International Conference on Learning Representations (ICLR), 2018.
Pong, V., Gu, S., Dalal, M., and Levine, S. Temporal Difference Models: Model-Free Deep RL For Model-Bascd Control. In International Conference on Learning Representations (ICLR), 2018
Rubin, D. B. Using the sir algorithm to simulate posterior distributions, Boyesian statistics, 3:395-402, 1988.

---

Under review as a conference paper at ICLR 2020
Savinov, N., Raichuk, A., Marinier, R., Vincent, D., Pollefeys, M., Lillierap. T., and Gelly, Episodic curiosity through reachability, arXiv preprint arxiv:1810.02274, 2018.
Schaul, T., Horgan, D.. Gregor, K., and Silver. D. Universal Value Function Approximators. In International Conference on Machine Learning (ICML), pp. 2312-1320, 2015. ISBN 9781510810587
Stadie, B. C., Levine, S., and Abbeel, P. Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models, International Conference on Learning Representations (ICLR), 2016.
Sutton, S., Precup, D., and Singh, S. Between midss and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2) (101112):81-21111 1999.
Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. #Exploration: A Study Count- Bouut-Baeed Exploration Deep Reinforcement Learning. Neural Information Processing Systems (NIPS), 2017.
Veeriah, V., Oh, J., and Singh, S. Many-goals reinforcement learning. arXiv preprint
2018.
Warde-Farley. D.. de Wiele, T. V., Kulkarni. T., Ionescu, C., Hansen. S., and Mnih, V. Unsupervised control through non-parametric discriminative rewards. CoRR, abs/1811.11359, 2018.
Zhao, R. and Tresp, V. Curiosity-driven experience prioritization via density estimation. CoRR, abs/ 190200999., 2019.
12

---

Under review as a conference paper at ICLR 2020
A PROOFS
A.1 PROOF OF LEMMA 3.1
Lemma A.1. Let S be a compact set. Define the of distributions {p= support of p S}. Let F Q + Q be a continuous function and such that H(F(p)) 2 H(p) with equality if and S, Ug. Define the sequence of distributions
only p is the uniform probability distribution P = (P1,P2, P.....) by starting with any E Q and recursively defining P++1 = F(pt).
The sequence P converges Ug.
Proof. The uniform distribution Us is well defined since S is compact. Because S is compact set, by Prokhorov's Theorem Billingsley (2013), the set is sequentially compact. Thus, P has convergent subsequence P' (Pk, P.a..... C P for k < h2 ... that converges distribution p" Q. Because F is continuous, p" must be a fixed point of F since by the convergence mapping theorem, have that
lim Ph. = pp = 1000
lim F(ps.) H(p")
1000
and
p" lim Pk:
i-x
lim F(Pk-1) 1-100
H(p").
The only fixed point of F is Us since for any distribution p that is not the uniform distribution, Us. we have that H(F(p)) > H(p) which implies that F(p) * Thus, P' converges to the only fixed point, Us. Since the entropy cannot decrease, then entropy of the distributions in P must also converge to the entropy of Us. Lastly, since entropy is continuous function distribution, must converge to Us.
A.2 PROOF OF LEMMA 3.2
Lemma A.2. Given two distribution p(x) and q(z) where < and Covpllog (X), log g( (X)
(6)
define the distribution Pa as
Pa (a(x) = p(x (xx))
Za
where a E IR and Za is the normalizing factor. Let Ha(aa) be the entropy of Pa. Then there exists constant a >> 0 such that for all a E [-a,0).
Ha(a) > (o) = =((p).
Proof. Observe that (Pa a (-1,0 0]} one-dimensional exponential family
(a(x) [())
with log carrier density l<x)) = log p(x), natural parameter a. sufficient statistic T(x) = log g(x),
and log-normalizer A(a:) = fx (<)))) dr. As shown in Nielsen & Nock (2010). the entropy of distribution from one-dimensional exponential family with parameter a given by:
(a) H(pa) = A(a) A'A(aa) EEp,
The derivative with respect to is then
d
Ha(a) = (aA" d
da Ep. [k()] -a) - E,, (TT((()(T(())) A'(a) =-aVar,. [Txx) [(x))))]T(xx)] Cov,. [+(x),
da
13

---

Under review as a conference paper at ICLR 2020
where the fact that the nth derivative of A(a) give the n central moment, i... A'(a) Ep. [x))] and Aa(a) = Varp,. [T())] The derivative of = 0 is
d da
Ha (0) = Co (xx),,(x))
= -oov p(x), log g(x)]
which is negative by assumption. Because the derivative at - 0 is negative, then there exists constant 0 such that for all [-a, 0J, (a) > H. (0) H(p).
A.3 SIMPLE CASE PROOF
We prove the convergence directly for the (even more) simplified case when po = p(S p&.) using similar technique:
Lemma A.3. Assume the set S has finite volume so that its uniform distribution Us is well defined and has finite entropy. Given any distribution p(s) whose support is recursively define pe with P1 and
P(+1(s) =
Z: P((s)), E
where Zy is the normalizing constant and a E 1).
The sequence (p1,P2, ..........) converges to Us, the uniform distribution
Proof. If a 0, then P2 (and all subsequent distributions) will clearly be the uniform distribution. We now study the case where a E (0, 1).
At each iteration define the one-dimensional exponential family {pá 0 E [0, 1]} where ph is
(s) = A(0)++))
with log carrier density k(s) = O, natural parameter 8 sufficient statistic T(s) = log (s), and log-
normalizer A()) = Ss e (T)))) ds. As shown in Nielsen & Nock (2010), the entropy of a distribution from a one-dimensaa exponential family with parameter is given by:
(0) H(pb) = A(0) - OA)
The derivative with respect to 0 is then
(0) (0)
d0
= -oarr [s))
Var [s)]
(8)
0
where the fact that the nth derivative of A(F) is the central moment, i.e. A²(0) Vars~pş [T(s)]. Since variance always non-negative, this means the entropy is monotonically
decreasing with 0. Note that p++1 is a member of this exponential family, with parameter 0 = a E (0, 1). So
H(p++1) Ha)aa) > H H(p.)
which implies
H(p:) < H(2)) S ....
This monotonically increasing sequence is upper bounded by the entropy of the uniform distribution. and so this sequence must converge.
The sequence can only converge if ()))) converges zero. However, because bounded away from 0, Equation 8 states that this can only happen if
Varsung [logpr Pr (s)] + 0.
Because p: has full support, then so does pá. Thus, Equation 9 is only true log (s) converges to a constant, i.e. pt converges to the uniform distribution.
14

---

Under review as a conference paper at ICLR 2020
Four Rooms Coverage Alpha Ablation
1,00
Covenge
0.75
alph--0
alpha-00.5
0.50
aighmm]
0.25
alpiz 2.5
200K
Episodes
Maze Coverage Alpha Ahlation
1.00
0.75
abph-00
alpham-0.55
0.50
alphz-11
a.phz=2255
200K
400K
Episodes
Figure 8: (Top) Coverage over time on the classic 4-room domain, shown the right. (Bottom) Coverage over time on a more challenging maze domain, shown on the right. In both cases, we see that not using Skew-Fit (a 0) results in significantly slower learning that primarily stays near the start (yellow star).
B ADDITIONAL EXPERIMENTS
B.1 SKEW-FIT FOR EXPLORING LOW-DIMENSIONAL SPACES
Skew-Fit is a general method that enables exploration when it is infeasible to sample goal states uniformly across the entire state space. While the experiments Section 6 focused on image-based state spaces, there exists many now-dimessional domains in which we know that the goal space is subset of dRd for some d n. but the exact goal space still unknown. This scenario is quite common in domains such as robotics: we know that we want agent to move the position of its center of mass (CoM), but we do not know the set of valid CoM positions, as this requires knowing the geometry of all potential obstacles priori. We conduct series of experiments that study whether Skew-Fit enable effectively exploration in these state spaces containing unknown obstacles.
2D Maze Navigation with Oracle Policy To study the impact of Skew-Fit on exploration in isolation of learning goal-reaching policy, our first set of experiments use near-perfect policy that reaches the goal state and then takes a step in a random direction (while taking wall-collisions into account). The random step size is Gaussian with standard deviation of 0.1 units, and the size of each square shown in Figure iis 1.8 units. Due to the relatively small step size, the agent cannot rely on random actions to explore the environment and must instead learn to set goals that progressively farther and farther from the initial state. The first environment is the Four Rooms environment (Sutton al.) 1999), shown in Figure (top). This environment requires policy explore four different rooms, each of which requires passing through a narrow doorway. The maze environment (Figure 8 bottom) presents more challenging exploration problem and consists of various long corridors that require setting goals progressively deeper into the maze. In both domains, setting goals near the state state (represented by the yellow star) and taking small actions will result in minimal exploration. To measure exploration, we discretize the space into squares (see sigure 8 for square sizes) and measure what fraction of the squares the agent has ever visited during exploration. We see (Figure 8 that using Skew-Fit significantly improves exploration, whereas training P4 samples drawn uniformly from the replay buffer (a = 0) results in little exploration.
2D Navigation with Learned Policy Next, we reproduce the 2D navigation environment ex- periment from SSection 6) and replace the oracle goal-reacher with goal-reaching policy that is simultaneously trained with the goal setter. The policy outputs velocities with maximum speed one. Evaluation goals are chosen uniformly over the valid states. The hyperparameters for this experiment are given in Table 2] In Figure Da| we can see that policy trained with a goal distribution trained by Skew-Fit consistently learns to reach all goals, whereas goal distribution trained with uniform sampling, labeled MLE, results in a policy that fails to reach states far from the starting position (the bottom left corner).
15

---

Under review as a conference paper at ICLR 2020
Final Distance to Goal
Pointmass: Goal Distance
Final Distance to Goal Skew-Fit
MLE
OK
10K
15K
20K
Timesteps
Skow-Fit
MLE
(a)
(b)
Figure 9: (a) Comparison of Skew-Fit vs MLE goal sampling on final distance to goal on version pointmaas environment. Skew-Fit consistently leamss solve the task, while MLE often fails. (b) Heatmaps of final distance to each possible goal location for Skew Fit and MLE. Skew-Fit Fit learns good policy over the entire state space, but MLE performs poorly for states far away from the starting position (the bottom left comner.
Ant
Final XY-Distance
Skew-Fit (Ours)
DISCERN
DISCERN-g
HER
AutoGoal GAN
Exploration
100K
200K
300K
400K
500K
Timesteps
Figure 10: (Left) Ant navigation environment. (Right) Evaluation reaching joint and XY position. Policies are trained from state. Reward n2-norm between current and target joint angle and XY position concatenated together. We use Skew-Fit to sample goals for relabeling and exploration, and compare other goal sampling methods. See main paper for description baselines.
Quadruped "Ant" Locomotion with Learned Policy Lastly, test Skew-Fit in exploration task that requires training a simulated quadruped "ant" robot to navigate to random XY positions in plane, shown in Figure TO The input to the policy the joint and velocity of each angle and the reward is the distance to the goal XY-position. While the goal space is known to reside in the X--plane, the agent does know about the location of the center obstacle, and it must still learn about the set of valid goals by controlling its 8 joint actuators. More details of the environment in Appendix D We see in Figure I0 that Skew-Fit outperforms prior methods both in terms of learning speed and final performance, demonstrating that Skew-Fit accelerates exploration in non-vision domains that contains unknown goal spaces.
B.2 SENSITIVITY ANALYSIS
Sensitivity to RL Algorithm In our experiments, we combined Skew-Fit with soft actor critic (SAC) (Haarnoja et al. 2018). We conduct set of experiments to test whether Skew-Fit may be used with other RL algorithms for training the goal-conditioned policy. To that end, we replaced SAC with twin delayed deep deterministic policy gradient (TD3) (Fujimoto et al.) 2018) and ran the same Skew-Fit experiments on Visual Door, Visual Pusher, and Visual Pickup. In Figure II we see that Skew-Fit performs consistently well with both SAC and TD3, demonstrating that Skew-Fit is beneficial across multiple RI. algorithms.
Sensitivity to Hyperparameter We study the sensitivity of the hyperparameter by testing values 1-1.50.75, -0.5, -0.25, 0) on the Visual Door and Visual Pusher task. The results are included in Figure 12 and shows that our method is robust different parameters of particularly for the more challenging Visual Pusher task. Also, the method consistently outperform a O, which equivalent to sampling uniformly from the replay buffer.
16

---

Under review as a conference paper at ICLR 2020
Final Angle Difference
Visual Door Opening
0.4
Skew-Fit-AC Sint-ACC(Ma (Ours)
Skew-Fit-TDS
20K
30K
40K
50K
60X
70K
Timesteps
Visual Object Pickup
Visual Puck Pushing
Final Object Distance
0.05
Final Puck Distance
0.12
Skow-Fil-SAC (Ours)
Skow-Fit- FAC (Ous)
0.04
Skow-Fil-TD3 TD3
0.10
Skew-Fit Fi--DD
0.03
0.08
0.02
0.06
0.01
100K
200K
300K
0.04
100K
200K Timestops
300K
Timestops
Figure 11: We compare using SAC (Haarnoja et al. 2018) and TD3 (Fujimoto et al. 2018 as the underlying RL algorithm Visual Door, Visual Pusher and Visual Pickup. We that Skew-Fi Fit works consistently well with both SAC and TD3, demonstrating that Skew-Fit may be used with various RI. algorithms.
Final Angle Difference
Visual Door Alpha Ablation
0.4
alpha-0 (No Skew-Fit)
alpha=-0.25
alpha=-0.5
alpha=.0.75
alpha=-1
60K
70K
Timestens
Visual Pickup Alpha Ablation
Visual Pusher Alpha Ablation
Final Object Distance
Final Pack Distance
0.12
0.04
0.10
0.08
0.02
0.06
OK
200K
400K Timesteps
600K
0.04
100K
200K Timesteps
300K
Figure 12: We sweep different values of Visual Door, Visual Pusher and Visual Pickup. Skew-Fi Fit helps the final performance on the Visual Door task, and outperforms No Skew-Fit (alpha=0) as seen in the zoomed in version of the plot. In the challenging Visual Pusher task, that Skew-Fit consistently helps and halves the final distance. Similarly, in we observe that Skew-Fit consistently outperforms No Skew-fit on Visual Pickup. Note that alpha=-1 is not always the optimal setting for each environment, but performs strongly in each case in terms of final performance.
17

---

Under review as a conference paper at ICLR 2020
Method
NLL
MLE uniform (oracle)
20175.4
Skew-Fit on unbalanced
20175.9
MLE on unbalanced
20178.03
Table Despite training on unbalanced Visual Door dataset (see Figure of paper), the negative log likelihood (NLL) of Skew-Fit evaluated on a uniform dataset matches that of VAE trained on uniform dataset.
B.3 VARIANCE ABLATION
20
10
0
-10
-5
-2.5
-0.75
-0.5
-0.25
IS
Skew-Fit (uses SIR)
MLE (constant across alpha)
Figure 13: Gradient variance averaged across paramcters in last epoch of training VAEs. Values of a less than -1 are numerically unstable for importance sampling (IS), but for Skew-Fit.
measure the gradient variance of training a VAE on an unbalanced Visual Door image dataset with Skew-Fit Skew-Fit with importance sampling (IS)
Skew-Fit (labeled MLE). We construct the imbalanced dataset by rolling out a random policy in the environment and collecting the visual observations. Most of the images contained the door in closed position; in few, the door was opened. In Figure 13] we see that the gradient variance for Skew-Fit with IS catastrophically large for large values of a. In contrast, for Skew-Fit with SIR, which is what variance is relatively similar to that of MLE. Additionally we trained three VAE's, one with MLE on uniform dataset of valid door opening images, one with Skew-Fit the unbalanced dataset from above, and one with MLE on the same unbalanced dataset. As expected, the VAE that has access to the uniform dataset gets the lowest negative log likelihood score. This is the oracle method, since in practice we would only have access to imbalanced data. As shown in Table 1] Skew-Fit considerably outperforms MIE,, getting a much closer to oracle log likelihood score.
in practice, the
B.4 GOAL AND PERFORMANCE VISUALIZATION
We visualize the goals sampled from Skew-Fit well as those sampled when using the prior method, RIG (Nair et al. 2018). As shown in Figure 14 and Figure 15] the generative model pş results in much more diverse samples when trained with Fit We see in Figure 16 this results in policy that more consistently reaches the goal image.
C IMPLEMENTATION DETAILS
C.1 RIG WITH SKEW-FIT SUMMARY
Algorithm provides detailed pseudo-code for how we combined our method with RIG. Steps that were removed from the base RIG algorithm are highlighted in blue and steps that were added are highlighted in red. The main differences between the two are (1) sampling exploration goals from the buffer using Pakowed instead of the VAE prior, (2) relabeling with replay buffer goals sampled using Pikewed instead of from the VAE prior, and (3) training the VAE on replay buffer data data sampled using Pskewed instead of uniformly.
18

---

Under review as a conference paper at ICLR 2020
Skew-Fis FFit
RIG
Pickup
Pusher
Door
Figure 14: Proposed goals from VAE for RIG and with Skew on the Visual Pickup. Visual Pusher, and Visual Door environments. Standard RIG produces goals where the door is closed and the object and puck is in same position, while RIG + Skew-Fit proposes goals with varied puck positions, occasional object goals the air, and both open and closed door angles.
19

---

Under review as a conference paper at ICLR 2020
Figure 15:
Proposed goals from the VAE for RIG (left) and with RIG Skew-Fit (right) on the Real World Visual Door enviroument. Standard RIG produces goals where the door is closed while RIG Skew-Fi proposes goals with both open and closed door angles.
Pickup
Pusher
Door
MIN
4
Figure Example reached gnals by Skew-Fit RIG. first column of each environment section specifies the target goal while the second and third columns show reached goals by Skew-Fit and RIG. Both methods learn how to reach goals close to the initial position, but only Skew-Fi learns to reach the more difficult goals.
20

---

Under review as a conference paper at ICLR 2020
C.2 LIKELIHOOD ESTIMATION USING B- 3-VAE
We estimate the density under the VAE by using a sample-wise approximation to the marginal over estimated using importance sampling:
p(z)
Pa. (z) = Ex-g0 (=|2)
Pve (cr l2)
go, 20.(2/1)
N
p(2)
qo (2 Por (x
N
where go is the encoder, Pv the decoder, and p(2) is the prior, which this case unit Gaussian. We found that sampling N = 10 latents for estimating the density worked well in practice.
C.3 IMPLEMENTATION OF PRIOR WORK
We replaced TD3 (Fujimoto al. 2018) with soft actor critic (SAC) from Haarnoja et al.) (2018) for all the methods that use RIG, including Skew-Fit. This is in contrast to the original RIG Nair al. (2018) paper which used TD3 Fujimoto et al. (2018). We found that maximum entropy policies in general improved the performance of RIG, and that we did not need to add noise top of the stochastic policy's noise. For our RL network architectures and training scheme, we use fully connected networks for the policy, Q-function and value networks with two hidden layers of size 400 and 300 each. We also delay training any of these networks for 10000 time steps in order to collect sufficient data for the replay buffer well as to ensure the latent space of the VAE is relatively stable (since we train the VAE online in this setting). As in RIG, we train goal-conditioned value functions (Schaul et al. 2015) using hindsight experience replay Andrychowicz al. (2017). relabelling 50% of exploration goals as goals sampled from the VAE prior N 1) and 30% from future goals in the trajectory. In the prior RIG method, the VAE was pre-trained sampling of images from the state space of each environment. In order to ensure a fair comparison to Skew-Fit, we forego pre-training and instead train the VAE alongside RL., using the variant described in the RIG paper.
a uniform
C.4 VISION-BASS CONTINUOUS CONTROL EXPERIMENTS
In our experiments, we use an image size of 48x48. For our VAE architecture, use a modified version of the architecture used in the original RIG paper Nair et al. (2018). Our VAE has three convolutional layers with kernel sizes: 5x5, 3x3, and 3x3, number of output filters: 16, 32, and 64 and strides: 3, 2, and 2. We then have a fully connected layer with the latent dimension number of units, and then reverse the architecture with de-convolution layers. We vary the latent dimension of the VAE, the ,8 term of the VAE and the term for Skew-Fit based the environment. Additionally, we vary the training schedule of the VAE based on the environment. See the table at the end of the appendix for more details. Our VAE has Gaussian decoder with identity variance, meaning that train the decoder with a mean-squared error loss.
When training the VAE alongside RL, found the following two schedules to be effective for different environments:
1. For first 5K steps: Train VAE using standard MLE training every 500 time steps for 1000 batches. After that, train VAE using Skew-Fit every 500 time steps for 200 batches.
2. For first 5K steps: Train VAE using standard MLE training every 500 time steps for 1000 batches. For the next 45K steps, train VAE using Skew-Fit every 500 steps for 200 batches. After that, train VAE using Skew-Fit every 1000 time steps for 200 batches.
We found that initially training the VAE without Skew-Fit improved the stability of the algorithm. This is due to the fact that density estimates under the VAE are constantly changing and inaccurate during the early phases of training. Therefore, it made little sense to use those estimates to prioritize goals early on in training. Instead, we simply train using MLE training for the first 5K timesteps, and after that we perform Skew-Fit according to the VAE schedules above. Table 3 lists the hyper- parameters that were shared across the continuous control experiments. Table paremetasameters specific to each environment. Additionally, Appendix C.I shows the combined RIG Skew-Fit algorithm.
21

---

Under review as a conference paper at ICLR 2020
Hyper-pe parameee
Value
Algorithm
TD3 Fujimoto et al. 2018
# training batches per time step
Q network hidden sizes
400, 300
Policy network hidden sizes
400, 300
Q network and policy activation
ReLU
Exploration Noise
None
RL Batch Size
1024
Discount Factor
0.99
Path length
25
Reward Scaling
100
Number of steps per epoch
5000
Table Hyper-parameters used for 2D RI. experiment (Figure 9a
WWo expect similar performance had we used SAC.
Value
Comments
# training batches per time step
Marginal improvements alter 2
Exploration Noise
None (SAC policy stochastic)
Did not tune
Batch Size
1024
smaller batch sizes work as well
VAE Batch Size
Did not
Discount Factor
0.99
Did not mune
Reward Scaling
Did not tune
Path length
100
Did not tune
Replay Bulter Size
100000
Did not tune
Number Latents for Estimating Density (N)
Marginal improvements beyond
Table 3: General hyper- parameters used for all continuous control experiments.
Hyper-parameter
Visual Pusher
Visual Door
Visual Pickup
Real World Visual Door
Path Length
50
100
50
100
for 3-VAE
20
20
60
Latent Dimension Size
4
16
16
16
for Skiw-Fit
-1
1/2
-1
1 1/2
VAF Training Schedule
2
Sample Goals From
Pe
Pskewed
Pikowed
Pakowed
Table Environment specific hyper- parameters
22

---

Under review as a conference paper at ICLR 2020
Algorithm RIG and RIG Skew-Fit. Blue text denotes RIG specific steps and red text denotes RIG + Skew-Fit specific steps
Require: VAE encoder 9s- VAE decoder pv. policy
16:
Compute reward
-11== [||a'= Eyll
Ne, goal-conditioned value function Qw a, VAE Training Schedule.
17:
Minimize
Bellman
Error
using
(z, a, z', (g:r).
1: Collect D { using exploration policy.
18:
end for
Train 3-VAE on data uniformly sampled from D.
19:
for f
for i O.... H steps do ...,
3: Fit p(z) to latent encodings (s)
20:
steps do Sample future state sh, 1 h4 2 Store (st, (se,ar, 8141, c(sh.)) into R.
4: for 0, 0..... N episodes do
21:
5: Sample latent goal from prior 2g ~ p(2).
22:
6: Sample latent goal e(s') from (s, a, 8 2gy
23:
end for end for
R using pe if R. not empty. Otherwise, use
24:
Zy .~pple p(z).
25:
Construct skewed replay buffer distribution pe
7:
Sample initial state so ~
using data from R with Equation g if total_steps < 5000 then
8:
fo O..., - steps do
26:
9:
Get action a, no (e())))), zg)
27:
Fine-tune 3-VAF on data uniformly sampled
10:
Get next state sc+1 p(- St, at).
from R. according to VAE Training Schedule.
11:
Store De, S141, 2g) into replay buffer R.
28:
else
12:
Sample transition (s, a, s'. 2.z))
R.
29:
Fine-tune B-VAE on data uniformly sampled from R according to VAE Training Schedule.
13:
Encode z = c(s), 2 e((').
14:
(Probability 0.5) replace 2g with 2 p(z).
30: Fine-tune 3-VAE on data sampled from R.
15:
(Probability 0.5) replace 2g with e(s') where
using ps according to VAE Training Schedule. 31: end if
a, 2g) using Pe
32: end for
C.5 ORACLE 2D NAVIGATION EXPERIMENTS
We initialize the VAE to the middle of the environment for Maze, and the bottom left corner of the environment for Four Rooms. Both the encoder and decoder have 2 hidden layers with [400, 300] units, ReLU hidden activations, and no output activations. The VAE has latent dimension of 8 and Gaussian decoder trained with mean-squared error loss, batch size of 256, and 1000 batches at each iteration. The VAE is trained on the exploration data buffer every 1000 rollouts.
D ENVIRONMENT DETAILS
In this environment, agent must learn navigate square-shaped corridor (see Figure B). The observation is the 2D position, and the agent must specify a velocity as the 2D action. The reward each time step is the negative distance between the achieved position and desired position.
Point-Mass:
Maze: A 20 20 2D pointmass environment in the shape of maze. The observation is the 2D position of the agent, and the agent must specify a target 2D position as the action. The dynamics of the environment are the following: first, the agent is teleported to the target position, specified by the action. Then a gaussian change in position with mean 0 and standard deviation 0.1 is then applied. If the action would result in the agent moving through or into a wall, then the agent will be stopped at the wall instead.
Four Rooms: A 20 x 20 2D pointmass environment in the shape of four rooms (Sutton et al. 1999). The observation space, actions space, and environment dynamics are the same as the Maze environment above.
Ant: A MuJoCo ant environment with the same corridor as the Point-Mass environment. The observation is position, orientation, joint angles, and velocity of the joint angles of the ant. The observation space is 29 dimensions. The agent controls the ant through the joints, which is dimensions. The goal a target 2D position, and the reward is the negative Euclidean distance between the achieved 2D position and target 2D position.
Visual Pusher: MuJoCo environment with a 7-DoF Sawyer arm and small puck on a table that the arm must push to a target position. The agent controls the arm by commanding æ, y position for the end effector (EE). The underlying state is the EE position, e and puck position p. The evaluation metric is the distance between the goal and final puck positions. The hand goal/state space is a 10x10
23

---

Under review as a conference paper at ICLR 2020
em2 box and the puck goal/state space is 30x20 em2 box. Both the hand and puck spaces centered around the origin. The action space ranges in the interval [-1, 1] in the x and y dimensions.
Visual Door: A MuJoCo environment with 7-DoF Sawyer and a door on a table that the arm must pull open to a target angle. Control is the same as in Visual Pusher. The evaluation metric is the distance between the goal and final door angle, measured in radians. In this environment, we do not reset the position of the hand or door at the end of each trajectory. The state/goal space is a 5x20x 15 em² box in the x, 3/ dimension respectively for the arm and an angle between [0,.83] radians. The action space ranges in the interval [-1, 1] in the x, y and dimensions.
Visual Pickup: A MuloCo environment with the same robot Visual Pusher, but now with different object. The object is cube-shaped, but larger intangible sphere is overlaid on top so that it is casier for the agent to see. Moreover, the robot is constrained to move in 2 dimension: it only controls the y. arm positions. The x position of both the arm and the object is fixed. The evaluation metric is the distance between the goal and final object position. For the purpose of evaluation, 75% of the goals have the object in the air and 25% have the object on the ground. The state/goal space for both the object and the arm 10cm in the y dimension and I 13mm the z dimension. The action space ranges in the interval [-1, 1] in the y and dimensions.
Real World Visual Door: A Rethink Sawyer Robot with a door on a table. The arm must pull the door open to target angle. The agent controls the arm by commanding the x, y, velocity of the EE. Our controller commands actions at a rate of up to 10Hz with the scale of actions ranging up to Icm in magnitude. The underlying state and goal is the same as in Visual Door. Again we do not reset the position of the hand or door at the end of each trajectory. We obtain images using a Kinect Sensor. The state/goal space for the environment is a 10x 1xx10x10 10 em box. The action space ranges in the interval [- (in cm) in the x. y and dimensions. The door angle lies in the range [0, 45] degrees.
E GOAL-CONDITIONED REINFORCEMENT LEARNING MINIMIZES H(G S)
Some goal-conditioned RL methods such as Warde-Farley et al. 2018): ; Nair et al. 2018 present
methods for minimizing a lower bound for HG S) by approximating log p( G S) and using
as the reward. Other coal-conditioned RL methods (Kaclbling) 1993 Lillicrap et al. 2016 Schaul
al. 2015 Andrychowicz al., 2017 Pong et al. 2018] et al. (018a) are not developed with the intention of minimizing the conditional entropy H(G S). Nevertheless, one can see that goal-conditioned RL generally minimizes H(G S) by noting that the optimal goal-conditioned policy will deterministically reach the goal. The corresponding conditional entropy of the goal given the state, (G S), would be zero, since given the current state, there would be no uncertainty over the goal (the goal must have been the current state since the policy is optimal). So, the objective of coal-conditioned RL can be interpreted as finding a policy such that H(G S) = O. Since zero is the minimum value of H(G S), then goal- coal-conditioned RL can be interpreted minimizing H(G S).
24


Reviews:
[{"review_id": "SJxzk8sitS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces SKEW-FIT, an exploration approach that maximizes the entropy of a distribution of goals such that the agent maximizes state coverage. \n\nThe paper is well-written and provides an interesting combination of reinforcement learning with imagined goals (RIG) and entropy maximization. The approach is well motivated and simulations are performed on several simulated and real robotics tasks.\n\nSome elements were unclear to me:\n- \"We also assume that the entropy of the resulting state distribution H(p(S | pφ)) is no less than the entropy of the goal distribution H(pφ(S)). Without this assumption, a policy could ignore the goal and stay in a single state, no matter how diverse and realistic the goals are.\" How do you ensure this in practice?\n- In the second paragraph of 2.2, it is written \"Note that this assumption does not require that the entropy of p(S | pφ) is strictly larger than the entropy of the goal distribution, pφ.\" Could you please clarify?\n\n\nThe experiments are interesting, yet some interpretations might be too strong (see below):\n- In the first experiment, \"Does Skew-Fit Maximize Entropy?\", it is empirically illustrated that the method does result in a high-entropy state exploration. However, it is only compared to one very naive way of exploring and it is not discussed whether other techniques also achieve the same entropy maximization. The last sentences seems to imply that only this technique ends up optimizing the entropy of the state coverage, while I believe that the claim (given the experiment) should only be about the fact it does so faster.\n- On the comments of Figure 6, the paper mentions that \"The other methods only rely on the randomness of the initial policy to occasionally pick up the object, resulting in a near-constant rate of object lifts.\" I'm unsure about the interpretation of this sentence given Figure 6 because other methods do not seem to fail entirely when given enough time.\n- In the experiment \"Real-World Vision-Based Robotic Manipulation\", It is written that \"a near-perfect success rate [is reached] after five and a half hours of interaction time\", while on the plot it is written 60% cumulative success after 5.5 hours and it is thus not clear where this \"5.5 hours\" comes from.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyeleOROFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary : \u000b\n\nThe paper proposes an exploratory objective that can maximize state coverage in RL. They show that a formal objective for maximizing state coverage is equivalent to maximizing the entropy of a goal distribution. The core idea is to propose a method to maximize entropy of a goal distribution, or a state distribution since goals are full states. They show that the proposed method to maximize the state or goal distribution can lead to diverse exploration behaviour sufficient for solving complex image based manipulation tasks. \n\n\nComments and Questions : \n\n\t- The core idea is to maximize the entropy of the state visitation frequency H(s). It is not clear from the paper whether the authors talk about the normalized discounted weighting of states (a distribution) or the stationary distribution? The entropy of the state visitation distribution only deals with valid states - but I am not sure what it means to maximize the entropy of this term exactly in terms of exploration, since it is neither the discounted weighting of states or the stationary distribution for an infinite horizon task? \n\t- The authors do mention that maximizing the entropy of H(s) is not sufficient - so instead suggests for maxmizing entropy of H(s|g). But why is this even sufficient for exploration - if I do not consider new tasks at test time but only the training task? How is this a sufficient exploration objective? Furthermore, since it is the conditional entropy given goal states, the fundamental idea of this is not clear from the paper. \n\t- Overall, I am not convinced that an objective based on H(s|g) is equivalent to an maximizing H(s), and why is this even a good objective for exploration? The meaning of H(s) to me is a bit vague from the text (due to reasons above) and therefore H(s|g) does not convince to be a good exploration objective either?\n\t- The paper then talks about the MI(S;G) to be maximized for exploration - what does this MI formally mean? I understand the breakdown from equation 1, but why is this a sufficient exploration objective? There are multiple ideas introduced at the same time - the MI(s;g) and talking about test time and training time exploration - but the idea itself is not convincing for a sufficient exploration objective. In light of this, I am not sure whether the core idea of the paper is convincing enough to me. \n\t- I think the paper needs more theoretical insights and details to show why this form of objective based on the MI(s;g) is good enough for exploration. Theoretically, there are a lot of details missing from the paper, and the paper simply proposes the idea of MI(s;g) and talks about formal or computationally tractable ways of computing this term. While the proposed solutuon to compute MI(s;g) seems reasonable, I don't think there is enough contribution or details as to why is maximizing H(s) good for exploration in the first place.\n\t- Experimentally, few tasks are proposed comparing skew-fit with other baselines like HER and AutoGoal GAN - but the differences in all the results seem negligible (example : Figure 5). \n\t- I am not sure why the discussion of goal conditioned policies is introduced rightaway. To me, a more convincing approach would have been to first discuss why H(s) and the entropy of this is good for exploration (discounted weighting or stationary state distribution and considering episodic and  infinite horizon tasks). If H(s) is indeed a difficult or not sufficient term to maximize the entropy for, then it might make sense to introduce goal conditioned policies? Following then, it might be convincing to discuss why goal conditioned policies are indeed required, and then tractable ways of computing MI(s;g). \n\t- Experimentally, I think the paper needs significantly more work - especially considering hard exploration tasks (it might be simple setups too like mazes to begin with), and then to propose a set of new experimental results, without jumping directly to image based tasks as discussed here and then comparing to all the goal conditioned policy baselines. \n\nOverall, I would recommend to reject this paper, as I am not convinced by the proposed solution, and there are lot of theoretical details missing from the paper. It skips a lot of theoretical insights required to propose a new exploration based objective, and the paper proposes a very specific solution for a set a very specific set of experimental setups. \n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BylJ3YZntB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduced a very interesting idea to facilitate exploration in goal-conditioned reinforcement learning. The key idea is to learn a generative model of goal distribution to match the weighted empirical distribution, where the rare states receive larger weights. This encourages the model to generate more diverse and novel goals for goal-conditioned RL policies to reach.\n\nPros:\nThe Skew-Fit exploration technique is independent of the goal-conditioned reinforcement learning algorithm and can be plugged in with any goal-conditioned methods. The experiments offer a comparison to several prior exploration techniques and demonstrate a clear advantage of the proposed Skew-Fit method. It is evaluated in a variety of continuous control tasks in simulation and a door opening task on a real robot. A formal analysis of the algorithm is provided under certain assumptions.\n\nCons:\nThe weakest part of this work is the task setup. The method has only been evaluated on simplistic short-horizon control tasks. It’d be interesting to see how this method is applied to longer-horizon multi-stage control tasks, where exploration is a more severe challenge. It is especially when the agent has no access to task reward and only explores the environment to maximize state coverage. It is unclear to me how many constraints are enforced in the task design in order for the robot to actually complete the full tasks through such exploration.\n\nI would also like to see how Skew-Fit works with different goal-conditioned RL algorithms, and how the performances of the RL policy in reaching the goals would affect the effectiveness of this method in exploring a larger set of states.\n\nSection E: it seems that there’s a logic jump before the conclusion “goal-conditioned RL methods effectively minimize H(G|S)”. More elaboration on this point is necessary.\n\nMinor:\nAppendix has several broken references.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the problem of exploration in RL. In order to maximize coverage of the state space, the authors introduce an approach where the agent attempts to reach some self-set goals. The empirically show that agents using this method uniformly visit all valid states under certain conditions. They also show that these agents are able to learn behaviours without providing a manually-defined reward function.

The drawback of this work is the combined lack of theoretical justification and limited (marginal) algorithmic novelty given other existing goal-directed techniques. Although they highlight the performance of the proposed approach, the current experiments do not convey a good enough understanding of why this approach works where other existing goal-directed techniques do not, which would be expected from a purely empirical paper. This dampers the contribution, hence I recommend to reject this paper.
Decision: Reject