SKEW-FIT: STATE-COVERING SELF-SUPERVISED REINFORCEMENT LEARNING
Year: 2019

Abstract:
Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manuallydesigned reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing the entropy of the goal distribution together with goal reaching performance, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. Skew-Fit enables self-supervised agents to autonomously choose and practice reaching diverse goals. We show that, under certain regularity conditions, our method converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that it can learn a variety of manipulation tasks from images, including opening a door with a real robot, entirely from scratch and without any manually-designed reward function.

text: 1 INTRODUCTION
Reinforcement learning (RL) provides an appealing formalism for automated learning of behavioral skills, but separately learning every potentially useful skill becomes prohibitively time consuming, both in terms of the experience required for the agent and the effort required for the user to design reward functions for each behavior. What if we could instead design an unsupervised RL algorithm that automatically explores the environment and iteratively distills this experience into general-purpose policies that can accomplish new user-specified tasks at test time? For an agent to learn autonomously, it needs an explo-
ration objective. In the absence of any prior knowledge about which states are more useful, an effective exploration scheme is one that visits as many states as possible, allowing a policy to autonomously prepare for user-specified task that it might see at test time. This objective has been formalized as maximizing the entropy of the learned policy’s visited state distribution 1 H(S) (Hazan et al., 2018a), since a policy that maximizes this objective should approach a uniform distribution over valid states. Unfortunately, directly optimizingH(S) requires an accurate model of the policy and environment (Hazan et al., 2018a). Moreover, even if this optimization were tractable, another short-coming of this objective is that the resulting policy cannot be used to solve new tasks: it only knows how to maximize state entropy. In other words, to develop principled unsupervised RL algorithms that result in useful policies, maximizingH(S) is not enough. We need a mechanism that allows us to control the resulting policy to achieve new tasks at test-time.
1We consider the distribution over terminal states in a finite horizon task and believe this work can be extended to infinite horizon stationary distributions.
We argue that this can be accomplished by performing goal-directed exploration. In addition to maximizing the state entropy, we should be able to control where the policy goes by giving it a goal G that corresponds to a state that it must reach. Mathematically, a goal-conditioned policy should minimize the conditional entropy over the states given a goal,H(S | G). This objective provides us with a principled way for training a policy to explore all states, by maximizingH(S), such that the state that is reached can be controlled by commanding goals, which means minimizingH(S | G). Directly using this objective is often intractable, since it requires optimizing the entropy of the marginal state distribution of the policy,H(S). However, we can sidestep this issue by noting that the objective is the mutual information between the state and the goal, I(S;G), which can be written as:
H(S)−H(S|G) = I(S;G) = H(G)−H(G|S). (1)
Equation 1 thus gives an equivalent objective for an unsupervised RL algorithm: the agent should set diverse goals, maximizingH(G), and learn how to reach them, minimizingH(G | S). While the second term is the typical objective studied in goal-conditioned RL (Kaelbling, 1993; Andrychowicz et al., 2017), maximizing the diversity of goals is crucial for effectively learning to reach all possible states. In a new environment, acquiring such a maximum-entropy goal distribution is challenging: how can an agent set diverse goals when it does not even know what states exist?
In this paper, we address this question via a new algorithm, Skew-Fit, which learns to model the uniform distribution over states, given only access to data collected by an autonomous goalconditioned policy. Our paper makes the following contributions. First, we propose a principled objective for unsupervised RL, based on Equation 1. While a number of prior works ignore theH(G) term, we argue that jointly optimizing the entire quantity is needed to develop effective and useful exploration. Second, we propose a method called Skew-Fit and prove that, under some regularity conditions, it learns a generative model that converges to a uniform distribution over the goal space, even when the set of valid states is unknown (e.g., as in the case of images). Third, we empirically demonstrate that, when combined with goal-conditioned RL, Skew-Fit allows us to autonomously train goal-conditioned policies that reach diverse states. We test this method on a variety of simulated vision-based robot tasks without any task-specific reward function. In these experiments, Skew-Fit reaches substantially better final performance than prior methods, and learns much more quickly. We also demonstrate that our approach solves a real-world manipulation task, which requires a robot to learn to open a door from scratch in about five hours, directly from images, and without any manually-designed reward function.

heading: 2 PROBLEM FORMULATION
text: To ensure that an unsupervised reinforcement learning agent learns to reach all possible states in a controllable way, we maximize the mutual information between the state S and the goal G, I(S;G), as stated in Equation 1. This section discusses how to optimize Equation 1 by splitting the optimization into two parts: minimizingH(G | S) and maximizingH(G).
2.1 MINIMIZING H(G | S): GOAL-CONDITIONED REINFORCEMENT LEARNING
Standard RL considers a Markov decision process (MDP), which has a state space S , action space A, and unknown dynamics ρ(st+1 | st,at) : S×S×A 7→ [0,+∞). Goal-conditioned RL also includes a goal space G. For simplicity, we will assume in our derivation that the goal space matches the state space, such that G = S , though the approach extends trivially to the case where G is a hand-specified subset of S , such as the global x-y position of a robot. A goal-conditioned policy π(a | s,g) maps a state s ∈ S and goal g ∈ S to a distribution over actions a ∈ A, and its objective is to reach the goal, i.e., to make the current state equal to the goal.
Goal-reaching can be formulated as minimizing H(G | S), and many practical goal-reaching algorithms (Kaelbling, 1993; Lillicrap et al., 2016; Schaul et al., 2015; Andrychowicz et al., 2017; Nair et al., 2018; Pong et al., 2018; Florensa et al., 2018a) can be viewed as approximations to this objective by observing that the optimal goal-conditioned policy will deterministically reach the goal, resulting in a conditional entropy of zero: H(G | S) = 0. See Appendix E for more details. Our method may thus be used in conjunction with any of these prior goal-conditioned RL methods in order to jointly minimizeH(G | S) and maximizeH(G).
2.2 MAXIMIZING H(G): SETTING DIVERSE GOALS
We now turn to the problem of setting diverse goals or, mathematically, maximizing the entropy of the goal distributionH(G). Let US be the uniform distribution over S , where we assume S has finite volume so that the uniform distribution is well-defined. Let pφ be the goal distribution from which goals G are sampled. Our goal is to maximize the entropy of pφ, which we write asH(G). Since the maximum entropy distribution over S is the uniform distribution US , maximizingH(G) may seem as simple as choosing the uniform distribution to be our goal distribution: pφ = US . However, this requires knowing the uniform distribution over valid states, which may be difficult to obtain when S is a subset of Rn, for some n. For example, if the states correspond to images viewed through a robot’s camera, S corresponds to the (unknown) set of valid images of the robot’s environment, while Rn corresponds to all possible arrays of pixel values of a particular size. In such environments, sampling from the uniform distribution Rn is unlikely to correspond to a valid image of the real world. Sampling uniformly from S would require knowing the set of all possible valid images, which we assume the agent does not know when starting to explore the environment.
While we cannot sample arbitrary states from S, we can sample states by performing goal-directed exploration. To derive and analyze our method, we introduce a simple model of this process: a goal G ∼ pφ is sampled from the goal distribution pφ, and then the agent attempts to achieve this goal, which results in a distribution of states S ∈ S seen along the trajectory. We abstract this entire process by writing the resulting marginal distribution over S as p(S | pφ). We assume that p(S | pφ) has full support, which can be accomplished with an epsilon-greedy goal reaching policy in a communicating MDP. We also assume that the entropy of the resulting state distribution H(p(S | pφ)) is no less than the entropy of the goal distributionH(pφ(S)). Without this assumption, a policy could ignore the goal and stay in a single state, no matter how diverse and realistic the goals are. Note that this assumption does not require that the entropy of p(S | pφ) is strictly larger than the entropy of the goal distribution, pφ. This simplified model allows us to analyze the behavior of our goal-setting scheme separately from any specific goal-reaching algorithm. We will however show in Section 6 that we can instantiate this approach into a practical algorithm that jointly learns the goal-reaching policy. In summary, our goal is to acquire a maximum-entropy goal distribution pφ over valid states S, while only having access to state samples from p(S | pφ).

heading: 3 SKEW-FIT: LEARNING A MAXIMUM ENTROPY GOAL DISTRIBUTION
text: Our method, Skew-Fit, learns a maximum entropy goal distribution pφ using samples collected from a goal-conditioned policy. We analyze the algorithm and show that Skew-Fit maximizes the entropy of the goal distribution, and present a practical instantiation for unsupervised deep RL.

heading: 3.1 SKEW-FIT ALGORITHM
text: To learn a uniform distribution over valid goal states, we present a method that iteratively increases the entropy of a generative model pφ. In particular, given a generative model pφt at iteration t, we would like to train a new generative model pφt+1 such that pφt+1 has higher entropy than pφt over the set of valid states. While we do not know the set of valid states S, we can sample states from p(S | pφt), resulting in an empirical distribution pempt over the states
pempt(s) , 1
N N∑ n=1 1{s = Sn}, Sn ∼ p(S | pφt), (2)
and use this empirical distribution to train the next generative model pφt+1 . However, if we simply train pφt+1 to model this empirical distribution, it may not necessarily have higher entropy than pφt .
The intuition behind our method is quite simple: rather than fitting a generative model to our empirical distribution, we skew the empirical distribution so that rarely visited states are given more weight. See Figure 2 for a visualization of this process. How should we skew the empirical distribution if we want to maximize the entropy of pφt+1? If we had access to the density of each state, pempt(S), then we could simply weight each state by 1/pempt(S). We could then perform maximum likelihood
estimation (MLE) for the uniform distribution by using the following loss to train φt+1:
L(φ) = ES∼US [log pφ(S)] = ES∼pempt
[ US(S)
pempt(S) log pφ(S)
] ∝ ES∼pempt [ 1
pempt(S) log pφ(S) ] where we use the fact that the uniform distribution US(S) has constant density for all states in S. However, computing this density pempt(S) requires marginalizing out the MDP dynamics, which requires an accurate model of both the dynamics and the goal-conditioned policy.
We avoid needing to model the entire MDP process by approximating pempt(S) with our previous learned generative model: pempt(S) ≈ p(S | pφt) ≈ pφt(S). We therefore weight each state by the following weight function
wt,α(S) , pφt(S) α, α < 0. (3)
where α is a hyperparameter that controls how heavily we weight each state. If our approximation pφt was exact, we could choose α = −1 and recover the exact importance sampling procedure described above. If α = 0, then this skew step has no effect. By choosing intermediate values of α, we can trade off the reliability of our estimate pφt(S) with the speed at which we want to increase the entropy of the goal distribution.
Variance Reduction As described, this procedure relies on importance sampling (IS), which can have high variance, particularly if pφt(S) ≈ 0. We therefore choose a class of generative models where the probabilities are prevented from collapsing to zero, as we will describe in Section 4. To further reduce the variance, we train pφt+1 with sampling importance resampling (SIR) (Rubin, 1988). Rather than sampling from pempt and weighting the update from each sample by wt,α, SIR explicitly defines a skewed distribution as
pskewedt(s) , 1
Zα pempt(s)wt,α(s), Zα = N∑ n=1 pempt(Sn)wt,α(Sn), (4)
where Zα is the normalizing coefficient and pempt is given by Equation 2. We note that computing Zα adds little computational overhead, since all of the weights already need to be computed. We then fit the generative model at the next iteration pφt+1 to pskewedt using standard MLE. We found that using SIR resulted in significantly lower variance than IS. See Appendix B.3 for this comparision.
Goal Sampling Alternative Because pφt+1 ≈ pskewedt , at iteration t + 1, one can sample goals from either pφt+1 or pskewedt . Sampling goals from pskewedt may be preferred if sampling from the learned generative model pφt+1 is computationally or otherwise challenging. In either case, one still needs to train the generative model pφt to create pskewedt . In our experiments, we found that both methods perform well.
Summary Overall, Skew-Fit samples data from the environment and weights different samples by their density under the generative model pφt. We prove in the next section conditions under which this weighting makes the generative model at the next iteration pφt+1 have higher entropy. With higher entropy, the pφt+1 is more likely to generate goals at the frontier of unseen states, which results in more uniform state coverage. Skew-Fit is shown in Figure 2 and summarized in Algorithm 1.
Algorithm 1 Skew-Fit 1: for Iteration t = 1, 2, ... do 2: Collect N states {Si}Ni=1 by sampling goals from pφt (or pskewedt) and running goal-
conditioned policy. 3: Construct skewed distribution pskewedt (Equation 3 and Equation 4). 4: Fit pφt+1 to skewed distribution pskewedt using MLE. 5: end for

heading: 3.2 SKEW-FIT ANALYSIS
text: In this section, we provide conditions under which pφt converges in distribution to the uniform distribution over the state space S. To make this analysis possible, we consider the case where N → ∞, which allows us to study the limit behavior of the goal distribution pskewedt . Our most general result is stated as follows:
Lemma 3.1. Let S be a compact set. Define the set of distributionsQ = {p : support of p is S}. Let F : Q 7→ Q be a continuous function and such thatH(F(p)) ≥ H(p) with equality if and only if p is the uniform probability distribution on S , US . Define the sequence of distributions P = (p1, p2, . . . ) by starting with any p1 ∈ Q and recursively defining pt+1 = F(pt).

heading: The sequence P converges to US .
text: Proof. See Appendix Section E.
We will apply Lemma 3.1 to be the map from pskewedt to pskewedt+1 to show that pskewedt converges to US . If we assume that the goal-conditioned policy and generative model learning procedure are well behaved ( i.e., the maps from pφt(S) to pempt and from pskewedt to pφt+1 are continuous ), then to apply Lemma 3.1, we only need to show that H(pskewedt) ≥ H(pempt) with equality if and only if pempt = US . For the simple case when pφt = pempt identically at each iteration, we prove the convergence of Skew-Fit true for any value of α ∈ [−1, 0) in Appendix A.3. However, in practice, pφt only approximates pempt . To address this more realistic situation, we prove the following result:
Lemma 3.2. Given two distribution pempt and pφt where pempt pφt 2 and
CovS∼pempt [ log pempt(S), log pφt(S) ] > 0, (5)
define the distribution pskewedt as in Equation 4. Let Hα(α) be the entropy of pskewedt for a fixed α. Then there exists a constant a < 0 such that for all α ∈ [a, 0),
H(pskewedt) = Hα(α) > H(pempt).
Proof. See Appendix Section E.
Thus, our generative model pφt does not need to exactly fit the empirical distribution. We merely need for the log densities of pφt and pempt to be correlated, which we expect to happen frequently with an accurate goal-conditioned policy, since pempt is the set of states seen when trying to reach goals from pφt . In this case, if we choose negative values of α that are small enough, then the entropy of pskewedt will be higher than that of pempt . Empirically, we found that α values as low as α = −1 performed well.
In summary, we see that under certain assumptions, pskewedt converges to US . Since we train each generative model pφt+1 by fitting it to pskewedt , we expect pφt to also converge to US .
2 p q means that p is absolutely continuous with respect to q, i.e. p(s) = 0 =⇒ q(s) = 0.

heading: 4 TRAINING GOAL-CONDITIONED POLICIES WITH SKEW-FIT
text: Thus far, we have presented and derived Skew-Fit assuming that we have access to a goal-reaching policy, allowing us to separately analyze how we can maximize H(G). However, in practice we do not have access to such a policy, and in this section we discuss how we concurrently train a goal-reaching policy.
Maximizing I(S;G) can be done by simultaneously performing Skew-Fit and training a goal conditioned policy to minimize H(G | S), or, equivalently, maximize −H(G | S). Maximizing −H(G | S) requires computing the density log p(G | S), which may be difficult to compute without strong modeling assumptions. However, for any distribution q, the following lower bound for −H(G | S) holds:
−H(G | S) = E(G,S)∼pφt ,π [log q(G | S)] +DKL(p | q) ≥ E(G,S)∼pφt ,π [log q(G | S)] ,
where DKL denotes Kullback–Leibler divergence as discussed by Barber & Agakov (2004). Thus, to minimizeH(G | S), we train a policy to maximize the following reward:
r(S,G) = log q(G | S).
For the RL algorithm, we use reinforcement learning with imagined goals (RIG) (Nair et al., 2018), though in principle any goal-conditioned method could be used. RIG is an efficient off-policy goalconditioned method that solves the vision-based RL problem in a learned latent space. In particular, RIG fits a β-VAE and uses it to encode all observations and goals into a latent space, which it uses as the state representation. RIG also uses the β-VAE to compute rewards, log q(G | S). Unlike RIG, we use the goal distribution from Skew-Fit to sample goals, both for exploration and for relabeling goals during training (Andrychowicz et al., 2017). Since RIG already trains a generative model over states, we reuse this β-VAE for the generative model pφ of Skew-Fit. To make the most use of the data, pφ is trained on all visited state rather than only the terminal states, which we found to work well in practice. In other words, our method uses the likelihood estimates from the β-VAE to choose the probability of sampling each state in Equation 3. To prevent these probabilities from collapsing to zero, we model the posterior of the β-VAE as a multivariate Gaussian distribution with a fixed variance and only learn the mean. We include a detailed summary of RIG and description our how we combine Skew-Fit and RIG in Appendix C.1.

heading: 5 RELATED WORK
text: Many prior methods for training goal-conditioned policies assume that a goal distribution is available to sample from during exploration (Kaelbling, 1993; Schaul et al., 2015; Andrychowicz et al., 2017; Pong et al., 2018). Other methods use data collected from a randomly initialized policy or heuristics based on data collected online to design a non-parametric (Colas et al., 2018b; Warde-Farley et al., 2018; Florensa et al., 2018a; Zhao & Tresp, 2019) or parametric (Péré et al., 2018; Nair et al., 2018) goal distribution. We remark that Warde-Farley et al. (2018) also motivate their work in terms of minimizing a lower bound forH(G | S). Our work is complementary to these goal-reaching methods: rather than focusing on how to train goal-reaching policies, we propose a principled method for maximizing the entropy of a goal sampling distribution,H(G). Our method learns without any task rewards, directly acquiring a policy that can be reused to reach user-specified goals. This stands in contrast to exploration methods that give bonus rewards based on state visitation frequency (Bellemare et al., 2016; Ostrovski et al., 2017; Tang et al., 2017; Savinov et al., 2018; Chentanez et al., 2005; Lopes et al., 2012; Stadie et al., 2016; Pathak et al., 2017; Burda et al., 2018; 2019; Mohamed & Rezende, 2015; Tang et al., 2017; Fu et al., 2017). While these methods can also be used without a task reward, they provide no mechanism for distilling the knowledge gained from visiting diverse states into flexible policies that can be applied to accomplish new goals at test-time: their policies visit novel states, and they quickly forget about them as other states become more novel.
Other prior methods extract reusable skills in the form of latent-variable-conditioned policies, where latent variables can be interpreted as options (Sutton et al., 1999) or abstract skills (Hausman et al., 2018; Gupta et al., 2018b; Eysenbach et al., 2019; Gupta et al., 2018a; Florensa et al., 2017). The
resulting skills may be diverse, but they have no grounded interpretation, while our method can be used immediately after unsupervised training to reach diverse user-specified goals.
Some prior methods propose to choose goals based on heuristics such as learning progress (Baranes & Oudeyer, 2012; Veeriah et al., 2018; Colas et al., 2018a), how off-policy the goal is (Nachum et al., 2018), level of difficulty (Florensa et al., 2018b) or likelihood ranking (Zhao & Tresp, 2019). In contrast, our approach provides a principled framework for optimizing a concrete and well-motivated exploration objective, and can be shown to maximize this objective under regularity assumptions. The work of Hazan et al. (2018b) also provably optimizes a well-motivated exploration objective, but is limited to tabular MDPs, while Skew-Fit is able to handle high dimensional settings such as vision-based continuous control.

heading: 6 EXPERIMENTS
text: Our experiments study the following questions: (1) Does Skew-Fit empirically result in a goal distribution with increasing entropy? (2) In image-based domains, how does Skew-Fit compare to prior work on choosing goals for goal-conditioned RL? (3) Can Skew-Fit be applied to a real-world, vision-based robot task?
Does Skew-Fit Maximize Entropy? To see the effects of Skew-Fit on goal distribution entropy in isolation of learning a goal-reaching policy, we begin by studying an idealized example where the policy is a near-perfect goal-reaching policy. The MDP is defined on a 2-by-2 unit square-shaped corridor (see Figure 3). At the beginning of an episode, the agent begins in the bottom-left corner and samples a goal from the goal distribution pφt . To simulate the stochasticity of the policy and environment, we add a Gaussian noise with standard deviation of 0.05 to this goal. The policy reaches the state that is closest to this noisy goal and inside the corridor, giving us a state S to add to our empirical distribution. We compare Skew-Fit to sampling uniformly from the replay buffer (labeled MLE). The β-VAE hyperparameters used to train pφt are given in Appendix C.5. As seen in Figure 3,
naively using previous experience to set goals results in a policy that primarily sets goal near the initial state distribution and only relies on the stochasticity of the policy and environment to explore. In contrast, Skew-Fit results in quickly learning a high entropy, near-uniform distribution over the state space.
Vision-Based Continuous Control Tasks We now evaluate Skew-Fit on a variety of continuous control tasks, where the policy must control a robot arm using only image observations, without access to any ground truth reward signal. We test our method on three different simulated continuous control tasks released by the authors of RIG (Nair et al., 2018): Visual Door, Visual Pusher, and Visual Pickup. To our knowledge, these are the only goal-conditioned, vision-based continuous control environments that are publicly available and used in experimental evaluations in prior work, making them a good point of comparison. See Figure 4 for visuals and Appendix C for details of these environments. The policies are trained in a completely unsupervised manner, without access to any prior information about the state-space or any pre-defined goal-sampling distribution. To evaluate their performance, we sample goal images from a uniform distribution over valid states and report the agent’s final distance to the corresponding simulator states (e.g., distance of the object
to the target object location), but the agent never has access to this true uniform distribution nor the ground-truth state information during training. While this evaluation method and metric is only practical in simulation, it provides us with a quantitative measure of a policy’s ability to reach a broad coverage of goals in a vision-based setting.
We use these domains to compare Skew-Fit to a number of existing methods on goal-sampling. We compare to Warde-Farley et al. (2018), a vision-based method which uses a non-parametric approach based on clustering to sample goals and an image discriminator to compute rewards. We denote this method as DISCERN. The other methods that we compare to were developed in non-vision, statebased environments. To ensure a fair comparison across methods, we combine these prior methods with a policy trained using RIG. First, we compare to RIG without Skew-Fit. We also compared to RIG using the relabeling scheme described in the hindsight experience replay (labeled HER). We compare to curiosity-driven prioritization (Ranked-Based Priority) (Zhao & Tresp, 2019), a variant of HER that samples goals for relabeling based on their ranked likelihoods. Florensa et al. (2018b) samples goals from a GAN based on the difficulty of reaching the goal. We compare against this method by replacing pφ with the GAN and label it AutoGoal GAN. We also separately compare to the goal proposal mechanism proposed by Warde-Farley et al. (2018) and otherwise train the policy with RIG, which we label DISCERN-g. Lastly, to demonstrate the difficulty of the exploration challenge in these domains, we compare to # Exploration (Tang et al., 2017), an exploration method that assigns bonus rewards based on the novelty of new states. Implementation details of the prior methods is given in Appendix C.3.
We see in Figure 5 that Skew-Fit significantly outperforms prior methods both in terms of task performance and sample complexity. The most common failure mode for prior methods is that the goal distributions collapse, resulting in the agent learning to reach only a fraction of the state space, as shown in Figure 1. For comparison, additional samples of pφ when trained with and without Skew-Fit are shown in Appendix B.4. Those images show that without Skew − Fit, pφ produces a small, non-diverse distribution for each environment: the object is in the same place for pickup, the puck is often in the starting position for pushing, and the door is always closed. In contrast, Skew-Fit proposes goals where the object is in the air and on the ground, where the puck positions are varied, and the door angle changes.
The direct effect of these goal choices can be seen by visualizing more example rollouts for RIG and Skew-Fit. Due to space constraints, these visuals are in Figure 16 in Appendix B.4. The figure shows that standard RIG only learns to reach states close to the initial position, while Skew-Fit learns to reach the entire state space. A quantitative comparison of the various methods on the pickup task can be seen in Figure 6, which gives the cumulative total exploration pickups for each method. From the graph, we can see that only Skew-Fit learns to pay attention to the object and therefore consistently increases the rate at which the policy picks up the object during exploration. In contrast, the other methods have near constant slopes past 40k steps, meaning that they do not continue to learning, and many methods have a near-constant rate of object lifts throughout all of training.
Real-World Vision-Based Robotic Manipulation We also demonstrate that Skew-Fit scales well to the real world with a door opening task, Real World Visual Door. See Figure 4 for a picture of this environment. While a number of prior works have studied RL-based learning of door opening Kalakrishnan et al. (2011); Chebotar et al. (2017), we demonstrate the first method for autonomous learning of door opening without a user-provided, task-specific reward function. As in simulation, we do not provide any goals to the agent and simply let it interact with the door to solve the door opening task from scratch, without any human guidance or reward signal. We train two agents using Skew-Fit with RIG and RIG alone. Unlike in simulation, we cannot measure the difference between the policy’s achieved and desired door angle since we do not have access to the true state of the world. Instead, we simply visually denote a binary success/failure for each
goal based on whether the last state in the trajectory achieves the target angle. Every seven and a half minutes of interaction time we evaluate on 5 goals and plot the cumulative successes for each method. As Figure 7 shows, standard RIG only starts to open the door after five hours of training. In contrast, Skew-Fit learns to occasionally open the door after three hours of training and achieves a near-perfect success rate after five and a half hours of interaction time, demonstrating that Skew-Fit is a promising technique for solving real world tasks without any human-provided reward function. Videos of Skew-Fit solving this task and the simulated tasks can be viewed on our website.3
3Anonymous while under review: https://sites.google.com/view/skew-fit-iclr-2020
Additional Experiments To study the sensitivity of our method to the hyperparameter α, we sweep α across the values [−1,−0.75,−0.5,−0.25, 0] on the simulated image-based tasks. Due to space constraints, the sensitivity analysis over the hyperparameter α is in Appendix B, and the results demonstrate that Skew-Fit works across a large range of values for α, and α = −1 consistently outperform α = 0, where the empirical distribution is not skewed. Additionally, Appendix C provides a complete description our method hyper-parameters, including network architecture and RL algorithm hyperparameters.

heading: 7 CONCLUSION
text: We presented a formal objective for self-supervised goal-directed exploration, allowing researchers to quantify progress and compare progress when designing algorithms that enable agents to autonomously learn. We also presented Skew-Fit, an algorithm for training a generative model to approximate a uniform distribution over valid states, using data obtained via goal-conditioned reinforcement learning, and our theoretical analysis gives conditions under which Skew-Fit converges to the uniform distribution. When such a model is used to choose goals for exploration and to relabeling goals for training, the resulting method results in much better coverage of the state space, enabling our method to explore effectively. Our experiments show that when we concurrently train a goal-reaching policy using self-generated goals, Skew-Fit produces quantifiable improvements on simulated robotic manipulation tasks, and can be used to learn a door opening skill to reach a 95% success rate directly on a real-world robot, without any human-provided reward supervision.

heading: A PROOFS

heading: A.1 PROOF OF LEMMA 3.1
text: Lemma A.1. Let S be a compact set. Define the set of distributions Q = {p : support of p is S}. Let F : Q 7→ Q be a continuous function and such that H(F(p)) ≥ H(p) with equality if and only if p is the uniform probability distribution on S, US . Define the sequence of distributions P = (p1, p2, . . . ) by starting with any p1 ∈ Q and recursively defining pt+1 = F(pt).

heading: The sequence P converges to US .
text: Proof. The uniform distribution US is well defined since S is compact. Because S is a compact set, by Prokhorov’s Theorem Billingsley (2013), the set Q is sequentially compact. Thus, P has a convergent subsequence P ′ = (pk1 , pk2 , . . . ) ⊂ P for k1 < k2 < . . . that converges to a distribution p∗ ∈ Q. Because F is continuous, p∗ must be a fixed point of F since by the convergence mapping theorem, we have that
lim i→∞
pki = p ∗ =⇒ lim i→∞ F(pki) = H(p∗)
and so
p∗ = lim i→∞ pki
= lim i→∞ F(pki−1)
= H(p∗). The only fixed point of F is US since for any distribution p that is not the uniform distribution, US , we have that H(F(p)) > H(p) which implies that F(p) 6= p. Thus, P ′ converges to the only fixed point, US . Since the entropy cannot decrease, then entropy of the distributions in P must also converge to the entropy of US . Lastly, since entropy is a continuous function of distribution, P must converge to US .

heading: A.2 PROOF OF LEMMA 3.2
text: Lemma A.2. Given two distribution p(x) and q(x) where p q and 0 < Covp[log p(X), log q(X)] (6)
define the distribution pα as
pα(x) = 1
Zα p(x)q(x)α
where α ∈ R and Zα is the normalizing factor. LetHα(α) be the entropy of pα. Then there exists a constant a > 0 such that for all α ∈ [−a, 0),
Hα(α) > Hα(0) = H(p). (7)
Proof. Observe that {pα : α ∈ [−1, 0]} is a one-dimensional exponential family
pα(x) = e αT (x)−A(α)+k(x)
with log carrier density k(x) = log p(x), natural parameter α, sufficient statistic T (x) = log q(x), and log-normalizer A(α) = ∫ X e
αT (x)+k(x)dx. As shown in Nielsen & Nock (2010), the entropy of a distribution from a one-dimensional exponential family with parameter α is given by:
Hα(α) , H(pα) = A(α)− αA′(α)− Epα [k(X)] The derivative with respect to α is then
d
dα Hα(α) = −αA′′(α)−
d
dα Epα [k(x)]
= −αA′′(α)− Eα[k(x)(T (x)−A′(α)] = −αVarpα [T (x)]− Covpα [k(x), T (x)]
where we use the fact that the nth derivative of A(α) give the n central moment, i.e. A′(α) = Epα [T (x)] and A′′(α) = Varpα [T (x)]. The derivative of α = 0 is
d
dα Hα(0) = −Covp0 [k(x), T (x)]
= −Covp[log p(x), log q(x)] which is negative by assumption. Because the derivative at α = 0 is negative, then there exists a constant a > 0 such that for all α ∈ [−a, 0],Hα(α) > Hα(0) = H(p).

heading: A.3 SIMPLE CASE PROOF
text: We prove the convergence directly for the (even more) simplified case when pθ = p(S | pφt) using a similar technique: Lemma A.3. Assume the set S has finite volume so that its uniform distribution US is well defined and has finite entropy. Given any distribution p(s) whose support is S, recursively define pt with p1 = p and
pt+1(s) = 1
Ztα pt(s)
α, ∀s ∈ S
where Ztα is the normalizing constant and α ∈ [0, 1). The sequence (p1, p2, . . . ) converges to US , the uniform distribution S.
Proof. If α = 0, then p2 (and all subsequent distributions) will clearly be the uniform distribution. We now study the case where α ∈ (0, 1). At each iteration t, define the one-dimensional exponential family {ptθ : θ ∈ [0, 1]} where ptθ is
ptθ(s) = e θT (s)−A(θ)+k(s)
with log carrier density k(s) = 0, natural parameter θ, sufficient statistic T (s) = log pt(s), and lognormalizer A(θ) = ∫ S e
θT (s)ds. As shown in Nielsen & Nock (2010), the entropy of a distribution from a one-dimensional exponential family with parameter θ is given by:
Htθ(θ) , H(ptθ) = A(θ)− θA′(θ) The derivative with respect to θ is then
d
dθ dHtθ(θ) = −θA′′(θ)
= −θVars∼ptθ [T (s)] = −θVars∼ptθ [log pt(s)] (8) ≤ 0
where we use the fact that the nth derivative of A(θ) is the n central moment, i.e. A′′(θ) = Vars∼ptθ [T (s)]. Since variance is always non-negative, this means the entropy is monotonically decreasing with θ. Note that pt+1 is a member of this exponential family, with parameter θ = α ∈ (0, 1). So
H(pt+1) = Htθ(α) ≥ Htθ(1) = H(pt) which implies
H(p1) ≤ H(p2) ≤ . . . . This monotonically increasing sequence is upper bounded by the entropy of the uniform distribution, and so this sequence must converge.
The sequence can only converge if ddθH t θ(θ) converges to zero. However, because α is bounded away from 0, Equation 8 states that this can only happen if
Vars∼ptθ [log pt(s)]→ 0. (9)
Because pt has full support, then so does ptθ. Thus, Equation 9 is only true if log pt(s) converges to a constant, i.e. pt converges to the uniform distribution.

heading: B ADDITIONAL EXPERIMENTS

heading: B.1 SKEW-FIT FOR EXPLORING LOW-DIMENSIONAL SPACES
text: Skew-Fit is a general method that enables exploration when it is infeasible to sample goal states uniformly across the entire state space. While the experiments in Section 6 focused on image-based state spaces, there exists many low-dimensional domains in which we know that the goal space is a subset of Rd for some d < n, but the exact goal space is still unknown. This scenario is quite common in domains such as robotics: we know that we want an agent to move the position of its center of mass (CoM), but we do not know the set of valid CoM positions, as this requires knowing the geometry of all potential obstacles a priori. We conduct a series of experiments that study whether Skew-Fit enable effectively exploration in these state spaces containing unknown obstacles.
2D Maze Navigation with Oracle Policy To study the impact of Skew-Fit on exploration in isolation of learning a goal-reaching policy, our first set of experiments use a near-perfect policy that reaches the goal state and then takes a step in a random direction (while taking wall-collisions into account). The random step size is Gaussian with a standard deviation of 0.1 units, and the size of each square shown in Figure 8 is 1.8 units. Due to the relatively small step size, the agent cannot rely on random actions to explore the environment and must instead learn to set goals that are progressively farther and farther from the initial state. The first environment is the Four Rooms environment (Sutton et al., 1999), shown in Figure 8 (top). This environment requires a policy to explore four different rooms, each of which requires passing through a narrow doorway. The maze environment (Figure 8, bottom) presents a more challenging exploration problem and consists of various long corridors that require setting goals progressively deeper into the maze. In both domains, setting goals near the state state (represented by the yellow star) and taking small actions will result in minimal exploration. To measure exploration, we discretize the space into squares (see Figure 8 for square sizes) and measure what fraction of the squares the agent has ever visited during exploration. We see in Figure 8 that using Skew-Fit significantly improves exploration, whereas training pφ on samples drawn uniformly from the replay buffer (α = 0) results in little exploration.
2D Navigation with Learned Policy Next, we reproduce the 2D navigation environment experiment from Section 6, and replace the oracle goal-reacher with a goal-reaching policy that is simultaneously trained with the goal setter. The policy outputs velocities with maximum speed of one. Evaluation goals are chosen uniformly over the valid states. The hyperparameters for this experiment are given in Table 2. In Figure 9a, we can see that a policy trained with a goal distribution trained by Skew-Fit consistently learns to reach all goals, whereas a goal distribution trained with uniform sampling, labeled MLE, results in a policy that fails to reach states far from the starting position (the bottom left corner).
Quadruped “Ant” Locomotion with Learned Policy Lastly, we test Skew-Fit in an exploration task that requires training a simulated quadruped “ant” robot to navigate to random XY positions in a plane, as shown in Figure 10. The input to the policy is the joint and velocity of each angle and the reward is the distance to the goal XY-position. While the goal space is known to reside in the XY-plane, the agent does not know about the location of the center obstacle, and so it must still learn about the set of valid goals by controlling its 8 joint actuators. More details of the environment are in Appendix D. We see in Figure 10 that Skew-Fit outperforms prior methods both in terms of learning speed and final performance, demonstrating that Skew-Fit accelerates exploration in non-vision domains that contains unknown goal spaces.

heading: B.2 SENSITIVITY ANALYSIS
text: Sensitivity to RL Algorithm In our experiments, we combined Skew-Fit with soft actor critic (SAC) (Haarnoja et al., 2018). We conduct a set of experiments to test whether Skew-Fit may be used with other RL algorithms for training the goal-conditioned policy. To that end, we replaced SAC with twin delayed deep deterministic policy gradient (TD3) (Fujimoto et al., 2018) and ran the same Skew-Fit experiments on Visual Door, Visual Pusher, and Visual Pickup. In Figure 11, we see that Skew-Fit performs consistently well with both SAC and TD3, demonstrating that Skew-Fit is beneficial across multiple RL algorithms.
Sensitivity to α Hyperparameter We study the sensitivity of the α hyperparameter by testing values of α ∈ [−1,−0.75,−0.5,−0.25, 0] on the Visual Door and Visual Pusher task. The results are included in Figure 12 and shows that our method is robust to different parameters of α, particularly for the more challenging Visual Pusher task. Also, the method consistently outperform α = 0, which is equivalent to sampling uniformly from the replay buffer.
B.3 VARIANCE ABLATION
We measure the gradient variance of training a VAE on an unbalanced Visual Door image dataset with Skew-Fit vs Skew-Fit with importance sampling (IS) vs no Skew-Fit (labeled MLE). We construct the imbalanced dataset by rolling out a random policy in the environment and collecting the visual observations. Most of the images contained the door in a closed position; in a few, the door was opened. In Figure 13, we see that the gradient variance for Skew-Fit with IS is catastrophically large for large values of α. In contrast, for Skew-Fit with SIR, which is what we use in practice, the variance is relatively similar to that of MLE. Additionally we trained three VAE’s, one with MLE on a uniform dataset of valid door opening images, one with Skew-Fit on the unbalanced dataset from above, and one with MLE on the same unbalanced dataset. As expected, the VAE that has access to the uniform dataset gets the lowest negative log likelihood score. This is the oracle method, since in practice we would only have access to imbalanced data. As shown in Table 1, Skew-Fit considerably outperforms MLE, getting a much closer to oracle log likelihood score.

heading: B.4 GOAL AND PERFORMANCE VISUALIZATION
text: We visualize the goals sampled from Skew-Fit as well as those sampled when using the prior method, RIG (Nair et al., 2018). As shown in Figure 14 and Figure 15, the generative model pφ results in much more diverse samples when trained with Skew-Fit. We we see in Figure 16, this results in a policy that more consistently reaches the goal image.
C IMPLEMENTATION DETAILS

heading: C.1 RIG WITH SKEW-FIT SUMMARY
text: Algorithm 2 provides detailed pseudo-code for how we combined our method with RIG. Steps that were removed from the base RIG algorithm are highlighted in blue and steps that were added are highlighted in red. The main differences between the two are (1) sampling exploration goals from the buffer using pskewed instead of the VAE prior, (2) relabeling with replay buffer goals sampled using pskewed instead of from the VAE prior, and (3) training the VAE on replay buffer data data sampled using pskewed instead of uniformly.
C.2 LIKELIHOOD ESTIMATION USING β-VAE
We estimate the density under the VAE by using a sample-wise approximation to the marginal over x estimated using importance sampling:
pφt(x) = Ez∼qθt (z|x) [ p(z)
qθt(z|x) pψt(x | z) ] ≈ 1 N N∑ i=1 [ p(z) qθt(z|x) pψt(x | z) ] .
where qθ is the encoder, pψ is the decoder, and p(z) is the prior, which in this case is unit Gaussian. We found that sampling N = 10 latents for estimating the density worked well in practice.
C.3 IMPLEMENTATION OF PRIOR WORK
We replaced TD3 (Fujimoto et al., 2018) with soft actor critic (SAC) from Haarnoja et al. (2018) for all the methods that use RIG, including Skew-Fit.. This is in contrast to the original RIG Nair et al. (2018) paper which used TD3 Fujimoto et al. (2018). We found that maximum entropy policies in general improved the performance of RIG, and that we did not need to add noise on top of the stochastic policy’s noise. For our RL network architectures and training scheme, we use fully connected networks for the policy, Q-function and value networks with two hidden layers of size 400 and 300 each. We also delay training any of these networks for 10000 time steps in order to collect sufficient data for the replay buffer as well as to ensure the latent space of the VAE is relatively stable (since we train the VAE online in this setting). As in RIG, we train a goal-conditioned value functions Schaul et al. (2015) using hindsight experience replay Andrychowicz et al. (2017), relabelling 50% of exploration goals as goals sampled from the VAE prior N (0, 1) and 30% from future goals in the trajectory. In the prior RIG method, the VAE was pre-trained on a uniform sampling of images from the state space of each environment. In order to ensure a fair comparison to Skew-Fit, we forego pre-training and instead train the VAE alongside RL, using the variant described in the RIG paper.
C.4 VISION-BASED CONTINUOUS CONTROL EXPERIMENTS
In our experiments, we use an image size of 48x48. For our VAE architecture, we use a modified version of the architecture used in the original RIG paper Nair et al. (2018). Our VAE has three convolutional layers with kernel sizes: 5x5, 3x3, and 3x3, number of output filters: 16, 32, and 64 and strides: 3, 2, and 2. We then have a fully connected layer with the latent dimension number of units, and then reverse the architecture with de-convolution layers. We vary the latent dimension of the VAE, the β term of the VAE and the α term for Skew-Fit based on the environment. Additionally, we vary the training schedule of the VAE based on the environment. See the table at the end of the appendix for more details. Our VAE has a Gaussian decoder with identity variance, meaning that we train the decoder with a mean-squared error loss.
When training the VAE alongside RL, we found the following two schedules to be effective for different environments:
1. For first 5K steps: Train VAE using standard MLE training every 500 time steps for 1000 batches. After that, train VAE using Skew-Fit every 500 time steps for 200 batches.
2. For first 5K steps: Train VAE using standard MLE training every 500 time steps for 1000 batches. For the next 45K steps, train VAE using Skew-Fit every 500 steps for 200 batches. After that, train VAE using Skew-Fit every 1000 time steps for 200 batches.
We found that initially training the VAE without Skew-Fit improved the stability of the algorithm. This is due to the fact that density estimates under the VAE are constantly changing and inaccurate during the early phases of training. Therefore, it made little sense to use those estimates to prioritize goals early on in training. Instead, we simply train using MLE training for the first 5K timesteps, and after that we perform Skew-Fit according to the VAE schedules above. Table 3 lists the hyperparameters that were shared across the continuous control experiments. Table 4 lists hyper-parameters specific to each environment. Additionally, Appendix C.1 shows the combined RIG + Skew-Fit algorithm.
aWe expect similar performance had we used SAC.
Algorithm 2 RIG and RIG + Skew-Fit. Blue text denotes RIG specific steps and red text denotes RIG + Skew-Fit specific steps
Require: VAE encoder qφ, VAE decoder pψ , policy πθ , goal-conditioned value function Qw, α, VAE Training Schedule.
1: Collect D = {s(i)} using exploration policy. 2: Train β-VAE on data uniformly sampled from D. 3: Fit prior p(z) to latent encodings {µφ(s(i))}. 4: for n = 0, ..., N − 1 episodes do 5: Sample latent goal from prior zg ∼ p(z). 6: Sample latent goal e(s′) from (s, a, s′, zg) ∼ R using pφ if R not empty. Otherwise, use zg ∼ p(z). 7: Sample initial state s0 ∼ E. 8: for t = 0, ..., H − 1 steps do 9: Get action at ∼ πθ(e(st), zg). 10: Get next state st+1 ∼ p(· | st, at). 11: Store (st, at, st+1, zg) into replay bufferR. 12: Sample transition (s, a, s′, zg) ∼ R. 13: Encode z = e(s), z′ = e(s′). 14: (Probability 0.5) replace zg with z′g ∼ p(z). 15: (Probability 0.5) replace zg with e(s′) where
(s, a, s′, zg) ∼ R using pφ
16: Compute new reward r = −||z′ − zg||. 17: Minimize Bellman Error using
(z, a, z′, zg, r). 18: end for 19: for t = 0, ..., H − 1 steps do 20: for i = 0, ..., k − 1 steps do 21: Sample future state shi , t < hi ≤ H − 1. 22: Store (st, at, st+1, e(shi)) intoR. 23: end for 24: end for 25: Construct skewed replay buffer distribution pφ
using data fromR with Equation 4 26: if total_steps < 5000 then 27: Fine-tune β-VAE on data uniformly sampled
fromR according to VAE Training Schedule. 28: else 29: Fine-tune β-VAE on data uniformly sampled
fromR according to VAE Training Schedule. 30: Fine-tune β-VAE on data sampled from R
using pφ according to VAE Training Schedule. 31: end if 32: end for

heading: C.5 ORACLE 2D NAVIGATION EXPERIMENTS
text: We initialize the VAE to the middle of the environment for Maze, and the bottom left corner of the environment for Four Rooms. Both the encoder and decoder have 2 hidden layers with [400, 300] units, ReLU hidden activations, and no output activations. The VAE has a latent dimension of 8 and a Gaussian decoder trained with mean-squared error loss, batch size of 256, and 1000 batches at each iteration. The VAE is trained on the exploration data buffer every 1000 rollouts.

heading: D ENVIRONMENT DETAILS
text: Point-Mass: In this environment, an agent must learn to navigate a square-shaped corridor (see Figure 3). The observation is the 2D position, and the agent must specify a velocity as the 2D action. The reward at each time step is the negative distance between the achieved position and desired position.
Maze: A 20 x 20 2D pointmass environment in the shape of a maze. The observation is the 2D position of the agent, and the agent must specify a target 2D position as the action. The dynamics of the environment are the following: first, the agent is teleported to the target position, specified by the action. Then a gaussian change in position with mean 0 and standard deviation 0.1 is then applied. If the action would result in the agent moving through or into a wall, then the agent will be stopped at the wall instead.
Four Rooms: A 20 x 20 2D pointmass environment in the shape of four rooms (Sutton et al., 1999). The observation space, actions space, and environment dynamics are the same as the Maze environment above.
Ant: A MuJoCo ant environment with the same corridor as the Point-Mass environment. The observation is a 2D position, orientation, joint angles, and velocity of the joint angles of the ant. The observation space is 29 dimensions. The agent controls the ant through the joints, which is 8 dimensions. The goal is a target 2D position, and the reward is the negative Euclidean distance between the achieved 2D position and target 2D position.
Visual Pusher: A MuJoCo environment with a 7-DoF Sawyer arm and a small puck on a table that the arm must push to a target position. The agent controls the arm by commanding x, y position for the end effector (EE). The underlying state is the EE position, e and puck position p. The evaluation metric is the distance between the goal and final puck positions. The hand goal/state space is a 10x10
cm2 box and the puck goal/state space is a 30x20 cm2 box. Both the hand and puck spaces are centered around the origin. The action space ranges in the interval [−1, 1] in the x and y dimensions. Visual Door: A MuJoCo environment with a 7-DoF Sawyer arm and a door on a table that the arm must pull open to a target angle. Control is the same as in Visual Pusher. The evaluation metric is the distance between the goal and final door angle, measured in radians. In this environment, we do not reset the position of the hand or door at the end of each trajectory. The state/goal space is a 5x20x15 cm3 box in the x, y, z dimension respectively for the arm and an angle between [0, .83] radians. The action space ranges in the interval [−1, 1] in the x, y and z dimensions. Visual Pickup: A MuJoCo environment with the same robot as Visual Pusher, but now with a different object. The object is cube-shaped, but a larger intangible sphere is overlaid on top so that it is easier for the agent to see. Moreover, the robot is constrained to move in 2 dimension: it only controls the y, z arm positions. The x position of both the arm and the object is fixed. The evaluation metric is the distance between the goal and final object position. For the purpose of evaluation, 75% of the goals have the object in the air and 25% have the object on the ground. The state/goal space for both the object and the arm is 10cm in the y dimension and 13cm in the z dimension. The action space ranges in the interval [−1, 1] in the y and z dimensions. Real World Visual Door: A Rethink Sawyer Robot with a door on a table. The arm must pull the door open to a target angle. The agent controls the arm by commanding the x, y, z velocity of the EE. Our controller commands actions at a rate of up to 10Hz with the scale of actions ranging up to 1cm in magnitude. The underlying state and goal is the same as in Visual Door. Again we do not reset the position of the hand or door at the end of each trajectory. We obtain images using a Kinect Sensor. The state/goal space for the environment is a 10x10x10 cm3 box. The action space ranges in the interval [−1, 1] (in cm) in the x, y and z dimensions. The door angle lies in the range [0, 45] degrees.
E GOAL-CONDITIONED REINFORCEMENT LEARNING MINIMIZES H(G | S)
Some goal-conditioned RL methods such as Warde-Farley et al. (2018); Nair et al. (2018) present methods for minimizing a lower bound forH(G | S), by approximating log p(G | S) and using it as the reward. Other goal-conditioned RL methods (Kaelbling, 1993; Lillicrap et al., 2016; Schaul et al., 2015; Andrychowicz et al., 2017; Pong et al., 2018; Florensa et al., 2018a) are not developed with the intention of minimizing the conditional entropyH(G | S). Nevertheless, one can see that goal-conditioned RL generally minimizes H(G | S) by noting that the optimal goal-conditioned policy will deterministically reach the goal. The corresponding conditional entropy of the goal given the state,H(G | S), would be zero, since given the current state, there would be no uncertainty over the goal (the goal must have been the current state since the policy is optimal). So, the objective of goal-conditioned RL can be interpreted as finding a policy such thatH(G | S) = 0. Since zero is the minimum value ofH(G | S), then goal-conditioned RL can be interpreted as minimizingH(G | S).

Review summaries:
The paper proposes an exploratory objective that can maximize state coverage in RL. They show that a formal objective for maximizing state coverage is equivalent to maximizing the entropy of a goal distribution. The core idea is to propose a method to maximize entropy of a goal distribution, or a state distribution since goals are full states. They show that the proposed method to maximize the state or goal distribution can lead to diverse exploration behaviour sufficient for solving complex image based manipulation tasks. 

This paper introduced a very interesting idea to facilitate exploration in goal-conditioned reinforcement learning. The key idea is to learn a generative model of goal distribution to match the weighted empirical distribution, where the rare states receive larger weights. This encourages the model to generate more diverse and novel goals for goal-conditioned RL policies to reach.