Title: SKEW-FIT: STATE-COVERING SELF-SUPERVISED REINFORCEMENT LEARNING

Abstract: Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manuallydesigned reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing the entropy of the goal distribution together with goal reaching performance, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. Skew-Fit enables self-supervised agents to autonomously choose and practice reaching diverse goals. We show that, under certain regularity conditions, our method converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that it can learn a variety of manipulation tasks from images, including opening a door with a real robot, entirely from scratch and without any manually-designed reward function.

1. INTRODUCTION:

Reinforcement learning (RL) provides an appealing formalism for automated learning of behavioral skills, but separately learning every potentially useful skill becomes prohibitively time consuming, both in terms of the experience required for the agent and the effort required for the user to design reward functions for each behavior. What if we could instead design an unsupervised RL algorithm that automatically explores the environment and iteratively distills this experience into general-purpose policies that can accomplish new user-specified tasks at test time?

For an agent to learn autonomously, it needs an exploration objective. In the absence of any prior knowledge about which states are more useful, an effective exploration scheme is one that visits as many states as possible, allowing a policy to autonomously prepare for user-specified task that it might see at test time. This objective has been formalized as maximizing the entropy of the learned policy's visited state distribution1 H(S) (Hazan et al., 2018a) , since a policy that maximizes this objective should approach a uniform distribution over valid states. Unfortunately, directly optimizing H(S) requires an accurate model of the policy and environment (Hazan et al., 2018a) . Moreover, even if this optimization were tractable, another short-coming of this objective is that the resulting policy cannot be used to solve new tasks: it only knows how to maximize state entropy. In other words, to develop principled unsupervised RL algorithms that result in useful policies, maximizing H(S) is not enough. We need a mechanism that allows us to control the resulting policy to achieve new tasks at test-time.

We argue that this can be accomplished by performing goal-directed exploration. In addition to maximizing the state entropy, we should be able to control where the policy goes by giving it a goal G that corresponds to a state that it must reach. Mathematically, a goal-conditioned policy should minimize the conditional entropy over the states given a goal, H(S | G). This objective provides us with a principled way for training a policy to explore all states, by maximizing H(S), such that the state that is reached can be controlled by commanding goals, which means minimizing H(S | G).

Directly using this objective is often intractable, since it requires optimizing the entropy of the marginal state distribution of the policy, H(S). However, we can sidestep this issue by noting that the objective is the mutual information between the state and the goal, I(S; G), which can be written as: H(S) -H(S|G) = I(S; G) = H(G) -H(G|S).

(1)

Equation 1 thus gives an equivalent objective for an unsupervised RL algorithm: the agent should set diverse goals, maximizing H(G), and learn how to reach them, minimizing H(G | S).

While the second term is the typical objective studied in goal-conditioned RL (Kaelbling, 1993; Andrychowicz et al., 2017) , maximizing the diversity of goals is crucial for effectively learning to reach all possible states. In a new environment, acquiring such a maximum-entropy goal distribution is challenging: how can an agent set diverse goals when it does not even know what states exist?

In this paper, we address this question via a new algorithm, Skew-Fit, which learns to model the uniform distribution over states, given only access to data collected by an autonomous goalconditioned policy. Our paper makes the following contributions. First, we propose a principled objective for unsupervised RL, based on Equation 1. While a number of prior works ignore the H(G) term, we argue that jointly optimizing the entire quantity is needed to develop effective and useful exploration. Second, we propose a method called Skew-Fit and prove that, under some regularity conditions, it learns a generative model that converges to a uniform distribution over the goal space, even when the set of valid states is unknown (e.g., as in the case of images). Third, we empirically demonstrate that, when combined with goal-conditioned RL, Skew-Fit allows us to autonomously train goal-conditioned policies that reach diverse states. We test this method on a variety of simulated vision-based robot tasks without any task-specific reward function. In these experiments, Skew-Fit reaches substantially better final performance than prior methods, and learns much more quickly. We also demonstrate that our approach solves a real-world manipulation task, which requires a robot to learn to open a door from scratch in about five hours, directly from images, and without any manually-designed reward function.

2. PROBLEM FORMULATION:

To ensure that an unsupervised reinforcement learning agent learns to reach all possible states in a controllable way, we maximize the mutual information between the state S and the goal G, I(S; G), as stated in Equation 1. This section discusses how to optimize Equation 1 by splitting the optimization into two parts: minimizing H(G | S) and maximizing H(G).

2.1. MINIMIZING H(G | S): GOAL-CONDITIONED REINFORCEMENT LEARNING:

Standard RL considers a Markov decision process (MDP), which has a state space S, action space A, and unknown dynamics ρ(s t+1 | s t , a t ) : S × S × A → [0, +∞). Goal-conditioned RL also includes a goal space G. For simplicity, we will assume in our derivation that the goal space matches the state space, such that G = S, though the approach extends trivially to the case where G is a hand-specified subset of S, such as the global x-y position of a robot. A goal-conditioned policy π(a | s, g) maps a state s ∈ S and goal g ∈ S to a distribution over actions a ∈ A, and its objective is to reach the goal, i.e., to make the current state equal to the goal.

Goal-reaching can be formulated as minimizing H(G | S), and many practical goal-reaching algorithms (Kaelbling, 1993; Lillicrap et al., 2016; Schaul et al., 2015; Andrychowicz et al., 2017; Nair et al., 2018; Pong et al., 2018; Florensa et al., 2018a) can be viewed as approximations to this objective by observing that the optimal goal-conditioned policy will deterministically reach the goal, resulting in a conditional entropy of zero: H(G | S) = 0. See Appendix E for more details. Our method may thus be used in conjunction with any of these prior goal-conditioned RL methods in order to jointly minimize H(G | S) and maximize H(G).

2.2. MAXIMIZING H(G): SETTING DIVERSE GOALS:

We now turn to the problem of setting diverse goals or, mathematically, maximizing the entropy of the goal distribution H(G). Let U S be the uniform distribution over S, where we assume S has finite volume so that the uniform distribution is well-defined. Let p φ be the goal distribution from which goals G are sampled. Our goal is to maximize the entropy of p φ , which we write as H(G). Since the maximum entropy distribution over S is the uniform distribution U S , maximizing H(G) may seem as simple as choosing the uniform distribution to be our goal distribution: p φ = U S . However, this requires knowing the uniform distribution over valid states, which may be difficult to obtain when S is a subset of R n , for some n. For example, if the states correspond to images viewed through a robot's camera, S corresponds to the (unknown) set of valid images of the robot's environment, while R n corresponds to all possible arrays of pixel values of a particular size. In such environments, sampling from the uniform distribution R n is unlikely to correspond to a valid image of the real world. Sampling uniformly from S would require knowing the set of all possible valid images, which we assume the agent does not know when starting to explore the environment.

While we cannot sample arbitrary states from S, we can sample states by performing goal-directed exploration. To derive and analyze our method, we introduce a simple model of this process: a goal G ∼ p φ is sampled from the goal distribution p φ , and then the agent attempts to achieve this goal, which results in a distribution of states S ∈ S seen along the trajectory. We abstract this entire process by writing the resulting marginal distribution over S as p(S | p φ ). We assume that p(S | p φ ) has full support, which can be accomplished with an epsilon-greedy goal reaching policy in a communicating MDP. We also assume that the entropy of the resulting state distribution H(p(S | p φ )) is no less than the entropy of the goal distribution H(p φ (S)). Without this assumption, a policy could ignore the goal and stay in a single state, no matter how diverse and realistic the goals are. Note that this assumption does not require that the entropy of p(S | p φ ) is strictly larger than the entropy of the goal distribution, p φ . This simplified model allows us to analyze the behavior of our goal-setting scheme separately from any specific goal-reaching algorithm. We will however show in Section 6 that we can instantiate this approach into a practical algorithm that jointly learns the goal-reaching policy. In summary, our goal is to acquire a maximum-entropy goal distribution p φ over valid states S, while only having access to state samples from p(S | p φ ).

3. SKEW-FIT: LEARNING A MAXIMUM ENTROPY GOAL DISTRIBUTION:

Our method, Skew-Fit, learns a maximum entropy goal distribution p φ using samples collected from a goal-conditioned policy. We analyze the algorithm and show that Skew-Fit maximizes the entropy of the goal distribution, and present a practical instantiation for unsupervised deep RL.

3.1. SKEW-FIT ALGORITHM:

To learn a uniform distribution over valid goal states, we present a method that iteratively increases the entropy of a generative model p φ . In particular, given a generative model p φt at iteration t, we would like to train a new generative model p φt+1 such that p φt+1 has higher entropy than p φt over the set of valid states. While we do not know the set of valid states S, we can sample states from p(S | p φt ), resulting in an empirical distribution p emp t over the states

EQUATION

and use this empirical distribution to train the next generative model p φt+1 . However, if we simply train p φt+1 to model this empirical distribution, it may not necessarily have higher entropy than p φt .

The intuition behind our method is quite simple: rather than fitting a generative model to our empirical distribution, we skew the empirical distribution so that rarely visited states are given more weight. See Figure 2 for a visualization of this process. How should we skew the empirical distribution if we want to maximize the entropy of p φt+1 ? If we had access to the density of each state, p emp t (S), then we could simply weight each state by 1/p emp t (S). We could then perform maximum likelihood Figure 2 : Our method, Skew-Fit, samples goals for goal-conditioned RL in order to induce a uniform state visitation distribution. We start by sampling from our replay buffer, and weighting the states such that rare states are given more weight. We then train a generative model p φ t+1 with the weighted samples. By sampling new states with goals proposed from this new generative model, we obtain a higher entropy distribution of states in our replay buffer at the next iteration.

estimation (MLE) for the uniform distribution by using the following loss to train φ t+1 :

L(φ) = E S∼U S [log p φ (S)] = E S∼pemp t U S (S) p emp t (S) log p φ (S) ∝ E S∼pemp t 1 p emp t (S) log p φ (S)

where we use the fact that the uniform distribution U S (S) has constant density for all states in S. However, computing this density p emp t (S) requires marginalizing out the MDP dynamics, which requires an accurate model of both the dynamics and the goal-conditioned policy.

We avoid needing to model the entire MDP process by approximating p emp t (S) with our previous learned generative model: p emp t (S) ≈ p(S | p φt ) ≈ p φt (S). We therefore weight each state by the following weight function w t,α (S) p φt (S) α , α < 0.

(

EQUATION

where α is a hyperparameter that controls how heavily we weight each state. If our approximation p φt was exact, we could choose α = -1 and recover the exact importance sampling procedure described above. If α = 0, then this skew step has no effect. By choosing intermediate values of α, we can trade off the reliability of our estimate p φt (S) with the speed at which we want to increase the entropy of the goal distribution.

Variance Reduction As described, this procedure relies on importance sampling (IS), which can have high variance, particularly if p φt (S) ≈ 0. We therefore choose a class of generative models where the probabilities are prevented from collapsing to zero, as we will describe in Section 4. To further reduce the variance, we train p φt+1 with sampling importance resampling (SIR) (Rubin, 1988) . Rather than sampling from p emp t and weighting the update from each sample by w t,α , SIR explicitly defines a skewed distribution as

EQUATION

where Z α is the normalizing coefficient and p emp t is given by Equation 2. We note that computing Z α adds little computational overhead, since all of the weights already need to be computed. We then fit the generative model at the next iteration p φt+1 to p skewedt using standard MLE. We found that using SIR resulted in significantly lower variance than IS. See Appendix B.3 for this comparision.

Goal Sampling Alternative Because p φt+1 ≈ p skewedt , at iteration t + 1, one can sample goals from either p φt+1 or p skewedt . Sampling goals from p skewedt may be preferred if sampling from the learned generative model p φt+1 is computationally or otherwise challenging. In either case, one still needs to train the generative model p φt to create p skewedt . In our experiments, we found that both methods perform well.

Summary Overall, Skew-Fit samples data from the environment and weights different samples by their density under the generative model p φ t . We prove in the next section conditions under which this weighting makes the generative model at the next iteration p φ t+1 have higher entropy. With higher entropy, the p φt+1 is more likely to generate goals at the frontier of unseen states, which results in more uniform state coverage. Skew-Fit is shown in Figure 2 and summarized in Algorithm 1.

Algorithm 1 Skew-Fit 1: for Iteration t = 1, 2, ... do 2:

Collect N states {S i } N i=1 by sampling goals from p φt (or p skewedt ) and running goalconditioned policy.

3::

Construct skewed distribution p skewedt (Equation 3 and Equation 4).

4::

Fit p φt+1 to skewed distribution p skewedt using MLE. 5: end for

3.2. SKEW-FIT ANALYSIS:

In this section, we provide conditions under which p φt converges in distribution to the uniform distribution over the state space S. To make this analysis possible, we consider the case where N → ∞, which allows us to study the limit behavior of the goal distribution p skewedt . Our most general result is stated as follows:

Lemma 3.1. Let S be a compact set. Define the set of distributions Q = {p : support of p is S}. Let F : Q → Q be a continuous function and such that H(F(p)) ≥ H(p) with equality if and only if p is the uniform probability distribution on S, U S . Define the sequence of distributions P = (p 1 , p 2 , . . . ) by starting with any p 1 ∈ Q and recursively defining p t+1 = F(p t ).

The sequence P converges to U S .

Proof. See Appendix Section E.

We will apply Lemma 3.1 to be the map from p skewedt to p skewedt+1 to show that p skewedt converges to U S . If we assume that the goal-conditioned policy and generative model learning procedure are well behaved ( i.e., the maps from p φt (S) to p emp t and from p skewedt to p φt+1 are continuous ), then to apply Lemma 3.1, we only need to show that H(p skewedt ) ≥ H(p emp t ) with equality if and only if p emp t = U S . For the simple case when p φt = p emp t identically at each iteration, we prove the convergence of Skew-Fit true for any value of α ∈ [-1, 0) in Appendix A.3. However, in practice, p φt only approximates p emp t . To address this more realistic situation, we prove the following result: Lemma 3.2. Given two distribution p emp t and p φt where p emp t p φt2 and

EQUATION

define the distribution p skewedt as in Equation 4. Let H α (α) be the entropy of p skewedt for a fixed α.

Then there exists a constant a < 0 such that for all α ∈ [a, 0),

H(p skewedt ) = H α (α) > H(p emp t ).

Proof. See Appendix Section E.

Thus, our generative model p φt does not need to exactly fit the empirical distribution. We merely need for the log densities of p φt and p emp t to be correlated, which we expect to happen frequently with an accurate goal-conditioned policy, since p emp t is the set of states seen when trying to reach goals from p φt . In this case, if we choose negative values of α that are small enough, then the entropy of p skewedt will be higher than that of p emp t . Empirically, we found that α values as low as α = -1 performed well.

In summary, we see that under certain assumptions, p skewedt converges to U S . Since we train each generative model p φt+1 by fitting it to p skewedt , we expect p φt to also converge to U S .

4. TRAINING GOAL-CONDITIONED POLICIES WITH SKEW-FIT:

Thus far, we have presented and derived Skew-Fit assuming that we have access to a goal-reaching policy, allowing us to separately analyze how we can maximize H(G). However, in practice we do not have access to such a policy, and in this section we discuss how we concurrently train a goal-reaching policy.

Maximizing I(S; G) can be done by simultaneously performing Skew-Fit and training a goal conditioned policy to minimize H(G | S), or, equivalently, maximize -H(G | S). Maximizing -H(G | S) requires computing the density log p(G | S), which may be difficult to compute without strong modeling assumptions. However, for any distribution q, the following lower bound for -H(G | S) holds:

-H(G | S) = E (G,S)∼p φ t ,π [log q(G | S)] + D KL (p | q) ≥ E (G,S)∼p φ t ,π [log q(G | S)] ,

where D KL denotes Kullback-Leibler divergence as discussed by Barber & Agakov (2004) . Thus, to minimize H(G | S), we train a policy to maximize the following reward:

r(S, G) = log q(G | S).

For the RL algorithm, we use reinforcement learning with imagined goals (RIG) (Nair et al., 2018) , though in principle any goal-conditioned method could be used. RIG is an efficient off-policy goalconditioned method that solves the vision-based RL problem in a learned latent space. In particular, RIG fits a β-VAE and uses it to encode all observations and goals into a latent space, which it uses as the state representation. RIG also uses the β-VAE to compute rewards, log q(G | S). Unlike RIG, we use the goal distribution from Skew-Fit to sample goals, both for exploration and for relabeling goals during training (Andrychowicz et al., 2017) . Since RIG already trains a generative model over states, we reuse this β-VAE for the generative model p φ of Skew-Fit. To make the most use of the data, p φ is trained on all visited state rather than only the terminal states, which we found to work well in practice. In other words, our method uses the likelihood estimates from the β-VAE to choose the probability of sampling each state in Equation 3. To prevent these probabilities from collapsing to zero, we model the posterior of the β-VAE as a multivariate Gaussian distribution with a fixed variance and only learn the mean. We include a detailed summary of RIG and description our how we combine Skew-Fit and RIG in Appendix C.1.

5. RELATED WORK:

Many prior methods for training goal-conditioned policies assume that a goal distribution is available to sample from during exploration (Kaelbling, 1993; Schaul et al., 2015; Andrychowicz et al., 2017; Pong et al., 2018) . Other methods use data collected from a randomly initialized policy or heuristics based on data collected online to design a non-parametric (Colas et al., 2018b; Warde-Farley et al., 2018; Florensa et al., 2018a; Zhao & Tresp, 2019) or parametric (Péré et al., 2018; Nair et al., 2018) goal distribution. We remark that Warde-Farley et al. ( 2018) also motivate their work in terms of minimizing a lower bound for H(G | S). Our work is complementary to these goal-reaching methods: rather than focusing on how to train goal-reaching policies, we propose a principled method for maximizing the entropy of a goal sampling distribution, H(G).

Our method learns without any task rewards, directly acquiring a policy that can be reused to reach user-specified goals. This stands in contrast to exploration methods that give bonus rewards based on state visitation frequency (Bellemare et al., 2016; Ostrovski et al., 2017; Tang et al., 2017; Savinov et al., 2018; Chentanez et al., 2005; Lopes et al., 2012; Stadie et al., 2016; Pathak et al., 2017; Burda et al., 2018; 2019; Mohamed & Rezende, 2015; Tang et al., 2017; Fu et al., 2017) . While these methods can also be used without a task reward, they provide no mechanism for distilling the knowledge gained from visiting diverse states into flexible policies that can be applied to accomplish new goals at test-time: their policies visit novel states, and they quickly forget about them as other states become more novel.

Other prior methods extract reusable skills in the form of latent-variable-conditioned policies, where latent variables can be interpreted as options (Sutton et al., 1999) or abstract skills (Hausman et al., 2018; Gupta et al., 2018b; Eysenbach et al., 2019; Gupta et al., 2018a; Florensa et al., 2017) . The resulting skills may be diverse, but they have no grounded interpretation, while our method can be used immediately after unsupervised training to reach diverse user-specified goals.

Some prior methods propose to choose goals based on heuristics such as learning progress (Baranes & Oudeyer, 2012; Veeriah et al., 2018; Colas et al., 2018a) , how off-policy the goal is (Nachum et al., 2018) , level of difficulty (Florensa et al., 2018b) or likelihood ranking (Zhao & Tresp, 2019) . In contrast, our approach provides a principled framework for optimizing a concrete and well-motivated exploration objective, and can be shown to maximize this objective under regularity assumptions.

The work of Hazan et al. (2018b) also provably optimizes a well-motivated exploration objective, but is limited to tabular MDPs, while Skew-Fit is able to handle high dimensional settings such as vision-based continuous control.

6. EXPERIMENTS:

Our experiments study the following questions: (1) Does Skew-Fit empirically result in a goal distribution with increasing entropy? (2) In image-based domains, how does Skew-Fit compare to prior work on choosing goals for goal-conditioned RL? (3) Can Skew-Fit be applied to a real-world, vision-based robot task?

Does Skew-Fit Maximize Entropy? To see the effects of Skew-Fit on goal distribution entropy in isolation of learning a goal-reaching policy, we begin by studying an idealized example where the policy is a near-perfect goal-reaching policy. The MDP is defined on a 2-by-2 unit square-shaped corridor (see Figure 3 ). At the beginning of an episode, the agent begins in the bottom-left corner and samples a goal from the goal distribution p φt . To simulate the stochasticity of the policy and environment, we add a Gaussian noise with standard deviation of 0.05 to this goal. The policy reaches the state that is closest to this noisy goal and inside the corridor, giving us a state S to add to our empirical distribution. We compare Skew-Fit to sampling uniformly from the replay buffer (labeled MLE). The β-VAE hyperparameters used to train p φt are given in Appendix C.5. As seen in Figure 3 , naively using previous experience to set goals results in a policy that primarily sets goal near the initial state distribution and only relies on the stochasticity of the policy and environment to explore.

In contrast, Skew-Fit results in quickly learning a high entropy, near-uniform distribution over the state space.

Vision-Based Continuous Control Tasks We now evaluate Skew-Fit on a variety of continuous control tasks, where the policy must control a robot arm using only image observations, without access to any ground truth reward signal. We test our method on three different simulated continuous control tasks released by the authors of RIG (Nair et al., 2018) : Visual Door, Visual Pusher, and Visual Pickup. To our knowledge, these are the only goal-conditioned, vision-based continuous control environments that are publicly available and used in experimental evaluations in prior work, making them a good point of comparison. See Figure 4 for visuals and Appendix C for details of these environments. The policies are trained in a completely unsupervised manner, without access to any prior information about the state-space or any pre-defined goal-sampling distribution. To evaluate their performance, we sample goal images from a uniform distribution over valid states and report the agent's final distance to the corresponding simulator states (e.g., distance of the object In the next two columns, we display final images reached by Skew-Fit and RIG respectively. Under each image is the final distance in state space to provide a notion of the behavior of each method in the plots.

to the target object location), but the agent never has access to this true uniform distribution nor the ground-truth state information during training. While this evaluation method and metric is only practical in simulation, it provides us with a quantitative measure of a policy's ability to reach a broad coverage of goals in a vision-based setting.

We use these domains to compare Skew-Fit to a number of existing methods on goal-sampling. We compare to Warde-Farley et al. ( 2018), a vision-based method which uses a non-parametric approach based on clustering to sample goals and an image discriminator to compute rewards. We denote this method as DISCERN. The other methods that we compare to were developed in non-vision, statebased environments. To ensure a fair comparison across methods, we combine these prior methods with a policy trained using RIG. First, we compare to RIG without Skew-Fit. We also compared to RIG using the relabeling scheme described in the hindsight experience replay (labeled HER). We compare to curiosity-driven prioritization (Ranked-Based Priority) (Zhao & Tresp, 2019) , a variant of HER that samples goals for relabeling based on their ranked likelihoods. Florensa et al. (2018b) samples goals from a GAN based on the difficulty of reaching the goal. We compare against this method by replacing p φ with the GAN and label it AutoGoal GAN. We also separately compare to the goal proposal mechanism proposed by Warde-Farley et al. ( 2018) and otherwise train the policy with RIG, which we label DISCERN-g. Lastly, to demonstrate the difficulty of the exploration challenge in these domains, we compare to # Exploration (Tang et al., 2017) , an exploration method that assigns bonus rewards based on the novelty of new states. Implementation details of the prior methods is given in Appendix C.3.

We see in Figure 5 that Skew-Fit significantly outperforms prior methods both in terms of task performance and sample complexity. The most common failure mode for prior methods is that the goal distributions collapse, resulting in the agent learning to reach only a fraction of the state space, as shown in Figure 1 . For comparison, additional samples of p φ when trained with and without Skew-Fit are shown in Appendix B.4. Those images show that without Skew -F it, p φ produces a small, non-diverse distribution for each environment: the object is in the same place for pickup, the puck is often in the starting position for pushing, and the door is always closed. In contrast, Skew-Fit proposes goals where the object is in the air and on the ground, where the puck positions are varied, and the door angle changes.

The direct effect of these goal choices can be seen by visualizing more example rollouts for RIG and Skew-Fit. Due to space constraints, these visuals are in Figure 16 in Appendix B.4. The figure shows that standard RIG only learns to reach states close to the initial position, while Skew-Fit learns to reach the entire state space. A quantitative comparison of the various methods on the pickup task can be seen in Figure 6 , which gives the cumulative total exploration pickups for each method. From the graph, we can see that only Skew-Fit learns to pay attention to the object and therefore consistently increases the rate at which the policy picks up the object during exploration. 

Real-World Vision-Based Robotic Manipulation:

We also demonstrate that Skew-Fit scales well to the real world with a door opening task, Real World Visual Door. See Figure 4 for a picture of this environment. While a number of prior works have studied RL-based learning of door opening Kalakrishnan et al. (2011); Chebotar et al. (2017) , we demonstrate the first method for autonomous learning of door opening without a user-provided, task-specific reward function. As in simulation, we do not provide any goals to the agent and simply let it interact with the door to solve the door opening task from scratch, without any human guidance or reward signal. We train two agents using Skew-Fit with RIG and RIG alone. Unlike in simulation, we cannot measure the difference between the policy's achieved and desired door angle since we do not have access to the true state of the world. Instead, we simply visually denote a binary success/failure for each goal based on whether the last state in the trajectory achieves the target angle. Every seven and a half minutes of interaction time we evaluate on 5 goals and plot the cumulative successes for each method. As Figure 7 shows, standard RIG only starts to open the door after five hours of training. In contrast, Skew-Fit learns to occasionally open the door after three hours of training and achieves a near-perfect success rate after five and a half hours of interaction time, demonstrating that Skew-Fit is a promising technique for solving real world tasks without any human-provided reward function. Videos of Skew-Fit solving this task and the simulated tasks can be viewed on our website. 3Additional Experiments To study the sensitivity of our method to the hyperparameter α, we sweep α across the values [-1, -0.75, -0.5, -0.25, 0] on the simulated image-based tasks. Due to space constraints, the sensitivity analysis over the hyperparameter α is in Appendix B, and the results demonstrate that Skew-Fit works across a large range of values for α, and α = -1 consistently outperform α = 0, where the empirical distribution is not skewed. Additionally, Appendix C provides a complete description our method hyper-parameters, including network architecture and RL algorithm hyperparameters.

7. CONCLUSION:

We presented a formal objective for self-supervised goal-directed exploration, allowing researchers to quantify progress and compare progress when designing algorithms that enable agents to autonomously learn. We also presented Skew-Fit, an algorithm for training a generative model to approximate a uniform distribution over valid states, using data obtained via goal-conditioned reinforcement learning, and our theoretical analysis gives conditions under which Skew-Fit converges to the uniform distribution. When such a model is used to choose goals for exploration and to relabeling goals for training, the resulting method results in much better coverage of the state space, enabling our method to explore effectively. Our experiments show that when we concurrently train a goal-reaching policy using self-generated goals, Skew-Fit produces quantifiable improvements on simulated robotic manipulation tasks, and can be used to learn a door opening skill to reach a 95% success rate directly on a real-world robot, without any human-provided reward supervision.

A PROOFS:

A.1 PROOF OF LEMMA 3.1

Lemma A.1. Let S be a compact set. Define the set of distributions Q = {p : support of p is S}.

Let F : Q → Q be a continuous function and such that H(F(p)) ≥ H(p) with equality if and only if p is the uniform probability distribution on S, U S . Define the sequence of distributions P = (p 1 , p 2 , . . . ) by starting with any p 1 ∈ Q and recursively defining p t+1 = F(p t ).

The sequence P converges to U S .

Proof. The uniform distribution U S is well defined since S is compact. Because S is a compact set, by Prokhorov's Theorem Billingsley (2013) , the set Q is sequentially compact. Thus, P has a convergent subsequence P = (p k1 , p k2 , . . . ) ⊂ P for k 1 < k 2 < . . . that converges to a distribution p * ∈ Q. Because F is continuous, p * must be a fixed point of F since by the convergence mapping theorem, we have that

lim i→∞ p ki = p * =⇒ lim i→∞ F(p ki ) = H(p * )

and so

p * = lim i→∞ p ki = lim i→∞ F(p ki-1 ) = H(p * ).

The only fixed point of F is U S since for any distribution p that is not the uniform distribution, U S , we have that H(F(p)) > H(p) which implies that F(p) = p. Thus, P converges to the only fixed point, U S . Since the entropy cannot decrease, then entropy of the distributions in P must also converge to the entropy of U S . Lastly, since entropy is a continuous function of distribution, P must converge to U S .

A.2 PROOF OF LEMMA 3.2

Lemma A.2. Given two distribution p(x) and q(x) where p q and 0 < Cov p [log p(X), log q(X)]

define the distribution p α as

p α (x) = 1 Z α p(x)q(x) α

where α ∈ R and Z α is the normalizing factor. Let H α (α) be the entropy of p α . Then there exists a constant a > 0 such that for all α ∈ [-a, 0),

EQUATION

Proof. Observe that

{p α : α ∈ [-1, 0]} is a one-dimensional exponential family p α (x) = e αT (x)-A(α)+k(x)

with log carrier density k(x) = log p(x), natural parameter α, sufficient statistic T (x) = log q(x), and log-normalizer A(α) = X e αT (x)+k(x) dx. As shown in Nielsen & Nock (2010) , the entropy of a distribution from a one-dimensional exponential family with parameter α is given by:

H α (α) H(p α ) = A(α) -αA (α) -E pα [k(X)]

The derivative with respect to α is then

d dα H α (α) = -αA (α) - d dα E pα [k(x)] = -αA (α) -E α [k(x)(T (x) -A (α)] = -αVar pα [T (x)] -Cov pα [k(x), T (x)]

where we use the fact that the nth derivative of A(α) give the n central moment, i.e.

A (α) = E pα [T (x)] and A (α) = Var pα [T (x)]. The derivative of α = 0 is d dα H α (0) = -Cov p0 [k(x), T (x)] = -Cov p [log p(x), log q(x)]

which is negative by assumption. Because the derivative at α = 0 is negative, then there exists a constant a > 0 such that for all α ∈ [-a, 0], H α (α) > H α (0) = H(p).

A.3 SIMPLE CASE PROOF:

We prove the convergence directly for the (even more) simplified case when p θ = p(S | p φt ) using a similar technique: Lemma A.3. Assume the set S has finite volume so that its uniform distribution U S is well defined and has finite entropy. Given any distribution p(s) whose support is S, recursively define p t with p 1 = p and

p t+1 (s) = 1 Z t α p t (s) α , ∀s ∈ S

where Z t α is the normalizing constant and α ∈ [0, 1). The sequence (p 1 , p 2 , . . . ) converges to U S , the uniform distribution S.

Proof. If α = 0, then p 2 (and all subsequent distributions) will clearly be the uniform distribution. We now study the case where α ∈ (0, 1).

At each iteration t, define the one-dimensional exponential family {p t θ : θ ∈ [0, 1]} where p t θ is p t θ (s) = e θT (s)-A(θ)+k(s) with log carrier density k(s) = 0, natural parameter θ, sufficient statistic T (s) = log p t (s), and lognormalizer A(θ) = S e θT (s) ds. As shown in Nielsen & Nock (2010) , the entropy of a distribution from a one-dimensional exponential family with parameter θ is given by:

H t θ (θ) H(p t θ ) = A(θ) -θA (θ)

The derivative with respect to θ is then

EQUATION

≤ 0

where we use the fact that the nth derivative of A(θ) is the n central moment, i.e. A (θ) = Var s∼p t θ [T (s)]. Since variance is always non-negative, this means the entropy is monotonically decreasing with θ. Note that p t+1 is a member of this exponential family, with parameter θ = α ∈ (0, 1). So

H(p t+1 ) = H t θ (α) ≥ H t θ (1) = H(p t ) which implies H(p 1 ) ≤ H(p 2 ) ≤ . . . .

This monotonically increasing sequence is upper bounded by the entropy of the uniform distribution, and so this sequence must converge.

The sequence can only converge if d dθ H t θ (θ) converges to zero. However, because α is bounded away from 0, Equation 8 states that this can only happen if

EQUATION

Because p t has full support, then so does p t θ . Thus, Equation 9 is only true if log p t (s) converges to a constant, i.e. p t converges to the uniform distribution. Skew-Fit is a general method that enables exploration when it is infeasible to sample goal states uniformly across the entire state space. While the experiments in Section 6 focused on image-based state spaces, there exists many low-dimensional domains in which we know that the goal space is a subset of R d for some d < n, but the exact goal space is still unknown. This scenario is quite common in domains such as robotics: we know that we want an agent to move the position of its center of mass (CoM), but we do not know the set of valid CoM positions, as this requires knowing the geometry of all potential obstacles a priori. We conduct a series of experiments that study whether Skew-Fit enable effectively exploration in these state spaces containing unknown obstacles.

2D Maze Navigation with Oracle Policy:

To study the impact of Skew-Fit on exploration in isolation of learning a goal-reaching policy, our first set of experiments use a near-perfect policy that reaches the goal state and then takes a step in a random direction (while taking wall-collisions into account). The random step size is Gaussian with a standard deviation of 0.1 units, and the size of each square shown in Figure 8 is 1.8 units. Due to the relatively small step size, the agent cannot rely on random actions to explore the environment and must instead learn to set goals that are progressively farther and farther from the initial state. The first environment is the Four Rooms environment (Sutton et al., 1999), shown in Figure 8 (top) . This environment requires a policy to explore four different rooms, each of which requires passing through a narrow doorway. The maze environment (Figure 8 , bottom) presents a more challenging exploration problem and consists of various long corridors that require setting goals progressively deeper into the maze. In both domains, setting goals near the state state (represented by the yellow star) and taking small actions will result in minimal exploration. To measure exploration, we discretize the space into squares (see Figure 8 for square sizes) and measure what fraction of the squares the agent has ever visited during exploration. We see in Figure 8 that using Skew-Fit significantly improves exploration, whereas training p φ on samples drawn uniformly from the replay buffer (α = 0) results in little exploration.

2D Navigation with Learned Policy Next, we reproduce the 2D navigation environment experiment from Section 6, and replace the oracle goal-reacher with a goal-reaching policy that is simultaneously trained with the goal setter. The policy outputs velocities with maximum speed of one. Evaluation goals are chosen uniformly over the valid states. The hyperparameters for this experiment are given in Table 2 . In Figure 9a , we can see that a policy trained with a goal distribution trained by Skew-Fit consistently learns to reach all goals, whereas a goal distribution trained with uniform sampling, labeled MLE, results in a policy that fails to reach states far from the starting position (the bottom left corner). Quadruped "Ant" Locomotion with Learned Policy Lastly, we test Skew-Fit in an exploration task that requires training a simulated quadruped "ant" robot to navigate to random XY positions in a plane, as shown in Figure 10 . The input to the policy is the joint and velocity of each angle and the reward is the distance to the goal XY-position. While the goal space is known to reside in the XY-plane, the agent does not know about the location of the center obstacle, and so it must still learn about the set of valid goals by controlling its 8 joint actuators. More details of the environment are in Appendix D. We see in Figure 10 that Skew-Fit outperforms prior methods both in terms of learning speed and final performance, demonstrating that Skew-Fit accelerates exploration in non-vision domains that contains unknown goal spaces.

B.2 SENSITIVITY ANALYSIS:

Sensitivity to RL Algorithm In our experiments, we combined Skew-Fit with soft actor critic (SAC) (Haarnoja et al., 2018) . We conduct a set of experiments to test whether Skew-Fit may be used with other RL algorithms for training the goal-conditioned policy. To that end, we replaced SAC with twin delayed deep deterministic policy gradient (TD3) (Fujimoto et al., 2018) and ran the same Skew-Fit experiments on Visual Door, Visual Pusher, and Visual Pickup. In Figure 11 , we see that Skew-Fit performs consistently well with both SAC and TD3, demonstrating that Skew-Fit is beneficial across multiple RL algorithms.

Sensitivity to α Hyperparameter We study the sensitivity of the α hyperparameter by testing values of α ∈ [-1, -0.75, -0.5, -0.25, 0] on the Visual Door and Visual Pusher task. The results are included in Figure 12 and shows that our method is robust to different parameters of α, particularly for the more challenging Visual Pusher task. Also, the method consistently outperform α = 0, which is equivalent to sampling uniformly from the replay buffer. We measure the gradient variance of training a VAE on an unbalanced Visual Door image dataset with Skew-Fit vs Skew-Fit with importance sampling (IS) vs no Skew-Fit (labeled MLE). We construct the imbalanced dataset by rolling out a random policy in the environment and collecting the visual observations. Most of the images contained the door in a closed position; in a few, the door was opened. In Figure 13 , we see that the gradient variance for Skew-Fit with IS is catastrophically large for large values of α. In contrast, for Skew-Fit with SIR, which is what we use in practice, the variance is relatively similar to that of MLE. Additionally we trained three VAE's, one with MLE on a uniform dataset of valid door opening images, one with Skew-Fit on the unbalanced dataset from above, and one with MLE on the same unbalanced dataset. As expected, the VAE that has access to the uniform dataset gets the lowest negative log likelihood score. This is the oracle method, since in practice we would only have access to imbalanced data. As shown in Table 1 , Skew-Fit considerably outperforms MLE, getting a much closer to oracle log likelihood score.

B.4 GOAL AND PERFORMANCE VISUALIZATION:

We visualize the goals sampled from Skew-Fit as well as those sampled when using the prior method, RIG (Nair et al., 2018) . As shown in Figure 14 and Figure 15 , the generative model p φ results in much more diverse samples when trained with Skew-Fit. We we see in Figure 16 , this results in a policy that more consistently reaches the goal image.

C IMPLEMENTATION DETAILS:

C.1 RIG WITH SKEW-FIT SUMMARY Algorithm 2 provides detailed pseudo-code for how we combined our method with RIG. Steps that were removed from the base RIG algorithm are highlighted in blue and steps that were added are highlighted in red. The main differences between the two are (1) sampling exploration goals from the buffer using p skewed instead of the VAE prior, (2) relabeling with replay buffer goals sampled using p skewed instead of from the VAE prior, and (3) training the VAE on replay buffer data data sampled using p skewed instead of uniformly. 

:

We consider the distribution over terminal states in a finite horizon task and believe this work can be extended to infinite horizon stationary distributions.

p q means that p is absolutely continuous with respect to q, i.e. p(s) = 0 =⇒ q(s) = 0.

Anonymous while under review: https://sites.google.com/view/skew-fit-iclr-2020

Figures and Tables:

Figure 1: Left: Robot learning to open a door with Skew-Fit, without any task reward. Right: Samples from a goal distribution when using (a) Skew-Fit and (b) unweighted (ie. uniform) sampling. When used as goals, the diverse samples from Skew-Fit encourage the robot to practice opening the door more frequently.

Figure 3: (Left) The set of final states visited by our agent and MLE over the course of training. In contrast to MLE, our method quickly approaches a uniform distribution over the set of valid states. (Right) The entropy of the sample data distribution, which quickly reaches its maximum for Skew-Fit. The entropy was calculated via discretization onto a 60 by 60 grid.

Figure 4: We evaluate on these continuous control environments. From left to right: Visual Pusher, a simulated pushing task; Visual Door, a door opening task; Visual Pickup, a picking task; and Real World Visual Door, a real world door opening task. All tasks are solved from images and without any task-specific reward. See Appendix D for details.

Figure 5: (Left) Learning curves for simulated continuous control experiments. Lower is better. For each environment and method, we show the mean and standard deviation of 6 seeds and smooth temporally across 25 epochs within each seed. Skew-Fit consistently outperforms RIG and various baselines. See the text for description of each method. (Right) The first column displays example test goal images for each environment.In the next two columns, we display final images reached by Skew-Fit and RIG respectively. Under each image is the final distance in state space to provide a notion of the behavior of each method in the plots.

Figure6: Cumulative total pickups during exploration for each method. The prior methods fail to pay attention to the object and only pick it up at the same rate as the initial policy. In contrast, after seeing the object picked up a few times, Skew-Fit practices picking up the object more often by sampling the appriopriate exploration goals.

Figure 7: Learning curve for Real World Visual Door environment. We visually label a success if the policy opens the door to the target angle by the last state of the trajectory. Skew-Fit results in considerable sample efficiency gains over prior work on this realworld task.

Figure 8: (Top) Coverage over time on the classic 4-room domain, shown on the right. (Bottom) Coverage over time on a more challenging maze domain, shown on the right. In both cases, we see that not using Skew-Fit (α = 0) results in significantly slower learning that primarily stays near the start (yellow star).B ADDITIONAL EXPERIMENTS

(a) Comparison of Skew-Fit vs MLE goal sampling on final distance to goal on RL version of the pointmass environment. Skew-Fit consistently learns to solve the task, while MLE often fails. (b) Heatmaps of final distance to each possible goal location for Skew-Fit and MLE. Skew-Fit learns a good policy over the entire state space, but MLE performs poorly for states far away from the starting position (the bottom left corner).

Figure 10: (Left) Ant navigation environment. (Right) Evaluation on reaching joint and XY position. Policies are trained from state. Reward is L2-norm between the current and target joint angle and XY position concatenated together. We use Skew-Fit to sample goals for relabeling and exploration, and compare to other goal sampling methods. See main paper for description of baselines.

Figure 13: Gradient variance averaged across parameters in last epoch of training VAEs. Values of α less than -1 are numerically unstable for importance sampling (IS), but not for Skew-Fit.

Figure 14: Proposed goals from the VAE for RIG and with Skew-Fit on the Visual Pickup, Visual Pusher, and Visual Door environments. Standard RIG produces goals where the door is closed and the object and puck is in the same position, while RIG + Skew-Fit proposes goals with varied puck positions, occasional object goals in the air, and both open and closed door angles.

Figure 15: Proposed goals from the VAE for RIG (left) and with RIG + Skew-Fit (right) on the Real World Visual Door environment. Standard RIG produces goals where the door is closed while RIG + Skew-Fit proposes goals with both open and closed door angles.

Figure 16: Example reached goals by Skew-Fit and RIG. The first column of each environment section specifies the target goal while the second and third columns show reached goals by Skew-Fit and RIG. Both methods learn how to reach goals close to the initial position, but only Skew-Fit learns to reach the more difficult goals.

In contrast, the other methods have near constant slopes past 40k steps, meaning that they do not continue to learning, and many methods have a near-constant rate of object lifts throughout all of training.

Table content: <table><tr><td>0 20 40 60 Exploration Pickups</td><td>20K</td><td>40K Visual Object Pickup 60K 80K 100K Timesteps</td><td>120K</td><td>RIG + Skew-Fit (Ours) RIG RIG + Rank-Based Priority DISCERN RIG + DISCERN-g RIG + HER RIG + AutoGoal GAN RIG + # Exploration</td></tr></table>

Hyper-parameters used for 2D RL experiment (Figure9a). We expect similar performance had we used SAC.

Table content: <table><tr><td colspan="2">Hyper-parameter</td><td>Value</td></tr><tr><td>Algorithm</td><td colspan="3">TD3 Fujimoto et al. (2018) a</td></tr><tr><td colspan="2"># training batches per time step</td><td>1</td></tr><tr><td colspan="2">Q network hidden sizes</td><td>400, 300</td></tr><tr><td colspan="2">Policy network hidden sizes</td><td>400, 300</td></tr><tr><td colspan="2">Q network and policy activation</td><td>ReLU</td></tr><tr><td colspan="2">Exploration Noise</td><td>None</td></tr><tr><td>RL Batch Size</td><td/><td>1024</td></tr><tr><td colspan="2">Discount Factor</td><td>0.99</td></tr><tr><td>Path length</td><td/><td>25</td></tr><tr><td colspan="2">Reward Scaling</td><td>100</td></tr><tr><td colspan="2">Number of steps per epoch</td><td>5000</td></tr><tr><td>Hyper-parameter</td><td>Value</td><td/><td>Comments</td></tr><tr><td># training batches per time step</td><td>2</td><td colspan="2">Marginal improvements after 2</td></tr><tr><td>Exploration Noise</td><td colspan="2">None (SAC policy is stochastic)</td><td>Did not tune</td></tr><tr><td>RL Batch Size</td><td>1024</td><td colspan="2">smaller batch sizes work as well</td></tr><tr><td>VAE Batch Size</td><td>64</td><td/><td>Did not tune</td></tr><tr><td>Discount Factor</td><td>0.99</td><td/><td>Did not tune</td></tr><tr><td>Reward Scaling</td><td>1</td><td/><td>Did not tune</td></tr><tr><td>Path length</td><td>100</td><td/><td>Did not tune</td></tr><tr><td>Replay Buffer Size</td><td>100000</td><td/><td>Did not tune</td></tr><tr><td>Number of Latents for Estimating Density (N )</td><td>10</td><td colspan="2">Marginal improvements beyond 10</td></tr></table>

General hyper-parameters used for all continuous control experiments.

Table content: <table><tr><td>Hyper-parameter</td><td colspan="4">Visual Pusher Visual Door Visual Pickup Real World Visual Door</td></tr><tr><td>Path Length</td><td>50</td><td>100</td><td>50</td><td>100</td></tr><tr><td>β for β-VAE</td><td>20</td><td>20</td><td>30</td><td>60</td></tr><tr><td>Latent Dimension Size</td><td>4</td><td>16</td><td>16</td><td>16</td></tr><tr><td>α for Skew-Fit</td><td>-1</td><td>-1/2</td><td>-1</td><td>-1/2</td></tr><tr><td>VAE Training Schedule</td><td>2</td><td>1</td><td>2</td><td>1</td></tr><tr><td>Sample Goals From</td><td>p φ</td><td>p skewed</td><td>p skewed</td><td>p skewed</td></tr></table>

Environment specific hyper-parameters

Table content: <table/>

Peer review (iclr2020):
The paper introduces SKEW-FIT, an exploration approach that maximizes the entropy of a distribution of goals such that the agent maximizes state coverage. The paper is well-written and provides an interesting combination of reinforcement learning with imagined goals (RIG) and entropy maximization. The approach is well motivated and simulations are performed on several simulated and real robotics tasks. Some elements were unclear to me: - *We also assume that the entropy of the resulting state distribution H(p(S | p?)) is no less than the entropy of the goal distribution H(p?(S)). Without this assumption, a policy could ignore the goal and stay in a single state, no matter how diverse and realistic the goals are.* How do you ensure this in practice? - In the second paragraph of 2.2, it is written *Note that this assumption does not require that the entropy of p(S | p?) is strictly larger than the entropy of the goal distribution, p?.* Could you please clarify? The experiments are interesting, yet some interpretations might be too strong (see below): - In the first experiment, *Does Skew-Fit Maximize Entropy?*, it is empirically illustrated that the method does result in a high-entropy state exploration. However, it is only compared to one very naive way of exploring and it is not discussed whether other techniques also achieve the same entropy maximization. The last sentences seems to imply that only this technique ends up optimizing the entropy of the state coverage, while I believe that the claim (given the experiment) should only be about the fact it does so faster. - On the comments of Figure 6, the paper mentions that *The other methods only rely on the randomness of the initial policy to occasionally pick up the object, resulting in a near-constant rate of object lifts.* I*m unsure about the interpretation of this sentence given Figure 6 because other methods do not seem to fail entirely when given enough time. - In the experiment *Real-World Vision-Based Robotic Manipulation*, It is written that *a near-perfect success rate [is reached] after five and a half hours of interaction time*, while on the plot it is written 60% cumulative success after 5.5 hours and it is thus not clear where this *5.5 hours* comes from.