Learning to Reach Goals Without Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) algorithms hold the promise of providing a broadly-applicable tool for automating control , and the combination of high-capacity deep neural network models with RL extends their applicability to settings with complex observations and that require intricate policies . However , RL with function approximation , including deep RL , presents a challenging optimization problem . Despite years of research , current deep RL methods are far from a turnkey solution : most popular methods lack convergence guarantees ( Baird , 1995 ; Tsitsiklis & Van Roy , 1997 ) or require prohibitive numbers of samples ( Schulman et al. , 2015 ; Lillicrap et al. , 2015 ) . Moreover , in practice , many commonly used algorithms are extremely sensitive to hyperparameters ( Henderson et al. , 2018 ) . Besides the optimization challenges , another usability challenge of RL is reward function design : although RL automatically determines how to solve the task , the task itself must be specified in a form that the RL algorithm can interpret and optimize . These challenges prompt us to consider whether there might exist a general method for learning behaviors without the need for complex , deep RL algorithms . Imitation learning is an alternative paradigm to RL that provides a simple and straightforward approach for training control policies via standard supervised learning methods . By maximizing the likelihood of good actions provided by an expert demonstrator , supervised imitation learning can produce effective policies without the algorithmic complexities and optimization challenges of RL . Supervised learning algorithms in deep learning have matured to the point of being robust and reliable , and imitation learning algorithms have demonstrated success in acquiring behaviors robustly and reliably from high-dimensional sensory data such as images ( Rajeswaran et al. , 2017 ; Lynch et al. , 2019 ) . The catch is that imitation learning methods require an expert demonstrator – typically a human – to provide a number of demonstrations of optimal behavior . Obtaining expert demonstrations can be challenging ; the large number of demonstrations required limits the scalability of such algorithms . In this paper , we ask : can we use ideas from imitation learning to train effective policies without any expert demonstrations , retaining the benefits of imitation learning , but making it possible to learn goal-directed behavior autonomously from scratch ? The key observation for making progress on this problem is that , in the multi-task setting , trajectories that are generated by a suboptimal policy can serve as optimal examples for other tasks . In particular , in the setting where the tasks correspond to reaching different goal states , every trajectory is a successful demonstration for the state that it actually reaches . Similar observations have been made in prior works as well ( Kaelbling , 1993 ; Andrychowicz et al. , 2017 ; Nair et al. , 2018 ; Mavrin et al. , 2019 ; Savinov et al. , 2018 ) , but have been used to motivate data reuse in off-policy RL or semiparametric methods . Our approach will leverage this idea to obtain near-optimal goal-conditioned policies without RL or reward functions . The algorithm that we study is , at its core , very simple : at each iteration , we run our latest goalconditioned policy , collect data , and then use this data to train a policy with supervised learning . Supervision is obtained by noting that each action that is taken is a good action for reaching the states that actually occurred in future time steps along the same trajectory . This algorithm resembles imitation learning , but is self-supervised . This procedure combines the benefits of goal-conditioned policies with the simplicity of supervised learning , and we theoretically show that this algorithm corresponds to a convergent policy learning procedure . While several prior works have proposed training goal-conditioned policies via imitation learning based on a superficially similar algorithm ( Ding et al. , 2019 ; Lynch et al. , 2019 ) , to our knowledge no prior work proposes a complete policy learning algorithm based on this idea that learns from scratch , without expert demonstrations . This procedure reaps the benefits of off-policy data re-use without the need for learning complex Q functions or value functions . Moreover , we can bootstrap our algorithm with a small number of expert demonstrations , such that it can continue to improve its behavior self supervised , without dealing with the challenges of combining imitation learning with off-policy RL . The main contribution of our work is a complete algorithm for learning policies from scratch via goal-conditioned imitation learning , and to show that this algorithm can successfully train goalconditioned policies . Our theoretical analysis of self-supervised goal-conditioned imitation learning shows that this method optimizes a lower bound on the probability that the agent reaches the desired goal . Empirically , we show that our proposed algorithm is able to learn goal reaching behaviors from scratch without the need for an explicit reward function or expert demonstrations . 2 RELATED WORK . Our work addresses the same problem statement as goal conditioned reinforcement learning ( RL ) ( Andrychowicz et al. , 2017 ; Held et al. , 2018 ; Kaelbling , 1993 ; Nair et al. , 2018 ) , where we aim to learn a policy via RL that can reach different goals . Learning goal-conditioned policies is quite challenging , especially when provided only sparse rewards . This challenge can be partially mitigated by hindsight relabeling approaches that relabel goals retroactively ( Kaelbling , 1993 ; Schaul et al. , 2015 ; Pong et al. , 2018 ; Andrychowicz et al. , 2017 ) . However , even with relabelling , the goalconditioned optimization problem still uses unstable off-policy RL methods . In this work , we take a different approach and leverage ideas from supervised learning and data relabeling to build offpolicy goal reaching algorithms which do not require any explicit RL . This allows GCSL to inherit the benefits of supervised learning without the pitfalls of off-policy RL . While , in theory , on-policy algorithms might be used to solve goal reaching problem as well , their inefficient use of data makes it challenging to apply these approaches to real-world settings . Our algorithm is based on ideas from imitation learning ( Billard et al. , 2008 ; Hussein et al. , 2017 ) via behavioral cloning ( Pomerleau , 1989 ) but it is not an imitation learning method . While it is built on top of ideas from supervised learning , we are not trying to imitate externally provided expert demonstrations . Instead , we build an algorithm which can learn to reach goals from scratch , without explicit rewards . A related line of work ( Hester et al. , 2018 ; Brown et al. , 2019 ) has explored how agents can leverage expert demonstrations to bootstrap the process of reinforcement learning . While GCSL is an algorithm to learn goal-reaching policies from scratch , it lends itself naturally to bootstrapping from demonstrations . As we show in Section 5.4 , GCSL can easily incorporate demonstrations into off-policy learning and continue improving , avoiding many of the challenges described in Kumar et al . ( 2019b ) . Recent imitation learning algorithms propose methods that are closely related to GCSL . Lynch et al . ( 2019 ) aim to learn general goal conditioned policies from “ play ” data collected by a human demonstrator , and Ding et al . ( 2019 ) perform goal-conditioned imitation learning where expert goal-directed demonstrations are relabeled for imitation learning . However , neither of these methods are iterative , and both require human-provided expert demonstrations . Our method instead iteratively performs goal-conditioned behavioral cloning , starting from scratch . Our analysis shows that performing such iterated imitation learning on the policy ’ s own sampled data actually optimizes a lower bound on the probability of successfully reaching goals , without the need for any expert demonstrations . The cross-entropy method ( Mannor et al. , 2003 ) , self-imitation learning ( Oh et al. , 2018 ) , rewardweighted regression ( Peters & Schaal , 2007 ) , path-integral policy improvement ( Theodorou et al. , 2010 ) , reward-augmented maximum likelihood ( Norouzi et al. , 2016 ; Nachum et al. , 2016 ) , and proportional cross-entropy method ( Goschin et al. , 2013 ) selectively weight policies or trajectories by their performance during learning , as measured by then environment ’ s reward function . While these may appear procedurally similar to GCSL , our method is fully self-supervised , as it does not require a reward function , and is applicable in the goal-conditioned setting . Additionally , our algorithm continues to perform well in the purely off-policy setting , where no new data is collected , a key difference from other algorithms ( Lynch et al. , 2019 ; Ding et al. , 2019 ) . A few works similar to ours in spirit study the problem of learning goal-conditioned policy without external supervision . Zero-shot visual imitation uses an inverse model with forward consistency to learn from novelty seeking behavior , but lacks convergence guarantees and requires learning a complex inverse model Pathak et al . ( 2018 ) . Semi-parametric methods ( Savinov et al. , 2018 ; Eysenbach et al. , 2019 ) learn a policy similar to ours but do so by building a connectivity graph over the visited states in order to navigate environments , which requires large memory storage and computation time that increases with the number of states . 3 PRELIMINARIES . Goal Reaching We consider the goal reaching in an environment defined by the tuple 〈S , A , T , ρ ( s0 ) , T , p ( g ) 〉 . S andA correspond to the state and action spaces respectively , T ( s′|s , a ) to the transition kernel , ρ ( s0 ) to the initial state distribution , T the horizon length , and p ( g ) to the distribution over goal states g ∈ S . We aim to find a time-varying goal-conditioned policy π ( ·|s , g , h ) : S × S × [ T ] → ∆ ( A ) , where ∆ ( A ) is the probability simplex over the action space A and h is the remaining horizon . We will say that a policy is optimal if it maximizes the probability the specified goal is reached at the end of the episode : J ( π ) = Eg∼p ( g ) [ Pπg ( sT = g ) ] . ( 1 ) It is important to note here that we are not considering the notion of optimality to be finding the shortest path to the goal , but merely saying that the trajectory must reach the goal at the end of T time-steps . This problem can equivalently be cast in reinforcement learning . The modified state space S ′ = S × S × [ T ] contains the current state , goal , and the remaining horizon ; the modified transition kernel T ′ ( ( s′ , g′ , h′ ) | ( s , g , h ) , a ) = T ( s′ | s , a ) · 1 ( d = d′ , h′ = h − 1 ) appropriately handles the modified state space ; and the reward function r ( ( s , g , h ) ) = 1 ( s = g , h = 0 ) depends on both the goal and the time step . Because of the special structure of this formulation , off-policy RL methods can relabel an observed transition ( ( s , g , h ) , a , ( s′ , g , h− 1 ) ) to that of a different goal g′ and different horizon h′ like ( ( s , g′ , h′ ) , a , ( s′ , g′ , h′ − 1 ) ) . A common approach is to relabel trajectories with the goal they actually reached instead of the commanded goal , and often referred to as hindsight experience replay ( Andrychowicz et al. , 2017 ; Nair et al. , 2018 ) . Imitation Learning We consider algorithms for goal-reaching that use behavior cloning , a standard method for imitation learning . In behavior cloning for goal-conditioned policies , an expert policy provides demonstrations for reaching some target goals at the very last timestep , and we aim to find a policy that best predicts the expert actions from the observations . More formally , given a dataset D = { { s00 , a00 , s01 , a01 , .... s0T } , { s10 , a10 , s11 , a11 , .... s1T } , ... } , and a set of stochastic , time-varying policies Π , the behavior-cloned policy corresponds to πBC = arg max π∈Π E ( sT−h , aT−h , sT ) ∼D [ log π ( aT−h|sT−h , sT , h ) ] . Goal-conditioned imitation learning ( Lynch et al. , 2019 ) investigate a similar formalism that makes an additional assumption on the quality of the expert demonstrations : that the expert is optimal not just for reaching sT , but also optimal for reaching all the states s1 , . . . sT−1 preceding it . This corresponding policy is πGCIL = arg max π∈Π E ( st , at , st+h ) ∼D [ log πθ ( at|st , st+h , h ) ] for t , h > 0 and t+ h ≤ T . ( 2 ) Note that Lynch et al . ( 2019 ) implement a special case of this objective where the policy is independent of the horizon . In the next section , we will discuss how repeatedly alternating between data collection and goal-conditioned imitation learning can be used to learn a goal-reaching policy . Perhaps surprisingly , this procedure optimizes the objective in Equation 1 , without relying on expert demonstrations .

Summary: This work presents the goal-conditioned supervised learning algorithm (GCSL), which learns goal conditioned policies using only behavioral-cloning of the agent's own actions.  The intuition behind the algorithm is the goal of an observed trajectory can be identified after the fact, by simply looking at the states reached during that trajectory.  GCSL treats each executed action as a sample from the expert policy conditioned on each of the states reached after that action is taken.  Given a distribution over goal states, GCSL alternates between executing its current goal-conditioned policy on randomly selected goals, and learning to imitate the generated actions conditioned on the states they actually reached.  Experimental results demonstrate superior performance against a base (non-goal conditioned) RL algorithm (TRPO), and against another approach to learning goal-conditioned polices (TD3-HER), on a relatively diverse set of control problems.