Learning to Reach Goals Without Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) algorithms hold the promise of providing a broadly-applicable tool for automating control , and the combination of high-capacity deep neural network models with RL extends their applicability to settings with complex observations and that require intricate policies . However , RL with function approximation , including deep RL , presents a challenging optimization problem . Despite years of research , current deep RL methods are far from a turnkey solution : most popular methods lack convergence guarantees ( Baird , 1995 ; Tsitsiklis & Van Roy , 1997 ) or require prohibitive numbers of samples ( Schulman et al. , 2015 ; Lillicrap et al. , 2015 ) . Moreover , in practice , many commonly used algorithms are extremely sensitive to hyperparameters ( Henderson et al. , 2018 ) . Besides the optimization challenges , another usability challenge of RL is reward function design : although RL automatically determines how to solve the task , the task itself must be specified in a form that the RL algorithm can interpret and optimize . These challenges prompt us to consider whether there might exist a general method for learning behaviors without the need for complex , deep RL algorithms . Imitation learning is an alternative paradigm to RL that provides a simple and straightforward approach for training control policies via standard supervised learning methods . By maximizing the likelihood of good actions provided by an expert demonstrator , supervised imitation learning can produce effective policies without the algorithmic complexities and optimization challenges of RL . Supervised learning algorithms in deep learning have matured to the point of being robust and reliable , and imitation learning algorithms have demonstrated success in acquiring behaviors robustly and reliably from high-dimensional sensory data such as images ( Rajeswaran et al. , 2017 ; Lynch et al. , 2019 ) . The catch is that imitation learning methods require an expert demonstrator – typically a human – to provide a number of demonstrations of optimal behavior . Obtaining expert demonstrations can be challenging ; the large number of demonstrations required limits the scalability of such algorithms . In this paper , we ask : can we use ideas from imitation learning to train effective policies without any expert demonstrations , retaining the benefits of imitation learning , but making it possible to learn goal-directed behavior autonomously from scratch ? The key observation for making progress on this problem is that , in the multi-task setting , trajectories that are generated by a suboptimal policy can serve as optimal examples for other tasks . In particular , in the setting where the tasks correspond to reaching different goal states , every trajectory is a successful demonstration for the state that it actually reaches . Similar observations have been made in prior works as well ( Kaelbling , 1993 ; Andrychowicz et al. , 2017 ; Nair et al. , 2018 ; Mavrin et al. , 2019 ; Savinov et al. , 2018 ) , but have been used to motivate data reuse in off-policy RL or semiparametric methods . Our approach will leverage this idea to obtain near-optimal goal-conditioned policies without RL or reward functions . The algorithm that we study is , at its core , very simple : at each iteration , we run our latest goalconditioned policy , collect data , and then use this data to train a policy with supervised learning . Supervision is obtained by noting that each action that is taken is a good action for reaching the states that actually occurred in future time steps along the same trajectory . This algorithm resembles imitation learning , but is self-supervised . This procedure combines the benefits of goal-conditioned policies with the simplicity of supervised learning , and we theoretically show that this algorithm corresponds to a convergent policy learning procedure . While several prior works have proposed training goal-conditioned policies via imitation learning based on a superficially similar algorithm ( Ding et al. , 2019 ; Lynch et al. , 2019 ) , to our knowledge no prior work proposes a complete policy learning algorithm based on this idea that learns from scratch , without expert demonstrations . This procedure reaps the benefits of off-policy data re-use without the need for learning complex Q functions or value functions . Moreover , we can bootstrap our algorithm with a small number of expert demonstrations , such that it can continue to improve its behavior self supervised , without dealing with the challenges of combining imitation learning with off-policy RL . The main contribution of our work is a complete algorithm for learning policies from scratch via goal-conditioned imitation learning , and to show that this algorithm can successfully train goalconditioned policies . Our theoretical analysis of self-supervised goal-conditioned imitation learning shows that this method optimizes a lower bound on the probability that the agent reaches the desired goal . Empirically , we show that our proposed algorithm is able to learn goal reaching behaviors from scratch without the need for an explicit reward function or expert demonstrations . 2 RELATED WORK . Our work addresses the same problem statement as goal conditioned reinforcement learning ( RL ) ( Andrychowicz et al. , 2017 ; Held et al. , 2018 ; Kaelbling , 1993 ; Nair et al. , 2018 ) , where we aim to learn a policy via RL that can reach different goals . Learning goal-conditioned policies is quite challenging , especially when provided only sparse rewards . This challenge can be partially mitigated by hindsight relabeling approaches that relabel goals retroactively ( Kaelbling , 1993 ; Schaul et al. , 2015 ; Pong et al. , 2018 ; Andrychowicz et al. , 2017 ) . However , even with relabelling , the goalconditioned optimization problem still uses unstable off-policy RL methods . In this work , we take a different approach and leverage ideas from supervised learning and data relabeling to build offpolicy goal reaching algorithms which do not require any explicit RL . This allows GCSL to inherit the benefits of supervised learning without the pitfalls of off-policy RL . While , in theory , on-policy algorithms might be used to solve goal reaching problem as well , their inefficient use of data makes it challenging to apply these approaches to real-world settings . Our algorithm is based on ideas from imitation learning ( Billard et al. , 2008 ; Hussein et al. , 2017 ) via behavioral cloning ( Pomerleau , 1989 ) but it is not an imitation learning method . While it is built on top of ideas from supervised learning , we are not trying to imitate externally provided expert demonstrations . Instead , we build an algorithm which can learn to reach goals from scratch , without explicit rewards . A related line of work ( Hester et al. , 2018 ; Brown et al. , 2019 ) has explored how agents can leverage expert demonstrations to bootstrap the process of reinforcement learning . While GCSL is an algorithm to learn goal-reaching policies from scratch , it lends itself naturally to bootstrapping from demonstrations . As we show in Section 5.4 , GCSL can easily incorporate demonstrations into off-policy learning and continue improving , avoiding many of the challenges described in Kumar et al . ( 2019b ) . Recent imitation learning algorithms propose methods that are closely related to GCSL . Lynch et al . ( 2019 ) aim to learn general goal conditioned policies from “ play ” data collected by a human demonstrator , and Ding et al . ( 2019 ) perform goal-conditioned imitation learning where expert goal-directed demonstrations are relabeled for imitation learning . However , neither of these methods are iterative , and both require human-provided expert demonstrations . Our method instead iteratively performs goal-conditioned behavioral cloning , starting from scratch . Our analysis shows that performing such iterated imitation learning on the policy ’ s own sampled data actually optimizes a lower bound on the probability of successfully reaching goals , without the need for any expert demonstrations . The cross-entropy method ( Mannor et al. , 2003 ) , self-imitation learning ( Oh et al. , 2018 ) , rewardweighted regression ( Peters & Schaal , 2007 ) , path-integral policy improvement ( Theodorou et al. , 2010 ) , reward-augmented maximum likelihood ( Norouzi et al. , 2016 ; Nachum et al. , 2016 ) , and proportional cross-entropy method ( Goschin et al. , 2013 ) selectively weight policies or trajectories by their performance during learning , as measured by then environment ’ s reward function . While these may appear procedurally similar to GCSL , our method is fully self-supervised , as it does not require a reward function , and is applicable in the goal-conditioned setting . Additionally , our algorithm continues to perform well in the purely off-policy setting , where no new data is collected , a key difference from other algorithms ( Lynch et al. , 2019 ; Ding et al. , 2019 ) . A few works similar to ours in spirit study the problem of learning goal-conditioned policy without external supervision . Zero-shot visual imitation uses an inverse model with forward consistency to learn from novelty seeking behavior , but lacks convergence guarantees and requires learning a complex inverse model Pathak et al . ( 2018 ) . Semi-parametric methods ( Savinov et al. , 2018 ; Eysenbach et al. , 2019 ) learn a policy similar to ours but do so by building a connectivity graph over the visited states in order to navigate environments , which requires large memory storage and computation time that increases with the number of states . 3 PRELIMINARIES . Goal Reaching We consider the goal reaching in an environment defined by the tuple 〈S , A , T , ρ ( s0 ) , T , p ( g ) 〉 . S andA correspond to the state and action spaces respectively , T ( s′|s , a ) to the transition kernel , ρ ( s0 ) to the initial state distribution , T the horizon length , and p ( g ) to the distribution over goal states g ∈ S . We aim to find a time-varying goal-conditioned policy π ( ·|s , g , h ) : S × S × [ T ] → ∆ ( A ) , where ∆ ( A ) is the probability simplex over the action space A and h is the remaining horizon . We will say that a policy is optimal if it maximizes the probability the specified goal is reached at the end of the episode : J ( π ) = Eg∼p ( g ) [ Pπg ( sT = g ) ] . ( 1 ) It is important to note here that we are not considering the notion of optimality to be finding the shortest path to the goal , but merely saying that the trajectory must reach the goal at the end of T time-steps . This problem can equivalently be cast in reinforcement learning . The modified state space S ′ = S × S × [ T ] contains the current state , goal , and the remaining horizon ; the modified transition kernel T ′ ( ( s′ , g′ , h′ ) | ( s , g , h ) , a ) = T ( s′ | s , a ) · 1 ( d = d′ , h′ = h − 1 ) appropriately handles the modified state space ; and the reward function r ( ( s , g , h ) ) = 1 ( s = g , h = 0 ) depends on both the goal and the time step . Because of the special structure of this formulation , off-policy RL methods can relabel an observed transition ( ( s , g , h ) , a , ( s′ , g , h− 1 ) ) to that of a different goal g′ and different horizon h′ like ( ( s , g′ , h′ ) , a , ( s′ , g′ , h′ − 1 ) ) . A common approach is to relabel trajectories with the goal they actually reached instead of the commanded goal , and often referred to as hindsight experience replay ( Andrychowicz et al. , 2017 ; Nair et al. , 2018 ) . Imitation Learning We consider algorithms for goal-reaching that use behavior cloning , a standard method for imitation learning . In behavior cloning for goal-conditioned policies , an expert policy provides demonstrations for reaching some target goals at the very last timestep , and we aim to find a policy that best predicts the expert actions from the observations . More formally , given a dataset D = { { s00 , a00 , s01 , a01 , .... s0T } , { s10 , a10 , s11 , a11 , .... s1T } , ... } , and a set of stochastic , time-varying policies Π , the behavior-cloned policy corresponds to πBC = arg max π∈Π E ( sT−h , aT−h , sT ) ∼D [ log π ( aT−h|sT−h , sT , h ) ] . Goal-conditioned imitation learning ( Lynch et al. , 2019 ) investigate a similar formalism that makes an additional assumption on the quality of the expert demonstrations : that the expert is optimal not just for reaching sT , but also optimal for reaching all the states s1 , . . . sT−1 preceding it . This corresponding policy is πGCIL = arg max π∈Π E ( st , at , st+h ) ∼D [ log πθ ( at|st , st+h , h ) ] for t , h > 0 and t+ h ≤ T . ( 2 ) Note that Lynch et al . ( 2019 ) implement a special case of this objective where the policy is independent of the horizon . In the next section , we will discuss how repeatedly alternating between data collection and goal-conditioned imitation learning can be used to learn a goal-reaching policy . Perhaps surprisingly , this procedure optimizes the objective in Equation 1 , without relying on expert demonstrations .

Summary: This paper proposes a method to learn to reach goals in an RL environment. The method is based on principles of imitation learning. For instance, beginning with an arbitrary policy that samples a sequence of state-action pairs, in the next iteration, the algorithm treats the previous policy as an expert by relabeling its ending state as a goal. The paper shows that the method is theoretically sound and effective empirically for goal-achieving tasks.