Generative predecessor models for sample-efficient imitation learning

1 INTRODUCTION . Training or programming agents to act intelligently in unstructured and sequential environments is a difficult and central challenge in the field of artificial intelligence . Imitation learning provides an avenue to tackle this challenge by allowing agents to learn from human teachers , which constitutes a natural way for experts to describe the desired behavior and provides an efficient learning signal for the agent . It is thus no surprise that imitation learning has enabled great successes on robotic ( Chernova and Thomaz , 2014 ) as well as software domains ( e.g . Aytar et al . ( 2018 ) ) . Yet , key challenges in the field are diverse and include questions such as how to learn from observations alone ( e.g . Aytar et al . ( 2018 ) ) , learning the correspondence between the expert ’ s demonstrations and the agent ’ s observations ( e.g . Sermanet et al . ( 2017 ) ) as well as the question of how to integrate imitation learning with other approaches such as reinforcement learning ( e.g . Vecerik et al . ( 2017 ) ) . However , at the core of the imitation learning problems lies the challenge of utilizing a given set of demonstrations to match the expert ’ s behavior as closely as possible . In this paper , we approach this problem considering the setting where the set of expert demonstrations is given up-front and the dynamics of the environment can only be observed through interaction . In principle , imitation learning could be seen as a supervised learning problem , where the demonstrations are used to learn a mapping from observed states to actions . This solution approach is known as behavioral cloning . However , it has long been known that the sequential structure of the task admits more effective solutions . In particular , the assumptions made in supervised learning are restrictive and don ’ t allow the agent to reason about the effect of its actions on it ’ s future inputs . As a result , errors and deviations from demonstrated behavior tend to accumulate over time as small mistakes lead the agent to parts of the observation space that the expert has not explored ( Ross and Bagnell , 2010 ) . In this work , we propose a novel imitation learning algorithm , Generative Predecessor Models for Imitation Learning ( GPRIL ) , based on a simple core insight : Augmenting the training set with state-action pairs that are likely to eventually lead the agent to states demonstrated by the expert is an effective way to train corrective behavior and to prevent accumulating errors . Recent advances in generative modeling , such as Goodfellow et al . ( 2014 ) ; Kingma and Welling ( 2013 ) ; Van Den Oord et al . ( 2016b ; a ) ; Dinh et al . ( 2016 ) , have shown great promise at modeling complex distributions and can be used to reason probabilistically about such state-action pairs . ∗This work was carried out at DeepMind . Specifically , we propose to utilize Masked Autoregressive Flows ( Papamakarios et al. , 2017 ) to model long-term predecessor distributions , i.e . distributions over state-action pairs which are conditioned on a state that the agent will see in the future . Predecessor models have a long history in reinforcement learning ( e.g . Peng and Williams ( 1993 ) ) with recent approaches using deep networks to generate off-policy transitions ( Edwards et al. , 2018 ; Pan et al. , 2018 ) or to reinforce behavior leading to high-value states ( Goyal et al. , 2018 ) . Here , we use predecessor models to derive a principled approach to state-distribution matching and propose the following imitation learning loop : 1 . Interact with the environment and observe state , action as well as a future state . To encode long-term corrective behavior , these states should be multiple steps apart . 2 . Train a conditional generative model to produce samples like the observed state-action pair when conditioned on the observed future state . 3 . Train the agent in a supervised way , augmenting the training set using data drawn from the model conditioned on demonstrated states . The additional training data shows the agent how to reach demonstrated states , enabling it to recover after deviating from expert behavior . In the above , we laid out the sketch of an algorithm that intuitively learns to reason about the states it will observe in the future . In section 3 , we derive this algorithm from first principles as a maximum likelihood approach to matching the state-action distribution of the agent to the expert ’ s distribution . In section 4 , we compare our approach to a state-of-the-art imitation learning method ( Ho and Ermon , 2016 ) and show that it matches or outperforms this baseline on our domains while being significantly more sample efficient . Furthermore , we show that GPRIL can learn using demonstrated states alone , allowing for a wider variety of methods to be used to record demonstrations . Together these properties are sufficient to allow GPRIL to be applied in real-world settings , which we demonstrate in section 4.3 . To our knowledge this is the first instance of dynamic , contact-rich and adaptive behavior being taught solely using the kinesthetic-teaching interface of a collaborative robot , without resorting to tele-operation , auxiliary reward signals , or manual task-decomposition . 2 BACKGROUND . 2.1 MARKOV DECISION PROCESSES WITHOUT REWARDS . As is usual , we model the problem as a Markov decision process without reward . That is , given state and action sets S , A , the agent is observing states s ∈ S and taking actions a ∈ A . In this work , we use s and a to refer to states and actions observed during self-supervision and s ∈ S and a ∈ A to refer to target and demonstration states and actions . We furthermore use superscripts s ( i ) , a ( i ) to refer to specific instances , e.g . specific demonstrated state-action pairs , and subscripts , e.g . st , at , to indicate temporal sequences . The observed transitions are guided by the Markovian dynamics of the environment and the probability of transitioning from state s to state s′ by taking action a is denoted as p ( st+1 = s′|st = s , at = a ) . The agent ’ s behavior is defined by a stationary parametric policy πθ ( a|s ) while the expert ’ s behavior is modeled by a stationary distribution π∗ ( a|s ) . We denote as dπt ( s ) the probability of observing state s at time-step t when following policy π . Under the usual ergodicity assumptions , each such policy induces a unique stationary distribution of observed states dπ ( s ) = limt→∞ dπt ( s ) as well as a stationary joint state-action distribution ρπ ( s , a ) : = π ( a|s ) dπ ( s ) . Furthermore , we use qπt to refer to the dynamics of the time reversed Markov chain induced by a particular policy π at time-step t qπt ( st = s , at = a|st+1 = s′ ) = dt+1 ( s′ ) −1dt ( s ) π ( at|st ) p ( st+1 = s′|st = s , at = a ) ( 1 ) and define qπ ( st = s , at = a|st+1 = s′ ) : = limt→∞ qπt ( st = s , at = a|st+1 = s′ ) . For the purposes of this work , we handle the episodic case with clear termination conditions by adding artificial transitions from terminal states to initial states . This creates a modified , ergodic MDP with identical state-distribution and allows us to assume arbitrarily large t such that qπt = q π . Finally , we extend this notation to multi-step transitions by writing qπ ( st = s , at = a|st+j = s′ ) . 2.2 IMITATION LEARNING . In this work , we are considering two settings of imitation learning . In the first setting , the agent is given a set of observed states s ( 1 ) , s ( 2 ) , · · · , s ( N ) and observed corresponding actions a ( 1 ) , a ( 2 ) , · · · , a ( N ) as expert demonstrations . The goal in this setting is to learn a policy πθ that matches the expert ’ s behavior as closely as possible . In the second setting , the agent is given the states observed by the expert but is not aware of the actions the expert has taken . Recent years have seen heightened interest in a related setting where to goal is to track expert state trajectories ( Zhu et al. , 2018 ; Peng et al. , 2018 ; Pathak et al. , 2018 ) . These approaches do not learn general policies that can adapt to unseen situations . A straightforward approach to train general policies in the first setting , usually referred to as behavioral cloning , is to treat the task as a supervised learning problem ( e.g . Pomerleau ( 1989 ) ) . However , as outlined in section 1 , predictions made by πθ ( a|s ) influence future observations thus violating a key assumption of supervised learning , which states that inputs are drawn from an i.i.d . distribution . This has formally been analyzed by Ross et al . who introduce a family of algorithms ( e.g . Ross and Bagnell ( 2010 ) ; Ross et al . ( 2011 ) ) that provably avoid this issue . However , these approaches require the expert to continuously provide demonstrations and thus are not applicable when the set of demonstrations is assumed to be fixed . A popular avenue of research that considers this setting is inverse reinforcement learning ( IRL ) . Inverse reinforcement learning ( Ng and Russell , 2000 ) aims to learn a reward function for which π∗ is optimal and thus captures the intent of the expert . The arguably most successful approach to IRL aims to match the state-action distribution to that of the demonstrations ( Ziebart et al. , 2008 ) with recent approaches extending these ideas to the model free case ( Boularias et al. , 2011 ; Finn et al. , 2016 ; Fu et al. , 2018 ) . However , inverse reinforcement learning is indirect and ill-defined as many reward functions induce the same behavior . Recently , methods have been proposed that aim to match state-action distributions directly and achieve state-of-the-art result without learning a reward function first . Generative Adversarial Imitation Learning ( GAIL ) ( Ho and Ermon , 2016 ) uses an adversarial objective , training a discriminator to identify demonstrations and using TRPO to train a policy that fools the discriminator . While GAIL is able to achieve impressive results , the adversarial objective can make the learning procedure unstable and unpredictable . This is especially true when parts of the state-space are not under the agent ’ s control , yielding a setting resembling a conditional GAN ( Mirza and Osindero , 2014 ) , which are prone to issues such mode collapse ( Odena et al. , 2017 ) . We compare our approach with GAIL in section 4 . State Aware Imitation Learning ( SAIL ) ( Schroecker and Isbell , 2017 ) is an alternative method that aims to learn the gradient of the state-action distribution using a temporal-difference update rule . This approach is able to avoid instabilities prompted by the adversarial learning rule but is only applicable to policies with a small number of parameters where learning a representation of the gradient is feasible . In this work , we follow a gradient descent approach similar to SAIL but estimate the gradient without representing it explicitly by a neural network . These methods can also be used to match state-distributions without actions . While the order of states is unspecified and the induced policy is therefore not unique , including transition information such as velocities can allow the agent to learn solely from expert state trajectories nonetheless .

Summary: The submission builds up on recent advances in neural density estimation to develop a new algorithm for imitation learning based on a probabilistic model for predecessor state dynamics. In particular, the method trains masked autoregressive flows as a probabilistic model for state action pairs conditioned on future states. This model is used to estimate the gradient of the stationary distribution of a policies visited states. Finally, the proposed objective uses this estimate and the gradient of the log likelihood of expert actions under the policy to maximise the similarity of the expert’s and agent’s stationary state-action distributions.