Imitation Learning by Reinforcement Learning

Imitation learning algorithms learn a policy from demonstrations of expert behavior . We show that , for deterministic experts , imitation learning can be done by reduction to reinforcement learning with a stationary reward . Our theoretical analysis both certifies the recovery of expert reward and bounds the total variation distance between the expert and the imitation learner , showing a link to adversarial imitation learning . We conduct experiments which confirm that our reduction works well in practice for continuous control tasks . 1 INTRODUCTION . Typically , reinforcement learning ( RL ) assumes access to a pre-specified reward and then learns a policy maximizing the expected average of this reward along a trajectory . However , specifying rewards is difficult for many practical tasks ( Atkeson & Schaal , 1997 ; Zhang et al. , 2018 ; Ibarz et al. , 2018 ) . In such cases , it is convenient to instead perform Imitation Learning ( IL ) , learning a policy from expert demonstrations . There are two major categories of Imitation Learning algorithms : behavioral cloning and inverse reinforcement learning . Behavioral cloning learns the policy by supervised learning on expert data , but is not robust to training errors , failing in settings where expert data is limited ( Ross & Bagnell , 2010 ) . Inverse reinforcement learning ( IRL ) achieves improved performance on limited data by constructing reward signals and calling an RL oracle to maximize these rewards ( Ng et al. , 2000 ) . The most versatile IRL method is adversarial IL ( Ho & Ermon , 2016 ; Li et al. , 2017 ; Ghasemipour et al. , 2020 ) , which minimizes a divergence between the distribution of data produced by the agent and provided by the expert . Adversarial IL learns a representation and a policy simultaneously by using a non-stationary reward obtained from a discriminator network . However , training adversarial IL combines two components which are hard to stabilize : a discriminator network , akin to the one used in GANs , as well as a policy , typically learned with an RL algorithm with actor and critic networks . This complexity makes the training process very brittle . There is a clear need for imitation learning algorithms that are simpler and easier to deploy . To address this need , Wang et al . ( 2019 ) proposed to reduce imitation learning to a single instance of reinforcement learning problem , where reward is defined to be one for state-action pairs from the expert trajectory and zero for other state-action pairs . A closely related , but not identical , algorithm has been proposed by Reddy et al . ( 2020 ) ( we describe the differences in Section 5 ) . However , while empirical performance of these approaches has been good , they enjoy no performance guarantees at all , even in the asymptotic setting where expert data is infinite . Contributions We fill in this missing justification for this algorithm , providing the needed theoretical analysis . Specifically , in Sections 3 and 4 , we show a total variation bound between the expert policy and the imitation policy , providing a high-probability performance guarantee for a finite dataset of expert data and linking the reduction to adversarial imitation learning algorithms . For stochastic experts , we describe how the reduction fails , completing the analysis . Moreover , in Section 6 , we empirically evaluate the performance of the reduction as the amount of available expert data varies . 2 PRELIMINARIES . Markov Decision Process An average-reward Markov Decision Process ( Puterman , 2014 ; Feinberg & Shwartz , 2012 ) is a tuple ( S , A , T , R , s1 ) , where S is a state space , A is the action space , T : S ×A→ P ( S ) is the transition model , R : S ×A→ [ 0 , 1 ] is a bounded reward function and s1 is the initial state . Here , we write P ( X ) to denote probability distributions over a set X . A stationary policy π : S → P ( A ) maps environment states to distributions over actions . A policy π induces a Markov chain Pπ . In the theoretical part of the paper , we treat MDPs with finite state and action spaces . Given a starting state s1 of the MDP , and a policy π , the limiting distribution over states is defined as ρπS ( s ) = 1 ( s1 ) > lim N→∞ 1 N N∑ i=0 P iπ , ( 1 ) where 1 ( s1 ) denotes the indicator vector . We adopt the convention that the subscript S indicates a distribution over states and a no subscript indicates a distribution over state-action pairs . We denote ρπ ( s , a ) = ρπS ( s ) π ( a|s ) . While the limit in equation 1 is guaranteed to exist for all finite MDPs ( Puterman , 2014 ) , without requiring ergodicity , in this paper we consider policies that induce ergodic chains . Correspondingly , we drop the dependence on the initial state from the notation . The expected per-step reward of π is defined as V π = lim N→∞ Eπ [ JN/N | s1 ] = EρπR ( s , a ) , ( 2 ) where JN = ∑N t=1R ( st , at ) is the total return . In this paper we consider undiscounted MDPs because they are enough to model the essential properties of the imitation learning problem while being conceptually simpler than discounted MDPs . However , we believe that similar results could be obtained for discounted MDPs . Expert Dataset Assume that the expert follows an ergodic policy πE . Denote the corresponding distribution over expert state-action pairs as ρE ( s , a ) = ρπE ( s , a ) , where we dropped the dependency on the initial state by ergodicity . Consider a finite trajectory s1 , a1 , s2 , a2 , . . . , sN , aN . Denote by ρ̂ ( s , a ) = 1N ∑N t=1 1 ( ( s , a ) = ( st , at ) ) the histogram ( empirical distribution ) of data seen along this trajectory and denote by ρ̂S ( s ) = 1N ∑N t=1 1 ( s = st ) the corresponding histogram over the states . Denote by D the support of ρ̂ , i.e . the set of state-action pairs visited by the expert . Total Variation Consider probability distributions defined on a discrete set X . The total variation distance between distributions is defined as ‖ρ1 − ρ2‖TV = sup M⊆X |ρ1 ( M ) − ρ2 ( M ) | . ( 3 ) In our application , the set X is either the set of MDP states S or the set of state state-action pairs S × A . Below , we restate in our notation standard results about the total variation distance . See appendix D for proofs . Lemma 1 . If for any vector v ∈ [ 0 , 1 ] |X| it holds that Eρ1 [ v ] − Eρ2 [ v ] ≤ , then we have ‖ρ1 − ρ2‖TV ≤ . Lemma 2 . If ‖ρ1 − ρ2‖TV ≤ , then we have |Eρ1 [ v ] − Eρ2 [ v ] | ≤ 2 for any vector v ∈ [ 0 , 1 ] |X| . Imitation Learning Learning to imitate means obtaining a policy that mimics expert behavior . This can be formalized in two ways . First , we can seek an imitation policy πI which obtains a expected per-step reward that is -close to the expert on any reward signal bounded in the interval [ 0 , 1 ] . Denote the limiting distribution of state-action tuples generated by the imitation learner with ρI ( s , a ) . Formally , we want to satisfy ∀R ∈ S ×A→ [ 0 , 1 ] . EρE [ R ] − EρI [ R ] ≤ , ( 4 ) where is a small constant . Second , we can seek to ensure that the distributions of state-action pairs generated by the expert and the imitation learner are similar ( Ho & Ermon , 2016 ; Finn et al. , 2016 ) . In particular , if we measure the divergence between distributions with total variation , we want to ensure ‖ρE − ρI‖TV ≤ . ( 5 ) Lemmas 1 and 2 show that the equations 4 and 5 are closely related . In other words , attaining expected per-step reward close to the expert ’ s is the same ( up to a multiplicative constant ) as closeness in total variation between the state-action distribution of the expert and the imitation learner . We provide further background on adversarial imitation learning and other divergences in Section 5 . Markov Chains When coupled with the dynamics of the MDP , a policy induces a Markov chain , In this paper , we focus our attention on chains which are irreducible and aperiodic . Crucially , such chains approach the stationary distribution at an exponential rate , governed by constants , commonly called C and α . In the following Lemma , we formalize this insight . The proof is given in Appendix D. Lemma 3 . Denote with PE the Markov chain induced by an irreducible and aperiodic expert policy . There are constants C > 0 and α ∈ ( 0 , 1 ) such that max′s ‖1 ( s′ ) > P tE − ρES ‖TV ≤ Cαt . Moreover , defining ρ̃s ′ t ( s , a ) = ( 1 ( s ′ ) > P tE ) ( s ) πE ( a|s ) , we have maxs′ ‖ρ̃s ′ t − ρE‖TV ≤ Cαt . The mixing time of the chain τmix , defined as the smallest t so that maxs′ ‖1 ( s′ ) > P tE − ρES ‖TV ≤ 14 is bounded as τmix ≤ logα C + logα ( 4 ) . McDiarmid ’ s inequality is a tool for measuring how fast functions of several random variables approach their expected value . We now recall a version of the inequality used when the variables form a Markov chain . The proof is given in Appendix D. Lemma 4 . Consider a Markov Chain s1 , s2 , . . . , sN . For any coefficients ci and a function f so that fs ( s1 , . . . , sN ) − fs ( s′1 , . . . , s′N ) ≤ ∑N i=1 ci1 ( si 6= s′i ) for all s1 , . . . , sN , s′1 , . . . , s′N , we have P ( |f ( s1 , s2 , . . . , sN ) − Ef ( s1 , s2 , . . . , sN ) | ≤ ) ≥ 1− 2 exp { − 2 4.5τmix‖c‖22 } , where τmix is the mixing time of the chain . 3 IMITATION LEARNING BY REINFORCEMENT LEARNING . Algorithm To perform imitation learning , we first obtain an expert dataset D and construct an intrinsic reward RI ( s , a ) = 1 ( ( s , a ) ∈ D ) . ( 6 ) We can then use any RL algorithm to solve for this reward . We note that for finite MDPs equation 6 exactly matches the algorithm proposed by Wang et al . ( 2019 ) as a heuristic . The reward defined in equation 6 is intrinsic , meaning we construct it artificially in order to elicit a certain behavior from the agent . This is in contrast to an extrinsic reward , which is what the agent gathers in the real world and what we want to recover . We will learn in Proposition 1 that the two are in fact closely related and solving for the intrinsic reward guarantees recovery of the extrinsic reward . Guarantee on Imitation Policy The main focus of the paper lies on showing theoretical properties of the imitation policy obtained when solving for the reward signal defined in equation 6 . Our formal guarantees hold under three assumptions . Assumption 1 . The expert policy is deterministic . Assumption 2 . The expert policy induces an irreducible and aperiodic Markov chain . Assumption 3 . The imitation learner policy induces an irreducible and aperiodic Markov chain . Assumption 1 is critical for our proof to go through and can not be relaxed . It is also essential to rationalize the approach of Wang et al . ( 2019 ) . In fact , no reduction involving a single call to an RL solver with stationary reward can exist for stochastic experts since for any MDP there is always an optimal policy which is deterministic . Assumptions 2 and 3 could in principle be relaxed , to allow for periodic chains , but it would complicate the reasoning , which we wanted to avoid . We note that we only interact with the expert policy via a finite dataset of N samples . Our main contribution is Proposition 1 , in which we quantify the performance of our imitation learner . We state it below and prove it in Section 4 . Proposition 1 . Consider an imitation learner trained on a dataset of size N , which attains the limiting distribution of state-action pairs ρI . Under Assumptions 1 , 2 and 3 , given we have N ≥ max { D η , log ( 2|S| δ ) |S|2 Bη2 } expert demonstrations , with probability at least 1− δ , the imitation learner attains total variation distance from the expert of at most ‖ρE − ρI‖TV ≤ η . ( 7 ) For brevity , we introduced D = 2C ( 1 + 2C1−α ) ( 1 − α ) −1 and B = 4.5τmix ( 1 + 2C1−α ) 2 , where the constants C , α and τmix reflect the mixing time of the expert policy and are defined in Section 2 . Moreover , with the same probability , for any extrinsic reward function , the imitation learner achieves expected per-step extrinsic reward of at least EρIRE ≥ EρERE − η . ( 8 ) Proposition 1 shows that the policy learned by imitation learner satisfies two important desiderata : we can guarantee both the closeness of the generated state-action distribution to the expert distribution as well as recovery of expert reward . Moreover , the total variation bound in equation 7 links the obtained policy to adversarial imitation learning algorithms . We describe this link in more detail in Section 5 .

Summary: The authors propose an algorithm for imitation learning which rewards the agent for observing state-action pairs that are part of the demonstration-set. The main contribution of the paper is the theoretical analysis which shows that the algorithm, given sufficient expert data, matches the expert’s occupancy distribution and expected reward. The proof comes in three parts: first, the authors relate expert rollouts to the limiting distribution of the expert’s policy. Second, the authors put a probabilistic bound on the expected agreement with expert-data based on the mixing time of the agent. Finally, the authors use this bound on expected agreement to bound the difference in expected extrinsic reward. The authors also provide an empirical evaluation in standard control benchmarks.