Imitation Learning by Reinforcement Learning

Imitation learning algorithms learn a policy from demonstrations of expert behavior . We show that , for deterministic experts , imitation learning can be done by reduction to reinforcement learning with a stationary reward . Our theoretical analysis both certifies the recovery of expert reward and bounds the total variation distance between the expert and the imitation learner , showing a link to adversarial imitation learning . We conduct experiments which confirm that our reduction works well in practice for continuous control tasks . 1 INTRODUCTION . Typically , reinforcement learning ( RL ) assumes access to a pre-specified reward and then learns a policy maximizing the expected average of this reward along a trajectory . However , specifying rewards is difficult for many practical tasks ( Atkeson & Schaal , 1997 ; Zhang et al. , 2018 ; Ibarz et al. , 2018 ) . In such cases , it is convenient to instead perform Imitation Learning ( IL ) , learning a policy from expert demonstrations . There are two major categories of Imitation Learning algorithms : behavioral cloning and inverse reinforcement learning . Behavioral cloning learns the policy by supervised learning on expert data , but is not robust to training errors , failing in settings where expert data is limited ( Ross & Bagnell , 2010 ) . Inverse reinforcement learning ( IRL ) achieves improved performance on limited data by constructing reward signals and calling an RL oracle to maximize these rewards ( Ng et al. , 2000 ) . The most versatile IRL method is adversarial IL ( Ho & Ermon , 2016 ; Li et al. , 2017 ; Ghasemipour et al. , 2020 ) , which minimizes a divergence between the distribution of data produced by the agent and provided by the expert . Adversarial IL learns a representation and a policy simultaneously by using a non-stationary reward obtained from a discriminator network . However , training adversarial IL combines two components which are hard to stabilize : a discriminator network , akin to the one used in GANs , as well as a policy , typically learned with an RL algorithm with actor and critic networks . This complexity makes the training process very brittle . There is a clear need for imitation learning algorithms that are simpler and easier to deploy . To address this need , Wang et al . ( 2019 ) proposed to reduce imitation learning to a single instance of reinforcement learning problem , where reward is defined to be one for state-action pairs from the expert trajectory and zero for other state-action pairs . A closely related , but not identical , algorithm has been proposed by Reddy et al . ( 2020 ) ( we describe the differences in Section 5 ) . However , while empirical performance of these approaches has been good , they enjoy no performance guarantees at all , even in the asymptotic setting where expert data is infinite . Contributions We fill in this missing justification for this algorithm , providing the needed theoretical analysis . Specifically , in Sections 3 and 4 , we show a total variation bound between the expert policy and the imitation policy , providing a high-probability performance guarantee for a finite dataset of expert data and linking the reduction to adversarial imitation learning algorithms . For stochastic experts , we describe how the reduction fails , completing the analysis . Moreover , in Section 6 , we empirically evaluate the performance of the reduction as the amount of available expert data varies . 2 PRELIMINARIES . Markov Decision Process An average-reward Markov Decision Process ( Puterman , 2014 ; Feinberg & Shwartz , 2012 ) is a tuple ( S , A , T , R , s1 ) , where S is a state space , A is the action space , T : S ×A→ P ( S ) is the transition model , R : S ×A→ [ 0 , 1 ] is a bounded reward function and s1 is the initial state . Here , we write P ( X ) to denote probability distributions over a set X . A stationary policy π : S → P ( A ) maps environment states to distributions over actions . A policy π induces a Markov chain Pπ . In the theoretical part of the paper , we treat MDPs with finite state and action spaces . Given a starting state s1 of the MDP , and a policy π , the limiting distribution over states is defined as ρπS ( s ) = 1 ( s1 ) > lim N→∞ 1 N N∑ i=0 P iπ , ( 1 ) where 1 ( s1 ) denotes the indicator vector . We adopt the convention that the subscript S indicates a distribution over states and a no subscript indicates a distribution over state-action pairs . We denote ρπ ( s , a ) = ρπS ( s ) π ( a|s ) . While the limit in equation 1 is guaranteed to exist for all finite MDPs ( Puterman , 2014 ) , without requiring ergodicity , in this paper we consider policies that induce ergodic chains . Correspondingly , we drop the dependence on the initial state from the notation . The expected per-step reward of π is defined as V π = lim N→∞ Eπ [ JN/N | s1 ] = EρπR ( s , a ) , ( 2 ) where JN = ∑N t=1R ( st , at ) is the total return . In this paper we consider undiscounted MDPs because they are enough to model the essential properties of the imitation learning problem while being conceptually simpler than discounted MDPs . However , we believe that similar results could be obtained for discounted MDPs . Expert Dataset Assume that the expert follows an ergodic policy πE . Denote the corresponding distribution over expert state-action pairs as ρE ( s , a ) = ρπE ( s , a ) , where we dropped the dependency on the initial state by ergodicity . Consider a finite trajectory s1 , a1 , s2 , a2 , . . . , sN , aN . Denote by ρ̂ ( s , a ) = 1N ∑N t=1 1 ( ( s , a ) = ( st , at ) ) the histogram ( empirical distribution ) of data seen along this trajectory and denote by ρ̂S ( s ) = 1N ∑N t=1 1 ( s = st ) the corresponding histogram over the states . Denote by D the support of ρ̂ , i.e . the set of state-action pairs visited by the expert . Total Variation Consider probability distributions defined on a discrete set X . The total variation distance between distributions is defined as ‖ρ1 − ρ2‖TV = sup M⊆X |ρ1 ( M ) − ρ2 ( M ) | . ( 3 ) In our application , the set X is either the set of MDP states S or the set of state state-action pairs S × A . Below , we restate in our notation standard results about the total variation distance . See appendix D for proofs . Lemma 1 . If for any vector v ∈ [ 0 , 1 ] |X| it holds that Eρ1 [ v ] − Eρ2 [ v ] ≤ , then we have ‖ρ1 − ρ2‖TV ≤ . Lemma 2 . If ‖ρ1 − ρ2‖TV ≤ , then we have |Eρ1 [ v ] − Eρ2 [ v ] | ≤ 2 for any vector v ∈ [ 0 , 1 ] |X| . Imitation Learning Learning to imitate means obtaining a policy that mimics expert behavior . This can be formalized in two ways . First , we can seek an imitation policy πI which obtains a expected per-step reward that is -close to the expert on any reward signal bounded in the interval [ 0 , 1 ] . Denote the limiting distribution of state-action tuples generated by the imitation learner with ρI ( s , a ) . Formally , we want to satisfy ∀R ∈ S ×A→ [ 0 , 1 ] . EρE [ R ] − EρI [ R ] ≤ , ( 4 ) where is a small constant . Second , we can seek to ensure that the distributions of state-action pairs generated by the expert and the imitation learner are similar ( Ho & Ermon , 2016 ; Finn et al. , 2016 ) . In particular , if we measure the divergence between distributions with total variation , we want to ensure ‖ρE − ρI‖TV ≤ . ( 5 ) Lemmas 1 and 2 show that the equations 4 and 5 are closely related . In other words , attaining expected per-step reward close to the expert ’ s is the same ( up to a multiplicative constant ) as closeness in total variation between the state-action distribution of the expert and the imitation learner . We provide further background on adversarial imitation learning and other divergences in Section 5 . Markov Chains When coupled with the dynamics of the MDP , a policy induces a Markov chain , In this paper , we focus our attention on chains which are irreducible and aperiodic . Crucially , such chains approach the stationary distribution at an exponential rate , governed by constants , commonly called C and α . In the following Lemma , we formalize this insight . The proof is given in Appendix D. Lemma 3 . Denote with PE the Markov chain induced by an irreducible and aperiodic expert policy . There are constants C > 0 and α ∈ ( 0 , 1 ) such that max′s ‖1 ( s′ ) > P tE − ρES ‖TV ≤ Cαt . Moreover , defining ρ̃s ′ t ( s , a ) = ( 1 ( s ′ ) > P tE ) ( s ) πE ( a|s ) , we have maxs′ ‖ρ̃s ′ t − ρE‖TV ≤ Cαt . The mixing time of the chain τmix , defined as the smallest t so that maxs′ ‖1 ( s′ ) > P tE − ρES ‖TV ≤ 14 is bounded as τmix ≤ logα C + logα ( 4 ) . McDiarmid ’ s inequality is a tool for measuring how fast functions of several random variables approach their expected value . We now recall a version of the inequality used when the variables form a Markov chain . The proof is given in Appendix D. Lemma 4 . Consider a Markov Chain s1 , s2 , . . . , sN . For any coefficients ci and a function f so that fs ( s1 , . . . , sN ) − fs ( s′1 , . . . , s′N ) ≤ ∑N i=1 ci1 ( si 6= s′i ) for all s1 , . . . , sN , s′1 , . . . , s′N , we have P ( |f ( s1 , s2 , . . . , sN ) − Ef ( s1 , s2 , . . . , sN ) | ≤ ) ≥ 1− 2 exp { − 2 4.5τmix‖c‖22 } , where τmix is the mixing time of the chain . 3 IMITATION LEARNING BY REINFORCEMENT LEARNING . Algorithm To perform imitation learning , we first obtain an expert dataset D and construct an intrinsic reward RI ( s , a ) = 1 ( ( s , a ) ∈ D ) . ( 6 ) We can then use any RL algorithm to solve for this reward . We note that for finite MDPs equation 6 exactly matches the algorithm proposed by Wang et al . ( 2019 ) as a heuristic . The reward defined in equation 6 is intrinsic , meaning we construct it artificially in order to elicit a certain behavior from the agent . This is in contrast to an extrinsic reward , which is what the agent gathers in the real world and what we want to recover . We will learn in Proposition 1 that the two are in fact closely related and solving for the intrinsic reward guarantees recovery of the extrinsic reward . Guarantee on Imitation Policy The main focus of the paper lies on showing theoretical properties of the imitation policy obtained when solving for the reward signal defined in equation 6 . Our formal guarantees hold under three assumptions . Assumption 1 . The expert policy is deterministic . Assumption 2 . The expert policy induces an irreducible and aperiodic Markov chain . Assumption 3 . The imitation learner policy induces an irreducible and aperiodic Markov chain . Assumption 1 is critical for our proof to go through and can not be relaxed . It is also essential to rationalize the approach of Wang et al . ( 2019 ) . In fact , no reduction involving a single call to an RL solver with stationary reward can exist for stochastic experts since for any MDP there is always an optimal policy which is deterministic . Assumptions 2 and 3 could in principle be relaxed , to allow for periodic chains , but it would complicate the reasoning , which we wanted to avoid . We note that we only interact with the expert policy via a finite dataset of N samples . Our main contribution is Proposition 1 , in which we quantify the performance of our imitation learner . We state it below and prove it in Section 4 . Proposition 1 . Consider an imitation learner trained on a dataset of size N , which attains the limiting distribution of state-action pairs ρI . Under Assumptions 1 , 2 and 3 , given we have N ≥ max { D η , log ( 2|S| δ ) |S|2 Bη2 } expert demonstrations , with probability at least 1− δ , the imitation learner attains total variation distance from the expert of at most ‖ρE − ρI‖TV ≤ η . ( 7 ) For brevity , we introduced D = 2C ( 1 + 2C1−α ) ( 1 − α ) −1 and B = 4.5τmix ( 1 + 2C1−α ) 2 , where the constants C , α and τmix reflect the mixing time of the expert policy and are defined in Section 2 . Moreover , with the same probability , for any extrinsic reward function , the imitation learner achieves expected per-step extrinsic reward of at least EρIRE ≥ EρERE − η . ( 8 ) Proposition 1 shows that the policy learned by imitation learner satisfies two important desiderata : we can guarantee both the closeness of the generated state-action distribution to the expert distribution as well as recovery of expert reward . Moreover , the total variation bound in equation 7 links the obtained policy to adversarial imitation learning algorithms . We describe this link in more detail in Section 5 .

Summary: In this paper, the authors show that, for deterministic experts, imitation learning can be done by reduction to reinforcement learning with a stationary reward. theoretical analysis both certifies the recovery of expert reward and bounds the total variation distance between the expert and the imitation learner, showing a link to adversarial imitation learning. Experiments are given to confirm that the reduction works well in practice for continuous control tasks.