Robust Imitation via Mirror Descent Inverse Reinforcement Learning

1 INTRODUCTION . Inverse reinforcement learning ( IRL ) is an algorithm of learning ground-truth rewards from demonstrations of an expert that acts optimally with respect to an unknown reward function . Traditional IRL approaches ( Ng & Russell , 2000 ; Abbeel & Ng , 2004 ; Ziebart et al. , 2008 ) solve the imitation problem based on iterative algorithms , alternating between the reward estimation process and a reinforcement learning ( RL ) algorithm . Recent adversarial imitation learning ( AIL ) studies ( Ho & Ermon , 2016 ; Fu et al. , 2017 ) focus on formulating a statistical divergence minimization with a fine-tuned representation of the target expert probability distribution ( Ghasemipour et al. , 2020 ) . One method to approximate the distance between probability distributions is using Bregman divergences , a family of metric-like functions induced by a strongly convex function . Jeon et al . ( 2020 ) generalized the AIL framework to an optimization problem with respect to a Bregman divergence between policy distributions where the method shares on the essence of regularized Markov decision processes ( Geist et al. , 2019 ) . Through the lens of differential geometries , this implies that a policy distribution and a reward function can be associated with geometric constraints specified by a convex regularizer ( Shima , 2007 ) . Apparently , the limitation of regularized IRL studies naturally comes from a geometric property in which minimizing a Bregman divergence does not guarantee unbiased progression due to the constraints of the underlying space ( Butnariu & Resmerita , 2006 ) . The success of machine learning has been the key to learning such divergences with high precision . When the demonstration size is sufficiently large , the discriminator approximation of AIL can model the entire expert policy distribution internally using nonlinear approximators such as neural networks ( Finn et al. , 2016 ; Fu et al. , 2017 ) . However , there are challenging tasks that require long action sequences to be solved . When states can only be rarely visited by imitation learning agents , the uncertainty of estimated expert policy often substantially affects the discrimination quality . Therefore , the development of a robust IRL method that is tolerant to unreliable discriminative signals induced by imperfect demonstrations is necessary . The reasonable question is how to learn with cost estimations that are not precisely determined throughout the learning process . There have been considerable achievements in dealing with temporal costs in the online learning domain ( Fiat & Woeginger , 1998 ; Hazan , 2019 ) . An online learning algorithm predicts a sequence of parameters { wt } Tt=1 ⊂ W that is incurred by a cost functionFt ( · ) at each iteration . The most ordinary approach in online learning is stochastic gradient descent ( SGD ) : wt+1 = wt − ηt∇Ft ( wt ) . SGD is a desirable algorithm when the setW is the Euclidean geometry since it can provide unbiased progression for minimizing Ft ( Raskutti & Mukherjee , 2015 ) . However , policies appear in manifolds of probability distributions ; a gradient may not be the direction of the steepest descent in this case due to geometric constraints ( Amari , 1998 ; 2016 ) . In this paper , we propose an iterative imitation learning framework that interprets the RL-IRL scheme similar to proximal optimization methods ( Boyd et al. , 2004 ; Amari , 2016 ) . We identify two issues in the AIL studies that have characteristics of unconstrained updates : ( 1 ) a divergence does not guarantee informative global directions to match the expert policy due to the geometry , and ( 2 ) a representation of divergence often can not be accurately obtained due to insufficient data . Our method is motivated by a template of optimization algorithms specified by a convex function , called mirror descent ( MD ; Nemirovsky & Yudin 1983 ) . For parameters sequences , cost functions , and step sizes , an MD update for a strongly convex function Ω is derived as ∇Ω ( wt+1 ) = ∇Ω ( wt ) − ηt∇Ft ( wt ) , ( 1 ) where ∇Ω is a bijective transformation that links the primal space and the dual space of gradients . Different from these standard MD formulations , our methodology draws a sequence of functions on a space formulated by a regularized reward operator which is an alternative to the dual space . Consequently , the reward functions are projected optimization targets for the space of policies . Our Contributions . We propose a novel IRL algorithm which facilitates the agent in robustly imitating the expert . Our work is complementary to previous regularized IRL studies ; we introduce NEW R1 , R2 a geometric perspective for optimizing rewards and derive solutions with theoretical guarantees . • Instead of a monolithic estimation process of a global solution , we derive a sequence of reward functions that provides local optimization targets for the space of policies ( Section 3 ) . • We prove that rewards derived by an MD algorithm guarantee convergent divergence minimization performance along with a rigorous regret bound ( Section 4 ) . • We propose mirror descent adversarial inverse reinforcement learning ( MD-AIRL ) , a novel IRL algorithm that can be easily implemented on top of the standard AIL framework ( Section 5 ) . • We validate the outperforming performance of MD-AIRL on benchmarks with large discrete ac- tion spaces and continuous action spaces ( Section 6 ) . 2 PRELIMINARIES . Notation . For finite setsX and Y , we define Y X as a set of functions fromX to Y . ∆X ( ∆YX ) is a set of ( conditional ) probabilities over X ( conditioned on Y ) . For a function f ∈ RS×A and a policy π ∈ ∆SA , fs and πs denote shorthand notation of f ( s , · ) and π ( ·|s ) , respectively . A Markov decision process ( MDP ) is defined as a tuple ( S , A , P , r , γ ) with the state space S , the action space A , the transition kernel P ∈ ∆S×AS , the reward function r ∈ RS×A and the discount factor γ ∈ [ 0 , 1 ) . A Bregman divergence with respect to a convex function Ω for a state s is defined as DΩ ( πs∥π̂s ) = Ω ( πs ) − Ω ( π̂s ) − ⟨∇Ω ( π̂s ) , πs − π̂s⟩A . We write Ω ( π ) = [ Ω ( πs ) ] s∈S and similarly for∇Ω . Regularized RL & IRL . We consider the RL-IRL framework in regularized MDPs ( Geist et al. , 2019 ) , where the policy is optimized along with a causal convex regularizer . The objective is to find π which maximizes the expected discounted sum of rewards with a strongly convex regularizer Ω : maximize π∈∆SA JΩ ( π , r ) : = Eπ [ ∑∞ i=0 γi { r ( si , ai ) − Ω ( π ( ·|si ) ) } ] , ( 2 ) where the subscript π on the expectation denotes samples generated from the MDP and policy π . Consider the convex conjugate of qsE = ∇Ω ( πsE ) : Ω∗ ( qsE ) = maxπs∈∆A⟨πs , qsE⟩A − Ω ( πs ) . Differentiating both sides with respect to qsE , the gradient of conjugate∇Ω∗ maps qE to the unique optimal policy πE . Let Π be a bounded , open set for ∆SA ⊂ Π and R : Π → RS×A be a reward operator which maps a policy to a reward function . The output r̂ = R ( πE ) is a representation of the expert ’ s behavior ; but finding such operator is an ill-posed problem because every function r̂ that makes the state-action value function qE is a valid solution of regularized IRL ( Geist et al. , 2019 ) . Recently , Jeon et al . ( 2020 ) proposed a tractable solution , which we refer to the function as the regularized reward function . We rewrite the previous notation by defining the regularized reward operator ΨΩ . Definition 1 ( Regularized reward operators ) . Define the regularized reward operator ΨΩ : π 7→ ψπ ψπ ( s , a ) : = Ω ′ ( s , a ; π ) − 〈 πs , ∇Ω ( πs ) 〉 A +Ω ( π s ) ∀s ∈ S , a ∈ A , ( 3 ) for Ω′ ( s , · ; π ) : = ∇Ω ( πs ) = [ ∇pΩ ( p ) ] p=π ( ·|s ) . By using the operator in discriminative networks , Jeon et al . ( 2020 ) proposed regularized adversarial IRL ( RAIRL ) as a generalization of AIL in terms of minimizing a specific Bregman divergence . Mirror Descent . Let w be a parameter on a set W and Ft : W → R be a convex cost function from a class of functions F . Replacing the L2 proximity term of proximal gradient descent with the Bregman divergence ( Gutman & Peña , 2018 ) , the proximal form of the MD update is given as wMDt+1 = argmin w∈W 〈 ∇Ft ( wMDt ) , w − wMDt 〉 W + αtDΩ ( w ∥∥wMDt ) , ( 4 ) where αt : = 1/ηt denotes an inverse of the step size ηt . MD was developed as a generalization of SGD , where the local geometry is specified by a Bregman divergence ( Gunasekar et al. , 2020 ) . 3 ITERATIVE RL-IRL AS A PROXIMAL OPTIMIZATION METHOD . We interpret the RL-IRL framework as a variant of proximal optimization methods and consider two sequences { πt } ∞t=1 and { ψt } ∞t=1 that denote the learning policies and reward functions , respectively . Associated reward function . An updated point in MD can be uniquely projected to the desired space ( ∆SA in our case ) using a Bregman projection operator PΩ that locates iterative points to the feasible region , i.e . πt+1 = PΩ ( π̃t+1 ) : = argminπ∈∆SA [ DΩ ( π∥π̃t+1 ) ] s∈S for π̃t+1 ∈ Π . To avoid these computations , we preemptively constrain a reward operator to satisfy the projection invariance with respect to regularized RL process of the associated state-action value function q̂ : PΩ ( ∇Ω∗ ( q̂ ) ) = ∇Ω∗ ( q̂ ) . According to Lemma 1 of Jeon et al . ( 2020 ) , a regularized reward function ψπ can replace q̂ , since the induced Bregman divergence allows the learning in a greedy manner . FIX R3 The projection invariance of the operator ΨΩ can be shown by a bijective relation with ∇Ω∗ in the space of ∆SA by the following lemma . Lemma 1 ( Natural isomorphism ) . Let ψ ∈ ΨΩ ( ∆SA ) for ΨΩ ( Π ) = { ψ | ψ ( s , a ) = ψπ ( s , a ) , ∀s∈ S , a∈A , π∈Π } . Then , ∇Ω∗ ( ψ ) is unique with respect to ψ , and for every π = ∇Ω∗ ( ψ ) , π ∈ ∆SA . Policy Space ∆SA Π πt πt+1 Reward Space ψt ψt+1 ΨΩ ∇Ω∗ FIX R3 As a result , we consider an updated reward function ψt+1 ∈ ΨΩ ( ∆SA ) as a projected target of MD , FIX R3 which can be associated by parameterization of ∆SA . For instance , a bijective mapping of ψt+1 can NEW R3 represent a softmax policy for a discrete space , or a Gaussian policy for a continuous space , using its parameters . The subsequent RL process at t-th step becomes finding the next iteration πt+1 by NEW R3 FIX R3maximize π∈∆SA JΩ ( π , ψt+1 ) = Eπ [ ∑∞ i=0 γiDΩ ( πsi ∥∥πsit+1 ) ] , πt+1 = ∇Ω∗ ( ψt+1 ) . ( 5 ) The equation shows that an arbitrary regularized RL algorithm with the regularizer Ω forms a cumulative discounted sum of Bregman divergences , thus the mapping to πt+1 can be achieved . Online imitation learning . The necessity of online learning setup comes from observing that our RL-IRL processes do not retain ψE = ΨΩ ( πE ) , a representation of πE , during training . Instead , consider a random process { π̄E , t } ∞t=1 where the estimation π̄E , t resides in a closed , convex neighborhood of πE , estimated by a separate estimation algorithm such as maximum likelihood methods . Substituting ψE to ψπ̄E , t in JΩ ( π , ψE ) turns the RL objective into an online learning problem : FIX R3minimize π∈∆SA Ft ( π ) = −JΩ ( π , ψπ̄E , t ) = Eπ [ ∑∞ i=0 γiDΩ ( πsi ∥∥π̄siE , t ) ] . ( 6 ) For a deeper understanding of our setup , we consider an example in Figure 2 . Assume that policies of the learning agent and the expert follow multivariate Gaussian distributions initially at π1 = N ( [ 0 , 0 ] T , I ) and πE = N ( [ 5 , 3 ] T , ΣE ) for |ΣE| < 1 . The policy πt is trained by a cost function DΩ ( ·∥π̄E , t ) with the proximity constraint 1ηDΩ ( ·∥πt ) where the Gaussian policy π̄E , t is also fitted using stochastic samples from πE at each iteration , starting from π̄E,1 = π1 . We first observe that the step size constant η affects the training speed in the early phase . Notably , the performance of certain cases exceeds the baselines of DΩ ( π̄E , t∥πE ) by choosing the step size effectively low as η < 1 . This suggests that there is a clear advantage of the online imitation learning setup for unreliable π̄E , t . MD update rules . Plugging each divergence of the cumulative cost Ft to the template of Equation ( 4 ) , the optimization process for the subsequent ψt+1 of the MD formulation is derived as1 ψt+1 : = ΨΩ ( πt+1 ) , π s t+1 = argmin πs∈∆A 〈 ∇DΩ ( πst ∥π̄sE , t ) ︸ ︷︷ ︸ ∇Ω ( πst ) −∇Ω ( π̄sE , t ) , πs − πst 〉 A + αtDΩ ( π s∥πst ) = argmin πs∈∆A DΩ ( π s∥π̄sE , t ) −DΩ ( πs∥πst ) + αtDΩ ( πs∥πst ) = argmin πs∈∆A ηtDΩ ( π s∥π̄sE , t ) ︸ ︷︷ ︸ estimated expert + ( 1− ηt ) DΩ ( πs∥πst ) ︸ ︷︷ ︸ learning agent , ηt = 1/αt , ∀s ∈ S , ( 7 ) where the gradient of DΩ is taken with respect to its first argument πst . The objective of reward NEW R3 learning is analogous to finding an interpolation at each iteration where the point is controlled by the step size ηt . Figure 3 shows that the region of πt ( defined by a norm ) gradually decreases 1Bregman divergences are generally intractable to be computed , unless the policy is a specific parametric model ( e.g . exponential families ) . See the works of Nielsen & Nock ( 2011 ) ; Jeon et al . ( 2020 ) and Appendix B. when ηt > ηt+1 . Note that solving the optimization of Ft requires interaction between πt and the dynamics of the given environment ; thus , the RL process in Equation ( 5 ) plays the essential role of sequential learning by the value measures of the reward function .

Summary: The paper proposes an algorithm MD-AIRL, where each iteration involves policy and reward updates, and an expert policy estimation. The algorithm is motivated by MD, especially the reward update is modeled as a projected update of an MD update. The sufficiency and necessities of the choices of stepsizes to guarantee convergence are theoretically justified. A practical algorithm with numerical experiments are also provided.