A Generalised Inverse Reinforcement Learning Framework

1 INTRODUCTION . Modelling the behaviours of rational agents is a long active research topic . From early attempts to decompose human and animal locomotion Muybridge ( 1979 ) to more recent approaches to simulate human movements Li & Todorov ( 2006 ) ; Mombaur ( 2009 ) ; Schultz & Mombaur ( 2009 ) , the common thread is an underlying assumption that the agents are acting according to some stationary policies . To rationalise these behaviours , it is natural to assume that they are optimal with respect to some objective function ( there are evidences to back this assumption in the case of animal conditioned learning Schmajuk & Zanutto ( 1997 ) ; Verschure & Althaus ( 2003 ) ; Maia ( 2010 ) ; Verschure et al . ( 2014 ) ) . The global objective of Inverse Reinforcement Learning ( IRL ) is inferring such objective function given measurements of the rational agent ’ s behaviour , its sensory inputs and a model of the environment Russell ( 1998 ) . IRL builds upon the standard Reinforcement Learning ( RL ) formulation , where the goal is to find the policy that minimises discounted cumulative costs of some Markov Decision Process Puterman ( 2014 ) . It aims at finding cost functions for which the observed behaviour is “ approximately optimal ” . However this simplistic formulation admits degenerate solutions Abbeel & Ng ( 2004 ) . This led to a series of innovative reformulations to lift this indeterminacy by favouring costs for which the observed behaviour is particularly better than alternative ones , namely maximum margin IRL Ratliff et al . ( 2006 ) and maximum entropy IRL Ziebart et al . ( 2008 ; 2010 ) . The latter formulation ended up as the building block of recent breakthroughs , with both tractable and highly performing algorithms Finn et al . ( 2016 ) ; Ho & Ermon ( 2016 ) ; Fu et al . ( 2017 ) . These improvements provided the ground for multiple practical real-life applications Ziebart et al . ( 2008 ) ; Bougrain et al . ( 2012 ) ; Sharifzadeh et al . ( 2016 ) ; Jarboui et al . ( 2019 ) ; Martinez-Gil et al . ( 2020 ) . We propose an orthogonal improvement to this literature . We question the very pertinence of characterising optimality w.r.t . the cumulative discounted costs as it induces a bias against policies with longer mixing times . We propose an extension of this criterion to alleviate this issue . From this novel objective , we derive reformulations for both the RL and IRL problems . We discuss the ability of existing RL algorithms to solve this new formulation and we generalise existing IRL algorithms to solve the problem under the new criterion . We back up our proposition with empirical evidence of improved performances in multiple OpenAI gym environments . 2 GENERALISED OPTIMALITY CRITERION . In this section , we introduce the classical settings of RL and IRL , as well as the new generalised settings we introduce to alleviate some inherent biases of current methods . 2.1 A CLASSICAL RL SETTING . Consider an infinite horizon Markov Decision Process ( MDP ) M = { S , A , P , c , γ , p0 } , where : – S is either a finite or a compact subset of Rd , for some dimension d ∈ N – A is either a finite or a compact subset of Rd′ , for d′ ∈ N – P is the state transition kernel , i.e. , a continuous mapping from S × A to ∆ ( S ) , where ∆ ( · ) denotes the set of probability measures1 over some set , – c : S ×A → R is a continuous non-negative cost function , – p0 ∈ ∆ ( S ) is the initial state distribution , and γ ∈ ( 0 , 1 ) is the discount factor . A policy π is a mapping indicating , at each time step t ∈ N , the action at to be chosen at the current state st ; it could depend on the whole past history of states/actions/rewards but it is well known that one can focus solely , at least under mild assumptions , on stationary policies π : S → ∆ ( A ) . The choice of a policy π , along with a kernel P and the initial probability p0 , generates a unique probability distribution over the sequences of states denoted by Pπ ( the solution to the forward Chapman–Kolmogorov equation ) . The expected cumulative discounted cost of this policy , in the MDP M is consequently equal to Ep0 , π [ ∑ t γ tc ( st , at ) ] = ∫ s0 p0 ( s0 ) ∑∞ t=0 ∫ st , at γtPπ ( st , at|s0 ) c ( st , at ) . Optimal policies are minimisers of this quantity ( existence is ensured under mild assumptions Puterman ( 2014 ) ) . A standard way to compute optimal policies , is to minimise the state-action value mapping defined as : Qcπ ( s , a ) = c ( s , a ) + ∑∞ t=1 ∫ st , at γtPπ ( st , at|s ) c ( st , at ) . Indeed , the expected cumulative discounted cost of a policy is the expectation of Q-function against p0 : Ep0 , π [ ∞∑ t=0 γtc ( st , at ) ] = ∫ s0 , a0 p0 ( s0 ) π ( a0|s0 ) Qcπ ( s0 , a0 ) 2.2 A BUILT-IN BIAS IN THE IRL FORMULATION . The problem gets more complicated in Inverse Reinforcement Learning where the objective is to learn an unknown cost function c whose associated optimal policy coincides with a given one πE ( referred to as the “ expert ” policy ) . This problem is unfortunately ill-posed as all policies are optimal w.r.t . a constant cost function Abbeel & Ng ( 2004 ) . In order to lift this indeterminacy , the most used alternative formulation is called maximum entropy inverse reinforcement learning Ziebart et al . ( 2008 ; 2010 ) that aims at finding a cost function c∗ such that the expert policy πE has a relatively small cumulative cost Ep0 , πE [ ∑∞ t=0 γ tc∗ ] while other policies incur a much higher cost . This implicitly boils down to learning an optimal policy ( associated to some learned cost ) that matches the expert ’ s future state occupancy measure ρπE marginalised over the initial state distribution , where ρπ ( s , a|s0 ) = ∑ t γ tPπ ( st = s , at = a|s0 ) . State of the art approaches Ho & Ermon ( 2016 ) ; Fu et al . ( 2017 ) consist , roughly speaking , in a two-step procedure . In the first step , given a cost function ĉ , an ( approximately ) optimal policy π̂ of M̂ ( the MDPM with ĉ for cost function ) , is learned . In the second step , trajectories generated by π̂ are compared to expert ones ( in the sense of ρπ ) ; then ĉ is updated to penalise states unvisited by the expert ( say , by gradient descent over some parameters ) . Obviously , those two steps can be repeated until convergence ( or until the generated and the original data-sets are close enough ) . However , the presence of a discount factor in the definition of ρπ has a huge undesirable effect : the total weight of the states in the far future ( say , after some stage t∗ ) is negligible in the global 1The σ-field is always the Borel one . loss , as it would be of the order of γt ∗ . So trying to match the future state occupancy measure will implicitly favours policies mimicking the behaviour in the short term . As a consequence , this would end up in penalising policies with longer mixing times even if their stationary distribution matches the experts on the long run . This built-in bias is a consequence of solving the reinforcement learning step with policies that optimise the cumulative discounted costs ( minimises the expectation of the Q-functions against p0 ) rather than policies that achieve the Bellman optimality criterion ( minimises the Q-function for any state action pairs ) . Unfortunately , there is no IRL framework solving the problem under the latter assumption . In order to bridge this gap , we introduce a more general optimality criterion for the reinforcement learning step ; it is still defined as the expectation of the Q-function , yet not against p0 as in traditional RL , but against both the initial and the future states distributions . To get some flexibility , we allow the loss to weight present and future states differently by considering a probability distribution η over N. Formally , we define the η-weighted future state measurement distribution : P ηπ ( s+ , a+|s0 ) : = ∞∑ n=0 η ( n ) Pπ ( sn = s+ , an = a+|s0 ) . Using P ηπ , the new criterion is defined as : Eηp0 , π [ Q c π ] : = ∫ s0 p0 ( s0 ) ∫ s+ , a+ P ηπ ( s+ , a+|s0 ) Qcπ ( s+ , a+ ) = Ep0 , π [ ∑ k η ( k ) ∑ t γtct+k ] where ct denotes the cost at the tth observation ( c ( st , at ) ) . Any policy that minimises Eηp0 , π [ Q c π ] will now be referred to as “ η-optimal ” ( w.r.t . the cost function c ) . As mentioned before , the inverse RL problem can be decomposed in two sub-problems , learning approximate optimal strategies ( given a candidate ĉ ) and optimizing over ĉ ( taking into account the expert distribution πE ) . In order to avoid over-fitting when learning optimal policies , the standard way is to regularize the optimization loss Geist et al . ( 2019 ) . As a consequence , we consider any mapping Ω : ∆ ( A ) S → R that is a concave over the space of policies . The associated regularised loss of adopting a policy π given the cost function c is defined as : LηΩ ( π , c ) = E η p0 , π [ Qcπ ] − Ω ( π ) ( 1 ) The generalised RL problem is then defined as : RLηΩ ( c ) : = arg min π LηΩ ( π , c ) ( 2 ) Similarly , in order to learn simpler cost functions Ho & Ermon ( 2016 ) , the optimization loss considered is in turn penalised by a convex ( over the space of cost functions ) regularizer ψ : R ( S×A ) → R. The problem of Generalised ( Maximum Entropy ) Inverse Reinforcement learning , whose objective is to learn an appropriate cost function c , is formally defined as : IRLηψ , Ω ( πE ) : = arg max c min π LηΩ ( π , c ) − L η Ω ( πE , c ) − ψ ( c ) ( 3 ) We emphasise that simply choosing δ0 ( a Dirac mass at 0 ) for the distribution η induces the classical definitions of both the RL and IRL problems Ho & Ermon ( 2016 ) . On the other hand , choosing η = Geom ( γ ) transforms the loss into the expectation of the sum of discounted Q-functions along the trajectory . Hypothetically , there could be other generalisations of discounted cost . However , preserving the compatibility of the Bellman criterion with the proposed generalisation for RL and duality properties for IRL is not trivial ( for example , polynomial decay γtn would break these properties ) . In the following , we prove that the η-optimality framework satisfies both properties .

Summary: This paper points out that the classical IRL approach has a tendency to match those occupancy measures that favor short-term behavior. To address this issue, a reformulation is proposed based on GAIL in order to put more emphasis on matching longer-term behavior. Specifically, the main difference is to replace the standard objective (i.e., the expectation of Q function over some initial state distribution) with the expectation of Q function over both the initial and an $\eta$-weighted future state distribution, where $\eta$ is some probability distribution over the support set of nonnegative integers. Built on this formulation, this paper proposes GIRL (and the resulting algorithm MEGAN), which follows the framework of GAIL to learn a policy that matches the $\eta$-weighted variant of occupancy measure of the expert policy. Experimental results on MuJoCo are provided to demonstrate the effectiveness of GIRL.