A Generalised Inverse Reinforcement Learning Framework

1 INTRODUCTION . Modelling the behaviours of rational agents is a long active research topic . From early attempts to decompose human and animal locomotion Muybridge ( 1979 ) to more recent approaches to simulate human movements Li & Todorov ( 2006 ) ; Mombaur ( 2009 ) ; Schultz & Mombaur ( 2009 ) , the common thread is an underlying assumption that the agents are acting according to some stationary policies . To rationalise these behaviours , it is natural to assume that they are optimal with respect to some objective function ( there are evidences to back this assumption in the case of animal conditioned learning Schmajuk & Zanutto ( 1997 ) ; Verschure & Althaus ( 2003 ) ; Maia ( 2010 ) ; Verschure et al . ( 2014 ) ) . The global objective of Inverse Reinforcement Learning ( IRL ) is inferring such objective function given measurements of the rational agent ’ s behaviour , its sensory inputs and a model of the environment Russell ( 1998 ) . IRL builds upon the standard Reinforcement Learning ( RL ) formulation , where the goal is to find the policy that minimises discounted cumulative costs of some Markov Decision Process Puterman ( 2014 ) . It aims at finding cost functions for which the observed behaviour is “ approximately optimal ” . However this simplistic formulation admits degenerate solutions Abbeel & Ng ( 2004 ) . This led to a series of innovative reformulations to lift this indeterminacy by favouring costs for which the observed behaviour is particularly better than alternative ones , namely maximum margin IRL Ratliff et al . ( 2006 ) and maximum entropy IRL Ziebart et al . ( 2008 ; 2010 ) . The latter formulation ended up as the building block of recent breakthroughs , with both tractable and highly performing algorithms Finn et al . ( 2016 ) ; Ho & Ermon ( 2016 ) ; Fu et al . ( 2017 ) . These improvements provided the ground for multiple practical real-life applications Ziebart et al . ( 2008 ) ; Bougrain et al . ( 2012 ) ; Sharifzadeh et al . ( 2016 ) ; Jarboui et al . ( 2019 ) ; Martinez-Gil et al . ( 2020 ) . We propose an orthogonal improvement to this literature . We question the very pertinence of characterising optimality w.r.t . the cumulative discounted costs as it induces a bias against policies with longer mixing times . We propose an extension of this criterion to alleviate this issue . From this novel objective , we derive reformulations for both the RL and IRL problems . We discuss the ability of existing RL algorithms to solve this new formulation and we generalise existing IRL algorithms to solve the problem under the new criterion . We back up our proposition with empirical evidence of improved performances in multiple OpenAI gym environments . 2 GENERALISED OPTIMALITY CRITERION . In this section , we introduce the classical settings of RL and IRL , as well as the new generalised settings we introduce to alleviate some inherent biases of current methods . 2.1 A CLASSICAL RL SETTING . Consider an infinite horizon Markov Decision Process ( MDP ) M = { S , A , P , c , γ , p0 } , where : – S is either a finite or a compact subset of Rd , for some dimension d ∈ N – A is either a finite or a compact subset of Rd′ , for d′ ∈ N – P is the state transition kernel , i.e. , a continuous mapping from S × A to ∆ ( S ) , where ∆ ( · ) denotes the set of probability measures1 over some set , – c : S ×A → R is a continuous non-negative cost function , – p0 ∈ ∆ ( S ) is the initial state distribution , and γ ∈ ( 0 , 1 ) is the discount factor . A policy π is a mapping indicating , at each time step t ∈ N , the action at to be chosen at the current state st ; it could depend on the whole past history of states/actions/rewards but it is well known that one can focus solely , at least under mild assumptions , on stationary policies π : S → ∆ ( A ) . The choice of a policy π , along with a kernel P and the initial probability p0 , generates a unique probability distribution over the sequences of states denoted by Pπ ( the solution to the forward Chapman–Kolmogorov equation ) . The expected cumulative discounted cost of this policy , in the MDP M is consequently equal to Ep0 , π [ ∑ t γ tc ( st , at ) ] = ∫ s0 p0 ( s0 ) ∑∞ t=0 ∫ st , at γtPπ ( st , at|s0 ) c ( st , at ) . Optimal policies are minimisers of this quantity ( existence is ensured under mild assumptions Puterman ( 2014 ) ) . A standard way to compute optimal policies , is to minimise the state-action value mapping defined as : Qcπ ( s , a ) = c ( s , a ) + ∑∞ t=1 ∫ st , at γtPπ ( st , at|s ) c ( st , at ) . Indeed , the expected cumulative discounted cost of a policy is the expectation of Q-function against p0 : Ep0 , π [ ∞∑ t=0 γtc ( st , at ) ] = ∫ s0 , a0 p0 ( s0 ) π ( a0|s0 ) Qcπ ( s0 , a0 ) 2.2 A BUILT-IN BIAS IN THE IRL FORMULATION . The problem gets more complicated in Inverse Reinforcement Learning where the objective is to learn an unknown cost function c whose associated optimal policy coincides with a given one πE ( referred to as the “ expert ” policy ) . This problem is unfortunately ill-posed as all policies are optimal w.r.t . a constant cost function Abbeel & Ng ( 2004 ) . In order to lift this indeterminacy , the most used alternative formulation is called maximum entropy inverse reinforcement learning Ziebart et al . ( 2008 ; 2010 ) that aims at finding a cost function c∗ such that the expert policy πE has a relatively small cumulative cost Ep0 , πE [ ∑∞ t=0 γ tc∗ ] while other policies incur a much higher cost . This implicitly boils down to learning an optimal policy ( associated to some learned cost ) that matches the expert ’ s future state occupancy measure ρπE marginalised over the initial state distribution , where ρπ ( s , a|s0 ) = ∑ t γ tPπ ( st = s , at = a|s0 ) . State of the art approaches Ho & Ermon ( 2016 ) ; Fu et al . ( 2017 ) consist , roughly speaking , in a two-step procedure . In the first step , given a cost function ĉ , an ( approximately ) optimal policy π̂ of M̂ ( the MDPM with ĉ for cost function ) , is learned . In the second step , trajectories generated by π̂ are compared to expert ones ( in the sense of ρπ ) ; then ĉ is updated to penalise states unvisited by the expert ( say , by gradient descent over some parameters ) . Obviously , those two steps can be repeated until convergence ( or until the generated and the original data-sets are close enough ) . However , the presence of a discount factor in the definition of ρπ has a huge undesirable effect : the total weight of the states in the far future ( say , after some stage t∗ ) is negligible in the global 1The σ-field is always the Borel one . loss , as it would be of the order of γt ∗ . So trying to match the future state occupancy measure will implicitly favours policies mimicking the behaviour in the short term . As a consequence , this would end up in penalising policies with longer mixing times even if their stationary distribution matches the experts on the long run . This built-in bias is a consequence of solving the reinforcement learning step with policies that optimise the cumulative discounted costs ( minimises the expectation of the Q-functions against p0 ) rather than policies that achieve the Bellman optimality criterion ( minimises the Q-function for any state action pairs ) . Unfortunately , there is no IRL framework solving the problem under the latter assumption . In order to bridge this gap , we introduce a more general optimality criterion for the reinforcement learning step ; it is still defined as the expectation of the Q-function , yet not against p0 as in traditional RL , but against both the initial and the future states distributions . To get some flexibility , we allow the loss to weight present and future states differently by considering a probability distribution η over N. Formally , we define the η-weighted future state measurement distribution : P ηπ ( s+ , a+|s0 ) : = ∞∑ n=0 η ( n ) Pπ ( sn = s+ , an = a+|s0 ) . Using P ηπ , the new criterion is defined as : Eηp0 , π [ Q c π ] : = ∫ s0 p0 ( s0 ) ∫ s+ , a+ P ηπ ( s+ , a+|s0 ) Qcπ ( s+ , a+ ) = Ep0 , π [ ∑ k η ( k ) ∑ t γtct+k ] where ct denotes the cost at the tth observation ( c ( st , at ) ) . Any policy that minimises Eηp0 , π [ Q c π ] will now be referred to as “ η-optimal ” ( w.r.t . the cost function c ) . As mentioned before , the inverse RL problem can be decomposed in two sub-problems , learning approximate optimal strategies ( given a candidate ĉ ) and optimizing over ĉ ( taking into account the expert distribution πE ) . In order to avoid over-fitting when learning optimal policies , the standard way is to regularize the optimization loss Geist et al . ( 2019 ) . As a consequence , we consider any mapping Ω : ∆ ( A ) S → R that is a concave over the space of policies . The associated regularised loss of adopting a policy π given the cost function c is defined as : LηΩ ( π , c ) = E η p0 , π [ Qcπ ] − Ω ( π ) ( 1 ) The generalised RL problem is then defined as : RLηΩ ( c ) : = arg min π LηΩ ( π , c ) ( 2 ) Similarly , in order to learn simpler cost functions Ho & Ermon ( 2016 ) , the optimization loss considered is in turn penalised by a convex ( over the space of cost functions ) regularizer ψ : R ( S×A ) → R. The problem of Generalised ( Maximum Entropy ) Inverse Reinforcement learning , whose objective is to learn an appropriate cost function c , is formally defined as : IRLηψ , Ω ( πE ) : = arg max c min π LηΩ ( π , c ) − L η Ω ( πE , c ) − ψ ( c ) ( 3 ) We emphasise that simply choosing δ0 ( a Dirac mass at 0 ) for the distribution η induces the classical definitions of both the RL and IRL problems Ho & Ermon ( 2016 ) . On the other hand , choosing η = Geom ( γ ) transforms the loss into the expectation of the sum of discounted Q-functions along the trajectory . Hypothetically , there could be other generalisations of discounted cost . However , preserving the compatibility of the Bellman criterion with the proposed generalisation for RL and duality properties for IRL is not trivial ( for example , polynomial decay γtn would break these properties ) . In the following , we prove that the η-optimality framework satisfies both properties .

Summary: The paper proposes a new formulation for inverse reinforcement learning that aims to address the _bias against policies with longer mixing times_. The key contribution of the paper is the proposal of an alternative optimality criterion that arguably addresses the aforementioned bias by considering, for a given policy, the value  $L(\pi,c)=E_{p_0,\pi}\left[\sum_{k=0}^\infty\eta(k)\sum_{t=0}^\infty\gamma^tc_{t+k}\right]$  instead of the more standard definition  $L(\pi,c)=E_{p_0,\pi}\left[\sum_{t=0}^\infty\gamma^tc_{t}\right],$  where $c$ is such that $E[c_t\mid s_t,a_t]=c(s_t,a_t)$.  To solve the IRL problem associated with this new optimality criterion (or, rather, a regularized version thereof) the paper proposes the use of maximum causal entropy IRL, which can roughly be broken down in two subproblems:  - Given a candidate cost function, $\hat{c}$, find a candidate policy, $\hat{\pi}$, that minimizes $L(\hat{\pi},\hat{c})$. In a sense, $\hat{\pi}$ is the "optimal" policy given the cost function $\hat{c}$. The paper proposes the use of soft actor-critic approach (although other value-based approaches could be used).  - Given the expert policy, $\pi_E$, and the candidate policy, $\hat{\pi}$, come up with a candidate cost function $\hat{c}$ such that the expert policy has a low cost and other policies have a high cost, which the paper shows that can be done by minimizing a measure of divergence between the (weighted) distributions induced by $\hat{\pi}$ and $\pi_E$.   The resulting algorithm is a variation of the generative adversarial IRL approach of Ho and Ermon, that the experiments suggest may be able to better recover the expert's policy.