A GENERALISED INVERSE REINFORCEMENT LEARN-
Year: 2021

Abstract:
The global objective of inverse Reinforcement Learning (IRL) is to estimate the unknown cost function of some MDP based on observed trajectories generated by (approximate) optimal policies. The classical approach consists in tuning this cost function so that associated optimal trajectories (that minimise the cumulative discounted cost, i.e. the classical RL loss) are “similar” to the observed ones. Prior contributions focused on penalising degenerate solutions and improving algorithmic scalability. Quite orthogonally to them, we question the pertinence of characterising optimality with respect to the cumulative discounted cost as it induces an implicit bias against policies with longer mixing times. State of the art value based RL algorithms circumvent this issue by solving for the fixed point of the Bellman optimality operator, a stronger criterion that is not well defined for the inverse problem. To alleviate this bias in IRL, we introduce an alternative training loss that puts more weights on future states which yields a reformulation of the (maximum entropy) IRL problem. The algorithms we devised exhibit enhanced performances (and similar tractability) than off-the-shelf ones in multiple OpenAI gym environments.

heading: 1 INTRODUCTION
text: Modelling the behaviours of rational agents is a long active research topic. From early attempts to decompose human and animal locomotion Muybridge (1979) to more recent approaches to simulate human movements Li & Todorov (2006); Mombaur (2009); Schultz & Mombaur (2009), the common thread is an underlying assumption that the agents are acting according to some stationary policies. To rationalise these behaviours, it is natural to assume that they are optimal with respect to some objective function (there are evidences to back this assumption in the case of animal conditioned learning Schmajuk & Zanutto (1997); Verschure & Althaus (2003); Maia (2010); Verschure et al. (2014)).
The global objective of Inverse Reinforcement Learning (IRL) is inferring such objective function given measurements of the rational agent’s behaviour, its sensory inputs and a model of the environment Russell (1998). IRL builds upon the standard Reinforcement Learning (RL) formulation, where the goal is to find the policy that minimises discounted cumulative costs of some Markov Decision Process Puterman (2014). It aims at finding cost functions for which the observed behaviour is “approximately optimal”. However this simplistic formulation admits degenerate solutions Abbeel & Ng (2004). This led to a series of innovative reformulations to lift this indeterminacy by favouring costs for which the observed behaviour is particularly better than alternative ones, namely maximum margin IRL Ratliff et al. (2006) and maximum entropy IRL Ziebart et al. (2008; 2010). The latter formulation ended up as the building block of recent breakthroughs, with both tractable and highly performing algorithms Finn et al. (2016); Ho & Ermon (2016); Fu et al. (2017). These improvements provided the ground for multiple practical real-life applications Ziebart et al. (2008); Bougrain et al. (2012); Sharifzadeh et al. (2016); Jarboui et al. (2019); Martinez-Gil et al. (2020).
We propose an orthogonal improvement to this literature. We question the very pertinence of characterising optimality w.r.t. the cumulative discounted costs as it induces a bias against policies with longer mixing times. We propose an extension of this criterion to alleviate this issue. From this novel objective, we derive reformulations for both the RL and IRL problems. We discuss the ability of existing RL algorithms to solve this new formulation and we generalise existing IRL algorithms to
solve the problem under the new criterion. We back up our proposition with empirical evidence of improved performances in multiple OpenAI gym environments.

heading: 2 GENERALISED OPTIMALITY CRITERION
text: In this section, we introduce the classical settings of RL and IRL, as well as the new generalised settings we introduce to alleviate some inherent biases of current methods.

heading: 2.1 A CLASSICAL RL SETTING
text: Consider an infinite horizon Markov Decision Process (MDP)M = {S,A,P, c, γ, p0}, where: – S is either a finite or a compact subset of Rd, for some dimension d ∈ N – A is either a finite or a compact subset of Rd′ , for d′ ∈ N – P is the state transition kernel, i.e., a continuous mapping from S × A to ∆(S), where ∆(·) denotes the set of probability measures1 over some set, – c : S ×A → R is a continuous non-negative cost function, – p0 ∈ ∆(S) is the initial state distribution, and γ ∈ (0, 1) is the discount factor.
A policy π is a mapping indicating, at each time step t ∈ N, the action at to be chosen at the current state st; it could depend on the whole past history of states/actions/rewards but it is well known that one can focus solely, at least under mild assumptions, on stationary policies π : S → ∆(A). The choice of a policy π, along with a kernel P and the initial probability p0, generates a unique probability distribution over the sequences of states denoted by Pπ (the solution to the forward Chapman–Kolmogorov equation). The expected cumulative discounted cost of this policy, in the MDP M is consequently equal to Ep0,π[ ∑ t γ tc(st, at)] = ∫ s0 p0(s0) ∑∞ t=0 ∫ st,at γtPπ(st, at|s0)c(st, at).
Optimal policies are minimisers of this quantity (existence is ensured under mild assumptions Puterman (2014)). A standard way to compute optimal policies, is to minimise the state-action value mapping defined as: Qcπ(s, a) = c(s, a) + ∑∞ t=1 ∫ st,at γtPπ(st, at|s)c(st, at). Indeed, the expected cumulative discounted cost of a policy is the expectation of Q-function against p0:
Ep0,π[ ∞∑ t=0 γtc(st, at)] = ∫ s0,a0 p0(s0)π(a0|s0)Qcπ(s0, a0)

heading: 2.2 A BUILT-IN BIAS IN THE IRL FORMULATION
text: The problem gets more complicated in Inverse Reinforcement Learning where the objective is to learn an unknown cost function c whose associated optimal policy coincides with a given one πE (referred to as the “expert” policy). This problem is unfortunately ill-posed as all policies are optimal w.r.t. a constant cost function Abbeel & Ng (2004). In order to lift this indeterminacy, the most used alternative formulation is called maximum entropy inverse reinforcement learning Ziebart et al. (2008; 2010) that aims at finding a cost function c∗ such that the expert policy πE has a relatively small cumulative cost Ep0,πE [ ∑∞ t=0 γ
tc∗] while other policies incur a much higher cost. This implicitly boils down to learning an optimal policy (associated to some learned cost) that matches the expert’s future state occupancy measure ρπE marginalised over the initial state distribution, where ρπ(s, a|s0) = ∑ t γ
tPπ(st = s, at = a|s0). State of the art approaches Ho & Ermon (2016); Fu et al. (2017) consist, roughly speaking, in a two-step procedure. In the first step, given a cost function ĉ, an (approximately) optimal policy π̂ of M̂ (the MDPM with ĉ for cost function), is learned. In the second step, trajectories generated by π̂ are compared to expert ones (in the sense of ρπ); then ĉ is updated to penalise states unvisited by the expert (say, by gradient descent over some parameters). Obviously, those two steps can be repeated until convergence (or until the generated and the original data-sets are close enough).
However, the presence of a discount factor in the definition of ρπ has a huge undesirable effect: the total weight of the states in the far future (say, after some stage t∗) is negligible in the global
1The σ-field is always the Borel one.
loss, as it would be of the order of γt ∗ . So trying to match the future state occupancy measure will implicitly favours policies mimicking the behaviour in the short term. As a consequence, this would end up in penalising policies with longer mixing times even if their stationary distribution matches the experts on the long run. This built-in bias is a consequence of solving the reinforcement learning step with policies that optimise the cumulative discounted costs (minimises the expectation of the Q-functions against p0) rather than policies that achieve the Bellman optimality criterion (minimises the Q-function for any state action pairs). Unfortunately, there is no IRL framework solving the problem under the latter assumption.
In order to bridge this gap, we introduce a more general optimality criterion for the reinforcement learning step; it is still defined as the expectation of the Q-function, yet not against p0 as in traditional RL, but against both the initial and the future states distributions. To get some flexibility, we allow the loss to weight present and future states differently by considering a probability distribution η over N. Formally, we define the η-weighted future state measurement distribution:
P ηπ (s+, a+|s0) := ∞∑ n=0 η(n)Pπ(sn = s+, an = a+|s0).
Using P ηπ , the new criterion is defined as:
Eηp0,π[Q c π] := ∫ s0 p0(s0) ∫ s+,a+ P ηπ (s+, a+|s0)Qcπ(s+, a+) = Ep0,π [∑
k η(k) ∑ t γtct+k ] where ct denotes the cost at the tth observation (c(st, at)). Any policy that minimises Eηp0,π[Q c π] will now be referred to as “η-optimal” (w.r.t. the cost function c). As mentioned before, the inverse RL problem can be decomposed in two sub-problems, learning approximate optimal strategies (given a candidate ĉ) and optimizing over ĉ (taking into account the expert distribution πE). In order to avoid over-fitting when learning optimal policies, the standard way is to regularize the optimization loss Geist et al. (2019). As a consequence, we consider any mapping Ω : ∆(A)S → R that is a concave over the space of policies. The associated regularised loss of adopting a policy π given the cost function c is defined as:
LηΩ(π, c) = E η p0,π [ Qcπ ] − Ω(π) (1)
The generalised RL problem is then defined as:
RLηΩ(c) := arg min π LηΩ(π, c) (2)
Similarly, in order to learn simpler cost functions Ho & Ermon (2016), the optimization loss considered is in turn penalised by a convex (over the space of cost functions) regularizer ψ : R(S×A) → R. The problem of Generalised (Maximum Entropy) Inverse Reinforcement learning, whose objective is to learn an appropriate cost function c, is formally defined as :
IRLηψ,Ω(πE) := arg max c min π LηΩ(π, c)− L η Ω(πE , c)− ψ(c) (3)
We emphasise that simply choosing δ0 (a Dirac mass at 0) for the distribution η induces the classical definitions of both the RL and IRL problems Ho & Ermon (2016). On the other hand, choosing η = Geom(γ) transforms the loss into the expectation of the sum of discounted Q-functions along the trajectory.
Hypothetically, there could be other generalisations of discounted cost. However, preserving the compatibility of the Bellman criterion with the proposed generalisation for RL and duality properties for IRL is not trivial (for example, polynomial decay γtn would break these properties). In the following, we prove that the η-optimality framework satisfies both properties.

heading: 2.3 GENERALISED REINFORCEMENT LEARNING
text: As in the classical setting, solving the generalised IRL problem (Equation 3), requires solving the generalised RL problem (Equation 2) as a sub-routine. Among the model free RL algorithms, valuebased vs. policy gradient-based methods can be distinguished. In this section, we focus on the first type of methods as they can easily be used for the search of η-optimal policies. We provide a detailed discussion of the limitations of current policy gradient-based methods in Appendix A, as they might be less adapted to solving RLηΩ.
Given a standard MDPM and policy π, the Bellman operator Tπ (from RS to RS) is defined as Geist et al. (2019):
[Tπ(v)](s) = Ea∼π [ c(s, a) + γEs′|s,a[v(s′)] ] ,
and its unique fixed point is called the associated value function vcπ . This concept is transposed to the regularised case as follows: Given a concave regularisation function Ω and a policy π, the associated regularised Bellman operator Tπ,Ω and the associated value function vcπ,Ω are respectively defined as: Tπ,Ω : v ∈ RS → Tπ,Ω(v) = Tπ(v)− Ω(π) ∈ RS , and as: vcπ,Ω = Tπ,Ω(v c π,Ω), its unique fixed point. As usual, the regularised Bellman optimality operator T∗,Ω is in turn defined as:
T∗,Ω : v ∈ RS 7→ T∗,Ω(v) ∈ RS
[T∗,Ω(v)](s) = min π
[Tπ,Ω(v)](s), ∀s ∈ S.
Notice that given v ∈ RS , the policy π̄v(·|s) = δā with ā = arg mina c(s, a) + γEs′|s,a[v(s′)] achieves the minimum in the overall equation for all state s ∈ S. The policy improvement theorem Sutton & Barto (2018) guarantees that if vcπ,Ω is the regularised value function of π, then π̄ := π̄vcπ,Ω dominates π (in the sense that v c π̄,Ω(s) ≤ vcπ,Ω(s) for any state s ∈ S).
If we denote by vc∗,Ω the unique fixed point of the regularised Bellman optimality operator T∗,Ω, then the policy π∗Ω := π̄vc∗,Ω is associated to the minimum regularised value function:
Proposition 1 Optimal regularised policy (Theorem 1 of Geist et al. (2019)) : The policy π∗Ω := π̄vc∗,Ω is the unique optimal regularised policy in the sense that, for all policies π, the following holds:
∀s ∈ S, vcπ∗Ω,Ω(s) = v c ∗,Ω(s) ≤ vcπ,Ω(s).
Notice how the optimal regularised policy π∗Ω (that minimizes the regularised cumulative discounted cost), is intuitively a good proxy of optimal policies in the sense of the regularised η-weighted Q-functions (as it optimises the Q-function for all state action pairs and therefor indirectly optimises the η-weighted Q-functions).
For this reason, we propose to exploit state of the art value base RL algorithms that optimise v∗,Ω (such as Soft Actor Critic (SAC) Haarnoja et al. (2018)) to approximately solve the generalised setting. Naturally, the optimality gap depends directly on the exact form of the distribution η. Unfortunately, there is no known performance bound for such approximation. However, we found out empirically that for Geometric and Poisson η distributions the derived IRL algorithms for the generalised setting (which we aim to solve in this paper) did not suffer from this approximation.

heading: 2.4 GENERALISED INVERSE REINFORCEMENT LEARNING
text: We recall that the global objective of IRL is to learn the cost function based on an expert policy πE . In this section, we illustrate that the solution of IRLηψ,Ω(πE) is a cost function ĉ, whose associated optimal policy RLηΩ(ĉ) matches the expert’s future state distributions P η πE marginalised against ρπE rather than simply matching the occupancy measure ρπE , as in usual IRL formulation. To alleviate notations, we denote the η-optimal policy π̂ = RLηΩ(ĉ) as RL η Ω ◦ IRL η ψ,Ω(πE). This policy minimises the worst-case cost weighted divergence dc(π̂‖πE) : S 7→ R averaged over p0, such that:
dc(π̂‖πE)(s0) := ∫ s,a,s+,a+ c(s+, a+) [ ρπ̂(s, a|s0)P ηπ̂ (s+, a+|s, a)− ρπE (s, a|s0)P η πE (s+, a+|s, a) ] This is formalised in the following proposition that requires the following notations. Given a policy π we denote by µπ(s+, a+|s0) = ∑ t,k γ
tη(k)Pπ(st+k = s+, at+k = a+|s0) the frequency of (s+, a+) in the η-weighted future steps of trajectories initialised according to ρπ(s, a|s0).
Proposition 2 For any convex penalty ψ, concave regulariser Ω (w.r.t. the future occupancy measure µπ) and any expert policy πE , if η is geometric, then:
RLηΩ ◦ IRL η ψ,Ω(πE) = arg min
π max c L(π, c) where: L(π, c) = −Ω(π)− ψ(c) + ∫ s0 p0(s0)dc(π‖πE)(s0)
As mentioned before, Proposition 2 states that in its generalised formulation, solving the IRL problem can be done by matching η-weighted future state distributions µπ (as opposed to matching ρπ in the classical case). This proves that the generalised setting preserves the duality properties of classical IRL. The solution of IRLηψ,Ω is a Nash-Equilibrium of a game between poicies and the cost functions:
Corollary 2.1 Under the assumptions of Proposition 2, (c̃, π̃) = (IRLηψ,Ω(πE),RL η Ω(c̃)) is a NashEquilibrium of the following game:
c̃ : maxc L(π, c) + Ω(π) ; π̃ : minπ E π + [ Qπ(c) ] − Ω(π)
A practical implication of Corollary 2.1 is a template algorithm dubbed GIRL(ψ,Ω,η) and illustrated in Algorithm 1 that can be used to solve this problem approximately.
Algorithm 1 GIRL(ψ,Ω,η) (Generalised IRL)
1: Input: Expert trajectories τE ∼ πE , initial policy πθ0 and initial cost function cw0 2: for e ∈ [1, N ] do 3: Sample trajectories τ ∼ πθi 4: Sample from τ policy state action (S+, A+) ∼ µηπ(τ) 5: Sample from τE expert state action (S+E , A + E) ∼ µηπE (τE)
6: Update the cost parameter wi to maximise −ψ(cw) + ∑ S+,A+ cw(s, a)− ∑ S+E ,A + E cw(s, a) 7: Update θi using a value-based reinforcement learning algorithm to minimise cwi+1 8: Return: (πθN , cwN )
We stress out now that the concavity of Ω w.r.t. µπ in Proposition 2 is not too restrictive in practical settings as the η-weighted entropy regulariser, amongst others, satisfies it:
Proposition 3 The η-weighted entropy regulariser H̄ηp0 defined by
H̄ηp0(µπ) := H η p0(π) = E η p0,π [∑ t −γt log [ π(at|st) ]] is concave with respect to the occupancy measure µπ .

heading: 3 TRACTABILITY
text: The tractability of GIRLψ,Ω,η is a crucial requirement for practical implementation. In this section, both the regulariser term Ω(·) and the penalty term ψ(·) are assumed to be tractably optimisable. For example the entropy, a widely used regulariser in the RL literature is efficiently tractable in practice. Indeed, Soft Actor Critic Haarnoja et al. (2018) uses a single sample approximation of the entropy to optimise the entropy regularised Bellman optimality operator. Similarly, using an indicator penalty over a subset C of possible cost functions (i.e., the penalty is infinite if c /∈ C and 0 otherwise) is also tractable with projected gradient updates if C is convex Abbeel & Ng (2004); Syed et al. (2008); Syed & Schapire (2007). As a consequence, establishing tractability of GIRLψ,Ω,η reduces to finding tractable sampling schemes from µπ . This is equivalent to sampling sequentially from the η-weighted future state distribution and the occupancy measure as:
µπ(s+, a+|s0) = ∫ s,a ρπ(s, a|s0)P ηπ (s+, a+|s, a) = ∫ s,a P ηπ (s, a|s0)ρπ(s+, a+|s, a)
Given a policy π, the simplest approach to sample from these distributions is to sample transitions from a set of π-generated trajectories, denoted by {(s(i)t , a (i) t )t∈{1,H}; i ∈ {1, N}}, where H is the horizon and N is the number of trajectories.
– For the occupancy measure ρπ: given a uniformly sampled index i ∼ U [1, N ] and a time sampled from truncated geometric distribution t ∼ Geom[1,H](γ), the associated pair of state/action (s (i) t , a (i) t ) is an (approximate) sample from the marginal of ρπ(.|s0) against p0. – For the future state distribution P ηπ : Given a state s (i) t sampled as above, a time k is sampled from a truncated η[1,H−t]; the state-action (s (i) t+k, a (i) t+k) is an approximate sample from P η π (.|st).
As a consequence, the above scheme shows that sampling from ρπ and P ηπ reduces to sampling indices from Geom(γ) and η, which is tractable from both the expert and the learned policies perspective. This proves that solving IRLηψ,Ω does not incur any additional computational burden.

heading: 4 MEGAN: MAXIMUM ENTROPY - GENERATIVE ADVERSARIAL NETWORK
text: This section introduces a new algorithm, called MEGAN, that will improve upon state of the art IRL algorithms. Recent progresses in the field propose variations of GAIL Ho & Ermon (2016) in order to solve a wide variety of problems. For example, AIRL Fu et al. (2017) uses a particular shape for the discriminator for better transferability of the learned rewards, EAIRL Qureshi et al. (2018) applies empowerment regulariser to policy updates to prevent over-fitting the expert demonstration, RAIRL Jeon et al. (2020) generalises AIRL for regularised MDPs (i.e. Ω is not necessarily the entropy), s-GAIL Kobayashi et al. (2019) generalises the formulation for multi-task RL, etc.
Their contributions were crucial to the progresses of IRL. However, we will actually focus on improving the core algorithm GAIL so that all the aforementioned approaches can be implemented with MEGAN instead of GAIL with improved performances.
We considered the rather classical penalty function Ho & Ermon (2016):
ψGAN (c) =
{ Eηp0,πE [g(c(s, a))] if c < 0 +∞ otherwise where: g(x) = { −x− log(1− ex) if x < 0 +∞ if x ≥ 0
The generalised problem boils down to using Eηπ instead of Eπ:
Proposition 4 Under the assumptions of Proposition 2, and for ψ = ψGAN (c), it holds:
RLηΩ ◦ IRL η ψ(πE) = arg min π −Ω(π) + max D∈(0,1)S×A Eηπ[logD]− EηπE [log(1−D))]
The algorithm MEGAN (Maximum Entropy - Generative Adversarial Network), is then equivalent to GIRLψGAN ,H,Geom(γ) and a generalisation of the corner stone in state of the art IRL Ho & Ermon (2016); its pseudo-code is given in Algorithm 2.
Algorithm 2 MEGAN 1: Input: Expert trajectories τE ∼ πE , initial policy πθ0 and initial discriminator function Dw0 2: for e ∈ [1, N ] do 3: Sample trajectories τ ∼ πθi 4: Sample states randomly (St, At) ∼ τ and (S+, A+) = (St+k, At+k) where k ∼ η 5: Sample states randomly (S′t, A ′ t) ∼ τE and (S+E , A + E) = (S ′ t+k, A ′ t+k) where k ∼ η
6: Update the cost parameter wi to maximise [ logDw(S +, A+)− log(1−Dw(S+E , A + E)) ]
7: Update θi using soft actor critic to minimise the cost [ logDwi+1 ] 8: Return: (πθN , DwN )

heading: 5 EXPERIMENTS
text: This section is devoted to experimental evidences that MEGAN achieves state of the art performances. It is compared to GAIL as all subsequent approaches build upon its formulation. The standard approach to compare IRL algorithms is to consider the best performing policies obtained during the training and evaluate their performances. This is an issue in practice as we do not have access to such cost function in order to implement a stopping rule once the learned policy reaches a certain performance threshold. A reasonable alternative criterion is to measure the divergence between generated and expert future state distributions (in the sense of ρπ or µπ). In this section, we propose to evaluate the divergence using the maximum mean discrepancy (MMD)2. We will tackle the following questions empirically:
2A formal reminder on the definition of MMD divergence is provided in Appendix E.1 for completeness.
-1 How does varying the parameter of a geometric η distribution affect performances? -2 How does alternative η distribution (e.g. a Poisson) compare to the use of a geometric one? -3 Does varying the discount factor γ produce similar performances?
Due to limited space, we only analyse single-task environments in this section. We provide in Appendix C and D further investigations for the multi-task setting. A summary of the used hyperparameters is also provided in Appendix E.2.
5.1 PERFORMANCE IMPROVEMENT USING A GEOMETRIC η DISTRIBUTION
Recall that solving the IRL problem essentially boils down to finding an equilibrium between a policy that matches the expert behaviour and a cost function that discriminates generated trajectories from expert ones. An important property of a given algorithm is the stability of the associated equilibrium. In order to take into account this aspect, we propose to evaluate performances using trajectories sampled during the last 100 iterations of training. We will refer to these trajectories as the remaining replay buffer. This procedure provides an evaluation of the policies toward which the algorithm converges, while factoring in their stability.
Notice that the goal of GAIL is to match the distribution ρπE while MEGAN matches the distribution µηπE . In order to take into account this difference, we propose to measure performances in terms of cumulative costs, MMDρ = MMD(ρπ, ρπE ) and MMDµ = MMD(µ GEOM(0.99) π , µ GEOM(0.99) πE ).
We evaluate the performances of MEGAN using a truncated3 geometric η distribution with different parameters (specifically {0, 0.25, 0.5, 0.75, 1}). Notice that using a geometric distribution with parameter 0 is equivalent to using a Dirac mass at 0 (or equivalently solving the IRL problem using GAIL). Similarly, using a geometric distribution with parameter 1 is equivalent to using a uniform η distribution. The remaining values can be seen as an interpolation between these extremes.
In Figure 1, each point reports the average performances obtained using the remaining replay buffer of 3 randomly seeded instances of the algorithm. The blue curves report the average MMDρ (divergence in the sense of the classical IRL formulation), the green curves report the average MMDµ (divergence in the sense of the generalised IRL formulation), and the red curves report the average cumulative costs (divergence in the sense of the environment’s ground truth). From left to right, we report the performances in three classical control settings with varying complexity from the MuJoCo based environments Plappert et al. (2018). In Figure 1a we used the Ant environment (a state action space of dimension 118), in Figure 1b we used the Half-Cheetah environment (a state action space of dimension 23) and in Figure 1c we used the Hopper environment (a state action space of dimension 14).
All the provided experiments confirmed a reduction of the average MMD divergence by 25% to 60% (in the sense of both classical IRL and generalised IRL formulations) as the parameter of the η distribution increased to 1. This confirms that using the η-optimality objective function improves both the stability and the ability of IRL algorithms to match faithfully the expert behaviour. Notice that despite the fact that GAIL explicitly optimises divergence in the sense of ρπ , it under-performs in the sense of MMDρ when compared to MEGAN (the blue curves decreases as the parameter of the geometric distribution increases). This confirms empirically that the η-optimality framework proposed in this paper does indeed bridge the gap between policy-based reinforcement learning (optimising cumulative discounted costs) and value-based reinforcement learning (achieving the Bellman optimality criterion) as it even improves performances in the sense of classical IRL.
Another important observation in Figure 1, is that for complex environments (Ant and Half-Cheetah) the decrease of the MMD divergence -as we increased the parameter of the geometric distribution to 1- was correlated with a decrease of the average cumulative costs by a factor of 2 to 4. This was not the case of the Hopper environment, as we obtained similar cumulative costs despite the reduction of the divergence by a factor of 3. This is explained by the fact that the ground truth cost function of the Hopper environment produces similar cumulative costs for a wider variety of policies. For this reason, the IRL solution does not need to achieve a faithful expert behaviour matching in order to achieve good performances. This illustrates the importance of evaluating IRL algorithm with respect to MMDρ and MMDµ when the goal is to mimic behaviors.
3Due to obvious computational limitations, the trajectories are finite
5.2 PERFORMANCE IMPROVEMENT USING A POISSON η DISTRIBUTION
Using the same experimental setting from the previous section, we evaluate MEGAN’s performances with non-geometric η distributions. In Figure 2, we plot the performances of the remaining replay buffer obtained with geometric η distributions in blue lines, and those obtained when using a Poisson distribution in red. To reduce clutter, we removed the cumulative costs and only provided the divergences MMDρ (represented with solid lines in Figure 2) and MMDµ (dashed lines in Figure 2).
Despite the weaker theoretical guarantees provided in our paper for the case of non-geometric η distributions, we observe that using a Poisson η can lead to comparable performances. Recall that the expectation of a Poisson distribution is equal to its parameter value. This implies that solving IRL
Poisson(λ) ψ,Ω searches for policies that match ρπE (.|s) for states s observed around the λth frame of the expert demonstrations. Now notice that the control tasks analysed in Figure 2, consist of movements cycles that are repeated perpetually. Quite interestingly, setting λ to a value around the length of an expert cycle (λ = 10 in the Ant environment, 25 in the Half-Cheetah, and 40 for the Hopper), ended up achieving the best performances.
In a sense, the proposed η-optimality criterion can be seen as an inductive bias: we successfully injected qualitative knowledge (the repetitive nature of the expert behaviour) by explicitly asking the agent to focus on matching ρπ(.|s) for states s observed within a single movement cycle of the expert demonstrations via careful parameterisation of the distribution η. In the case where such higher understanding/representation of the expert behaviour is unavailable, using a uniform η distribution (or a geometric η with a parameter close to 1) is a safe bet. Notice that in Figure 2, the both the blue and red curves have comparable minimum values.
5.3 EFFECT OF VARYING THE DISCOUNT FACTOR γ
Recall that in the classical problem formulation, the discount factor γ can be interpreted as the weight of future observations. In this section, we investigate whether changing this parameter can overcome the buit-in bias against policies with longer mixing times, without resorting to the η-optimality criterion. In practice, the discount factor γ is used separately in two building blocks of IRL algorithms (including GAIL and MEGAN). The first instance is in the Bellman updates when solving the RL problem under a given cost function: we refer to this parameter as γRL. The second instance is in the discrimination problem when approximately sampling from ρπ: we refer to this parameter as γIRL.
Due to the finite nature of the expert demonstrations, the standard approach is to approximate future state distributions by setting γIRL to 1 (or equivalently, sampling transitions uniformly). This can be seen as an asymptotic behaviour of the classical problem formulation: as the discount factor approaches 1, the associated truncated geometric distribution will converge to a uniform one. Reducing this parameter will only accentuate the discussed issues as it entails up-sampling the early stages of the collected demonstrations, which will inevitably favor short term imitation. On the other hand, it is not possible to set γRL = 1 (the Bellman operator is no longer guaranteed to admit a fixed point, and state of the art value based RL algorithms become extremely unstable). For this reason, most practitioners set this parameter to a value reasonably close to 1.
In this section, we evaluated the remaining replay buffers obtained using GAIL as we vary the discount factors values (γRL ∈ [0.9, 0.99, 0.999]). We emphasize that in all the reported empirical evaluations (including previous ones, i.e. Figures 1 and 2), we fixed γIRL to 1.
In Figure 3, we report the average MMDρ divergence of the remaining replay buffer in solid lines, and the average MMDµ divergence in dashed lines. GAIL’s performances as we vary the discount factor are reported in blue and the best performances obtained with MEGAN are reported in red. Reducing γRL to 0.9 accentuated the bias against policies with longer mixing times, and on the other hand increasing it to 0.999 lead to a less reliable RL algorithm. As expected, we observe in Figure 3 that both tweaks did not entail performances on par to what we obtained using MEGAN.

heading: 6 CONCLUSION
text: In this paper, we generalised the classical criterion of optimality in the reinforcement learning literature by putting more weights onto future observations. Using this novel criterion, we reformulated both the regularised RL and the maximum-entropy IRL problems. We reviewed existing RL algorithms and discussed their ability to search for η-optimal policies. We also generalised classical IRL solutions. The derived algorithm produced stable solutions with enhanced expert matching properties.
In practice, the main difference between MEGAN and GAIL is the discriminator’s sampling procedure. This implies that it can easily replace the latter algorithm in all subsequent contributions. An interesting future direction of research consists in evaluating the margin of improvement that can be gained from this modification.
BROADER IMPACT STATEMENT
Inverse reinforcement learning provides a framework to explain observed complex behaviors in sophisticated environments. However, the inherent biases of current formulations againt policies with longer mixing times due to the use of geometrically discounted optimality criterion, prevent existing algorithms from reaching the frameworks full potential. By constructing approximate solutions of the IRL problem (in the sense of the η-optimality criterion), we believe that we take a step towards more applicable IRL algorithms. In that spirit, we also commit ourselves to releasing our code soon in order to allow the wider community to extend our work in the future.

Review summaries:
This paper points out that the classical IRL approach has a tendency to match those occupancy measures that favor short-term behavior. To address this issue, a reformulation is proposed based on GAIL in order to put more emphasis on matching longer-term behavior. Specifically, the main difference is to replace the standard objective (i.e., the expectation of Q function over some initial state distribution) with the expectation of Q function over both the initial and an $\eta$-weighted future state distribution, where $\eta$ is some probability distribution over the support set of nonnegative integers. Built on this formulation, this paper proposes GIRL (and the resulting algorithm MEGAN), which follows the framework of GAIL to learn a policy that matches the $\eta$-weighted variant of occupancy measure of the expert policy. Experimental results on MuJoCo are provided to demonstrate the effectiveness of GIRL.

The paper proposes a new formulation for inverse reinforcement learning that aims to address the _bias against policies with longer mixing times_. The key contribution of the paper is the proposal of an alternative optimality criterion that arguably addresses the aforementioned bias by considering, for a given policy, the value  $L(\pi,c)=E_{p_0,\pi}\left[\sum_{k=0}^\infty\eta(k)\sum_{t=0}^\infty\gamma^tc_{t+k}\right]$  instead of the more standard definition  $L(\pi,c)=E_{p_0,\pi}\left[\sum_{t=0}^\infty\gamma^tc_{t}\right],$  where $c$ is such that $E[c_t\mid s_t,a_t]=c(s_t,a_t)$.  To solve the IRL problem associated with this new optimality criterion (or, rather, a regularized version thereof) the paper proposes the use of maximum causal entropy IRL, which can roughly be broken down in two subproblems:  - Given a candidate cost function, $\hat{c}$, find a candidate policy, $\hat{\pi}$, that minimizes $L(\hat{\pi},\hat{c})$. In a sense, $\hat{\pi}$ is the "optimal" policy given the cost function $\hat{c}$. The paper proposes the use of soft actor-critic approach (although other value-based approaches could be used).  - Given the expert policy, $\pi_E$, and the candidate policy, $\hat{\pi}$, come up with a candidate cost function $\hat{c}$ such that the expert policy has a low cost and other policies have a high cost, which the paper shows that can be done by minimizing a measure of divergence between the (weighted) distributions induced by $\hat{\pi}$ and $\pi_E$.   The resulting algorithm is a variation of the generative adversarial IRL approach of Ho and Ermon, that the experiments suggest may be able to better recover the expert's policy.