A Generalised Inverse Reinforcement Learning Framework
Authors: Firas Jarboui
Vianney Perchet
Venue: ICLR 2022 Submitted (2022)
Keywords: IRL

Abstract: The global objective of inverse Reinforcement Learning (IRL) is to estimate the unknown cost function of some MDP based on observed trajectories generated by (approximate) optimal policies. The classical approach consists in tuning this cost function so that associated optimal trajectories (that minimise the cumulative discounted cost, i.e. the classical RL loss) are “similar” to the observed ones. Prior contributions focused on penalising degenerate solutions and improving algorithmic scalability. Quite orthogonally to them,  we question the pertinence of characterising optimality with respect to the cumulative discounted cost as it induces an implicit bias against policies with longer mixing times. State of the art value based RL algorithms circumvent this issue by solving for the fixed point of the Bellman optimality operator, a stronger criterion that is not well defined for the inverse problem.
To alleviate this bias in IRL, we introduce an alternative training loss that puts more weights on future states which yields a reformulation of the (maximum entropy) IRL problem. The algorithms we devised exhibit enhanced performances (and similar tractability) than off-the-shelf ones in multiple OpenAI gym environments.

Under review as a conference paper at ICLR 2022
A GENERALISED INVERSE REINFORCEMENT LEARN- ING FRAMEWORK
Anonymous authors Paper under double-blind review
ABSTRACT
The global objective of inverse Reinforcement Learning (IRL) is to estimate the unknown cost function of some MDP based on observed trajectories generated by (approximate) optimal policies. The classical approach consists in tuning this cost function so that associated optimal trajectories (that minimise the cumulative discounted cost, i.e. the classical RI. loss) are "similar" to the observed ones. Prior contributions focused on penalising degenerate solutions and improving algorithmic scalability. Quite orthogonally to them, we question the pertinence of characterising optimality with respect to the cumulative discounted cost as it induces implicit bias against policies with longer mixing times. State of the art value based RL algorithms circumvent this issue by solving for the fixed point of the Bellman optimality operator, stronger criterion that not well defined for the inverse problem. To alleviate this bias in IRI., we introduce an alternative training loss that puts more weights on future states which yields reformulation of the (maximum entropy) IRL problem. The algorithms we devised exhibit enhanced performances (and similar tractability) than off-the-shelf ones in multiple OpenAl gym environments.
INTRODUCTION
Modelling the behaviours of rational agents is a long active research topic. From early attempts to decompose human and animal locomotion Muybridge/ (1979) to more recent approaches simulate human movements Li & Todorov (2006); Mombaur (2009); (chultz & Mombaur (2009), the common thread an underlying assumption that the agents are acting according to some stationary policies. To rationalise these behaviours, it is natural to assume that they are optimal with respect to some objective function (there are evidences to back this assumption in the case of animal conditioned learning Schmajuk & Zanutto 1997); Verschure & Althaus (2003); Maia (010); Verschure et al. (2044))
The global objective of Inverse Reinforcement Learning (IRL) is inferring such objective function given measurements of the rational agent's 's behaviour, its sensory inputs and a model of the envi- ronment |ussell (1998). IRI. builds upon the standard Reinforcement Learning (RI.) formulation, where the goal is to find the policy that minimises discounted cumulative costs of some Markov Decision Process Puterman (2014). It aims finding cost functions for which the observed behaviour is "approximately optimal". However this simplistic formulation admits degenerate solutions Abbeel Ng (2004). This led to series of innovative reformulations lift this indeterminacy by favouring costs for which the observed behaviour is particularly better than alternative ones, namely maximum margin IRL Ratliff et al. (206)) and maximum entropy IRL Ziebart et (2008 2010). The latter formulation ended up the building block of recent breakthroughs, with both tractable and highly performing algorithms Finn al. (2016); (Ho & Ermon (0T6); : Fu et al. (2017). These improvements provided the ground for multiple practical real-life applications Ziebart et al. (2008); Bougrain et al. (2012); Sharifzadeh et al. (2016); Jarboui et al. (2019); Martinez- et al. (2020).
We propose an orthogonal improvement to this literature. We question the very pertinence of characterising optimality w.r.t. the cumulative discounted costs as induces a bias against policies with longer mixing times. We propose extension of this criterion to alleviate this issue. From this novel objective, we derive reformulations for both the RL and IRL problems. We discuss the ability of existing RI. algorithms to solve this new formulation and generalise existing IRL algorithms to

---

Under review as a conference paper at ICLR 2022
solve the problem under the new criterion. We back up our proposition with empirical evidence of improved performances in multiple OpenAl gym environments.
GENERALISED OPTIMALITY CRITERION
In this section, we introduce the classical settings of RL and IRL, as well as the new generalised settings we introduce to alleviate some inherent biases of current methods.
2.1 A CLASSICAL RL SETTING
Consider an infinite horizon Markov Decision Process (MDP) M = A, 7, where: S is either finite or a compact subset of R for some dimension d E: IN
A is either a finite or a compact subset of RR for d' E
P is the state transition kernel, i.e., continuous mapping from 8 xx A to A(S), where A(-) denotes the set of probability measures] over some set,
S x A + a continuous non-negative cost function,
Po A(S) is the initial state distribution, and y (0, 1) is the discount factor.
A policy ι is a mapping indicating, at each time step t E IN, the action a to be chosen at the current state s;; it could depend on the whole past history of states/actionssrewarrd but it is well known that one can focus solely, at least under mild assumptions, on stationary policies : S + A(A). The choice of policy æ, along with kernel P and the initial probability Po- generates a unique probability distribution over the sequences of states denoted by Ir (the solution to the forward equation). The expected cumulative discounted cost of this policy, in the MDP M is consequently equal to Epo.= (P. 7 ('c(Ss, 0t) = Je, Po (o())) Ei= J (8t, ( az).
Ghapman-Kolmoog
Optimal policies are minimisers of this quantity (existence is ensured under mild assumptions Puterman (2014)). A standard way to compute optimal policies, to minimise the state-action value mapping defined as: (s, a) c(s,a) + Er= Jepaf x (st, at (els))(s,, a.). Indeed, the expected cumulative discounted cost of policy is the expectation of ffnucction against po:
/ (aos)) dn)
E
p
r'c(s,,a)]
20
t=
2.2 A BUILT-IN BIAS IN THE IRL FORMULATION
The problem gets more complicated in Inverse Reinforcement Learning where the objective is to learn unknown cost function whose associated optimal policy coincides with given one me (referred to as the "expert" policy). This problem is unfortunately ill-posed as all policies are optimal w.r.t. constant cost function Abbeel & Ng (2004). In order lift this indeterminacy. the most used alternative formulation is called maximum entropy inverse reinforcement learning Ziebarr al. (2008; 2010) that aims finding a cost function such that the expert policy Rg has relatively small cumulative cost E 12 7'6²| while other policies incur a much higher cost. This implicitly boils down to learning an optimal policy (associated to some learned cost) that matches the expert's future state occupancy measure Pxg marginalised over the initial state distribution, where Pa (s, also) = E. P'P (s; = 8, a, = also).
State of the art approaches Ho & Ermon (0016); Fu et al. (017)) consist, roughly speaking, in two-step procedure. In the first step, given a cost function c, an (approximately) optimal policy ń of M (the MDP M with for cost function), is learned. In the second step, trajectories generated by î compared to expert ones (in the sense of Px)) ): then é is updated to penalise states unvisited by the expert (say, by gradient descent over some parameters). Obviously, those two steps can be repeated until convergence (or until the generated and the original data-sets are close enough).
However, the presence of a discount factor in the definition of ps has a huge undesirable effect: the total weight of the states in the far future (say, after some stage t*) is negligible in the global
The a-field always the Borel one.
2

---

Under review as a conference paper at ICLR 2022
loss, it would be of the order of y So trying to match the future state occupancy measure will implicitly favours policies mimicking the behaviour in the short term. As a consequence, this would end up in penalising policies with longer mixing times even if their stationary distribution matches the experts on the long run. This built-in bias is consequence of solving the reinforcement learning step with policies that optimise the cumulative discounted costs (minimises the expectation of the Q-functions against Po) rather than policies that achieve the Bellman optimality criterion (minimises the Q-function for any state action pairs). Unfortunately, there is IRL framework solving the problem under the latter assumption.
In order to bridge this gap, we introduce a more general optimality criterion for the reinforcement learning step; it is still defined as the expectation of the Q-function, yet not against as in traditional RI., but against both the initial and the future states distributions. To get some flexibility, we allow the loss to weight present and future states differently by considering probability distribution over N. Formally, we define the y7- weigteted future state measurement distribution:
Pa (8+ (80) = Σ (n))Pp(s = 84, = =a+180)
n=0
Using PT, the new criterion is defined as:
Po. 10. = / Po((o)
/ 24.00
Q2 a+) Epo. Σ [nkk)
Σ Ereech
where e denotes the cost at the 1hh observation (c(se, at)). Any policy that minimises F now be referred to "n-optimall (w.r.t. the cost function c). As mentioned before, the inverse RL problem can be decomposed in two sub-problems. learning approximate optimal strategies (given
"Po.n [ will candidate and optimizing over è (taking into account the expert distribution æg). In order to avoid over- wer-fitting when learning optimal policies, the standard way is to regularize the optimization loss Geist et al. (019)). As consequence, we consider any mapping 5 ((A) - I that is a concave over the space of policies. The associated regularised loss of adopting policy given the cost function e is defined as:
Ch c) E7 E [2=] [((T))
The generalised RL problem is then defined as:
(LÍ(c) arg min CT,(T, (T)
Similarly, in order to learn simpler cost functions Ho & Ermon (2016), the optimization loss consid- ered is in penalised by convex (over the space of cost functions) regularizer dé ((SxxA) IR. The problem of Generalised (Maximum Entropy) Inverse Reinforcement learning, whose objective is learn an appropriate cost function c, is formally defined as
IRL (RL",n((T) = arg max min (T, c) CZ (c)
We emphasise that simply choosing du (a Dirac mass at 0) for the distribution r) induces the classical definitions of both the RL and IRL problems Ho & Ermon (2016). On the other hand, choosing " Geom(7) transforms the loss into the expectation of the sum of discounted Q-functions along the trajectory.
Hypothetically, there could be other generalisations of discounted cost. However, preserving the compatibility of the Bellman criterion with the proposed generalisation for RL and duality properties for IRL is not trivial (for example, polynomial decay -z would break these properties). In the following, we prove that the -ooptimaltty framework satisfies both properties.
2.3 GENERALISED REINFORCEMENT LEARNING
As in the classical setting, solving the generalised IRL problem (Equation 3) requires solving the generalised RL problem (Equation 2 as sub-routine. Among the model free RL algorithms, value- based vs. policy gradient-basse type of methods as they can easily be used for the search of 7-optimll policies. We provide a detailed discussion of the limitations of current policy gradient-bard based methods in Appendix A as they might be less adapted to solving RLY.
methods can be distinguished. In this section, we focus on the first
3

---

Under review as a conference paper at ICLR 2022
Given a standard MDP .M and policy the Bellman operator T7 (from R$ to R$) is defined as Geist et al. (2019):
(s) = Buwn (c(s,) a) + "
and its unique fixed point called the associated value function vg. This concept is transposed the regularised case as follows: Given concave regularisation function Si and a policy the associated regularised Bellman operator Tx,2 and the associated value function o are respectively defined as:
T v E R + (v) = (u) (((T) E R
and as: 02 (v its unique fixed point. As usual, the regularised Bellman optimality operator T=,0 is in turn defined as:
T. Ta: + T(,)()) E R [Ta(v)](ss) =in[(=,,(()(()))) vs E S.
Notice that given E R$, the policy *u(-|s) 6a with à arg min, c(s, a) + t. achieves the minimum in the overall equation for all state s E S. The policy improvement the- orem Sutton & Barto (2188) guarantees that if a is the regularised value function of a, then π. == Ref dominates x (in the sense that Us ( S (s) for any state s E S).
If we denote by " the unique fixed point of the regularised Bellman optimality operator T.n then the policy mh := Mof is associated to the minimum regularised value function:
Proposition 1 Optimal regularised policy (Theorem Geist et al. (2019) The policy wh Teg is the unique optimal regularised policy in the sense that, for all policies n, the following holds:
vs S. olg))(8)) = (1 S (x,)(s).
Notice how the optimal regularised policy u (that minimizes the regularised cunulative discounted cost), is intuitively good proxy of optimal policies in the sense of the regularised w7-weihhted Q-functions (as it optimises the Q-function for all state action pairs and therefor indirectly optimises the weieteeted Q-functions).
For this reason, we propose to exploit state of the art value base RL algorithms that optimise v..0 (such as Soft Actor Critic (SAC) Haarnojj et (2018)) to approximately solve the generalised setting. Naturally, the optimality gap depends directly the exact form of the distribution 1). Unfortunately, there is no known performance bound for such approximation. However, we found out empirically that for Geometric and Poisson y distributions the derived IRL algorithms for the generalised setting (which we aim to solve in this paper) did not suffer from this approximation.
2.4 GENERALISED INVERSE REINFOR LEARNING
We recall that the global objective of IRL is to learn the cost function based on an expert policy TE- In this section, we illustrate that the solution of IRL S. (Tg) is a cost function č, whose associated optimal policy (LQ(C) matches the expert's 's future state distributions P marginalised against PTo rather than simply matching the occupancy measure PTg. as in usual IRL formulation. To alleviate notations, we denote the ooppimal policy ग = (c) as RLG IRL in (T)). This policy minimises the worst-case cost weighted divergence de TE) S R averaged over Po. such that:
de(# (_(f[I=g@@a) (so) - c(s+ a+) pa (s, al
a+ |s, a) - PaR (s, alsu) (s+. a+ a.
-..0 2,0,84.0..
This is formalised in the following proposition that requires the following notations. Given a policy т we denote by Mn (84, (41=0) = Σ., T'n(k)P (*++ = 84, as+* = (=))) the frequency of (84, a.) in the we-whgtt future steps of trajectories initialised according to (r also).
Proposition 2 For any convex penalty bb concave regulariser Q (w.rt. the future occupancy measure (kz) and any expert policy TE, if is geometric, then:
RLA IRL (2 arg min max L(T,c)
where: - - v())) + f (T()))d((TT(
4

---

Under review as a conference paper at ICLR 2022
As mentioned before, Proposition states that in its generalised formulation, solving the IRL problem can be done by matching 77-weitteed future state distributions jln (as opposed to matching Pa in the classical case). This proves that the generalised setting preserves the duality properties of classical is a Nash-Equilibrum of a game between poicies and the cost functions:
IRL. The solution of TRL n ee a
Corollary 2.1 Under the assumptions of Proposition Equilibrium of the following game:
(c. #) ( (C)( is Nash-
ē: max, L(T, + (((T)
#: min, E7 (c) (R(T)
practical implication of Corollary 2.1 is template algorithm dubbed (F. and illustrated in Algorithm that can be used to solve this problem approximately.
Algorithm 1 (2, (Generalised IRL)
1: Input: Expert trajectories Tg ~ Tg, initial policy Te, and initial cost function Cure
2: for E [N do
3: Sample trajectories T To
4: Sample from policy state action (St,A+)= (T)
5: Sample from тæ expert state action (S A 12 (Tg)
6: Update the cost parameter w; to maximise (c)(cw) + + (w, - Est.ng (s
+
7: Update 0; using a value-based reinforcement learning algorithm to minimise Cwi+1
8: Return: (Te Cwn)
We stress out now that the concavity of Ω w.r.t. j4x in Proposition 2 is not too restrictive in practical settings as the n--weighted entropy regulariser, amongst others, satisfies it:
Proposition 3 The n-weighted entropy regulariser H% defined by
H. (wx) = H7 (T) Ep Σ - log [T(OOSS)]]
concave with respect to the occupancy measure Hxz
TRACTABILITY
The tractability of GIRLg is crucial requirement for practical implementation. In this section, both the regulariser term (1-) and the penalty term (.) are assumed to be tractably optimisable. For example the entropy, widely used regulariser in the RL literature is efficiently tractable in practice. Indeed, Soft Actor Critie Haarnoja et (2018) uses single sample approximation of the entropy to optimise the entropy regularised Bellman optimality operator. Similarly, using indicator penalty over a subset of possible cost functions (i.c., the penalty is infinite if & C and 0 otherwise) is also tractable with projected gradient updates if is convex Abbeel & Ng (2004) [yed al. (2008); 'Syed & Schapire (2007). As consequence, establishing tractability of GIRL,.0. reduces to finding tractable sampling schemes from Ma. This equivalent to sampling sequentially from the 17- weightte future state distribution and the occupancy measure as:
[ P (7 al (= (418,
|r (*+,,,0
(8 (/so)) (*+ a+ (1,
Given a policy x, the simplest approach to sample from these distributions to sample transitions from a set of T-generated trajectories, denoted by {( ,a!" " (0))en,,n)s [1, where H is the horizon and N is the number of trajectories.
- For the occupancy measure Px: given uniformly sampled index ~ 2111, N] and a time sampled from truncated geometric distribution Geomp, (7). the associated pair of state/action (s!²) a (i) an (approximate) sample from the marginal of pa (=(-ss) against Po-
For the future state distribution PP: Given a state s sampled as above, a time is sampled from truncated - the state-action (st4ke k ) is approximate sample from PM ( (x).
(8)
5

---

Under review as a conference paper at ICLR 2022
As consequence, the above scheme shows that sampling from Pa and Pa reduces sampling indices from Gyom(y) and 17, which is tractable from both the expert and the learned policies perspective. This proves that solving IRL" 2,2 does not incur any additional computational burden.
MEGAN: MAXIMUM ENTROPY GENERATIVE ADVERSARIAL NETWORK
This section introduces a new algorithm. called MEGAN, that will improve upon state of the art IRL algorithms. Recent progresses in the field propose variations of GAIL Ho & Ermon (2016) in order to solve wide variety of problems. For example, AIRL Fu et al. (2017 uses a particular shape for the discriminator for better transferability of the learned rewards, EAIRL Qureshi et al. (2018) applies empowerment regulariser to policy updates to prevent over-fitting the expert demonstration, RAIRL Jeon et al. (2020) generalises AIRL for regularised MDPs (i.e. Ω is not necessarily the entropy), s s-GAIL Kobayashi et al. (2019) generalises the formulation for multi-task RL, etc.
Their contributions were crucial to the progresses of IRL. However, will actually focus improving the core algorithm GAIL so that all the aforementioned approaches can be implemented with MEGAN instead of GAIL with improved performances.
We considered the rather classical penalty function Ho & Ermon 2016
(C)A(()= = {
Eg Po me (o(c(s,a)) +00
<
where: g(x) =
-r
log(1 e*)
if < 0
otherwise
t00
if
The generalised problem boils down to using Eg instead of Egg:
Proposition 4 Under the assumptions of Proposition 2 and UGGN (c), it holds:
RLg (TTE) arg min (I(
max DE(0.1)SxA
[og D] 1 Dlg(( D))]
The algorithm MEGAN (Maximum Entropy- Generative Adversarial Network), is then equivalent to GIRL Grom(r) and generalisation of the corner stone in state of the art IRL. Ho & Ermon (2016); its pseudo-code is given in Algorithm 2
Algorithm 2 MEGAN
1: Input: Expert trajectories TE ~ Tg, initial policy me, and initial discriminator function Dan
2: for E N] do
3: Sample trajectories T ~ Te:
4: Sample states randomly (St, A.) and (S+ .A+)= (S14k. Actk) where 17
5: Sample states randomly (S,, A1) Te and (St A (8 Al Al(te) where h: 7
6: Update the cost parameter w; to maximise [log (St A+) log(1 Dw (St, A
7: Update 0g using soft actor critic to minimise the cost [log Dw
8: Return: (To Dwx)
EXPERIMENTS
This section is devoted to experimental evidences that MEGAN achieves state of the art performances. It is compared to GAIL as all subsequent approaches build upon its formulation. The standard approach to compare IRL. algorithms is to consider the best performing policies obtained during the training and evaluate their performances. This is an issue in practice such cost function in order to implement stopping rule once the learned policy reaches a certain performance threshold. A reasonable alternative criterion is to measure the divergence between generated and expert future state distributions (in the sense of or (t)). ). In this section, we propose to evaluate the divergence using the maximum mean discrepancy (MMD We will tackle the following questions empirically:
do not have access to
A formal reminder the definition of MMD divergence is provided in Appendix F.1 for completeness.

---

Under review as a conference paper at ICLR 2022
-1 How does varying the parameter of geometric 7, distribution affect performances?
-2 How does alternative distribution (e.g. a Poisson) compare to the use of a geometric one?
-3 Does varying the discount factor y produce similar performances?
Due to limited space, we only analyse single-task environments in this section. We provide in Appendix Can further investigations for the multi-task setting. A summary of the used hyper- parameters also provided in Appendix F.2
5.1 PERFORMANCE IMPROVEMENT USING GEOMETRIC 7y DISTRIBUTION
Recall that solving the IRL problem essentially boils down to finding an equilibrium between policy that matches the expert behaviour and a cost function that discrininates generated trajectories from expert ones. An important property of given algorithm is the stability of the associated equilibrium. In order to take into account this aspect, we propose to evaluate performances using trajectories sampled during the last 100 iterations of training. We will refer to these trajectories the remaining replay buffer. This procedure provides an evaluation of the policies toward which the algorithm converges, while factoring in their stability.
Notice that the goal of GAIL is to match the distribution PaR while MEGAN matches the distribution g In order to take into account this difference, we propose to measure performances in terms of cumulative costs, MMD, > MMD (MD: PFg) and MMD, = MMD( GEOM( MPT
(B.M()
We evaluate the performances of MEGAN using truncated" geometric 1) distribution with different parameters (specifically {0, 0.25, 0.5, 0.75, 1}). Notice that using a geometric distribution with parameter 0 is equivalent to using Dirac mass at 0 (or equivalently solving the IRL problem using GAIL). Similarly, using a geometric distribution with parameter 1 is equivalent to using a uniform distribution. The remaining values can be seen an interpolation between these extremes.
In Figure D each point reports the average performances obtained using the remaining replay buffer of 3 randomly seeded instances of the algorithm. The blue curves report the average MMD,, (divergence in the sense of the classical IRL formulation), the green curves report the average MMD), y. (divergence in the sense of the generalised IRL formulation), and the red curves report the average cumulative costs (divergence in the sense of the environment's s ground truth). From left to right, we report the performances in three classical control settings with varying complexity from the MuJoCo based environments Plappert et al.] 2018). In Figure Ia we used the Ant environment (a state action space of dimension 118), in Figure [b] used the Half-Cheetah environment (a state action space of dimension 23) and in Figure Ic we used the Hopper environment (a state action space of dimension 14).
All the provided experiments confirmed a reduction of the average MMD divergence by 25% to 60% (in the sense of both classical IRL and generalised IRL formulations) as the parameter of the n distribution increased to 1. This confirms that using the n-opimmality objective function improves both the stability and the ability of IRL algorithms to match faithfully the expert behaviour. Notice that despite the fact that GAIL explicitly optimises divergence in the sense of (r. it under-performs in the sense of MMD, when compared to MEGAN (the blue curves decreases as the parameter of the geometric distribution increases). This confirms empirically that the oottimallyy framework proposed in this paper does indeed bridge the gap between policy-based reinforcement learning (optimising cumulative discounted costs) and value-based reinforcement learning (achieving the Bellman optimality criterion) as even improves performances in the sense of classical IRL.
Another important observation in Figure is that for complex environments (Ant and Half- salf-Cheettah) the decrease of the MMD divergence aa we increased the parameter of the geometric distribution to 1- was correlated with decrease of the average cumulative costs by factor of not the case of the Hopper environment, as we obtained similar cumulative costs despite the reduction of the divergence by a factor of 3. This is explained by the fact that the ground truth cost function of the Hopper environment produces similar cumulative costs for wider variety of policies. For this reason, the IRL solution does not need to achieve a faithful expert behaviour matching in order to achieve good performances. This illustrates the importance of evaluating IRL algorithm with respect to MMD, and MMD,, when the goal is to mimic behaviors.
This was
DDue obvious computational limitations, the trajectories are finite
7

---

Under review as a conference paper at ICLR 2022
c079
5908
6009
05.
92
(a) Ant environment
(b) Half-Cheetah environment
(c) Hopper environment
Figure 1: Performances of the policies obtained during the last 100 iteration of MEGAN: as the
parameter of Geom(K) grows from 0 (equivalent to instance of GAIL) to (equivalent an instance of MEGAN with a uniform 11), we observe that the learned policies' generated trajectories are increasingly similar with those generated by the expert in the sense of P. (classical IRL criterion), JA (Generalised IRL criterion) and the cumulative discounted costs (The environment's ground truth).
mey " TUE
00'00
0038
E07
0032
6/00
24
(a) Ant environment
(b) Half-Cheetah environment
(c) Hopper environment
Figure 2: Performances of MEGAN using a Poisson v) distribution: Setting the parameter of " Poisson(A) value around the length of the expert's movement cycle achieved similar/better performances than those obtained using a uniform 17 distribution (Geom ()). The expert cycle is roughly 10 frames long the Ant environment, 25 in the Salf-Cheetah, and 40 the Hopper.
5.2 PERFORMANCE IMPROVEMENT USING A POISSON DISTRIBUTION
Using the same experimental setting from the previous section, we evaluate MEGAN's performances with non- non-geomen men-eeemttric 17 distributions. In Figure 2 we plot the performances of the remaining replay buffer obtained with geometric distributions in blue lines, and those obtained when using a Poisson distribution in red. To reduce clutter, we removed the cumulative costs and only provided the divergences MMD, (represented with solid lines in Figure 2 and MMD,, y (dashed lines in Figure Z).
Despite the weaker theoretical guarantees provided our paper for the case of non-geometric distributions, we observe that using Poisson 7) can lead to comparable performances. Recall that the expectation of a Poisson distribution is equal to its parameter value. This implies that solving IRI. Poieson(A) searches for policies that match PTr (-Is) for states observed around the A frame ...
of the expert demonstrations. Now notice that the control tasks analysed in Figure 2 consist of movements cycles that are repeated perpetually. Quite interestingly, setting value around the length of an expert cycle (A = 10 in the Ant environment, 25 in the Half-Cheetah, and 40 for the Hopper), ended up achieving the best performances.
In a sense, the proposed 7/ootimality criterion can be seen as inductive bias: we successfully injected qualitative knowledge (the repetitive nature of the expert behaviour) by explicitly asking the agent to focus matching Pa (.1s) for states observed within single movement cycle of the expert demonstrations via careful parameterisation of the distribution 77. In the case where such higher undersanding/rprresttatton of the expert behaviour is unavailable, using uniform r distribution (or a geometric with a parameter close to 1) is a safe bet. Notice that in Figure the both the blue and red curves have comparable minimum values.
8

---

Under review as a conference paper at ICLR 2022
MFL Iilene
MC (BR.
Kize bnseemmmant
0030
6030
600
0000
(a) Ant environment
(b)
environment
(c) Hopper environment
Figure 3: Performances of GAIL as we vary the discount factor Yे Neither increasing nor de- creasing the discount factor resulted in improved performances. Agnostic of the used parameter, GAIL was not able match the expert behavior as well as MEGAN.
5.3 EFFECT OF VARYING THE DISCOUNT FACTOR 7
Recall that in the classical problem formulation, the discount factor 7 can be interpreted as the weight of future observations. In this section, we investigate whether changing this parameter can overcome the buit-in bias against policies with longer mixing times, without resorting to the 77-ptimmlity criterion. In practice, the discount factor y is used separately in two building blocks of IRL algorithms (including GAIL and MEGAN). The first instance is in the Bellman updates when solving the RL problem under given cost function: we refer to this parameter as RRR, The second instance is in the discrimination problem when approximately sampling from Px: refer to this parameter as TIRL.
Due to the finite nature of the expert demonstrations, the standard approach is to approximate future state distributions by setting IIR 1 (or equivalently, sampling transitions uniformly). This can be seen as an asymptotic behaviour of the classical problem formulation: as the discount factor approaches 1, the associated truncated geometric distribution will converge to a uniform one. Reducing this parameter will only accentuate the discussed issues as it entails up-sampling the early stages of the collected demonstrations, which will inevitably favor short term imitation. On the other hand, it is not possible to set 7RI. = 1 (the Bellman operator is no longer guaranteed to admit fixed point, and state of the art value based RL algorithms become extremely unstable). For this reason, most practitioners set this parameter to a value reasonably close to 1.
In this section, we evaluated the remaining replay buffers obtained using GAIL as we vary the discount factorr values (YRLL E [0.9, 0.99, 0.999].. We emphasize that in all the reported empirical evaluations (including previous ones, i.e. Figures
and 2), we fixed "ARL to 1.
In Figure 3 report the average MMD, divergence of the remaining replay buffer in solid lines, and the average MMD,, y divergence dashed lines. GAIL's 's performances as we vary the discount factor are reported in blue and the best performances obtained with MEGAN are reported in red. Reducing TR. to 0.9 accentuated the bias against policies with longer mixing times, and on the other hand increasing it to 0.999 lead less reliable RL algorithm. As expected, we observe Figure that both tweaks did not entail performances on par to what we obtained using MEGAN.
CONCLUSION
In this paper, we generalised the classical criterion of optimality in the reinforcement learning litera- ture by putting more weights onto future observations. Using this novel criterion, we reformulated both the regularised RL and the maximum-entropy IRL problems. We reviewed existing RL algo- rithms and discussed their ability to search for 17-optimal policies. We also generalised classical IRL solutions. The derived algorithm produced stable solutions with enhanced expert matching properties.
In practice, the main difference between MEGAN and GAIL is the discriminator's 's sampling procedure. This implies that it can easily replace the latter algorithm in all subsequent contributions. An interesting future direction of research consists in evaluating the margin of improvement that can be gained from this modification.

---

Under review as a conference paper at ICLR 2022
BROADER IMPACT STATEMENT
Inverse reinforcement learning provides framework explain observed complex behaviors in sophisticated environments, However, the inherent biases of current formulations againt policies with longer mixing times due the use of geometrically discounted optimality criterion, prevent existing algorithms from reaching the frameworks full potential. By constructing approximate solutions of the IRL. problem (in the sense of the 7-optimality criterion), we believe that we take step towards more applicable IRL algorithms. In that spirit, also commit ourselves to releasing code soon in order to allow the wider community to extend our work in the future.
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning. pp. 2004.
Laurent Bougrain, Matthieu Duvinage, and Edouard Klein. Inverse reinforcement learning to control robotic arm using a brain-computer interface. eNTERFACE Summer Workshop, 2012.
Thomas Degris, Patrick M Pilarski, and Richard Sutton. Model-frce reinforcement learning with continuous action in practice. In 2012 American Control Conference (ACC), pp. 2177-2182. IEEE,
2012.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pp. 49-58. PMLR, 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. arXiv preprint 1rXiv:1710.1124:: 2017.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. arXiv preprint 1275 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661 2014.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.0129: 2018.
Jean-Bate CanntBaptitt Hiriart- Urruty and Claude Lemaréchal. Convex analysis and minimization algorithms I: Fundamentals, volume 305. Springer science & business media, 2013.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4565-4573, 2016.
Firas Jarboui, Célya Gruson- Dansel, Alain Durmus, Vincent Rocchisani, Sophie- Helene Goulet Ebongue, Anneliese Depoux, Wilfried Kirschenmann, and Vianney Perchet. Markov decision process for mooe users behavioral inference. In European MOOCs Stakeholders Summit, pp- 70-80. Springer, 2019.
Wonseok Jeon, Chen-Ye Chen-Yang Su, Paul Barde, Thang Doan, Derek Nowrouzezahrai, and Joelle Pineau. Regularized inverse reinforcement learning. arXiv preprint arXiv:2010003691,.000 2020.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In ICML, volume 2, pp. 267-274, 2002.
Kyoichiro Kobayashi, Takato Horii, Ryo Iwaki, Yukie Nagai, and Minoru Asada. Situated gail: Multitask imitation using cuskcconditioned adversarial inverse reinforcement learning. arXiv preprint arXiv: 190020 2019.
10

---

Under review as a conference paper at ICLR 2022
Weiwei Li and Emanuel Todorov. An iterative optimal control and estimation design for nonlinear stochastic system. In Proceedings of the 45th IEEE Conference on Decision and Control, pp- 3242-3247. IEEE, 2006.
Tiago V Maia. Two-factor theory, the actor-critic model, and conditioned avoidance. Learning & behavior, 38(1):50-0)) 67, 2010.
Francisco Martinez-Gil, Miguel Lozano, Ignacio García-Femánddez Pau Romero, Dolors Serra, and Rafael Sebastián. Using inverse reinforcement learning with real trajectories to get more trustworthy pedestrian simulations. Mathematics, 8(9):1479, 2020..
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv:1411.1784 2014.
arXiv preprint
Katja Mombaur. Using optimization to create self-stable human-like running, Robotica, 27(3):321, 2009.
Eadweard Muybridge. Muybridges Complete human and Animal locomotion: all 781 plates from the 1887 Animal locomotion. Dover Publications, 1979.
Matthias Plappert, Marcin Andrychowiez, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej. Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Ahmed H Qureshi, Byron Boots, and Michael C Yip. Adversarial imitation via variational inverse reinforcement learning. arXiv preprint 1rXiv: 1809.06404 2018.
Nathan D Ratliff, Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In
Proceedings of the 23rd international conference on Machine learning. pp. 729-736, 2006.
Stuart Russell. Learning agents for uncertain environments. Proceedings of the eleventh annual conference on Computational learning theory, pp. 101-103, 1998.
Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: source of platcaus in deep reinforcement learning. arXiv preprint (eti:v1904.114555 2019.
Nestor A Schmajuk and B Silvano Zanutto. Escape, avoidance, and imitation: A neural network approach. Adaptive Behavior, 6(1):63-129, 1997.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
Gerrit Schultz and Katja Mombaur. Modeling and optimal control of human-like running. JEFE/ASME Transactions on mechatronics, 15(5):783-792, 2009.
Sahand Sharifzadeh, Ioannis Chiotellis, Rudolph Triebel, and Daniel Cremers. Learning to drive using inverse reinforcement learning and deep nettwwrr.s. arXiv preprint arXiv:J612.03355. 2016.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference machine learning, pp. 387-395, PMLR, 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. Policy gradient methods for reinforcement learning with function approximation. NIPs, volume 99, pp. 1057- 1063. Citeseer, 1999.
11

---

Under review as a conference paper at ICLR 2022
Umar Syed and Robert E Schapire. A game-theoretic approach apprenticeship learning. Advances in neural information processing systems, 20: 111114441555 1456, 2007.
Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear program- ming. In Proceedings of the 25th international conference Machine learning, pp. 1032-1039, 2008.
Paul FMJ Verschure and Philipp Althaus. A real-world rational agent: unifying old and new ai. Cognitive science, 27(4):561-595 2003.
Paul FMJ Verschure, Cyriel MA Pennartz, and Giovanni Pezzulo. The why, what, where, when and how of goal-directed choice: neuronal and computational principles. Philosophical Transactions of the Royal Society B: Biological Sciences, (69(1655):20134453 2014.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine learning, 8(3-4):229-256, 1992.
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, II., USA, 2008
Brian D Ziebart, Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 1255-1263 Omnipress, 2010.
12


Reviews:
[{"review_id": "OH-d2Z_0MP", "reviewer": "Reviewer_j2NT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The issue raised in this paper is novel as far as I know. The proposed solution is interesting and seems to lead to good experimental performance. However, the explanation for why penalizing penalizing longer mixing times is an issue is not clear to me, since two policies regardless of their mixing times that match the stationary distribution of the expert demonstrations should have the same cumulative costs. Does this issue come up because of finite-length trajectories used in RL/IRL? If this explanation was true, shouldn't the experiments with varying discount factors suggest that higher discount factors lead to better results?\n\nThe paper should be checked for typos. For instance:\n\nCitations should be in parenthesese, e.g.:\nName et al., (year) -> (Name et al., year)\n\nIn Section 2.1, there are some issues with the notations:\nIn Q^c_\\pi, the expectation with respect to P(s_1 | s, a) is missing.\n\nIs matching state occupancy really related to maximum entropy IRL?\n\nPage 3, line 1-2: will implicitly favours -> will implicitly favour\n\nPage 4: therefor", "summary_of_the_paper": "The paper extends the maximum entropy inverse reinforcement learning (IRL) framework by changing the optimal criterion used in reinforcement learning (RL). This novel criterion is an expectation of the Q-values over a weighted distribution over states and actions induced by a policy, which is in contrast to the standard criterion that is an expectation over the initial state dsitribution.", "summary_of_the_review": "Novel issue in IRL with interesting solution to tackle it, although the proposition sounds a bit heuristics. A better explanation for it would have made the paper stronger.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NgmuaaaAAD5", "reviewer": "Reviewer_hoAL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-Originality: GIRL provides a new and interesting perspective for extending the GAIL framework to flexibly balance the importance of short-term and longer-term behavior. I agree that the vanilla GAIL could largely ignore the benefits of longer-term imitation and the distribution $\\eta$ could provide a tractable way to address this. One aspect that remains unclear to me is how to properly choose $\\eta$ to find the sweet spot of balancing short-term and long-term imitation. The empirical results show that geometric and Poisson distributions work quite well in three MuJoCo tasks, but it remains unknown whether these two types of distributions are universal in improving the matching of occupancy measures. A more rigorous discussion on the choice of $\\eta$ is needed.\n\n-Technicity: Though I can appreciate this new perspective, the overall technical novelty is somewhat limited as GIRL mostly follows the algorithmic framework of GAIL.\n\n-Significance: The empirical results show that the proposed GIRL framework does indeed match the state-action occupancy measure of the experts better than GAIL by a large margin. \nWhile the empirical results look very promising, there are several claims that require further explanation:\n- As mentioned in Section 5.1, despite that GAIL explicitly optimizes divergence in the standard occupancy measure, GAIL still underperforms MEGAN in this metric. This is indeed quite counterintuitive and would require further explanation.\n- Section 5.3 shows that even setting the discount factor $\\gamma$ of the RL problem close to 1 could not address long-term imitation. What is the fundamental reason behind this phenomenon (in addition to simply stating “increasing it to 0.999 leads to a less reliable RL algorithm”)?\n\nA more in-depth explanation of the experimental results would be very much appreciated.\n\n-Clarity: The paper is well-written and well-organized. The proposed method is in most places well-explained.\n\n-Other comments:\n- There are many candidate metrics for measuring the difference between occupancy measures. Is there any specific reason why MMD is selected?\n- The integral in the definition of $E_{p_0,\\pi}^{\\eta}[Q^{c}_{\\pi}]$ is missing the terms $ds_0$, $ds+$, and $da+$?\n", "summary_of_the_paper": "This paper points out that the classical IRL approach has a tendency to match those occupancy measures that favor short-term behavior. To address this issue, a reformulation is proposed based on GAIL in order to put more emphasis on matching longer-term behavior. Specifically, the main difference is to replace the standard objective (i.e., the expectation of Q function over some initial state distribution) with the expectation of Q function over both the initial and an $\\eta$-weighted future state distribution, where $\\eta$ is some probability distribution over the support set of nonnegative integers. Built on this formulation, this paper proposes GIRL (and the resulting algorithm MEGAN), which follows the framework of GAIL to learn a policy that matches the $\\eta$-weighted variant of occupancy measure of the expert policy. Experimental results on MuJoCo are provided to demonstrate the effectiveness of GIRL.", "summary_of_the_review": "This paper proposes an extension of GAIL that addresses longer-term imitation. I like the insight provided by this paper, but the overall technical novelty is somewhat limited. The empirical results look promising, but there are several places that would require better explanations. As a result, I lean towards rejection in its current form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "42mktx3ppmu", "reviewer": "Reviewer_Msxm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found the premise of the paper intriguing. Although I am not fully up to date with the current IRL literature, the problem addressed in the paper strikes me as novel, and the proposed method (whose derivations I did not check in detail) follows naturally from the formulation proposed. The results also seem promising, although I am not sure that I was able to read them properly (more on this below).\n\nThis said, there are several aspects in which I believe the paper could be improved.\n\n- First and foremost is, perhaps, the motivation for the proposed setting. Although the paper makes an argument for the proposed setting based on the bias introduced by the standard optimality criterion underlying IRL, I do not fully grasp how relevant the impact of such bias truly is. In other words, although I understand the technical argument, I am not sure that this translates into an actual problem in practice, and some more discussion on this would be welcome. \n\n- Following on the previous item, and if I understood correctly, it seems to me that the introduced weighting is tantamount to considering a policy-dependent initial distribution $p_0$ in the standard formulation. The different ways of weighting the future state-action pairs correspond to different dependencies on the policy, and perhaps it may be easier to make a case for the proposed approach from that interpretation (just an idea, I don't know if that is the case or not).\n\n- The notation is also a bit overloaded and not always in a very intuitive way. This makes parsing the technical derivations sometimes quite difficult. For example, the symbol \"P\" appears in several different shapes and sizes with different meanings. As another example, the value function associated with the optimal policy computed from a regularized RL objective is denoted as $v^c_{\\pi^*_\\Omega,\\Omega}$, with all the sub- and super-scripts. Also, I understand that treating the state and action spaces as continuous is convenient for the sake of generality, but the notation used in the resulting integrals is often sloppy and does not add much to the understanding of the paper.\n\n* Finally, the experimental results are somewhat hard to read. Although the plots suggest that, indeed, the performance seems to improve as $\\eta$ increases, the differences (as measured in the $yy$ axis of the plots) seem to be quite small. Additionally, except for the half-cheetah environment, the difference between the MMD on $\\mu_\\pi$ for $\\eta=0$ and $\\eta=1$ does not seem to be statistically significant. Some more discussion on this aspect would be welcome.\n\n*Minor aspects:*\n\n- Throughout the paper the references do not seem to be properly typeset, which at points is somewhat disruptive.\n\n- Maybe I missed something, but the definition of $\\mu_\\pi(s_+a_+\\mid s_0)$ in the beginning of Section 2.4 is not clear: the paper defines it as the frequency of $(s_+,a_+)$ in the $\\eta$-weighted future steps of trajectories initialized according to $\\rho_\\pi(s,a\\mid s_0)$, but I do not see where $\\rho_\\pi(s,a\\mid s_0)$ come into play in the definition.", "summary_of_the_paper": "The paper proposes a new formulation for inverse reinforcement learning that aims to address the _bias against policies with longer mixing times_. The key contribution of the paper is the proposal of an alternative optimality criterion that arguably addresses the aforementioned bias by considering, for a given policy, the value\n\n$L(\\pi,c)=E_{p_0,\\pi}\\left[\\sum_{k=0}^\\infty\\eta(k)\\sum_{t=0}^\\infty\\gamma^tc_{t+k}\\right]$\n\ninstead of the more standard definition\n\n$L(\\pi,c)=E_{p_0,\\pi}\\left[\\sum_{t=0}^\\infty\\gamma^tc_{t}\\right],$\n\nwhere $c$ is such that $E[c_t\\mid s_t,a_t]=c(s_t,a_t)$.\n\nTo solve the IRL problem associated with this new optimality criterion (or, rather, a regularized version thereof) the paper proposes the use of maximum causal entropy IRL, which can roughly be broken down in two subproblems:\n\n- Given a candidate cost function, $\\hat{c}$, find a candidate policy, $\\hat{\\pi}$, that minimizes $L(\\hat{\\pi},\\hat{c})$. In a sense, $\\hat{\\pi}$ is the \"optimal\" policy given the cost function $\\hat{c}$. The paper proposes the use of soft actor-critic approach (although other value-based approaches could be used).\n\n- Given the expert policy, $\\pi_E$, and the candidate policy, $\\hat{\\pi}$, come up with a candidate cost function $\\hat{c}$ such that the expert policy has a low cost and other policies have a high cost, which the paper shows that can be done by minimizing a measure of divergence between the (weighted) distributions induced by $\\hat{\\pi}$ and $\\pi_E$. \n\nThe resulting algorithm is a variation of the generative adversarial IRL approach of Ho and Ermon, that the experiments suggest may be able to better recover the expert's policy.", "summary_of_the_review": "Although the premise of the paper is intriguing, it is not clear to me the relevance of the proposed approach, since the motivation is not convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper extends the maximum entropy inverse reinforcement learning (IRL) framework by changing the optimal criterion used in reinforcement learning (RL). This novel criterion is an expectation of the Q-values over a weighted distribution over states and actions induced by a policy, which is in contrast to the standard criterion that is an expectation over the initial state distribution.

All the reviewers agree that the topic addressed in this paper is interesting and novel. On the other hand, there are some concerns about the technical novelty and relevance of the paper. Since the authors have not provided any feedback, the reviewers did not solve their concerns and they reach a consensus on rejecting this paper.
Decision: Reject