Regularized Inverse Reinforcement Learning
Authors: Wonseok Jeon
Chen-Yang Su
Paul Barde
Thang Doan
Derek Nowrouzezahrai
Joelle Pineau
Venue: ICLR 2021 Spotlight (2021)
Keywords: inverse reinforcement learning
reward learning
regularized markov decision processes
reinforcement learning

Abstract: Inverse Reinforcement Learning (IRL) aims to facilitate a learner’s ability to imitate expert behavior by acquiring reward functions that explain the expert’s decisions. Regularized IRLapplies strongly convex regularizers to the learner’s policy in order to avoid the expert’s behavior being rationalized by arbitrary constant rewards, also known as degenerate solutions. We propose tractable solutions, and practical methods to obtain them, for regularized IRL. Current methods are restricted to the maximum-entropy IRL framework, limiting them to Shannon-entropy regularizers, as well as proposing solutions that are intractable in practice.  We present theoretical backing for our proposed IRL method’s applicability to both discrete and continuous controls, empirically validating our performance on a variety of tasks.

Published as a conference paper at ICLR 2021
REGULARIZED INVERSE REINFORCEMENT LEARNING
Wonseok Jeon .1,2, Chen-Yang Su. Paul Ba Thang Doan 1.2
Derek Nowrouzezahrai Joelle Mila, Quebec AI Institute
Joelle Pineau 1.233
Schooo of Computer Science, McGill University
Facebook AI Research
ABSTRACT
Inverse Reinforcement Learning (IRL) aims to facilitate a learner's ability to imitate expert behavior by acquiring reward functions that explain the expert's 's decisions. Regularized IRL applies strongly convex regularizers to the learner's 's policy in order avoid the expert's behavior being rationalized by arbitrary constant rewards, also known as degenerate solutions. We propose tractable solutions, and practical methods to obtain them, for regularized IRI.. Current methods restricted to the maximum-entre entropy IRL framework, limiting them to Shannon-entropy regularizers, well as proposing solutions that are intractable in practice. We present theo- retical backing for our proposed IRL method's ' applicability to both discrete and continuous controls, empirically validating our performance on a variety of tasks.
INTRODUCTION
Reinforcement learning (RL) has been successfully applied to many challenging domains including
games (Mnih et al. 2013 | and robot control (Schulman et al.) 2015 Fujimoto et al. 2018
Haarnoja et al.) 2018). Advanced RL methods often employ policy regularization motivated by. e.g..
boosting exploration (Haarnoja et al. 2018) or safe policy improvement (Schulman et al. 2015).
While Shannon entropy often used policy regularizer (Ziebart et al. (2008), (Geist et al. (2019)
recently proposed theoretical foundation of regularized Markov decision processes (MDPs)
framework that uses strongly convex functions as policy regularizers. Here, one crucial advantage is that an optimal policy is shown to uniquely exist, whereas multiple optimal policies may exist in the absence of policy regularization.
Meanwhile, since RL requires a given or known reward function (which can often involve non-trivial reward engineering). Inverse Reinforcement Learning (IRI.) (Russell 1998 Ng et al., 2000) -te problem of acquiring reward function that promotes expert-like behavior is more generally adopted practical scenarios like robotic manipulation (Finn et al.) 2016b), autonomous driving (Sharifzadeh) al. 2016 Wu et al. 2020) and clinical motion analysis (Li al. 2018). In these scenarios, defining a reward function beforehand is particularly challenging and IRL is simply more pragmatic. However, complications with IRL in unregularized MDPs relate to the issue of degeneracy, where any constant function can rationalize the expert's 's behavior (g et al. 2000).
Fortunately, Geist al. (2019) show that IRL in regularized MDPs
does not
contain such degenerate solutions due to the uniqueness of the optimal policy for regularized MDPs. Despite this, no tractable solutions of regularized IRI. other than SawiiiunnSSannon-enrrooon
IRL
(MaXEmIRL) (Ziebart et al. 2008 Ziebart 2010 Ho & Ermon 2016
Finn et al. 201 6a
Fu et al.
2018 have been proposed.
In Geist al. (2019), solutions for regularized IRL. were introduced. However, they are generally intractable since they require closed-form relation between the policy and optimal value function and the knowledge on model dynamics. Furthermore, practical algorithms for solving regularized IRL problems have not yet been proposed.
We summarize our contributions as follows. Unlike the solutions in Geist et al. 2011)) we propose tractable solutions for regularized IRL problems that can be derived from policy regularization and
"Correspondence Wonseok Jeon <jeonwons@mi quebec>>

---

Published as a conference paper at ICLR 2021
its gradient in discrete control problems (Section 3.1 We additionally show that solutions are tractable for Tsallis entropy regularization with multi-variate variitte Gaussian policies in continuous control problems (Section 3.2 We devise Regularized Adversarial Inverse Reinforcement Learning (RAIRL), a practical sample-based method for policy imitation and reward learning in regularized MDPs, which generalizes adversarial IRL. (AIRL,
(2018) (Section ( Finally, empirically validate our RAIRL method on both discrete and continuous control tasks, evaluating RAIRL via episodic scores and from divergence minimization perspective (Ke al. 2019 Ghasemipour et al. 2019
Dadashi et al. 2020) (Section 5).
2
PRELIMINARIES
Notation For finite sets X and Y, y is set of functions from X to Y. A (At) is set of (conditional) probabilities over X (conditioned
Y). Especially for the conditional probabilities Pxiy E Ay, we say Pxly(-ly) E A for E Y. R is the set of real numbers. For functions f1, E x² we define f2), == ExEx fi (z) f2(x).
Regularized Markov Decision Processes and Reinforcement Learning We consider sequential decision making problems where agent sequentially chooses its action after observing the state of the environment, and the environment in turn emits a reward with state transition. Such an interaction between the agent and the environment is modeled as an infinite-horizon Markov Decision Process (MDP), M := (S, A, Po, P,r, and the agent's policy π E A The terms within the MDP are defined as follows: S is a finite state space, A is a finite action space, Po E A is an initial state distribution, P E A 4 is state transition probability, r E S is reward function, and Y E [0, 1) is the discount factor. We also define an MDP without reward as M := (S, Po, P, 7) The normalized state-action visitation distribution, d₂ E A SxA associated with π is defined as the expected discounted action visitation of æ, i.e., (s, a) == (1-7) E, DE= < = s, a, a}}. where the subscript π on E means that trajectory (su, $1, A1....) is randomly generated from M and π, and I{-} is an indicator function. Note that dx satisfies the transposed Bellman recurrence (Boularias & Chaib-Draa 2010 Zhang et al. 2019)
da (s, a) = (1 P) + (als)) Σ (sd,
We consider RL in regularized MDPs (Geist et al. 2019), where the policy optimized with a causal policy regularizer. Mathematically for MDP M and strongly convex function Ω A - R, the objective in regularized MDPs to seek n that maximizes the expected discounted sum of rewards, or return in short, with policy regularizer Ω:
arg max (n(r, = Ex Σ =EAS
rs (r(ss, a.) - (++(+ss)))
B Bls,a)nd, [r(s, ((T(-1s)).
T-
(1)
turns out that the optimal solution of Eq. unique (Geist et al. 2019), whereas multiple optimal policies may exist in unregularized MDPs (See Appendix A for detailed explanation). In later work YYngg et al., 2019), st(p)
XEE=pp(p(a)), E A was considered for X > and & (0, I2 satisfying some mild conditions. For example, RL with Shannon entropy regularization (Haarnoja et al. 2018) can be recovered by d(x) = log x, while RL with Tsallis entropy regularization (Lee et 2020) can be recovered from ((x) g-1 x²-1) for k > 0, q > The optimal policy म for Eq. with from Yang et al. 2019 is shown to be
(als)) max Do (s((s) Q"(s,a)
).)]
X
(s) r(8, a) + 7 P( V2 V'6),, (s))) A Σ
(a (TT
aEA
where d'(x)= a ((x), go is an inverse function of f for fo(x) = (0(x), x E (0, 1]. and μ² is th normalization term such that E.eA (a(as) = 1. Note that we still need to find out μ to acquire closed-form relation between optimal policy x² and value function Q" However, such relations
2

---

Published as a conference paper at ICLR 2021
have not been discovered except for Shannon-entropy regularization (Haarnoja et al. 2018) and specific instances (q = 1,2, oo) of Tsallis-ent Tsallis-entropy regularization (Lee et al. 2019) to the best of our knowledge.
Inverse Reinforcement Learning Given set of demonstrations from expert policy Tg- IRL (Russell, 1998 Ng et al. 2000) is the problem of seeking reward function from which we can recover T through RL. However, IRL in unregularized MDPs has been shown be an ill-defined problem since (1) any constant reward function can rationalize every expert and (2) multiple rewards meet the criteria of being a solution (Ng al. 2000). Maximum entropy IRL (MaxEntIRL) (Ziebart) et al. 2008 Ziebarti 2010) is capable of solving the first issue by seeking a reward function that maximizes the expert's return along with Shannon entropy of expert policy. Mathematically, for the RL objective Jp2 in Eq. ① and Ω - H for negative Shannon entropy H(p) Ea-pl- log p(a)
(Ho Ermon 2016), the objective of MaxEntIRL is
(T arg max rERAXA
J_ J_ max J_ (1; xEAg
(4)
Another commonly used IRL method is Adversarial Inverse Reinforcement Learning (AIRL) (Fu
et al. 2018) which involves generative adversarial training (Goodfellow et al.] 2014 Ho & Ermon
2016) to acquire a solution of MaxEntIRL. AIRL considers the structured discriminator (Finn et al.
2016a) (s,,a) (r(s, a) log m[aas))
("(o.w) for (x)) 1/(1 + e=*) and iteratively
erlo.e) e-to.a) (aals)
optimizes the following objective:
rERSXA ((,,))=d [log Dr, (s, a)] + (B)a)) [log(1 Dr, (5, (s,
max (ss,d))d, [log Dr (s, - log(1 - Drar (s, = max (a [r(s, a) - log (a|s)] rEdg
xEAg
(5)
turns out that AIRL minimizes the divergence between visitation distributions da and dz, by solving min (L |drg)) for Bullback-Leible (KL) divergence KL(-II-) (Ghasemipour et al. 2019
INVERSE REINFORCEMENT LEARNING IN REGULARIZED MDPs
In this section, propose the solution of IRL in regularized IDDPs-regularzer' IRL-and relevant properties in Section 3.1 We then discuss a specific instance of our proposed solution in Section 3.2 where Tsallis entropy regularizers and multi-variate Gaussian policies are used in continuous action spaces.
3.1 SOLUTIONS OF REGULARIZED IRL
We consider regularized IRL that generalizes MaxEntIRL in Eq. to IRL with class of strongly convex policy regularizers:
(RL9(Tg) := arg max nERSXA
Jo(r, max J SEA3
(6)
For any strongly convex policy regularizer Ω2 regularized IRL does not suffer from degenerate solutions since there is unique optimal policy any regularized MDP (Geist et al.) 2019). While Geist et al. (2019) proposed solutions of regularized IRI., those are intractable solutions (See Appendix F.T for detailed explanation). In the following lemma, we propose tractable solutions that only requires the evaluation of the policy regularizer (!) and its gradient (VΩ) which are more manageable practice. Our solution is motivated from figuring out a reward function that is capable of converting regularized RL into equivalent divergence minimization problem associated with and ,.
Lemma 1. For policy regularizer Ω: A + R. let us define
1(8, a; == S2' a; m) - E Easwatle) (s, m)) + ((T(-1s))
(7)
for (s, 3 () V[(((- == [ E RA,, E Then, t(s, a; ग.. for expert policy Tg is solution of regularized IRI. with Ω.
3

---

Published as a conference paper at ICLR 2021
Proof. (Abbreviated. See Appendix B for the full version.) With r(s.a) t(s, a; ng). the RL objective in [ becomes equivalent to the problem of minimizing the discounted sum of Bregman divergences between and Te
o0 Σ
arg min Ex rEAg
(s)(( (+1s))
(8)
where DĀ is the Bregman divergence (Bregman 1967 defined by D (pillp)) ((pp) (2(p))
(VD(p2),P) A for p1,p2 E A Due to the non-negativity of the Bregman divergence, TE is a solution of (8) and is unique since Eq.T] T has the unique solution for arbitrary reward functions (Geist et al.) 2019).
In particular, for any policy regularizer Ω represented by an expectation over the policy Yang et al. 2019), Lemma [ be reduced to the following solution in Corollary [
Corollary 1. For a(p) XIE (ppa))((p(aa) with p E A (Ynng et al. 2019) Eq. 7 becomes
((s,a:)) = - (5 - E (1) - <(<(a')))))}
(9)
for (z) = 0z (x)(x)) The proof is in Appendix
Throughout the paper, we denote reward baseline by the expectation E (f) ((T(a))))]- Note that for continuous control tasks with (1p) = - (p(a)) we can obtain the same form of the reward in Eq. (The proof is in Appendix D Although the reward baseline is generally not intractable in continuous control tasks, we derive a tractable reward baseline for a special case (See Section 3.2). Additionally, when = 1 and ((x) = log x, can shown that (ss, a; T = log π (a) for both discrete and continuous control problems-aa whicch was used as a reward objective in previous work (Fu et al. 2018), and that the Bregman divergence in Eq. ( becomes the KI. divergence (T](s(+ Tg (-1)))-
Additional solutions of regularized IRL. can be found by shaping t(s, a; Tg) as stated the following lemma:
Lemma 2 (Potential- reward shaping). Let x² be the solution of Eq. 47) regularized MDP M" with a regularizer Ω A + R and a reward function r E IRSAA Then for & E IRS, using (s) as a reward does not change
either r(8, a) ( (s)) or ((r,s) + P(ls,a
the solution of Eq. 7.. The proof is in Appendix
From Lemma and Lemma 2 we prove the sufficient condition of rewards being solutions of the IRL problem. However, the necessary condition- set of those solutions the only possible solutions for an arbitrary MDP-is not proved (Ng et al. 1999).
In the following lemma, we check how the proposed solution is related the normalized state- visitation distribution which can be discussed in the line of distribution matching perspective on imitation learning problems (Ho Ermon 2016 Fu et al., 2018 Ke al. 2019. Ghasemipour et al.)
2019):
Lemma 3. Given the policy regularizer Ω, let us define ②(d) == (( for an arbitrary normalized state-action visitation distribution d A and the policy Ra(als)
Ewee d(s.a') induced by d. Then, Eq. 7) is equal to
dis,a)
a() (V)(d)(ss.)).
(10)
When (d) strictly convex and a solution 1(s,a; T = vit(d=)) of IRL in Eq. used, the RL. objective in Eq. 11 is equal to
arg min Addlldd) #EA
where DSXA is the Bregman divergence among visitation distributions defined by DSx (ld2) ((d))- (((d)) -VV(dd) di dz) for visitation distributions dy and dz. The proof is in Appendix
4

---

Published as a conference paper at ICLR 2021
Note that the strict convexity of Zt is required for S to become valid Bregman divergence. Although the strict convexity of a policy regularizer 12 does not guarantee the assumption on the strict convexity of 52. it has been shown to be true for Shannon entropy regularizer (f2 - of Lemma 3.1 in Ho & Ermon (016)) and Tsallis entropy regularizer with its constants k = 2. (S2 = W of Theorem 3 in Lee et al. 2018)).
3.2
IRL WITH TSALLIS ENTROPY REGULARIZATION AND GAUSSIAN POLICIES
For continuous controls, multi-variate Gaussian policies are often used in practice (Schulman et al. 2013 2017) and we consider IRL. problems with those policies in this subsection. In particular, we
consider IRL with Tsallis entropy regularizer (p) = -T,² (p) -aapp 9- p(a)9 (Lee
et al. 2018 8 Yang et al. 2019 Lee et al. 2020 for multi-variate Gaussian policy (-|
N ( s), ( (s) with p(s)
|n (s) Ma [3]] (s) ( .... (aa(s))²). In such
case, we can obtain tractable forms of the following quantities:
Tsallis entropy. The tractable form Tsallis entropy for multi-variate Gaussian policy is
T (T(1)) k[1 - el -(1-g))((((()))) (,((((1))))
log q 2(1
log( (og(V2na,(s))
for Renyi entropy Rg Its derivation is given in Appendix !
Reward baseline. The reward baseline term Eg (122 - (( in Corollary is
generally intractable except for either discrete control problems or Shannon entropy regularization where the reward baseline equal to 1. Interestingly, as long the tractable form of Tsallis entropy can be derived, that of the corresponding reward baseline can also be derived since the reward baseline satisfies
Ea ( (a(ss) - o(T((a)))] = - (1)-))Ea-aea (1))(((()) - k = (q L)T((())))) - k.
Here, the first equality holds with (x) q- 9x--11 {0(()) for Tsallis entropy regularization.
Bregman divergence associated with Tsallis entropy. For two different multivariate Gaussian policies, derive the tractable form of the Bregman divergence (associated with Tsallis entropy) between two policies. The resultant divergence has a complicated form, so we leave its derivation in Appendix L.
9-2.0
-1
logo
-2
-2
-2 -1
-2 -1 1 2
-2
-1
0
3
2
P
Figure Bregman divergence Da(T) TT) associated with Tsallis entropy (32(p) = -T
T (p)
between two uni-variate Gaussian distributions = NN,, and T. = =NO00, (e= (green point each subplot). In each subplot, we normalized the Bregman divergence that the maximum value becomes 1. Note that for q = 1, [ becomes the KL divergence KL( [
For deeper understanding of Tsallis entropy and its associated Bregman divergence in regularized MDPs, we consider an example in Figure [[] We first assume that both learning agents' and ex- perts' policies follow uni-variate Gaussian distributions respectively. We then evaluate the Bregman divergence Figure by using its tractable form and varying from 1.0 1.0-which corresponds to the KL divergence- 2.0. We observe that the constant q from the Tsallis entropy affects the sensitivity the associated Bregman divergence w.r.t the mean and standard deviation of the learning agent's policy Specifically, as q increases, the size of the valley-relativan relatively red region in Figure I across the μ-axii and log a-axis decreases. This suggests that for larger q, minimizing the Bregman divergence requires more tightly matching means and variances π and Tg.
. NN(,, a²) and Tg A00 (e- (e=3)
5

---

Published as a conference paper at ICLR 2021
Algorithm I: Regularized Adversarial Inverse Reinforcement Learning (RAIRL)
Input: A set Dr expert demonstration generated by expert policy Tg. a reward approximator
ro, a policy Te for neural network parameters 0 and
2: for each iteration do
3: Sample rollout trajectories by using the learners policy Tel
4: Optimize with the discriminator Dre.kg and the learning objective in Eq.(1]
5: Optimize db with Regularized Actor Critic by using re as reward function.
6: end for
7: Output: Te TR. ro(s.a) t(s, a; nT) solution of the IRL problem in Lemma
ALGORIITMICC CONSIDERATION
Based on solution for regularized IRL in the previous section, we focus on developing an IRL
algorithm in this section. Particularly to recover the reward function {(s, a; 7e in Lemma I we design an adversarial training objective as follows. Motivated by AIRL (Fu et al. 2018). we consider the following structured discriminator associated with I,, and in Lemma
(r, a) = (r(s,, a) t(s, a; (aa;z=)))) x)), (() I+e-z R.
Note that we can recover the discriminator of AIRL in Eq. when t(ss, a) = log π (als) (ő(r) = log x and À = 1). Then, consider the following optimization objective of the discriminator which is the same as that of AIRL:
f(s, a; π) == arg max (ss,))ddzg [log Dr, (s, a)] + ((s,))dd, [log(1 Dr, (s, a))] rERSXA
(11)
Since the function x + (z) + b log(1 (x)) attains its maximum at (xx) a+8 or cquivalently at æ log i
GGoodfelow et al. 2014 Mescheder et al. 2017). it can be shown that
a(s,a; = a(s,a; + log dxg (8,a) dx (s,
(12)
When Tg in 12). we have f(s, a(s,a;TT) = t(s, a; TT) since dx dag which means the maximizer i becomes the solution of IRL. after the agent successfully imitates the expert policy ge- To do so, consider the following iterative algorithm. Assuming that we find out the optimal reward approximator f(s, a; (()) in Eq. 12) for the policy (+)) of the i-th iteration, we get the policy (+++1) I) by optimizing the following objective with gradient ascent:
maximize ((sa))dd, [8(s, (()) ((M(11s))
(13)
HEAS
The above expectation in Eq. [3) can be decomposed into the following two terms
E( [(ss, (0) (((((ss)) = ((,,)ad,, [(ss, ((s,a;z*i (0) (((((1)))
(=( - KL(dxlldr
(
(II
(14)
where the second equality follows since Lemma tells us that f(s, a; ()) is a reward function that makes (00) optimal policy in the O-regularized MDP. Minimizing term (II) in Eq.(14) makes ((++)) close to Tg while minimizing term (I) can be regarded conservative policy optimization around the policy (0) (Schulman al. 2015).
In practice, we parameterize our reward and policy approximations with neural networks and train them using off-policy Regularized Actor-Critic (RAC) (Ynng et al. 2019) described in Algo- [ we evaluate our Regularized Adversarial Inverse Reinforcement Learning (RAIRL) approach across various scenarios.

---

Published as a conference paper at ICLR 2021
5 EXPERIMENTS
We summarize the experimental setup follows. In our experiments, consider a(p) (lEa-plo(pp()) with the following regularizers from Yang et al. (2019): (1) Shannon entropy
(6(x) = log x), (2) Tsallis entropy regularizer (δ(x) 1 x- 1), (3) exp regularizer (0(x) = e²), (4) cos regularizer (d(x)) = cos( =x)), ) (5) sin regularizer (0(x) = - sin 5x). The above regularizers were chosen since other regularizers have not been empirically validated to the best of our knowledge. We chose these regularizers to make
empirical analysis more tractable. In addition, we model the reward approximator of RAIRL as a neural network with either one of the following models: (1) Non-structured model (NSM)-a simple feed-forward neural network that outputs real values used in AIRL (Fu et al., 2018) -nd (2) Density-based model (DBM) model using neural network for (softmax for discrete controls and multi-variate Gaussian model for continuous controls) of the solution in Eq.(T (See Appendix y.2 for a detailed explanation). For the RL algorithm of RAIRL, we implement Regularized Actor Critic (RAC) (Yang et al.) 2019) on top of the SAC implementation from Ripyt (Stooke & Abbeel 2019). Other settings are summarized in Appendix [ For all experiments, we use runs and report 95% confidence intervals.
5.1 EXPERIMENT 1: MULTI-ARMED BANDIT (DISCRETE ACTION)
We consider a 4-armed bandit environment as shown in Figure 2 (left). An expert policy Te is assumed to be either dense (with probability 0.1, 0.2, 0.3, 0.4 for = 0. 2, 3) or sparse (with probability 0, O, 1/3, 2/3 for a = 0, 1, 2,3). For those experts, we use RAIRL with actions sampled from Fg and compare learned rewards with the ground truth reward ((s, a; mg) in Lemmal 1 When Tg is dense, RAIRL successfully acquires the ground truth rewards irrespective of the reward model choices. When sparse g is used, however, RAIRL with non-structured model (RAIRL-NSM) fails to recover the rewards for a = 0, 1-where Tg (a) = 0-due to the lack of samples at the end of the imitation. On the other hand, RAIRL with density- based model (RAIRL-DBM) the correct rewards due to the softmax layer which maintains the sum over the outputs equal to Therefore, we argue that using DBM is necessary for correct reward acquisition since set of demonstrations is generally sparse. In the following experiment, we show that the choice of reward models indeed affects the performance rewards.
recover
EXPERT
SHANNON
TSALLIS
SIN
0.4
4.7
0.3
lwward
0.
-..8
408
-1.
4.9
40.7
0.1
-2.3
-1.1
-1.9
-12
-1.1
0.8
0.0
4.5
5.0
-2.5
3
action
action
action
adtiio
Ground truth
RAIRL-DBM
RAIRL-NSM
Figure 2: Expert policy (Leff) and reward learned by RAIRL with different types of policy regularizers (Right) in Multi-armed Bandit. Either one of dense (Top row) or sparse (Bottom row) expert policies Th is considered.
5.2 EXPERIMENT 2: BERMUDA WORLD (CONTINUOUS OBSERVATION, DISCRETE ACTION)
We consider an environment with a 2-dimensional continuous state space as described in Figure 3 At each episode, the learning agent is initialized uniformly on the r-axis between -5 and 5, and there are 8 possible actions-an angle in {-T,- - - T, ... ",, m that determines the direction of movement. An expert in Bermuda World considers target positions (-5, 10), (0. 10). (5. 10) and behaves stochastically. We state how we mathematically define the expert policy g. in Appendix J.3 During RAIRL's training (Figure Top row), we use 1000 demonstrations sampled from the expert and periodically measure mean Bregman divergence, i.e., for DAA Ilp2) Ka-m 55 (P2(a))
7

---

Published as a conference paper at ICLR 2021
((P
Bewpe (F {(p(()))).
1 (T(+ss) (+1s)).
N
Here, the states $1...., SN come from 30 evaluation trajectories that are stochastically sampled from the agent's policy w-whiih is fixed during evaluation- separate evaluation environment. During the evaluation of learned reward (Figure Bottom row), we train randomly initialized agents with RAC and rewards acquired from RAIRL'S training and check if the mean Bregman divergence is properly minimized. We measure the mean Bregman divergence as was done in RAIRL's training.
RAIRL-DBM is shown to minimize the target divergence more effectively compared to RAIRL-NSM during reward evaluation, although both achieve comparable performances during RAIRL's training. Moreover, we substitute with 1, 5, 10 and observe that learning with larger than returns better rewards-only À = 1 was considered in AIRL (Fu et al., 2018). Note that in all cases, the minimum divergence achieved by RAIRL is comparable with that of behavioral cloning (BC). This is because BC performs sufficiently well when many demonstrations may be the near-optimal divergence that can be achieved with our policy neural network model.
given. We think the divergence of BC
SHANNON
TSALLIS
EXP
COS
SIN
Bregman Div.
RAIRL-DBM (1 1)
(Uaa)
RAIRL-DBM (1 5)
RAIRL-DBM (Λ 10)
RAIRL-NSM (4
RAIRL-NSM (A 5)
Bregman Div.
RAIRL-NSM (A = 10)
(eval)
BC
Random
Expert
0
0
0
250K
500K0
250K
500K0
250K
500K0
250K
500K0
250K
500K
Time Step
Time Step
Time Step
Time Step
Time Step
Figure 3: Mean Bregman divergence during training (Top row) and the divergence during reward evaluation (Bottom row) in Bermuda World, In each column, different policy regularizers and their respective target divergences are considered. The results are reported after normalization with the divergence of uniform random policy, and that of behavioral cloning (BC) is reported for comparison.
5.3 EXPERIMENT 3: MuJoCo (CONTINUOUS OBSERVATION AND ACTION)
We validate RAIRL on MuJoCo continuous control tasks (Hopper-v2, Walker-v2, HalfCheetah-v2, Ant-v2) as follows. We assume multivariate-Gaussia policies (with diagonal covariance matrices) for both learner's policy and expert policy Tg. Instead of tanh-squashed policy in Soft-Actor where
Critic (Haarnoja et al., 2018), we use hyperbolized environments wheree tanh regarded as a part of the environment- with additional engineering the policy networks (See Appendix I.4 for details). We use 100 demonstrations stochastically sampled from Fe to validate RAIRI. In MuJoCo experiments, focus set of Tsallis entropy regularizer (Ω - -Tg) with g = 1,,,,, 2-where Tsallis entropy becomes Shannon entropy for 1. We then exploit the tractable quantities for multi-variate Gaussian distributions Scction 3.2 to stabilize RAIRL and check its performance in terms of its mean Bregman divergence similar to the previous experiment. Note that since both π and n. are
Gaussian and can be evaluated, we can evaluate the individual Bregman divergence DØA (-1s) Te (-Is)) on 8 by using the derivation in Appendix L.3
The performances during RAIRL's training are described as follows. We report ग with both an episodic score (Figure and mean Bregman divergences with respect to three types of Tsallis entropics (Figure 5 Ω = -T with g' 1,5, 2. Note that the objective of RAIRL with = -T,1 is to minimize the corresponding mean Bregman divergence with g' = In Figure 4 both RAIRL- DBM and RAIRL-NSM are shown to achieve the expert performance, irrespective of q, in Hopper- v2, Walker-v2, and HalfCheetah-v2. In contrast, RAIRL in Ant-v2 fails to achieve the expert's performance within 2,000,000 steps and RAIRL-NSM highly outperforms RAIRL-DBM in
8

---

Published as a conference paper at ICLR 2021
setting. Although the episodic scores are comparable for all methods in Hopper-v2. Walker-v2, and HalfCheetah--2, respective divergences are shown to be highly different from one another as shown in Figure 5 RAIRL with 2 in most cases achieves the minimum mean Bregman divergence (for all three divergences with y' = 1, 1.5, 2), whereas RAIRL with q = 1 w-which corresponds to AIRL (Fu et al. 2018) achieves the maximum divergence in most cases. This result is in alignment with our intuition from Section 3.2 as innreases, minimizing the Bregman divergence requires much tighter matching between and Te. Unfortunately, while evaluating the acquired reward-RAy with a randomly initialized agent and acquired reward- the target divergence is not properly decreased in continuous controls. We believe this is because ι probability density function in continuous controls and causes large variance during training, while is mass function and is well-bounded in discrete control problems.
Hopper-v2
Walker2d-v2
HaltCheetah-v2
Ant-v2
Episodic Score
3.6K
8K
10K
6K
1.8K
3K
5K
3K
0
0
0
0
0.5M
1M
0.5M
1M
0
0.5M
1M
0
1M
2M
RAIRL-NMM (q 1)
RAIL-NEM (q 1.5)
RAILLNSM (q 2)
Expert
RARL- 1.5)
RARL-DBM
Figure 4: Averaged episodic score of RAIRL's training in MuJoCo environments. RAIRL with T regularizer with = 1, 1.5, 2 considered.
Hopper-v2
Walker2d-v2
HalfCheetah-v2
Ant-v2
Bregman q= 2.0
9 1.5
(q' 2) q= 1.0
12
24e3
0
3
6e8
35
70a7
0
35
70e10
q 2
Bregman
Div.
= 1.5
(q' = 1.5) q 1.0
0
15
30e1
0
2
4e4
25
50e3
0
14e5
2.0
Bregman
Div.
q 1.5
(q' 1) q 1.0
1
2e1
0
11
22e3
2
4e1
0
8e4
1)
RAIRLNNMM (q 1.5)
RAIRL-NSM 2)
RAIRL-DBM 1)
RAIRL-DBM (q 1.5)
RAIRL-DBM 2)
Figure 5: Bregman divergences with Tsallis entropy T with q' 1, 1.5,2 during RAIRL's training
in MuJoCo environments. We consider RAIRL with Tsallis entropy regularizer Ty with g = 1, 1.1.5,2 2.
DISCUSSION AND FUTURE WORKS
We consider the problem of IRL in regularized MDPs Geist et al. 2019). assuming a class of strongly convex policy regularizers. We theoretically derive its solution (a set of reward functions) and show that learning with these rewards is equivalent to a specific instance of imitation learning-i.. one that minimizes the Bregman divergence associated with policy regularizers. We propose RAIRL-a practical sampled-based IRL. algorithm in regularized MDPs and evaluate its applicability on policy imitation (for discrete and continuous controls) and reward acquisition (for discrete control).
Finally, recent advances in imitation learning and IRL. are built from the perspective of regarding imitation learning as statistical divergence minimization problems (Ke et al. 2019 Ghasemipour et al.) 2019). Although Bregman divergence is known cover various divergences, it does not include some divergence families such as f-divergence (Csiszár) 1963 Amari 2009). Therefore, we believe that considering RL with policy regularization different from Geist al. (2019) and its inverse problem is possible way of finding the links between imitation learning and various statistical distances.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We would like to thank Daewoo Kim at Waymo for his valuable comments on this work.
REFERENCES
Shun-Ichi Amari. a-divergence is unique, belonging both f-divergence and bregman divergence classes. IEEE Transactions on Information Theory, 55(11):4925- 4931, 2009.
Abdeslam Boularias and Brahim Chaib-Draa. Bootstrapping apprenticeship learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 289-297, 2010.
Lev M Bregman. The relaxation method of finding the common point of convex sets and its application the solution of problems convex programming. USSR computational mathematics and mathematical physics, 7(3):200-2177 1967.
Imre Csiszár. Eine informatiinsshhoortisshe ungleichung und ihre anwendung auf den beweis der ergodizitat von markoffschen ketten. Magyar. Tud. Akad. Mat. Kutató Int. Közl, 8:85-108, 1963.
Robert Dadashi. Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. arXiv preprint arXiv:2006. 04678, 2020.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. connection between generative adversarial networks, inverse reinforcement learning, and energy -aased models. arXiv preprint arXiv:1611.03852 2016a.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the 33rd International Conference on Machine Learning (ICML), pp. 49-58, 2016b.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforce- ment learning. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 1582 1591, 2018
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In Proceedings of the 36th International Conference Machine Learning (ICML), pp. 2160-2169, 2019.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Proceedings of the 3rd Conference on Robot Learning (CoRL), 2019.
Ian Goodfellow, Jean Pouget Abadie, Mehdi Mirza, Bing Xu, David Warde Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2672-2680, 2014.
Xin Guo, Johnny Hong, and Nan Yang. Ambiguity set and learning via Bregman and Wasserstein. arXiv preprint arXiv:/705.0000 08050, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning (ICML), pp. 1861-1870, 2018
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 3565-4577. 2016.
Lee K Jones and Charles L Byrne General entropy criteria for inverse problems, with applications to data compression, pattern classification, and cluster analysis. IEEE Transactions on Information Theory, 36(1):23-30, 1990.
10

---

Published as a conference paper at ICLR 2021
Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha Srinivasa. Imitation learning as f f-divergence minimization. arXiv preprint
2019.
Kyungjae Lee, Sungjoon Choi, and Songhwai Oh. Maximum causal tsallis entropy imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4403-4413, 2018.
Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, and Songhwai Oh. Tsallis reinforcement learning: A unified framework for maximum entropy reinforcement learning. arXiv preprint arXiv: (Mt::/902.00333 2019.
Kyungjae Lee, Sungyub Kim. Sungbin Lim, Sungjoon Choi, Mineui Hong, Jaein Kim, Yong-Lae Park, and Songhwai Oh. Generalized Tsallis entropy reinforcement learning and its application to soft mobile robots. In Robotics: Science and Systems (RSS), 2020.
Kun Li, Mrinal Rath, and Joel W Burdick. Inverse reinforcement learning via function approximation for clinical motion analysis. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 610-617, 617. IEEE, 2018.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational Bayes: Unify- ing variational autoencoders and generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 2391-24000 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):52-533) 2015.
Volodymyr Mnih Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML), pp. 1937.1937, 2016.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the 16th International Conference on Machine Learning (ICML), pp. 278-287, 1999.
Andrew Y Ng. Stuart J Russell, al. Algorithms for inverse reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning (ICML), pp. 663-670, 2000.
Frank Nielsen and Richard Nock. On Rényi and Tsallis entropies and divergences for exponential families. arXiv preprint arXiv:1105.32299 2011.
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the 11th Annual Conference on Computational Learning Theory, pp. 101-103, 1998.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
Sahand Sharifzadeh, Ioannis Chiotellis, Rudolph Triebel, and Daniel Cremers. Learning to drive using inverse reinforcement learning and deep Q-networks. arXiv preprint arXiv:1612.03653, 2016.
Adam Stooke and Pieter Abbeel. ripyt: A research code base for deep reinforcement learning in pytorch. arXiv preprint arXiv: 1909:::9000055500 015000 2019.
Umar Syed, Michael Bowling, and Robert E. Schapire. Apprenticeship learning using linear pro- gramming. In Proceedings of the 25th International Conference on Machine Learning (ICML), pp. 1039, 2008.
Zheng Wu. Liting Sun. Wei Zhan, Chenyu Yang, and Masayoshi Tomizuka. Efficient sampling-basee maximum entropy inverse reinforcement learning with application to autonomous driving. IEEE Robotics and Automation Letters, 5(4):5355-5362, 2020.

---

Published as a conference paper at ICLR 2021
Wenhao Yang, Xiang Li, and Zhihua Zhang. A regularized approach to sparse optimal policy in reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), pp- 5938-5948, 2019.
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. GenDice: Generalized offline estimation of Stationary values. In International Conference on Learning Representations, 2019.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
Brian Zicbart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd AAAI Conference Artificial Intelligence, volume 8, pp. 1433-1438, 2008
12

---

Published as a conference paper at ICLR 2021
A BELLMAN OPERATORS, VALUE FUNCTIONS IN REGULARIZED MDPs
Let a policy regularizer Q: A + IR be strongly convex, and define the convex conjugate of 92 is
Q* RA
as
D- (Q(
max -(-ss)AA
e ESxA, E S.
(15)
Then, Bellman operators, equations and value functions in regularied MDPs are defined as follows. Definition 1 (Regularized Bellman Operators). For V Rs, let us define Q(s,a) r(s,a) +
(((g)))(()))))) (V())). The regularized Bellman evaluation operator is defined as
[T*V((s) = (T({)s), ((s,))))))))))))))())))) (((T()))),,EE.
and T V called the regularized Bellman equation. Also, the regularized Bellman optimality operator is defined as
(V)] = max
(T*]](>) = = (QQ(s))))))
*(-19)EAA
and T = V is called the regularized Bellman optimality equation.
Definition 2 (Regularized value functions). The unique fixed point V of the operator T is called the state value function, and (= a) r(s, a) + PLss- value function. Also, the unique fixed point V* of the operator T is called the optimal state value function, and Q" (s,a) a( + Ppeee( (s')) ())] is called the optimal mate-action value function.
is called the state-action
It should be noted that Proposition 1 in Geist et al. uniquely maximizes Eq. (5). For example, when (2
2019 tells us (Q 1)) is policy that
[ss) Σ (a-x(1e) log (als) (negative Shan- exp(Q(x.))
non entropy). (Q }) is a softmax policy, i.e., (Q )) = Evea exp(Q(s,a")) Due to this property, the optimal policy x' of regularized MDP is uniquely found for the optimal state-action value function Q* which is also uniquely defined as the fixed point T"*
B PROOF OF LEMMA 1
Let us define n² = x-1ss. For ((s, = =t(s,a; a(
n. the RL objective Eq.
satisfies
B
(r(s), a:) ((T
Ex
yd -
E
12' (si - E Ba D' ('s Tpp) +((==)) -((())))))))))))))))
(ii)
E
E Banny 22 (T) + 1((T) - s((T()))
(i))
((V[(T=)) ,, (V -,0))) ((T [(x+"1))
Exr
Σ
- ", (V ) TE)
E
72 ( - [(( ([( )* (V )
E
7' (1((7*)) - (((=)) (V(())) )
( _
Σ 7/ D T
(16)
13

---

Published as a conference paper at ICLR 2021
where (i) follows from the assumption r(s, a) t(s, a; Tp.) in Lemma [ (ii) follows from the definition of t(s, a; π) in Eq.(7). (ii)) follows since taking the inner expectation first does not change the overall expectation, (iv) follows from the definition of S2 in Lemma | and Eaea P(a) [V)())(()))) = (VR(p): and (v) follows from the definition of Bregman diver- gence, i.e. Dá (pı llp)) a(pr)) R(pp) - (VD(p2). P1 P2) Due to the non-negativity of D ( is less than or equal zero which can be achieved when Rg.
C PROOF OF COROLLARY I
Let a E (1....,|A4) and Tr = (a) for simplicity. For
((x)= = -XEan - Σ Too(Ta) = = Σ Fa(TT)
aEA
aed
with fo(x) = xœ(x), we have
VΩ(T) = -Y- Σ fa(TT) = (..
Al
aEA
for (z) = dz (xσ(x)). Therefore, for x(-18) we have
t(s, a) = [V[[TT)] - >V) +
= - - Σ Tid (- + (->
Σ w'Ed
Ti Ta.0(TT))
W'EA
(Ti)
Σ
Ted (ol + [ T()((T)))
loed
Enh Tha - ,(T
d'EA
- ((((()))) Ealia ( - )(x)))}-
D PROOF OF OPTIMAL REWARDS ON CONTINUOUS CONTROLS
Note that for two continuous distributions P1 and P2 having probability density functions p1 (z) and (2)),, respectively, the Bregman divergence can be defined as (Guo et al. 2017 Jones & Byrne 1990)
(P) == /// (w(pi(x)) - (p2((x)) - w/ (p)) - p2(x)) dz,
(17)
where w/ wxxx) = wx)) and the divergence is measured point-wisely on E X. Let assume
((T)) / w(F(a))da
(18)
for a probability density function æ on A and define
((5,;))) = ('()"(()) - - wwF"(d')))
(19)
14

---

Published as a conference paper at ICLR 2021
for x- = =--ss) For ((s, a) t(ss, a; TT)), the RL objective in Eq satisfies
E
(r(8i, as) - 2(T-1)]
Ex 7 (18 az (gj - S((T))}}
(1)
w' (a:)) - /A ( (a)w (TT (a)) (T) da / W(T (a))aa)
E
En
T (a)w (T (a))da (x) w (a)) w da
/1 (ati (a)) bo (a) (T
E
w (a))]} da
/ (a)) (0 (a) (a)) da
B
/A (w(T (a)) ( (a)) (a) (a)) (a) (5T (a))} du
E
1=
/ (Tr)(() w/ (TT (a)) (T (a) The (a)) da
E
(a))
1=
Ex
(20)
where (i) follows from r(s, a) = t(s. Tg)). (ii) follows from Eq. 18 and Eq 19 and (iii) follows from the definition of Bregman divergence in T7 Due to the non-negativity of Das (24 is less than or equal to zero which can be achieved when g.. Also, π Tg is unique solution since Eq. ( has unique solution for arbitrary reward functions.
E PROOF OF LEMMA 2
Since Lemma was mentioned but proved in Geist al. (2019), we derive the rigorous proof for Lemma 2 in this subsection. Note that we follow the proof idea in Ng et al. 1999).
First, let us assume an MDP M with reward r and corresponding optimal policy x². From Definition
1 the optimal state-value function V*. and its corresponding state-action value function (V*T(s') should satisfy the regularized Bellman optimality (**"(sa) a) r(s,a) + P
equation
(*) TVV 28 (Q0 ((T(-18))
max
max (..lss, + BBr a.0)) or (V))( ((T(-1s)),
(21)
*(H1))AM
where we explicitize the dependencies on r. Also, the optimal policy x is a unique maximizer for the above maximization.
o() Now, let us consider the shaped rewards r(s, a) ((s') (('))((s))))))))) and r88, a) E (08) (s).
Please note that for both rewards, the expectation over s' for given s, a is equal to
F(s, = r(s, + E. ExNP(Is ((5) 4ss).
15

---

Published as a conference paper at ICLR 2021
and thus, is sufficient only consider the optimality for F. By subtracting ((s) from the regularized optimality Bellman equation for in Eq. (2I) we have
(s) (s)
max (T(-)s) B - ((s) p( P( 17000 -(('))) =(|))EA
- [[(((()s))
(T(-)s), t + (((((ss))
= max
[T (V (s))
That is, V*r + is the fixed point of the regularized Bellman optimality operator T associated with the shaped reward Also, a maximizer for the above maximization is π since subtracting ((s) from 21 does not change the unique maximizer n²"
F COMPARISON BETWEEN OUR SOLUTION AND EXISTING SOLUTION IN GEIST ET AL. (2019)
F.1
ISSUES WITH THE SOLUTIONS IN GEIST ET AL.. (2019)
In Proposition of Geist et al. 2019)) solution of regularized IRL is given, and we rewrite the relevant theorem in this subsection. Let us consider a regularized IRL problem, where Tg A is an expert policy. Assuming that both the model (dynamics, discount factor and regularizer) and the expert policy are known, Geist et al. (2019) proposed solution of regularized IRL as follows:
Lemma (A solution of regularized IRL from Geist et al. 2019)). Ler On E SxA be a function
such that (-)s))) (Q, (s, )),,
S. Also, define
Ne (s, a) == Qe (8, a) - yB (() (22 (e
= Q₂ (s, a) - (Br-P()aa) (TT (-Is'), Q (s', - 10x
Then, in the D-regularized MDP with the reward rg. Tg is optimal policy.
Proof. Although the brief idea of the proof is given Geist et al. (201)) we rewrite the proof in
more rigorous way follows. Let define Vg (s) (TT (1s). Qu (s. )) 22 (-1s)). Then,
(s, a) = Qg (8, a) - P(- [V (s')], and thus, Qg (8, a) Tg (8, a) + + (V (+)).
By using this and regularized Bellman optimality operator, we have
[T' Vo)s ( (Q (s,
max (T Qs (s, )) ((=(+1s)
=(-1))
(ii))
(TT (-ss) Og (s, M (-1=) = (s),,,
where (i) and (ii) follow from Definition 1 and (iii) follows since Th is unique maximizer. Thus, Ve is the fixed point of T'*, and (- V2* (Q= (s,,))) E S, becomes an optimal policy.
For example, when negative Shannon entropy is used as regularizer, we can get r. (s, a)
log mg (als) by setting Qg (s, a) = log mg (a|s). However, solution proposed in Lemma has two crucial issues:
Issue 1. It requires the knowledge on the model dynamics, which is generally intractable.
Issue We need to figure out Qr that satisfies Tg (-|s) = VΩ (Q (s. (s.))) .)), which comes from the
relationship between the optimal value function and optimal policy (Geist et al. 2019).
In the following subsection, we show how our solution in Lemma is related to the solution from Geist et al. (2019) in Lemma
F.2 RELATION BETWEEN THE SOLUTION OF GEIST ET AL. (2019) AND OUR TRACTABLE SOLUTION
Let us consider the expert policy Th and functions Q, and re satisfying the conditions in Lemmal From Lemma regularized MDP with the shaped reward
ig (s, a) := Tg (s, a) + (ero))+BrrPPpp..c BB D(s') (bss
16

---

Published as a conference paper at ICLR 2021
for 1 I has its optimal policy as ng. Since d can be arbitrarily chosen, let assume 4(s)
0 (Q (s, Then, we have
Eg (s)a)
-T{(8, a) + P(aa) ((5)) (s8)
(8,( - P( [12" (s', + TBp a. (Qe (0) (Q -
-
= - (Q (s.
(22)
Note that the reward in Eq. 22 does require the knowledge on the model dynamics, which addresses Issue 1 in Appendix F.I Also, by using (s) = 52" (Qp (s, in the proof of Lemma
the reward in Eq. (22 can be written as
Fg (s, (s, (s),
which means ig (s, a) is advantage function for the optimal policy Te in the P-regularized MDP.
However, we still have Issue in Appendix F.1 since
since Q in Eq. (2. is generally intractable (Geist
al. 2019), which prevents from finding the appropriate Og (s. a). Interestingly, we show that for all 5 E S and (s.)) = V (-1s))
(Q
arg max (IT+s) (x,, 4 - ((=(11))) =(-1a)Ea>
arg max (T(sss) V. (+5))) ( (+(((+)))
=(-1))aa
arg min ((T(-1s)) (11))
=--1))aa
arg min ((T((1s)) - 11x (-1s) (V (1|s)) o (-1s))
*(-+))=
min (T(((( (s)(-(s)][TTe (+ss) = TT (-1s),
=(-1))AA
where the last equality holds since the Bregman divergence DØ (-((s) [s) mg (-1s) is greater than or equal to zero and its lowest value is achieved when π (-118) = Te (-18). This means that when Qg (8.)) (-(sm)(--s)) is used, the condition Tg (-1s) VQ" (Qg (s) in Lemma is satisfied without knowing the tractable form of S2" or VΩ. Thus, Issue 2 in Appendix F.llis addressed; we instead require the knowledge on the gradient VΩ of the policy regularizer Q. which is practically
more tractable. Finally, by substituting Qe (s,
(5)) for (''s, == V[((T(-(s) S,
Eq. we have
fs a) = (' a; mgg (1'
12'(s, a: TT)) ( (-1), (2 3 2(T, (+s)))
S2 12'(s, (2(ss;;) TT)) - Eg' (1 (1)) (s, R.) 2(T (-1s)
f(s, a; mg);
where t(s, a; nT) is our proposed solution in Lemmal 1
G PROOF OF LEMMA 3
RL objective in Regularized MDPs w.r.. normalized visitation distributions. For reward
function E SxA and strongly convex function Ω A + R, the RL objetive Ja(r, 77) in [q equivalent to
arg max Ja dx) (r, d=)xx - (((k.)
(23)
where for a set D of normalized visitation distributions (Syed et al. 2008)
D D: de de RSKA, > (1-)))))(1-)))8666 + Σ P(s)1s..)d(s..))
5.a
17

---

Published as a conference paper at ICLR 2021
define (2dd) ((s [s)]] and #d(+|s)
d(a))
A for d D and use
E. d(s,a))
Tde (-|s) (--ss) for all E S. For i D + R, its convex conjugate ft" is
(r)r) = max (r, d) dED
max (r, -
max (r, dx)xsx- (((dx)
TEAg
max REa$
Σ
dr -
(1 7) max Jn( T
(24)
where (i) follows from using the one-to-one correspondence between policies and visitation distribu- tions (Syed et al. 2008 Ho Ermon 2016). Note that Eq. (4) is equal to the optimal discounted
average return in regularized MDPs.
IRL objective in regularized MDPs w.r.t. normalized visitation distributions. By using the RL rewrite the IRL objective in 6 w.rt. the normalized visitation distributions as the maximization the following objective over r E RSXA;
objective in Eq. 23
(1-)) Julr, TT)) max Ju(r; nEAS
Ja(r, dag)
max Jn(r, dED [Ja(T (1g) [1(r,d)}
min dED
{ ((r, )8x -((d=))) )) - ((rd
min deD
-min (i(d) 0(drg) (r, dog )SxA].
(25)
deD
Note that if Vi((d) is well-defined and r= Vi((d=g) for any strictly convex Q Eq. 25 is equal to
min
(((d) (2(dg)) (Vā(d= ), dig )sx min DSxA
(dlld.g)
dED
dED
where the equality comes from the definition of Bregman divergence.
a). For simpler notation, we use matrix- notation for the proof when discrete state and action spaces S = {1,..., SI} and A = {1,..., |AI} are considered. For normalized visitation distribution D, let us define
Proof of t(s, a; ma) (s)
d= d(s, a), A. d' [df..... .... d
l s
d
A
= d dSlT
E SSA
/S|
[
π(x) 1/
21....
L|A||
IRA = E1...., [|AI] "|ll" | RAA
ExEA La
where 1 14 = [1,.... 1)7 E R an IAl-dimensional all-one vector. By using these notations, the
original Ω can be rewritten
π(D) = Σ ((((((d)) = Σ (ddd((((d)))
5.8
SES
18

---

Published as a conference paper at ICLR 2021
The gradient of Ω w.r.t D (using cotoominnto-llyyoou notation) is
(i((D)
D
VDΩ(D)
E S
0d
0d
where each element of VDD(D) satisfies
(Q(D)
(0((D)
(
0d
0dj
0d
1 Id (d'((T(d"))
0d
= =((T(d")) + 'dd' (dd))
0d
On (d') (d)))
= =((T(d))) + dd
(26)
Od
(d
for
Üu (d*)
(d')
Oi A (d*)
Od
Od
Od
FF (d*)
d
dd
d(1
(1 T
+ d%
0d
Od
1 dd
dd
Od
Note that each element of (x.
satisfies
öd
0n (d')
Od
(1 d2 + 0% 0(12dd)
Odz
ad
Od
0% laa = a' d1)(1") d d1)
Ita = a' d( -d' =
= -1' [Ifa a'} - (d'))
and thus,
(d
- (Lax
(27)
0d
By substituting Eq. 27 into (2. we have
(i((D)
On (d)) (d()()'))
((F(d")) d
Od
0d Ōπ
-(F((d)) d' d1 -1 {LAXA 1 (d
θा (d.)
- 1 )l(T(d')) =(({(d")) + -
OF (d')
0 (d')) OE (d+)
| (d')
1
On (d=)
(d'))
1 + ((T
(28)
θग (d=)
θt (d=)
we use the function notation, Eq. (4.28 can be written as
v(((d)((..) V[((Ta((Is))(a) E, Em [V)(aa()s))(aa) (((a((1ss)
((,,a; #a)
for of Eq. in Lemma 1
19

---

Published as a conference paper at ICLR 2021
H DERIVATION OF BREGMAN-DIVERRENCEEBB MEASURE IN CONTINUOUS CONTROLS
In Eq. 17. the Bregman divergence the control task defined as
D (PPIIP) = / (w(p)(x)) -((p(=)) - (12 - p2(2))) dx.
(29)
Note that consider ?(pp) w((())) fx (xl=f((((x)) de for fo(x) xQ(x), which
makes Eq. 29) equal to
+ [ ( fal - -p(())) da
f (x) (f)()2(x)) ((1((=))) dr / pa(x)) (f)(((())) (x)())))) de -Eropi (p2((())) ((r(())) Eu-pp (p2()())) (x)2)
Thus, by considering a learning agent's policy π" = (-1ss) expert policy गh. = Tg (-|s), and the objective in (8) characterized by the Bregman divergence, we can think of the following measure between expert and agent policies:
Byndy (+(((=
.ond, EaNn (F)(T - ((T - Eawny (f)( - ((
(30)
I TSALLIS ENTROPY AND ASSOCIATED BREGMAN DIVERGENCE AMONG MULTI-VARIATE GAUSSIAN DISTRIBUTIONS
Based on the derivation in Nielsen Nock (201 )) we derive the Tsallis entropy and associated
Bremgan divergence as follows. We first consider the distributions in the exponential family
exp ((0,(((())) F(0) + k(x)).
(31)
Note that for
Σ
[ 02 :
0
Σ
t(x)
xx
1
(0)
0
01
+
log -
-T0,²
2 --1
+ log(2x)) |1) [
4
2
2
k(z) 0,
we can recover the multi-variate Gaussian distribution (Nielsen & Nock 201T):
(xp((0,((x)) (() + k(x))
(32)
exp
112 xy--la - tr(2 - 'xx7) ( - 7/
- --111
log(2
(33)
2
1
/2 exp
T Σ -1 2
x
T
-1
x z//² - -ー |1
(34)
2
T)TT - (x 11
exp
(I - u)
(35)
d2m) E 1/2
For two distributions with k(x) 0.
=xx cxp( t(x)) F(0)), f(x) exp((0,t(x)) F(0))
20

---

Published as a conference paper at ICLR 2021
that share F, and k, it can be shown that
at, (()))
= exp F(aa + 30) - -oF(0) -BF())
since
{x))"((z)ddz
/ exp (o(0,t(z)) - -FF(0) - -BP(0)) dr
= / exp (la0 (6, t(x)) - F(o0 301) exp (F(at + -B) -F(() - BF(() dr
exp (F(a0 + 30) aF 3F) /
exp 30, f(x)) F(a0 + +B0) dr
exp (F(a0 + 30) FF 3F())
1.1 TSALLIS ENTROPY
For x0x) 9-1 (1 x0-1) and 1, the Tsallis entropy of π can be written as
(T) = Eenn o(x) = π (z))--1
dr
(xx
1 (1 [[T,T:q.
1 - exp exp (F(g0) - (F)
9
9-1
If is multivariate Gaussian distribution, have
F(q0)
2 2
T μ
log(2 [=]
log d
2
2
qF(0)=
2"
11"
- +
+
)
2
F(g0) - gF()
log (2 log(2 )
log 9d
2
d
1
d dlogg 9
(1 - 9)
log 2n +
log |>]
2
2
2(1 ②)
For Σ = diag(o? (oi. 0ā), we have
dlog q
F(g0) - qF()) (1 - 9)
log 2T +
log [
2(1 ②)
d
d logg 2(1 ①)
(1 4)
log 2T
+
of log
2
2
1=1
log 2t
log ạ
(1 9)
+ log 0,
2
2(1 - 9) 5
i=1
21

---

Published as a conference paper at ICLR 2021
1.2 TRACTABLE FORM OF BASELINE
A For <(x) q-1 (1 1) we have
(x)
(1 920
k
(g qxx-- - -a - 1))
q qk
q (1 - x" 1 - k
g0(x) k.
Therefore, the baseline can be rewritten as
B [ (x) + ((x) = En - gb(xx) <bxx) = (1 - (TT) + k.
For a multivariate Gaussian distribution π, the tractable form of Eggnn 1- (x) + x(xx) can be derived by using that of Tsallis entropy T (T)) of x.
1.3 BREGMAN DIVERGENCE WITH TSALLIS ENTROPY REGULARIZATION
In Eq.(3)) we consider the following form of the Bregman divergence:
f (()(((((())) (((())) f - - (+(())))dx
For ((x:)
19 شfم (1 -.----111 g0(x) k. and le 1, the above form is
equal
1- -+(x))) 9-
dx - (T) - (q - (+) +1
q
1
q
(q - (t)
q 6-11
q
--1
q
(x (g (f)
9 4-1
q
For multivariate Gaussians
(2) Nax; μ, Σ) / = [Vi...., V/] vu)T Σ diag o 02),
A (xx) = Nax; i. E) f = [01, Dz)" Σ = diag( ( 02),
we have
/ (()))())) I(T, #; 1,9 - 1) = exp (F(07) - F(B) (q 1))())
1
where
0
-
0
E 2
Σ μ + (g - 1) A
[8]
0' 0 + (q
1)0
(
+ (g - 11
0
81
(g
0
V/8 02
+ (g
1) id
o
07
03.
1
1
8/
diag
(q
1)
(g
1)
[o
87
03
22

---

Published as a conference paper at ICLR 2021
and
v22
log 2T
F(0) = 2" "E''E'11 + log(2
= Σ
log oi
2
20²
2
1=1
F(O) = 1 -- 1og(2x)
= Σ
i2
log 2T
log 8i
2
2
20
2
i=1
1
P 1)0) = 7 (01) (02) 10 + logl - (0,) -1
-0,
2
(q 1)
Σ
log 2m
log
2
1) 03
2
+
(g
1) 8
of
0
23

---

Published as a conference paper at ICLR 2021
J EXPERIMENT SETTING
J.1 POLICY REGULARIZERS IN EXPERIMENTS
Table 1: Policy regularizers & and their corresponding fø (Yang et al. 2019).
reg. type.
condition
((x)
fg(()
Shannon
log
log æ
Tsallis
1: 0>0, >1
9-1 (1 wa-1)
- (1 qu=-!)
Exp
1: 0. 2 I
x" 9"'
x² n² (l logs
Cos
0.<, 0 x/2
cos(9z) cos(O))
cos(0) cos((1a)) Por (0n))))
0. S x/2
sin(0) (0x)
sin(O) sin(0x) On cos((x)
J.2 DENSITY-BASE BASED MODEL
By exploiting the knowledge on the reward in Corollary 1
(F((ss)) (o ((T(a'|s)))
we consider the density-based model (DBM) which is defined by
a)( R - (TTo (a|s)) + (s)
for 0 = (0),02) Here, f is a function that can be known priorly, and Te, is a neural network which defined separately from the policy neural network тØ. The DBM for discrete control problems are depicted in Figure 6 (Left). The model outputs rewards over all actions in parallel, where softmax is used for Te, (-1s) and - f' is elementwisely applied to those softmax outputs followed by elementwisely adding Bog (x). For continuous control (Figure Right), we use the network architecture similar that in discrete control, where multivariate Gaussian distribution is used instead of a softmax layer.
Te (s
[a a)
Ba
Bug (s)
To (a|s)
To, (-1s)
softmax
linear
linear
Po, (8)
Eo (s)
FC
FC
FC
FC
Figure 6: Density-based model for discrete (Leff) and continuous control (Right)
J.3 EXPERT IN BERMUDA WORLD ENVIRONMENT
We assume stochastic expert defined by
(als) -1a)) Proj(O(
g(t) -, - d
|| s112 + = 1,2,3,
24

---

Published as a conference paper at ICLR 2021
for (a, y). s (5(1) / (-5,10), 502, = (I(2) (0, = (x(3) (3) (5, 10), < 10 and an operator Proj(0) :R - AA that maps to the closest angle in A. Figure 7
depicted the expert policy.
Figure 7: Visualization of the expert policy
J.4 MUJOCO EXPERIMENT SETTING
Instead of directly using MuloCo environments with tanh-squashed policies proposed in Soft Actor- Critic (SAC) (Haarnoja al. 2018), we move tanh to part the environment -amedd hyperbolized environments in short and assume Gaussian policies. Specifically, after an action a is sampled from policies, we pass Lanh(a) to environment. We then consider multi-variate Gaussian policy
(-|ss) N (n(s),
with (s) =
(s)) diag (oi (aa(s))²), where
a0.9aan((0.999) S (s(s) S (0.99))(0.999), log(0001)))) S log (s(s) S log(2)
for all 1,..,, d. Instead of using clipping, we tanh-activated outputs and scale them fit in the ahove ranges, which empirically improves the performance. Also, instead of using potential-based reward shaping used in AIRL (Fu et al. 20188), we update the moving mean of intermediate reward values and update the value network with mean-subtracted rewards-skt that the value network gets approximately mean-zero reward- to stabilize the RL part RAIRL. Note that this is motivated by Lemma 22 from which we can guarantee that any constant shift of reward functions does not change optimality.
25

---

Published as a conference paper at ICLR 2021
J.5 HYPERPARAMETERS
Table 2] Table 3 and Table list the parameters used in our Bandit, Bermuda World, and MuJoCo experiments, respectively.
Table 2: Hyperparameters for Bandit environments.
Hyper-paramerts
Bandit
Batch size
500
Initial exploration steps
10,000
Replay size
500,000
Target update rate (T)
0.0005
Learning rate
0.0005
À
5
4
(Tsallis entropy T
2.0
1 (Tsallis entropy T₂
1.0
Number trajectories
1,000
Reward learning rate
0.0005
Steps per update
50
Total environment steps
500,000
Table 3: Hyperparameters for Bermuda World environment.
Hyper-parameter
Bermuda World
Batch size
500)
Initial exploration steps
10,000
Replay size
500,000
Target update rate (r)
0.0005
Learning rate
0.0005
q (Tsallis entropy T,A
2.0
k (Tsallis entropy T
1.0
Number of trajectories
1,000
Reward learning rate
0.0005
(For evaluation)
(For evaluation) Learning rate
0.001
(For evaluation) Target update rate (7)
0.0005
Steps per update
50
Number steps
500,000
Table 4: Hyperparameters for MuJoCo environments.
Hyper-parameter
Hopper
Walker2d
HalfCheetah
Ant
Batch size
256
256
256
Initial exploration steps
10,000
10,000
10,000
10,000
Replay size
1,000,000
1,000,000
1,000,000
1,000,000
Target update rate (r)
0.005
0.005
0.005
0.005
Learning rate
0.001
0.001
0.001
0.001
0.0001
0.000001
0.0001
0.000001
k (Tsallis entropy ,
1.0
1.0
1.0
1.0
Number trajectories
100
100
100
100
Reward learning rate
0.001
0.001
0.001
0.001
Steps per update
1
Number of steps
1,000,000
1,000,000
1,000,000
2,000,000
26


Reviews:
[{"review_id": "xV6bV0Hnbs4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1. this paper studies an interesting problem called regularized inverse reinforcement learning, which is novel to me and brings me some knowledge.\n2. this paper first proposes a general solution for regularized IRL; then, TSALLIS entropy is proposed with IRL, as well as an adversary learning-based training strategy. The methodology part seems reasonable and the contribution is good.\n3. the paper is well-organized. The experiments are convincing, trying to illustrate the performance under different scenarios, including continuous and discrete reinforcement learning settings.\n\nCons:\n1. in the experiments, much comparison is internal comparison of proposed methods. In the beginning, the authors mention Shannon-entropy regularizers' limitations. The authors should conduct more experiments to prove the statement. Now, only Experiment 3 mentions it. \n2. I don't see many comparisons with other baselines. Adding more baselines is better.\n\n\nIn summary, this paper studies a novel problem, regularized inverse reinforcement learning. The paper proposes several techniques to solve the regularization in different aspects. The experiments are conducted under different settings, but some cons are needed to revise.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ocT-r_GeUBP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=====POST-REBUTTAL COMMENTS======== \n\nI thank the authors for the response and the efforts in the updated draft. I think the paper is stronger and should be accepted.\n\n\n########################################################################## \nSummary: \nThis paper examines the problem of regularized IRL. As opposed to standard IRL, which can have degenerate solutions, regularized IRL has a unique optimal solution. The authors examine different forms of regularization and derive an efficient IRL algorithm which generalizes AIRL and is applicable to continuous control tasks.\n\n########################################################################## \nReasons for score: \nThe  idea of regularized IRL is a nice contribution to the field. It ties in nicely with recent work on regularized RL and overcomes some of the challenges of classical IRL. The mathematical foundation is rigorous and the experimental results are promising.\n\n########################################################################## \nPros: 1. Nice general mathematical framework that generalizes prior work. 2. Tractable algorithms that work in continuous state and action domains. 3. The empirical analysis of the actual divergence is interesting. \n\n########################################################################## \nCons: 1. The paper is a bit notation heavy and that makes it hard to follow. 2. Some undefined notation such as \\mathcal{D} on page 2.  3. Lacking in justification for the choice of entropy regularizers. Why Tsallis? Why follow Yang 2019 in choice of regularizer? 4. In general a lot of the theoretical results are given without any intuitive explanation for what they mean or how to interpret them. 5. It's unclear how this is different from the theory for GAIL which is also in terms of a general regularizer.\n\n##########################################################################\nOther comments:\n\nThe sentences before the discussion mention that RAIRL is good for safe imitation learning. This is unclear since safety is undefined. Also in the same sentences the authors mention that RAIRL fails to minimize divergence, but it seems to work in the figure for some values of q'. What do the authors mean by fail?\n\nFigure 1 seems to transition from having high divergence in the top to low divergence (for  small log \\sigma). Why?\n\nRather than using a single dash for parentheticals, it is usually better to use latex em-dash for parentheticals by using three dashes --- with no space before and after \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WIx0rccmV6T", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows a formulation of regularized Markov Decision Processes (MDPs), which is slightly different from that of Geist et al. (2019). Then, the authors propose a novel inverse reinforcement learning under regularized MDPs. One of the contributions is that policy regularization considered here is more general than that of Yang et al. (2019). \n\nThis paper is written very well and is of publishing quality. I think it is sufficiently significant to be accepted. Still, I have the following questions. \n\n1. The proposed method is based on the relationship between imitation learning and statistical divergence minimization. If my understanding is correct, Bregman divergence plays a role in generalizing generalized adversarial imitation learning. However, as the authors mentioned in Section 6, Bregman divergence does not include f-divergence, which is also studied in imitation learning. Would you discuss the connection to the formulation using f-divergence in more detail?\n\n2. I am interested in the relationship between the proposed method and Lee et al. (NeurIPS2018). Is the proposed method nearly the same as Lee et al. (2018) when Tsallis entropy is selected as regularization? If not, does the proposed method outperform Lee et al. (2018) in the MuJoCo control tasks? \n\n3. The authors claim that the solutions provided by Geist et al. (2019) are intractable in the Introduction. However, it is shown that the reward baseline term in Corollary 1 is intractable except for some well-studied setups. Does it imply that the proposed method faces the same difficulty when applied with arbitrary policy regularization?\n\n4. The experimental results shown in Figure 3 is interesting, but I have a few concerns. In some cases, the averaged Bregman divergence of RAIRL-NSM (\\lambda = 1) was larger than that of Random. Would you show the example of the learned policy for the readers’ understanding? Besides, is the same policy regularization used in Behavior Cloning? Finally, are exp, cos, and sin the meaningful regularizer? \n\n5. To derive the practical algorithms, the authors consider the same form of the policy regularization used by Yang et al. (2019), which is given by - \\lambda E[\\phi(\\pi(a))]. Is it possible to derive the algorithm in which the regularizer is given by \\Omega(\\pi)?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ebg0gVmee6F", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method for regularized inverse RL. The paper builds upon work by Geist et al. who studied regularized MDPs with convex policy regularizers. The Shannon entropy is a special case of such a policy regularizer. The paper extends the analysis of Geist et al. for regularized IRL and devises tractable solutions to regularized IRL that only depend on the analytic knowledge of the regularizer. The paper further proposes regularized adversarial IRL (RAIRL), an extension of AIRL by Fu et al., as an algorithm for IRL in regularized MDPs. The algorithm is validated on a number of domains.\n\nThe paper is generally clearly written. I believe that the paper is technically correct (and I appreciated that the derivations are well explained in the appendix). The paper is novel to the best of my knowledge and I think better forms of regularization could improve the state of the art in inverse RL and will be of interest to the ICLR community. \n\nI think it's cool that this paper allows us to solve inverse RL problems in Tsallis entropy regularized MDPs and a great technical contribution. However I think what is less clear to me is why we would want to. As it is I'm leaning towards acceptance because I enjoyed the paper and appreciate the technical contribution but I think the paper would be substantially stronger if it made a better case for why regularizers other than the normal Shannon entropy are relevant for practical IRL problems (perhaps even with an example application demonstrating a clear benefit over AIRL as well as other imitation learning baselines).\n\nMinor comments/questions/typos:\n- Would RAIRL-NSM eventually reach expert performance on the ant-v2 task?\n- I find the results in figure 4 a little surprising. I would have expected the divergence to be minimal for q=q'. Could you comment further on this empirical result?\n- I'm wondering why the exponential/cosine/sine regularizers were chosen in experiment 1. Are these just meant to demonstrate the flexibility of the framework or is there a motivation for using them in practice?\n- I appreciate the need to stay within the page limit but would encourage the authors to decompress the paper a bit for a potential camera-ready version. \n- I think it would be nice to include an algorithm box as a summary of the proposed algorithm. \n- Figures 3 and 4 are a bit small and hard to parse. \n- The paper might benefit from some further proofreading for typos and small grammatical errors such as missing articles\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8ql8xpyhOV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work considers a regularized IRL setup, where instead of the entropy regularization used in maximum entropy IRL, an arbitrary convex regularizer $\\Omega$ is used. The work presents a number of theoretical results for this general setting, and it is shown that when $\\Omega$ is Tsallis entropy, the $RL \\cdot IRL$ is equivalent to minimizing a Bregman divergence defined based on the Tsallis entropy and the expert state-action distribution. A practical algorithm is presented for IRL with the Tsallis entropy. A number of experiments are performed to obtain understanding of various components.\n\nI hope that the following questions can be resolved during the discussion period as some things are a bit unclear to me which is preventing me from providing a more detailed analysis of the work.\n\n* Section 3\n  * Section 3.1, paragraph after equation 6: Can you clarify (for example with concrete examples or equations) what you mean by terms like \"functional-form\" and \"intractable\"?\n* Section 4\n  * __Could you clarify your structured discriminator? What exactly are you using for $t(s,a,\\pi)$? How does this relate to the Tsallis entropy baseline equations derived earlier?__\n  * Equation 13: Should this be argmax? I think maybe this should instead be just a small gradient update using this objective since $\\hat{t}(s,a;\\pi^{(i)})$ is the correct objective locally around $\\pi^{(i)}$?\n* Section 5\n  * __Based on the description in H.2 I don't understand what the Density-Based Model is. Please clarify.__\n  * Section 5.1: Can you clarify what you mean by \"acquires the ground truth rewards\"? Figure 2 is showing different reward curves for each method, so how can they all be the ground truth?\n  * Section 5.2: Can you explain how you are computing the divergences between the joint state-action distributions $p(s,a)$ of the expert and trained policy? Also how do you compute these divergences for the Mujoco experiments in section 5.3?\n  * Section 5.3: Could you elaborate what you mean by \"Unfortunately, RAIRL fails to acquire a reward function that effectively minimizes the target divergence in continuous controls.\"?\n\nYou may also want to cite:\n@article{ke2019imitation,\n  title={Imitation Learning as $ f $-Divergence Minimization},\n  author={Ke, Liyiming and Barnes, Matt and Sun, Wen and Lee, Gilwoo and Choudhury, Sanjiban and Srinivasa, Siddhartha},\n  journal={arXiv preprint arXiv:1905.12888},\n  year={2019}\n}", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies inverse reinforcement learning through the prism of regularized Markov decision processes, by generalizing MaxEntIRL from the negative entropy to any strongly convex regularizer (as a side note, strict convexity might be enough for many results).
The reviewers appreciated the clarity, the mathematical rigor and the empirical evaluation of this paper. They asked some questions and raised some concerns, that were mostly addressed in the rebuttal and the revision provided by the authors.
This is a strong paper, for which the AC recommends acceptance.

Decision: Accept (Spotlight)