REGULARIZED INVERSE REINFORCEMENT LEARNING
Year: 2021

Abstract:
Inverse Reinforcement Learning (IRL) aims to facilitate a learner’s ability to imitate expert behavior by acquiring reward functions that explain the expert’s decisions. Regularized IRL applies strongly convex regularizers to the learner’s policy in order to avoid the expert’s behavior being rationalized by arbitrary constant rewards, also known as degenerate solutions. We propose tractable solutions, and practical methods to obtain them, for regularized IRL. Current methods are restricted to the maximum-entropy IRL framework, limiting them to Shannon-entropy regularizers, as well as proposing solutions that are intractable in practice. We present theoretical backing for our proposed IRL method’s applicability to both discrete and continuous controls, empirically validating our performance on a variety of tasks.

heading: 1 INTRODUCTION
text: Reinforcement learning (RL) has been successfully applied to many challenging domains including games (Mnih et al., 2015; 2016) and robot control (Schulman et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018). Advanced RL methods often employ policy regularization motivated by, e.g., boosting exploration (Haarnoja et al., 2018) or safe policy improvement (Schulman et al., 2015). While Shannon entropy is often used as a policy regularizer (Ziebart et al., 2008), Geist et al. (2019) recently proposed a theoretical foundation of regularized Markov decision processes (MDPs)—a framework that uses strongly convex functions as policy regularizers. Here, one crucial advantage is that an optimal policy is shown to uniquely exist, whereas multiple optimal policies may exist in the absence of policy regularization.
Meanwhile, since RL requires a given or known reward function (which can often involve non-trivial reward engineering), Inverse Reinforcement Learning (IRL) (Russell, 1998; Ng et al., 2000)—the problem of acquiring a reward function that promotes expert-like behavior—is more generally adopted in practical scenarios like robotic manipulation (Finn et al., 2016b), autonomous driving (Sharifzadeh et al., 2016; Wu et al., 2020) and clinical motion analysis (Li et al., 2018). In these scenarios, defining a reward function beforehand is particularly challenging and IRL is simply more pragmatic. However, complications with IRL in unregularized MDPs relate to the issue of degeneracy, where any constant function can rationalize the expert’s behavior (Ng et al., 2000).
Fortunately, Geist et al. (2019) show that IRL in regularized MDPs—regularized IRL—does not contain such degenerate solutions due to the uniqueness of the optimal policy for regularized MDPs. Despite this, no tractable solutions of regularized IRL—other than maximum-Shannon-entropy IRL (MaxEntIRL) (Ziebart et al., 2008; Ziebart, 2010; Ho & Ermon, 2016; Finn et al., 2016a; Fu et al., 2018)—have been proposed.
In Geist et al. (2019), solutions for regularized IRL were introduced. However, they are generally intractable since they require a closed-form relation between the policy and optimal value function and the knowledge on model dynamics. Furthermore, practical algorithms for solving regularized IRL problems have not yet been proposed.
We summarize our contributions as follows. Unlike the solutions in Geist et al. (2019), we propose tractable solutions for regularized IRL problems that can be derived from policy regularization and
∗Correspondence to: Wonseok Jeon <jeonwons@mila.quebec>
its gradient in discrete control problems (Section 3.1). We additionally show that our solutions are tractable for Tsallis entropy regularization with multi-variate Gaussian policies in continuous control problems (Section 3.2). We devise Regularized Adversarial Inverse Reinforcement Learning (RAIRL), a practical sample-based method for policy imitation and reward learning in regularized MDPs, which generalizes adversarial IRL (AIRL, Fu et al. (2018)) (Section 4). Finally, we empirically validate our RAIRL method on both discrete and continuous control tasks, evaluating RAIRL via episodic scores and from divergence minimization perspective (Ke et al., 2019; Ghasemipour et al., 2019; Dadashi et al., 2020) (Section 5).

heading: 2 PRELIMINARIES
text: Notation For finite sets X and Y , Y X is a set of functions from X to Y . ∆X (∆XY ) is a set of (conditional) probabilities over X (conditioned on Y ). Especially for the conditional probabilities pX|Y ∈ ∆XY , we say pX|Y (·|y) ∈ ∆X for y ∈ Y . R is the set of real numbers. For functions f1, f2 ∈ RX , we define 〈f1, f2〉X := ∑ x∈X f1(x) f2(x).
Regularized Markov Decision Processes and Reinforcement Learning We consider sequential decision making problems where an agent sequentially chooses its action after observing the state of the environment, and the environment in turn emits a reward with state transition. Such an interaction between the agent and the environment is modeled as an infinite-horizon Markov Decision Process (MDP), Mr := 〈S,A, P0, P, r, γ〉 and the agent’s policy π ∈ ∆AS . The terms within the MDP are defined as follows: S is a finite state space, A is a finite action space, P0 ∈ ∆S is an initial state distribution, P ∈ ∆SS×A is a state transition probability, r ∈ RS×A is a reward function, and γ ∈ [0, 1) is the discount factor. We also define an MDP without reward asM− := 〈S,A, P0, P, γ〉. The normalized state-action visitation distribution, dπ ∈ ∆S×A, associated with π is defined as the expected discounted state-action visitation of π, i.e., dπ(s, a) := (1−γ) ·Eπ[ ∑∞ i=0 γ
iI{si = s, ai = a}], where the subscript π on E means that a trajectory (s0, a0, s1, a1, ...) is randomly generated fromM− and π, and I{·} is an indicator function. Note that dπ satisfies the transposed Bellman recurrence (Boularias & Chaib-Draa, 2010; Zhang et al., 2019):
dπ(s, a) = (1− γ)P0(s)π(a|s) + γπ(a|s) ∑ s̄,ā P (s|s̄, ā)dπ(s, a).
We consider RL in regularized MDPs (Geist et al., 2019), where the policy is optimized with a causal policy regularizer. Mathematically for an MDPMr and a strongly convex function Ω : ∆A → R, the objective in regularized MDPs is to seek π that maximizes the expected discounted sum of rewards, or return in short, with policy regularizer Ω:
arg max π∈∆AS
JΩ(r, π) := Eπ [ ∞∑ i=0 γi{r(si, ai)− Ω(π(·|si))} ] = 1 1− γ E(s,a)∼dπ [r(s, a)− Ω(π(·|s)] .
(1)
It turns out that the optimal solution of Eq.(1) is unique (Geist et al., 2019), whereas multiple optimal policies may exist in unregularized MDPs (See Appendix A for a detailed explanation). In later work (Yang et al., 2019), Ω(p) = −λEa∼pφ(p(a)), p ∈ ∆A was considered for λ > 0 and φ : (0, 1] → R satisfying some mild conditions. For example, RL with Shannon entropy regularization (Haarnoja et al., 2018) can be recovered by φ(x) = − log x, while RL with Tsallis entropy regularization (Lee et al., 2020) can be recovered from φ(x) = kq−1 (1 − x
q−1) for k > 0, q > 1. The optimal policy π∗ for Eq.(1) with Ω from Yang et al. (2019) is shown to be
π∗(a|s) = max { gφ ( µ∗(s)−Q∗(s, a)
λ
) , 0 } , (2)
Q∗(s, a) = r(s, a) + γEs′∼P (·|s,a)V ∗(s′), V ∗(s) = µ∗(s)− λ ∑ a∈A π∗(a|s)2φ′(π∗(a|s)), (3)
where φ′(x) = ∂∂xφ(x), gφ is an inverse function of f ′ φ for fφ(x) := xφ(x), x ∈ (0, 1], and µ∗ is a normalization term such that ∑ a∈A π
∗(a|s) = 1. Note that we still need to find out µ∗ to acquire a closed-form relation between optimal policy π∗ and value function Q∗. However, such relations
have not been discovered except for Shannon-entropy regularization (Haarnoja et al., 2018) and specific instances (q = 1, 2,∞) of Tsallis-entropy regularization (Lee et al., 2019) to the best of our knowledge.
Inverse Reinforcement Learning Given a set of demonstrations from an expert policy π E , IRL (Russell, 1998; Ng et al., 2000) is the problem of seeking a reward function from which we can recover πE through RL. However, IRL in unregularized MDPs has been shown to be an ill-defined problem since (1) any constant reward function can rationalize every expert and (2) multiple rewards meet the criteria of being a solution (Ng et al., 2000). Maximum entropy IRL (MaxEntIRL) (Ziebart et al., 2008; Ziebart, 2010) is capable of solving the first issue by seeking a reward function that maximizes the expert’s return along with Shannon entropy of expert policy. Mathematically, for the RL objective JΩ in Eq.(1) and Ω = −H for negative Shannon entropy H(p) = Ea∼p[− log p(a)] (Ho & Ermon, 2016), the objective of MaxEntIRL is
MaxEntIRL(π E ) := arg max r∈RS×A
{ J−H(r, πE )− max
π∈∆AS J−H(r, π)
} . (4)
Another commonly used IRL method is Adversarial Inverse Reinforcement Learning (AIRL) (Fu et al., 2018) which involves generative adversarial training (Goodfellow et al., 2014; Ho & Ermon, 2016) to acquire a solution of MaxEntIRL. AIRL considers the structured discriminator (Finn et al., 2016a) D(s, a) = σ(r(s, a)− log π(a|s)) = e r(s,a)
er(s,a)+π(a|s) for σ(x) := 1/(1 + e −x) and iteratively
optimizes the following objective: max
r∈RS×A E(s,a)∼dπ E [logDr,π(s, a)] + E(s,a)∼dπ [log(1−Dr,π(s, a))] ,
max π∈∆AS E(s,a)∼dπ [logDr,π(s, a)− log(1−Dr,π(s, a))] = max π∈∆AS E(s,a)∼dπ [r(s, a)− log π(a|s)] .
(5) It turns out that AIRL minimizes the divergence between visitation distributions dπ and dπ
E by
solving minπ∈∆AS KL(dπ||dπE ) for Kullback-Leibler (KL) divergence KL(·||·) (Ghasemipour et al., 2019) .

heading: 3 INVERSE REINFORCEMENT LEARNING IN REGULARIZED MDPS
text: In this section, we propose the solution of IRL in regularized MDPs—regularized IRL—and relevant properties in Section 3.1. We then discuss a specific instance of our proposed solution in Section 3.2 where Tsallis entropy regularizers and multi-variate Gaussian policies are used in continuous action spaces.

heading: 3.1 SOLUTIONS OF REGULARIZED IRL
text: We consider regularized IRL that generalizes MaxEntIRL in Eq.(4) to IRL with a class of strongly convex policy regularizers:
IRLΩ(πE ) := arg max r∈RS×A
{ JΩ(r, πE )− max
π∈∆AS JΩ(r, π)
} . (6)
For any strongly convex policy regularizer Ω, regularized IRL does not suffer from degenerate solutions since there is a unique optimal policy in any regularized MDP (Geist et al., 2019). While Geist et al. (2019) proposed solutions of regularized IRL, those are intractable solutions (See Appendix F.1 for a detailed explanation). In the following lemma, we propose tractable solutions that only requires the evaluation of the policy regularizer (Ω) and its gradient (∇Ω) which are more manageable in practice. Our solution is motivated from figuring out a reward function that is capable of converting regularized RL into an equivalent divergence minimization problem associated with π and π
E :
Lemma 1. For a policy regularizer Ω : ∆A → R, let us define t(s, a;π) := Ω′(s, a;π)− Ea′∼π(·|s)[Ω′(s, a′;π)] + Ω(π(·|s)) (7)
for Ω′(s, ·;π) := ∇Ω(π(·|s)) := [∇pΩ(p)]p=π(·|s) ∈ RA, s ∈ S. Then, t(s, a;πE ) for expert policy π E is a solution of regularized IRL with Ω.
Proof. (Abbreviated. See Appendix B for the full version.) With r(s, a) = t(s, a;π E ), the RL objective in Eq.(1) becomes equivalent to the problem of minimizing the discounted sum of Bregman divergences between π and π
E
arg min π∈∆AS
Eπ [ ∞∑ i=0 γiDAΩ (π(·|si)||πE (·|si)) ] , (8)
where DAΩ is the Bregman divergence (Bregman, 1967) defined by D A Ω (p1||p2) = Ω(p1)−Ω(p2)− 〈∇Ω(p2), p1− p2〉A for p1, p2 ∈ ∆A. Due to the non-negativity of the Bregman divergence, π = πE is a solution of Eq.(8) and is unique since Eq.(1) has the unique solution for arbitrary reward functions (Geist et al., 2019).
In particular, for any policy regularizer Ω represented by an expectation over the policy (Yang et al., 2019), Lemma 1 can be reduced to the following solution in Corollary 1: Corollary 1. For Ω(p) = −λEa∼pφ(p(a)) with p ∈ ∆A (Yang et al., 2019) , Eq.(7) becomes
t(s, a;π) = −λ · { f ′φ(π(a|s))− Ea′∼π(·|s)[f ′φ(π(a′|s))− φ(π(a′|s))] } (9)
for f ′φ(x) = ∂ ∂x (xφ(x)). The proof is in Appendix B.
Throughout the paper, we denote reward baseline by the expectation Ea∼π(·|s)[f ′φ(π(a|s)) − φ(π(a|s))]. Note that for continuous control tasks with Ω(p) = −λEa∼pφ(p(a)), we can obtain the same form of the reward in Eq.(9) (The proof is in Appendix D). Although the reward baseline is generally not intractable in continuous control tasks, we derive a tractable reward baseline for a special case (See Section 3.2). Additionally, when λ = 1 and φ(x) = − log x, it can be shown that t(s, a;π) = log π(a|s)—for both discrete and continuous control problems—which was used as a reward objective in previous work (Fu et al., 2018), and that the Bregman divergence in Eq.(8) becomes the KL divergence KL(π(·|s)||π
E (·|s)).
Additional solutions of regularized IRL can be found by shaping t(s, a;π E ) as stated in the following lemma:
Lemma 2 (Potential-based reward shaping). Let π∗ be the solution of Eq.(1) in a regularized MDP Mr with a regularizer Ω : ∆A → R and a reward function r ∈ RS×A. Then for Φ ∈ RS , using either r(s, a) + γΦ(s′)− Φ(s) or r(s, a) + Es′∼P (·|s,a)Φ(s′)− Φ(s) as a reward does not change the solution of Eq.(1). The proof is in Appendix E.
From Lemma 1 and Lemma 2, we prove the sufficient condition of rewards being solutions of the IRL problem. However, the necessary condition—a set of those solutions are the only possible solutions for an arbitrary MDP—is not proved (Ng et al., 1999).
In the following lemma, we check how the proposed solution is related to the normalized statevisitation distribution which can be discussed in the line of distribution matching perspective on imitation learning problems (Ho & Ermon, 2016; Fu et al., 2018; Ke et al., 2019; Ghasemipour et al., 2019):
Lemma 3. Given the policy regularizer Ω, let us define Ω̄(d) := E(s,a)∼d[Ω(π̄d(·|s))] for an arbitrary normalized state-action visitation distribution d ∈ ∆S×A and the policy π̄d(a|s) := d(s,a)∑ a′∈A d(s,a ′) induced by d. Then, Eq.(7) is equal to
t(s, a; π̄d) = [∇Ω̄(d)](s, a). (10)
When Ω̄(d) is strictly convex and a solution t(s, a;π E ) = ∇Ω̄(dπ E ) of IRL in Eq.(10) is used, the RL objective in Eq.(1) is equal to
arg min π∈∆AS
DS×A Ω̄ (dπ||dπ E ),
where DS×A Ω̄ is the Bregman divergence among visitation distributions defined by DS×A Ω̄ (d1||d2) = Ω̄(d1)−Ω̄(d2)−〈∇Ω̄(d2), d1−d2〉 for visitation distributions d1 and d2. The proof is in Appendix G.
Note that the strict convexity of Ω̄ is required for DS×A Ω̄
to become a valid Bregman divergence. Although the strict convexity of a policy regularizer Ω does not guarantee the assumption on the strict convexity of Ω̄, it has been shown to be true for Shannon entropy regularizer (Ω̄ = −H̄ of Lemma 3.1 in Ho & Ermon (2016)) and Tsallis entropy regularizer with its constants k = 12 , q = 2 (Ω̄ = −W̄ of Theorem 3 in Lee et al. (2018)).

heading: 3.2 IRL WITH TSALLIS ENTROPY REGULARIZATION AND GAUSSIAN POLICIES
text: For continuous controls, multi-variate Gaussian policies are often used in practice (Schulman et al., 2015; 2017) and we consider IRL problems with those policies in this subsection. In particular, we consider IRL with Tsallis entropy regularizer Ω(p) = −T kq (p) = −Ea∼p[ kq−1 (1− p(a)
q−1)] (Lee et al., 2018; Yang et al., 2019; Lee et al., 2020) for a multi-variate Gaussian policy π(·|s) = N (µ(s),Σ(s)) with µ(s) = [µ1(s), ..., µd(s)]T , Σ(s) = diag{(σ1(s))2, ..., (σd(s))2}. In such a case, we can obtain tractable forms of the following quantities:
Tsallis entropy. The tractable form of Tsallis entropy for a multi-variate Gaussian policy is
T kq (π(·|s)) = k(1− e(1−q)Rq(π(·|s)))
q − 1 ,Rq(π(·|s)) = d∑ i=1 { log( √ 2πσi(s))− log q 2(1− q) } for Renyi entropyRq . Its derivation is given in Appendix I. Reward baseline. The reward baseline term Ea∼π(·|s)[f ′φ(π(a|s)) − φ(π(a|s))] in Corollary 1 is generally intractable except for either discrete control problems or Shannon entropy regularization where the reward baseline is equal to −1. Interestingly, as long as the tractable form of Tsallis entropy can be derived, that of the corresponding reward baseline can also be derived since the reward baseline satisfies Ea∼π(·|s)[f ′φ(π(a|s))− φ(π(a|s))] = (q − 1)Ea∼π(·|s)[φ(π(a|s)]− k = (q − 1)T kq (π(·|s))− k.
Here, the first equality holds with f ′φ(x) = k q−1 (1 − qx q−1) = qφ(x) − k for Tsallis entropy regularization.
Bregman divergence associated with Tsallis entropy. For two different multivariate Gaussian policies, we derive the tractable form of the Bregman divergence (associated with Tsallis entropy) between two policies. The resultant divergence has a complicated form, so we leave its derivation in Appendix I.3.
For deeper understanding of Tsallis entropy and its associated Bregman divergence in regularized MDPs, we consider an example in Figure 1. We first assume that both learning agents’ and experts’ policies follow uni-variate Gaussian distributions π = N (µ, σ2) and π
E = N (0, (e−3)2),
respectively. We then evaluate the Bregman divergence in Figure 1 by using its tractable form and varying q from 1.0—which corresponds to the KL divergence—to 2.0. We observe that the constant q from the Tsallis entropy affects the sensitivity of the associated Bregman divergence w.r.t. the mean and standard deviation of the learning agent’s policy π. Specifically, as q increases, the size of the valley—relatively red region in Figure 1—across the µ-axis and log σ-axis decreases. This suggests that for larger q, minimizing the Bregman divergence requires more tightly matching means and variances of π and π
E .
Algorithm 1: Regularized Adversarial Inverse Reinforcement Learning (RAIRL) 1: Input: A set D
E of expert demonstration generated by expert policy π E , a reward approximator
rθ, a policy πψ for neural network parameters θ and ψ 2: for each iteration do 3: Sample rollout trajectories by using the learners policy πψ . 4: Optimize θ with the discriminator Drθ,πψ and the learning objective in Eq.(11). 5: Optimize ψ with Regularized Actor Critic by using rθ as a reward function. 6: end for 7: Output: πψ ≈ πE , rθ(s, a) ≈ t(s, a;πE ) (a solution of the IRL problem in Lemma 1).

heading: 4 ALGORITHMIC CONSIDERATION
text: Based on a solution for regularized IRL in the previous section, we focus on developing an IRL algorithm in this section. Particularly to recover the reward function t(s, a;π
E ) in Lemma 1, we
design an adversarial training objective as follows. Motivated by AIRL (Fu et al., 2018), we consider the following structured discriminator associated with π, r and t in Lemma 1:
Dr,π(s, a) = σ(r(s, a)− t(s, a;π)), σ(z) = 1
1 + e−z , z ∈ R.
Note that we can recover the discriminator of AIRL in Eq.(5) when t(s, a) = log π(a|s) (φ(x) = log x and λ = 1). Then, we consider the following optimization objective of the discriminator which is the same as that of AIRL:
t̂(s, a;π) := arg max r∈RS×A E(s,a)∼dπ E [logDr,π(s, a)] + E(s,a)∼dπ [log(1−Dr,π(s, a))] . (11)
Since the function x 7→ a log σ(x) + b log(1 − σ(x)) attains its maximum at σ(x) = aa+b , or equivalently at x = log ab (Goodfellow et al., 2014; Mescheder et al., 2017), it can be shown that
t̂(s, a;π) = t(s, a;π) + log dπ E (s, a)
dπ(s, a) . (12)
When π = π E in Eq.(12), we have t̂(s, a;π E ) = t(s, a;π E ) since dπ = dπ E , which means the maximizer t̂ becomes the solution of IRL after the agent successfully imitates the expert policy π
E .
To do so, we consider the following iterative algorithm. Assuming that we find out the optimal reward approximator t̂(s, a;π(i)) in Eq.(12) for the policy π(i) of the i-th iteration, we get the policy π(i+1) by optimizing the following objective with gradient ascent:
maximize π∈∆AS
E(s,a)∼dπ [ t̂(s, a;π(i))− Ω(π(·|s)) ] . (13)
The above expectation in Eq.(13) can be decomposed into the following two terms E(s,a)∼dπ [ t̂(s, a;π(i))− Ω(π(·|s)) ] = E(s,a)∼dπ [ t(s, a;π(i))− Ω(π(·|s)) ] −KL(dπ||dπ E )
= −E(s,a)∼dπ [ DAΩ (π(·|s)||π(i)(·|s)) ] ︸ ︷︷ ︸
(I)
−KL(dπ||dπ E )︸ ︷︷ ︸ (II) ,
(14)
where the second equality follows since Lemma 1 tells us that t(s, a;π(i)) is a reward function that makes π(i) an optimal policy in the Ω-regularized MDP. Minimizing term (II) in Eq.(14) makes π(i+1) close to π
E while minimizing term (I) can be regarded as a conservative policy optimization
around the policy π(i) (Schulman et al., 2015).
In practice, we parameterize our reward and policy approximations with neural networks and train them using an off-policy Regularized Actor-Critic (RAC) (Yang et al., 2019) as described in Algorithm 1.Below, we evaluate our Regularized Adversarial Inverse Reinforcement Learning (RAIRL) approach across various scenarios.

heading: 5 EXPERIMENTS
text: We summarize the experimental setup as follows. In our experiments, we consider Ω(p) = −λEa∼p[φ(p(a)] with the following regularizers from Yang et al. (2019): (1) Shannon entropy (φ(x) = − log x), (2) Tsallis entropy regularizer (φ(x) = kq−1 (1 − x
q−1)), (3) exp regularizer (φ(x) = e − ex), (4) cos regularizer (φ(x) = cos(π2x)), (5) sin regularizer (φ(x) = 1 − sin π 2x). The above regularizers were chosen since other regularizers have not been empirically validated to the best of our knowledge. We chose these regularizers to make our empirical analysis more tractable. In addition, we model the reward approximator of RAIRL as a neural network with either one of the following models: (1) Non-structured model (NSM)—a simple feed-forward neural network that outputs real values used in AIRL (Fu et al., 2018)—and (2) Density-based model (DBM)—a model using a neural network for π (softmax for discrete controls and multi-variate Gaussian model for continuous controls) of the solution in Eq.(1) (See Appendix J.2 for a detailed explanation). For the RL algorithm of RAIRL, we implement Regularized Actor Critic (RAC) (Yang et al., 2019) on top of the SAC implementation from Rlpyt (Stooke & Abbeel, 2019). Other settings are summarized in Appendix J. For all experiments, we use 5 runs and report 95% confidence intervals.

heading: 5.1 EXPERIMENT 1: MULTI-ARMED BANDIT (DISCRETE ACTION)
text: We consider a 4-armed bandit environment as shown in Figure 2 (left). An expert policy π E is assumed to be either dense (with probability 0.1, 0.2, 0.3, 0.4 for a = 0, 1, 2, 3) or sparse (with probability 0, 0, 1/3, 2/3 for a = 0, 1, 2, 3). For those experts, we use RAIRL with actions sampled from π
E and compare learned rewards with the ground truth reward t(s, a;π E ) in Lemma 1. When
π E is dense, RAIRL successfully acquires the ground truth rewards irrespective of the reward model choices. When sparse π
E is used, however, RAIRL with a non-structured model (RAIRL-NSM) fails
to recover the rewards for a = 0, 1—where π E (a) = 0—due to the lack of samples at the end of the imitation. On the other hand, RAIRL with a density-based model (RAIRL-DBM) can recover the correct rewards due to the softmax layer which maintains the sum over the outputs equal to 1. Therefore, we argue that using DBM is necessary for correct reward acquisition since a set of demonstrations is generally sparse. In the following experiment, we show that the choice of reward models indeed affects the performance of rewards.

heading: 5.2 EXPERIMENT 2: BERMUDA WORLD (CONTINUOUS OBSERVATION, DISCRETE ACTION)
text: We consider an environment with a 2-dimensional continuous state space as described in Figure 3. At each episode, the learning agent is initialized uniformly on the x-axis between −5 and 5, and there are 8 possible actions—an angle in {−π,− 34π, ..., 1 2π, 3 4π} that determines the direction of movement. An expert in Bermuda World considers 3 target positions (−5, 10), (0, 10), (5, 10) and behaves stochastically. We state how we mathematically define the expert policy π
E in Appendix J.3.
During RAIRL’s training (Figure 3, Top row), we use 1000 demonstrations sampled from the expert and periodically measure mean Bregman divergence, i.e., for DAΩ (p1||p2) = Ea∼p1 [f ′φ(p2(a)) −
φ(p1(a))]− Ea∼p2 [f ′φ(p2(a))− φ(p2(a))],
1
N N∑ i=1 DAΩ (π(·|si)||πE (·|si)).
Here, the states s1, ..., sN come from 30 evaluation trajectories that are stochastically sampled from the agent’s policy π—which is fixed during evaluation—in a separate evaluation environment. During the evaluation of learned reward (Figure 3, Bottom row), we train randomly initialized agents with RAC and rewards acquired from RAIRL’s training and check if the mean Bregman divergence is properly minimized. We measure the mean Bregman divergence as was done in RAIRL’s training.
RAIRL-DBM is shown to minimize the target divergence more effectively compared to RAIRL-NSM during reward evaluation, although both achieve comparable performances during RAIRL’s training. Moreover, we substitute λ with 1, 5, 10 and observe that learning with λ larger than 1 returns better rewards—only λ = 1 was considered in AIRL (Fu et al., 2018). Note that in all cases, the minimum divergence achieved by RAIRL is comparable with that of behavioral cloning (BC). This is because BC performs sufficiently well when many demonstrations are given. We think the divergence of BC may be the near-optimal divergence that can be achieved with our policy neural network model.

heading: 5.3 EXPERIMENT 3: MUJOCO (CONTINUOUS OBSERVATION AND ACTION)
text: We validate RAIRL on MuJoCo continuous control tasks (Hopper-v2, Walker-v2, HalfCheetah-v2, Ant-v2) as follows. We assume multivariate-Gaussian policies (with diagonal covariance matrices) for both learner’s policy π and expert policy π
E . Instead of tanh-squashed policy in Soft-Actor
Critic (Haarnoja et al., 2018), we use hyperbolized environments—where tanh is regarded as a part of the environment—with additional engineering on the policy networks (See Appendix J.4 for details). We use 100 demonstrations stochastically sampled from π
E to validate RAIRL. In MuJoCo
experiments, we focus on a set of Tsallis entropy regularizer (Ω = −T 1q ) with q = 1, 1.5, 2—where Tsallis entropy becomes Shannon entropy for q = 1. We then exploit the tractable quantities for multi-variate Gaussian distributions in Section 3.2 to stabilize RAIRL and check its performance in terms of its mean Bregman divergence similar to the previous experiment. Note that since both π and π
E are multi-variate Gaussian and can be evaluated, we can evaluate the individual Bregman
divergence DAΩ (π(·|s)||πE (·|s)) on s by using the derivation in Appendix I.3. The performances during RAIRL’s training are described as follows. We report π with both an episodic score (Figure 4) and mean Bregman divergences with respect to three types of Tsallis entropies (Figure 5) Ω = −T 1q′ with q′ = 1, 1.5, 2. Note that the objective of RAIRL with Ω = −T 1q is to minimize the corresponding mean Bregman divergence with q′ = q. In Figure 4, both RAIRLDBM and RAIRL-NSM are shown to achieve the expert performance, irrespective of q, in Hopperv2, Walker-v2, and HalfCheetah-v2. In contrast, RAIRL in Ant-v2 fails to achieve the expert’s performance within 2,000,000 steps and RAIRL-NSM highly outperforms RAIRL-DBM in our
setting. Although the episodic scores are comparable for all methods in Hopper-v2, Walker-v2, and HalfCheetah-v2, respective divergences are shown to be highly different from one another as shown in Figure 5. RAIRL with q = 2 in most cases achieves the minimum mean Bregman divergence (for all three divergences with q′ = 1, 1.5, 2), whereas RAIRL with q = 1—which corresponds to AIRL (Fu et al., 2018)—achieves the maximum divergence in most cases. This result is in alignment with our intuition from Section 3.2; as q increases, minimizing the Bregman divergence requires much tighter matching between π and π
E . Unfortunately, while evaluating the acquired reward—RAC with
a randomly initialized agent and acquired reward—the target divergence is not properly decreased in continuous controls. We believe this is because π is a probability density function in continuous controls and causes large variance during training, while π is a mass function and is well-bounded in discrete control problems.

heading: 6 DISCUSSION AND FUTURE WORKS
text: We consider the problem of IRL in regularized MDPs (Geist et al., 2019), assuming a class of strongly convex policy regularizers. We theoretically derive its solution (a set of reward functions) and show that learning with these rewards is equivalent to a specific instance of imitation learning—i.e., one that minimizes the Bregman divergence associated with policy regularizers. We propose RAIRL—a practical sampled-based IRL algorithm in regularized MDPs—and evaluate its applicability on policy imitation (for discrete and continuous controls) and reward acquisition (for discrete control).
Finally, recent advances in imitation learning and IRL are built from the perspective of regarding imitation learning as statistical divergence minimization problems (Ke et al., 2019; Ghasemipour et al., 2019). Although Bregman divergence is known to cover various divergences, it does not include some divergence families such as f -divergence (Csiszár, 1963; Amari, 2009). Therefore, we believe that considering RL with policy regularization different from Geist et al. (2019) and its inverse problem is a possible way of finding the links between imitation learning and various statistical distances.

heading: ACKNOWLEDGMENTS
text: We would like to thank Daewoo Kim at Waymo for his valuable comments on this work.

heading: A BELLMAN OPERATORS, VALUE FUNCTIONS IN REGULARIZED MDPS
text: Let a policy regularizer Ω : ∆A → R be strongly convex, and define the convex conjugate of Ω is Ω∗ : RA → R as
Ω∗(Q(s, ·)) = max π(·|s)∈∆A 〈π(·|s), Q(s, ·)〉A − Ω(π(·|s)), Q ∈ RS×A, s ∈ S. (15)
Then, Bellman operators, equations and value functions in regularied MDPs are defined as follows.
Definition 1 (Regularized Bellman Operators). For V ∈ RS , let us define Q(s, a) = r(s, a) + γEs′∼P (·|s,a)[V (s′)]. The regularized Bellman evaluation operator is defined as
[TπV ](s) := 〈π(·|s), Q(s, ·)〉A − Ω(π(·|s)), s ∈ S,
and TπV = V is called the regularized Bellman equation. Also, the regularized Bellman optimality operator is defined as
[T ∗V ](s) := max π(·|s)∈∆A [TπV ](s) = Ω∗(Q(s, ·)), s ∈ S,
and T ∗V = V is called the regularized Bellman optimality equation.
Definition 2 (Regularized value functions). The unique fixed point V π of the operator Tπ is called the state value function, and Qπ(s, a) = r(s, a) + γEs′∼P (·|s,a)[V π(s′)] is called the state-action value function. Also, the unique fixed point V ∗ of the operator T ∗ is called the optimal state value function, and Q∗(s, a) = r(s, a) + γEs′∼P (·|s,a)[V ∗(s′)] is called the optimal state-action value function.
It should be noted that Proposition 1 in Geist et al. (2019) tells us ∇Ω∗(Q(s, ·)) is a policy that uniquely maximizes Eq.(15). For example, when Ω(π(·|s)) = ∑ a∼π(·|s) log π(a|s) (negative Shannon entropy),∇Ω∗(Q(s, ·)) is a softmax policy, i.e.,∇Ω∗(Q(s, ·)) = exp(Q(s,·))∑ a′∈A exp(Q(s,a
′)) . Due to this property, the optimal policy π∗ of a regularized MDP is uniquely found for the optimal state-action value function Q∗ which is also uniquely defined as the fixed point of T ∗.
B PROOF OF LEMMA 1
Let us define πs = π(·|s). For r(s, a) = t(s, a;π E ), the RL objective Eq.(1) satisfies
Eπ [ ∞∑ i=0 γi {r(si, ai)− Ω(πsi)} ] (i) = Eπ [ ∞∑ i=0 γi {t(s, a;π E )− Ω(πsi)} ] (ii) = Eπ
[ ∞∑ i=0 γi { Ω′(si, ai;πE )− Ea∼πsiE Ω ′(si, a;πE ) + Ω(π si E )− Ω(πsi) }]
(iii) = Eπ [ ∞∑ i=0 γi { Ea∼πsiΩ′(si, a;πE )− Ea∼πsiE Ω ′(si, a;πE ) + Ω(π si E )− Ω(πsi) }]
(iv) = Eπ [ ∞∑ i=0 γi { 〈∇Ω(πsi E ), πsi〉A − 〈∇Ω(πsiE ), π si E 〉A + Ω(πsiE )− Ω(π si) }]
= Eπ [ ∞∑ i=0 γi { Ω(πsi E )− Ω(πsi) + 〈∇Ω(πsi E ), πsi〉A − 〈∇Ω(πsiE ), π si E 〉A }]
= −Eπ [ ∞∑ i=0 γi { Ω(πsi)− Ω(πsi E )− 〈∇Ω(πsi E ), πsi〉A + 〈∇Ω(πsiE ), π si E 〉A }]
= −Eπ [ ∞∑ i=0 γi { Ω(πsi)− Ω(πsi E )− 〈∇Ω(πsi E ), πsi − πsi E 〉A }] (v) = −Eπ [ ∞∑ i=0 γiDAΩ (π si ||πsi E )) ] ,
(16)
where (i) follows from the assumption r(s, a) = t(s, a;π E ) in Lemma 1, (ii) follows from the definition of t(s, a;π) in Eq.(7), (iii) follows since taking the inner expectation first does not change the overall expectation, (iv) follows from the definition of Ω′ in Lemma 1 and∑ a∈A p(a)[∇Ω(p)](a) = 〈∇Ω(p), p〉A, and (v) follows from the definition of Bregman divergence, i.e., DAΩ (p1||p2) = Ω(p1)−Ω(p2)− 〈∇Ω(p2), p1 − p2〉A. Due to the non-negativity of DAΩ , Eq.(16) is less than or equal to zero which can be achieved when π = π
E .
C PROOF OF COROLLARY 1
Let a ∈ {1, ..., |A|} and πa = π(a) for simplicity. For
Ω(π) = −λEa∼πφ(πa) = −λ ∑ a∈A πaφ(πa) = −λ ∑ a∈A fφ(πa)
with fφ(x) = xφ(x), we have
∇Ω(π) = −λ∇π1,...,π|A| ∑ a∈A fφ(πa) = −λ[f ′φ(π1), ..., f ′φ(π|A|)]T
for f ′φ(x) = ∂ ∂x (xφ(x)). Therefore, for π s = π(·|s) we have
t(s, a;π) = [∇Ω(πs)](a)− 〈∇Ω(πs), πs〉A + Ω(πs)
= −λf ′φ(πsa)− (∑ a′∈A πsa′(−λf ′φ(πsa′)) ) + ( −λ ∑ a′∈A πsa′φ(π s a′) )
= −λ { f ′φ(π s a)− (∑ a′∈A πsa′f ′ φ(π s a′) ) + (∑ a′∈A πsa′φ(π s a′) )}
= −λ { f ′φ(π s a)− ∑ a′∈A πsa′ ( f ′φ(π s a′)− φ(πsa′) )} = −λ { f ′φ(π s a)− Ea′∼πs [ f ′φ(π s a′)− φ(πsa′) ]} .

heading: D PROOF OF OPTIMAL REWARDS ON CONTINUOUS CONTROLS
text: Note that for two continuous distributions P1 and P2 having probability density functions p1(x) and p2(x), respectively, the Bregman divergence can be defined as (Guo et al., 2017; Jones & Byrne, 1990)
DXω (P1||P2) := ∫ X {ω(p1(x))− ω(p2(x))− ω′(p2(x))(p1(x)− p2(x))} dx, (17)
where ω′(x) := ∂∂xω(x) and the divergence is measured point-wisely on x ∈ X . Let us assume
Ω(π) = ∫ A ω(π(a))da (18)
for a probability density function π on A and define
t(s, a;π) := ω′(πs(a))− ∫ A [πs(a′)ω′(πs(a′))− ω(πs(a′))] da′. (19)
for πs = π(·|s). For r(s, a) = t(s, a;π E ), the RL objective in Eq.(1) satisfies
Eπ [ ∞∑ i=0 γi {r(si, ai)− Ω(πsi)} ] (i) = Eπ [ ∞∑ i=0 γi {t(si, ai;πE )− Ω(πsi)} ] (ii) = Eπ
[ ∞∑ i=0 γi { ω′(πsi E (ai))− ∫ A [ πsi E (a)ω′(πsi E (a))− ω(πsi E (a)) ] da− ∫ A ω(πsi(a))da }]
= Eπ [ ∞∑ i=0 γi {∫ A πsi(a)ω′(πsi E (a))da− ∫ A [ πsi E (a)ω′(πsi E (a))− ω(πsi E (a)) + ω(πsi(a)) ] da }]
= Eπ [ ∞∑ i=0 γi ∫ A { πsi(a)ω′(πsi E (a))− [ πsi E (a)ω′(πsi E (a))− ω(πsi E (a)) + ω(πsi(a)) ]} da ]
= Eπ [ ∞∑ i=0 γi ∫ A { ω(πsi E (a))− ω(πsi(a)) + πsi(a)ω′(πsi E (a))− πsi E (a)ω′(πsi E (a)) } da ]
= −Eπ [ ∞∑ i=0 γi ∫ A { ω(πsi(a))− ω(πsi E (a))− πsi(a)ω′(πsi E (a)) + πsi E (a)ω′(πsi E (a)) } da ]
= −Eπ [ ∞∑ i=0 γi ∫ A { ω(πsi(a))− ω(πsi E (a))− ω′(πsi E (a)) ( πsi(a)− πsi E (a) )} da ] (iii) = −Eπ
[ ∞∑ i=0 γiDAω (π si ||πsi E )) ] , (20)
where (i) follows from r(s, a) = t(s, a;π E ), (ii) follows from Eq.(18) and Eq.(19), and (iii) follows from the definition of Bregman divergence in Eq.(17). Due to the non-negativity of Dω, Eq.(24) is less than or equal to zero which can be achieved when π = π
E . Also, π = π E is a unique solution
since Eq.(1) has a unique solution for arbitrary reward functions.
E PROOF OF LEMMA 2
Since Lemma 2 was mentioned but not proved in Geist et al. (2019), we derive the rigorous proof for Lemma 2 in this subsection. Note that we follow the proof idea in Ng et al. (1999).
First, let us assume an MDP Mr with a reward r and corresponding optimal policy π∗. From Definition 1, the optimal state-value function V ∗,r and its corresponding state-action value function Q∗,r(s, a) := r(s, a) + γEs′∼P (·|s,a)[V ∗,r(s′)] should satisfy the regularized Bellman optimality equation
V ∗,r(s) = T ∗,rV ∗,r(s) = Ω∗(Q∗,r(s, a))
= max π(·|s)∈∆A
〈π(·|s), Q∗,r(s, a)〉A − Ω(π(·|s))
= max π(·|s)∈∆A
〈π(·|s), r(s, a) + γEs′∼P (·|s,a)[V ∗,r(s′)]〉A − Ω(π(·|s)), (21)
where we explicitize the dependencies on r. Also, the optimal policy π∗ is a unique maximizer for the above maximization.
Now, let us consider the shaped rewards r(s, a)+Φ(s′)−Φ(s) and r(s, a)+Es′∼P (·|s,a)Φ(s′)−Φ(s). Please note that for both rewards, the expectation over s′ for given s, a is equal to
r̃(s, a) = r(s, a) + Es′∼P (·|s,a)Φ(s′)− Φ(s),
and thus, it is sufficient to only consider the optimality for r̃. By subtracting Φ(s) from the regularized optimality Bellman equation for r in Eq.(21), we have V ∗,r(s)− Φ(s) = max π(·|s)∈∆A 〈π(·|s), r(s, a) + γEs′∼P (·|s,a)Φ(s′)− Φ(s) + γEs′∼P (·|s,a)[V ∗,r(s′)− Φ(s′)]〉A
− Ω(π(·|s)) = max π(·|s)∈∆A 〈π(·|s), r̃(s, a) + γEs′∼P (·|s,a)[V ∗,r(s′)− Φ(s′)]〉A − Ω(π(·|s))
= [T ∗,r̃(V ∗,r − Φ)](s). That is, V ∗,r − Φ is the fixed point of the regularized Bellman optimality operator T ∗,r̃ associated with the shaped reward r̃. Also, a maximizer for the above maximization is π∗ since subtracting Φ(s) from Eq.(21) does not change the unique maximizer π∗.

heading: F COMPARISON BETWEEN OUR SOLUTION AND EXISTING SOLUTION IN GEIST
text: ET AL. (2019)
F.1 ISSUES WITH THE SOLUTIONS IN GEIST ET AL. (2019)
In Proposition 5 of Geist et al. (2019), a solution of regularized IRL is given, and we rewrite the relevant theorem in this subsection. Let us consider a regularized IRL problem, where π
E ∈ ∆AS is
an expert policy. Assuming that both the model (dynamics, discount factor and regularizer) and the expert policy are known, Geist et al. (2019) proposed a solution of regularized IRL as follows: Lemma 4 (A solution of regularized IRL from Geist et al. (2019)). Let Q
E ∈ RS×A be a function
such that πE(·|s) = ∇Ω∗(QE (s, ·)), s ∈ S. Also, define r E (s, a) := Q E
(s, a)− γEs′∼P (·|s,a)[Ω∗(QE (s′, ·))] = Q
E (s, a)− γEs′∼P (·|s,a)[〈πE (·|s′), QE (s′, ·)〉A − Ω(πE (·|s′))].

heading: Then, in the Ω-regularized MDP with the reward r
text: E , π E is an optimal policy.
Proof. Although the brief idea of the proof is given in Geist et al. (2019), we rewrite the proof in a more rigorous way as follows. Let us define V
E (s) = 〈π E (·|s), Q E (s, ·)〉A − Ω(πE (·|s)). Then,
r E (s, a) = Q E (s, a)− γEs′∼P (·|s,a)[VE (s′)], and thus, QE (s, a) = rE (s, a) + γEs′∼P (·|s,a)[VE (s′)]. By using this and regularized Bellman optimality operator, we have
[T ∗V E ](s) (i) = Ω∗(Q E (s, ·)) (ii)= max
π(·|s)∈∆A 〈π(·|s), Q E (s, ·)〉A − Ω(π(·|s))
(iii) = 〈π
E (·|s), Q E (s, ·)〉A − Ω(πE (·|s)) = VE (s),
where (i) and (ii) follow from Definition 1, and (iii) follows since π E is a unique maximizer. Thus, V E is the fixed point of T ∗, and πE(·|s) = ∇Ω∗(QE (s, ·)), s ∈ S, becomes an optimal policy.
For example, when negative Shannon entropy is used as a regularizer, we can get r E (s, a) = log π
E (a|s) by setting Q E (s, a) = log π E (a|s). However, a solution proposed in Lemma 4 has two
crucial issues: Issue 1. It requires the knowledge on the model dynamics, which is generally intractable. Issue 2. We need to figure out Q
E that satisfies π E (·|s) = ∇Ω∗(Q E (s, ·)), which comes from the
relationship between the optimal value function and optimal policy (Geist et al., 2019).
In the following subsection, we show how our solution in Lemma 1 is related to the solution from Geist et al. (2019) in Lemma 4.

heading: F.2 RELATION BETWEEN THE SOLUTION OF GEIST ET AL. (2019) AND OUR TRACTABLE SOLUTION
text: Let us consider the expert policy π E and functions Q E and r E satisfying the conditions in Lemma 4. From Lemma 2, a regularized MDP with the shaped reward
r̃ E (s, a) := r E (s, a) + γEs′∼P (·|s,a)Φ(s′)− Φ(s)
for Φ ∈ RS has its optimal policy as π E
. Since Φ can be arbitrarily chosen, let us assume Φ(s) = Ω∗(Q
E (s, ·)). Then, we have
r̃ E (s, a)
= r E (s, a) + γEs′∼P (·|s,a)Φ(s′)− Φ(s) = { Q E (s, a)− γEs′∼P (·|s,a)[Ω∗(QE (s′, ·))] }
+ γEs′∼P (·|s,a)Ω∗(QE (s′, ·))− Ω∗(QE (s, ·)) = Q
E (s, a)− Ω∗(Q E (s, ·)). (22)
Note that the reward in Eq.(22) does not require the knowledge on the model dynamics, which addresses Issue 1 in Appendix F.1. Also, by using V
E (s) = Ω∗(Q E (s, ·)) in the proof of Lemma 4,
the reward in Eq.(22) can be written as
r̃ E (s, a) = Q E (s, a)− V E (s),
which means r̃ E (s, a) is an advantage function for the optimal policy π E in the Ω-regularized MDP.
However, we still have Issue 2 in Appendix F.1 since Ω∗ in Eq.(22) is generally intractable (Geist et al., 2019), which prevents us from finding the appropriate Q
E (s, a). Interestingly, we show that for
all s ∈ S and Q E (s, ·) = ∇Ω(π E (·|s)),
∇Ω∗(Q E (s, ·)) = arg max π(·|s)∈∆A 〈π(·|s), Q E (s, ·)〉A − Ω(π(·|s))
= arg max π(·|s)∈∆A 〈π(·|s),∇Ω(π E (·|s))〉A − Ω(π(·|s))
= arg min π(·|s)∈∆A Ω(π(·|s))− 〈∇Ω(π E (·|s)), π(·|s)〉A
= arg min π(·|s)∈∆A Ω(π(·|s))− Ω(π E (·|s))− 〈∇Ω(π E (·|s)), π(·|s)− π E (·|s)〉A
= arg min π(·|s)∈∆A
DAΩ (π(·|s)||πE (·|s)) = πE (·|s),
where the last equality holds since the Bregman divergence DAΩ (π(·|s)||πE (·|s)) is greater than or equal to zero and its lowest value is achieved when π(·|s) = π
E (·|s). This means that when
Q E (s, ·) = ∇Ω(π E (·|s)) is used, the condition π E (·|s) = ∇Ω∗(Q E (s, ·)) in Lemma 4 is satisfied without knowing the tractable form of Ω∗ or ∇Ω∗. Thus, Issue 2 in Appendix F.1 is addressed; we instead require the knowledge on the gradient ∇Ω of the policy regularizer Ω, which is practically more tractable. Finally, by substituting Q
E (s, ·) = Ω′(s, ·;π E ) for Ω′(s, ·;π) := ∇Ω(π(·|s)), s ∈ S,
to Eq.(22), we have
r̃ E (s, a) = Ω′(s, a;π E )− Ω∗(Ω′(s, ·;π E ))
= Ω′(s, a;π E )− {〈π E (·|s),Ω′(s, ·;π E )〉A − Ω(πE (·|s))} = Ω′(s, a;π
E )− Ea′∼π E (·|s)[Ω ′(s, a′;π E )] + Ω(π E (·|s)) = t(s, a;π
E ),
where t(s, a;π E ) is our proposed solution in Lemma 1.
G PROOF OF LEMMA 3
RL objective in Regularized MDPs w.r.t. normalized visitation distributions. For a reward function r ∈ RS×A and a strongly convex function Ω : ∆A → R, the RL objetive JΩ(r, π) in Eq.(1) is equivalent to
arg max π J̄Ω̄(r, dπ) := 〈r, dπ〉S×A − Ω̄(dπ), (23)
where for a set D of normalized visitation distributions (Syed et al., 2008)
D := { d ∈ RS×A :
∑ a′ d(s′, a′) = (1− γ)P0(s′) + γ ∑ s,a P (s′|s, a)d(s, a),∀s′ ∈ S
} ,
we define Ω̄(d) := E(s,a)∼d[Ω(π̄d(·|s))] and π̄d(·|s) := d(s,·)∑ a′ d(s,a ′) ∈ ∆ A S for d ∈ D and use π̄dπ (·|s) = π(·|s) for all s ∈ S. For Ω̄ : D → R, its convex conjugate Ω̄∗ is
Ω̄∗(r) : = max d∈D J̄Ω̄(r, d)
= max d∈D 〈r, d〉S×A − Ω̄(d)
(i) = max
π∈∆AS 〈r, dπ〉S×A − Ω̄(dπ)
= max π∈∆AS ∑ s,a dπ(s, a) [r(s, a)− Ω(π(a|s))]
= (1− γ) · max π∈∆AS JΩ(r, π), (24)
where (i) follows from using the one-to-one correspondence between policies and visitation distributions (Syed et al., 2008; Ho & Ermon, 2016). Note that Eq.(24) is equal to the optimal discounted average return in regularized MDPs.
IRL objective in regularized MDPs w.r.t. normalized visitation distributions. By using the RL objective in Eq.(23), we can rewrite the IRL objective in Eq.(6) w.r.t. the normalized visitation distributions as the maximization of the following objective over r ∈ RS×A:
(1− γ) · { JΩ(r, πE )− max
π∈∆AS JΩ(r, π) } = J̄Ω̄(r, dπE )−maxd∈D J̄Ω̄(r, d)
= min d∈D
{ J̄Ω̄(r, dπE )− J̄Ω̄(r, d) } = min
d∈D
{( 〈r, dπ
E 〉S×A − Ω̄(dπ E ) ) − ( 〈r, d〉S×A − Ω̄(d) )} = min
d∈D
{ Ω̄(d)− Ω̄(dπ
E )− 〈r, d− dπ E 〉S×A
} . (25)
Note that if∇Ω̄(d) is well-defined and r = ∇Ω̄(dπ E ) for any strictly convex Ω̄, Eq.(25) is equal to
min d∈D
{ Ω̄(d)− Ω̄(dπ
E )− 〈∇Ω̄(dπ E ), d− dπ E 〉S×A
} = min
d∈D DS×A Ω̄ (d||dπ E ),
where the equality comes from the definition of Bregman divergence.
Proof of t(s, a;πd) = ∇[Ω̄(d)](s, a). For simpler notation, we use matrix-vector notation for the proof when discrete state and action spaces S = {1, ..., |S|} and A = {1, ..., |A|} are considered. For a normalized visitation distribution d ∈ D, let us define
dsa := d(s, a), s ∈ S, a ∈ A, ds := [ds1, ..., d s |A|] T ∈ RA, s ∈ S,
D := [d1, ..., d|S|]T =  d11 · · · d1|A| ... . . . ...
d |S| 1 · · · d |S| |A|  ∈ RS×A, π̄(x) := x
1 TAx = 1∑ a∈A xa
[ x1, ..., x|A| ]T ∈ RA,x := [x1, ..., x|A|]T ∈ RA, where 1A = [1, ..., 1]T ∈ RA is an |A|-dimensional all-one vector. By using these notations, the original Ω̄ can be rewritten as
Ω̄(D) = ∑ s,a dsaΩ(π̄(d s)) = ∑ s∈S 1 TAd sΩ(π̄(ds)).
The gradient of Ω̄ w.r.t. D (using denominator-layout notation) is ∇DΩ̄(D) = [ ∂Ω̄(D)
∂d1 , ...,
∂Ω̄(D)
∂d|S|
]T ∈ RS×A,
where each element of∇DΩ̄(D) satisfies
∂Ω̄(D)
∂ds =
[ ∂Ω̄(D)
∂ds1 , ...,
∂Ω̄(D)
∂ds|A|
]T
= ∂
∂ds {∑ s∈S 1 TAd sΩ(π̄(ds)) }
= Ω(π̄(ds))1A + 1 T Ad
s ∂Ω(π̄(d s))
∂ds
= Ω(π̄(ds))1A + 1 T Ad
s ∂π̄(d s) ∂ds ∂Ω(π̄(ds)) ∂π̄(ds) . (26)
for
∂π̄(ds)
∂ds =
[ ∂π̄1(d s)
∂ds , ...,
∂π̄|A|(d s)
∂ds
] ,
∂π̄a(d s)
∂ds =
∂
∂ds
[ dsa
1 TAd s
] = ∂dsa ∂ds (1 TAd s)−1 + dsa ∂(1 TAd s)−1 ∂ds .
Note that each element of ∂π̄a(d s)
∂ds satisfies
∂π̄a(d s)
∂dsa′ = ∂dsa ∂dsa′ (1 TAd s)−1 + dsa ∂(1 TAd s)−1 ∂dsa′
= I{a = a′}(1 TAds)−1 − dsa(1 TAds)−2 = I{a = a′}(1 TAds)−1 − π̄a(ds)(1 TAds)−1 = (1 TAd s)−1 [I{a = a′} − π̄a(ds)] ,
and thus,
∂π̄(ds)
∂ds = (1 TAd
s)−1 { IA×A − 1A[π̄(ds)]T } . (27)
By substituting Eq.(27) into Eq.(26), we have
∂Ω̄(D)
∂ds = Ω(π̄(ds))1A + 1 T Ad
s ∂π̄(d s) ∂ds ∂Ω(π̄(ds)) ∂π̄(ds)
= Ω(π̄(ds))1A + 1 T Ad s [ (1 TAd s)−1 { IA×A − 1A[π̄(ds)]T }] ∂Ω(π̄(ds)) ∂π̄(ds)
= Ω(π̄(ds))1A + { IA×A − 1A[π̄(ds)]T } ∂Ω(π̄(ds)) ∂π̄(ds)
= Ω(π̄(ds))1A + ∂Ω(π̄(ds)) ∂π̄(ds) − [π̄(ds)]T ∂Ω(π̄(d s)) ∂π̄(ds) 1A
= ∂Ω(π̄(ds)) ∂π̄(ds) − [π̄(ds)]T ∂Ω(π̄(d s)) ∂π̄(ds) 1A + Ω(π̄(d s))1A. (28)
If we use the function notation, Eq.(28) can be written as
∇[Ω̄(d)](s, a) = ∇Ω(π̄d(·|s))(a)− Ea′∼π̄d(·|s) [∇Ω(π̄d(·|s))(a ′)] + Ω(π̄d(·|s))
= t(s, a; π̄d)
for t of Eq.(7) in Lemma 1.

heading: H DERIVATION OF BREGMAN-DIVERGENCE-BASED MEASURE IN CONTINUOUS CONTROLS
text: In Eq.(17), the Bregman divergence in the control task is defined as DAω (P1||P2) := ∫ X {ω(p1(x))− ω(p2(x))− ω′(p2(x))(p1(x)− p2(x))} dx. (29)
Note that we consider Ω(p) = ∫ X ω(p(x))dx = ∫ X [−fφ(p(x))] dx for fφ(x) = xφ(x), which makes Eq.(29) equal to∫ X { −p1(x)φ(p1(x)) + p2(x)φ(p2(x)) + f ′φ(p2(x))(p1(x)− p2(x)) } dx
= ∫ X p1(x) { f ′φ(p2(x))− φ(p1(x)) } dx− ∫ X p2(x) { f ′φ(p2(x))− φ(p2(x)) } dx
= Ex∼p1 [ f ′φ(p2(x))− φ(p1(x)) ] − Ex∼p2 [ f ′φ(p2(x))− φ(p2(x)) ] .
Thus, by considering a learning agent’s policy πs = π(·|s), expert policy πs E = π E (·|s), and the objective in Eq.(8) characterized by the Bregman divergence, we can think of the following measure between expert and agent policies:
Es∼dπ [ DAΩ (π s||πs E ) ]
= Es∼dπ [ Ea∼πs [ f ′φ(π s E (a))− φ(πs(a)) ] − Ea∼πs
E
[ f ′φ(π s E (a))− φ(πs E (a)) ]] . (30)

heading: I TSALLIS ENTROPY AND ASSOCIATED BREGMAN DIVERGENCE AMONG MULTI-VARIATE GAUSSIAN DISTRIBUTIONS
text: Based on the derivation in Nielsen & Nock (2011), we derive the Tsallis entropy and associated Bremgan divergence as follows. We first consider the distributions in the exponential family
exp (〈θ, t(x)〉 − F (θ) + k(x)) . (31)
Note that for
θ = [ Σ−1µ − 12Σ −1 ] = [ θ1 θ2 ] ,
t(x) = [ x xxT ] ,
F (θ) = −1 4 θT1 θ −1 2 θ1 + 1 2 log | − πθ−12 | = 1 2 µTΣ−1µ+ 1 2 log(2π)d|Σ|,
k(x) = 0,
we can recover the multi-variate Gaussian distribution (Nielsen & Nock, 2011):
exp(〈θ, t(x)〉 − F (θ) + k(x)) (32)
= exp ( µTΣ−1x− 1
2 tr(Σ−1xxT )− 1 2 µTΣ−1µ− 1 2 log(2π)d|Σ|
) (33)
= 1
(2π)d/2|Σ|1/2 exp
( µTΣ−1x− 1
2 xTΣ−1x− 1 2 µTΣ−1µ
) (34)
= 1
(2π)d/2|Σ|1/2 exp
( 1
2 (x− µ)TΣ−1(x− µ)
) . (35)
For two distributions with k(x) = 0,
π(x) = exp(〈θ, t(x)〉 − F (θ)), π̂(x) = exp(〈θ̂, t(x)〉 − F (θ̂))
that share t, F , and k, it can be shown that
I(π, π̂;α, β) = ∫ π(x)απ̂(x)βdx
= exp ( F (αθ + βθ̂)− αF (θ)− βF (θ̂) ) since ∫
π(x)απ̂(x)βdx
= ∫ exp ( α〈θ, t(x)〉 − αF (θ) + β〈θ̂, t(x)〉 − βF (θ̂) ) dx
= ∫ exp ( 〈αθ + βθ̂, t(x)〉 − F (αθ + βθ̂) ) exp ( F (αθ + βθ̂)− αF (θ)− βF (θ̂) ) dx = exp ( F (αθ + βθ̂)− αF (θ)− βF (θ̂) )∫ exp ( 〈αθ + βθ̂, t(x)〉 − F (αθ + βθ̂) ) dx
= exp ( F (αθ + βθ̂)− αF (θ)− βF (θ̂) ) .

heading: I.1 TSALLIS ENTROPY
text: For φ(x) = kq−1 (1− x q−1) and k = 1, the Tsallis entropy of π can be written as
Tq(π) := Ex∼πφ(x) = ∫ π(x) 1− π(x)q−1
q − 1 dx
= 1−
∫ π(x)qdx
q − 1
= 1 q − 1 (1− I(π, π; q, 0)) = 1− exp (F (qθ)− qF (θ)) q − 1 .
If π is a multivariate Gaussian distribution, we have
F (qθ) = q
2 µTΣ−1µ+
1 2 log(2π)d|Σ| − 1 2 log qd,
qF (θ) = q
2 µTΣ−1µ+
q 2 log(2π)d|Σ|,
F (qθ)− qF (θ) = 1− q 2 log(2π)d|Σ| − 1 2 log qd
= (1− q) { d
2 log 2π +
1 2 log |Σ| − d log q 2(1− q)
} .
For Σ = diag{σ21 , ..., σ2d}, we have
F (qθ)− qF (θ) = (1− q) { d
2 log 2π +
1 2 log |Σ| − d log q 2(1− q) } = (1− q) { d
2 log 2π +
1 2 log d∏ i=1 σ2i − d log q 2(1− q)
}
= (1− q) d∑ i=1 { log 2π 2 + log σi − log q 2(1− q) } .

heading: I.2 TRACTABLE FORM OF BASELINE
text: For φ(x) = kq−1 (1− x q−1), we have
f ′φ(x) = k
q − 1 (1− qxq−1)
= k
q − 1 (q − qxq−1 − (q − 1))
= qk
q − 1 (1− xq−1)− k
= qφ(x)− k. Therefore, the baseline can be rewritten as
Ex∼π[−f ′φ(x) + φ(x)] = Ex∼π[k − qφ(x) + φ(x)] = (1− q)Tq(π) + k.
For a multivariate Gaussian distribution π, the tractable form of Ex∼π[−f ′φ(x)+φ(x)] can be derived by using that of Tsallis entropy Tq(π) of π.

heading: I.3 BREGMAN DIVERGENCE WITH TSALLIS ENTROPY REGULARIZATION
text: In Eq.(30), we consider the following form of the Bregman divergence:∫ π(x){f ′φ(π̂(x))− φ(π(x))}dx− ∫ π̂(x){f ′φ(π̂(x))− φ(π̂(x))}dx.
For φ(x) = kq−1 (1− x q−1), f ′φ(x) = k q−1 (1− qx q−1) = qφ(x)− k, and k = 1, the above form is equal to ∫
π(x)
[ 1− qπ̂(x)q−1
q − 1
] dx− Tq(π)− (q − 1)Tq(π̂) + 1
= 1 q − 1 − q q − 1
∫ π(x)π̂(x)q−1dx− Tq(π)− (q − 1)Tq(π̂) + 1
= q q − 1 − q q − 1
∫ π(x)π̂(x)q−1dx− Tq(π)− (q − 1)Tq(π̂).
For multivariate Gaussians
π(x) = N (x;µ,Σ), µ = [ν1, ..., νd]T ,Σ = diag(σ21 , ..., σ2d), π̂(x) = N (x; µ̂, Σ̂), µ̂ = [ν̂1, ..., ν̂d]T , Σ̂ = diag(σ̂21 , ..., σ̂2d),
we have ∫ π(x)π̂(x)q−1dx = I(π, π̂; 1, q − 1) = exp ( F (θ′)− F (θ)− (q − 1)F (θ̂) ) ,
where
θ = [ Σ−1µ − 12Σ −1 ] ,
θ̂ =
[ Σ̂−1µ̂
− 12 Σ̂ −1
] ,
θ′ = θ + (q − 1)θ̂ = [
Σ−1µ+ (q − 1)Σ̂−1µ̂ − 12 (Σ −1 + (q − 1)Σ̂−1)
] = [ θ′1 θ′2 ] ,
θ′1 = [ ν1 σ21 + (q − 1) ν̂1 σ̂21 , ..., νd σ2d + (q − 1) ν̂d σ̂2d ]T ,
θ′2 = − 1
2 diag
{ 1
σ21 + (q − 1) 1 σ̂21 , ..., 1 σ2d + (q − 1) 1 σ̂2d
} ,
and
F (θ) = 1
2 µTΣ−1µ+
1 2 log(2π)d|Σ| = d∑ i=1 { ν2i 2σ2i + log 2π 2 + log σi } ,
F (θ̂) = 1
2 µ̂T Σ̂−1µ̂+
1 2 log(2π)d|Σ̂| = d∑ i=1 { ν̂2i 2σ̂2i + log 2π 2 + log σ̂i } ,
F (θ + (q − 1)θ̂) = −1 4 (θ′1) T (θ′2) −1θ′1 + 1 2 log | − π(θ′2)−1|
= d∑ i=1 12 ( νi σ2i + (q − 1) ν̂i σ̂2i )2 1 σ2i + (q − 1) 1 σ̂2i + log 2π 2 + log 1 1 σ2i + (q − 1) 1 σ̂2i  .

heading: J EXPERIMENT SETTING

heading: J.1 POLICY REGULARIZERS IN EXPERIMENTS

heading: J.2 DENSITY-BASED MODEL
text: By exploiting the knowledge on the reward in Corollary 1 −f ′φ(π(a|s))− Ea′∼π(·|s)[f ′φ(π(a′|s))− φ(π(a′|s))],
we consider the density-based model (DBM) which is defined by
rθ(s, a) ≈ −f ′φ(πθ1(a|s)) +Bθ2(s)
for θ = (θ1, θ2). Here, f ′φ is a function that can be known priorly, and πθ1 is a neural network which is defined separately from the policy neural network πψ. The DBM for discrete control problems are depicted in Figure 6 (Left). The model outputs rewards over all actions in parallel, where softmax is used for πθ1(·|s) and −f ′φ is elementwisely applied to those softmax outputs followed by elementwisely adding Bθ2(s). For continuous control (Figure 6, Right), we use the network architecture similar to that in discrete control, where a multivariate Gaussian distribution is used instead of a softmax layer.

heading: J.3 EXPERT IN BERMUDA WORLD ENVIRONMENT
text: We assume a stochastic expert defined by
π E
(a|s) = ∑3 t=1(d
(t))−1I{a = Proj(θ(t))}∑3 t=1(d (t))−1 ,
θ(t) = arctan2(ȳ(t) − y, x̄(t) − x), d(t) = ‖s̄(t) − s‖42 + , t = 1, 2, 3,
for s = (x, y), s̄(1) = (x̄(1), ȳ(1)) = (−5, 10), s̄(2) = (x̄(2), ȳ(2)) = (0, 10), s̄(3) = (x̄(3), ȳ(3)) = (5, 10), = 10−4 and an operator Proj(θ) : R→ A that maps θ to the closest angle inA. In Figure 7, we depicted the expert policy.

heading: J.4 MUJOCO EXPERIMENT SETTING
text: Instead of directly using MuJoCo environments with tanh-squashed policies proposed in Soft ActorCritic (SAC) (Haarnoja et al., 2018), we move tanh to a part of the environment—named hyperbolized environments in short—and assume Gaussian policies. Specifically, after an action a is sampled from the policies, we pass tanh(a) to the environment. We then consider multi-variate Gaussian policy
π(·|s) = N (µ(s),Σ(s))
with µ(s) = [µ1(s), ..., µd(s)]T , Σ(s) = diag{(σ1(s))2, ..., (σd(s))2}, where
−arctanh(0.99) ≤ µi(s) ≤ arctanh(0.99), log(0.01) ≤ log σi(s) ≤ log(2)
for all i = 1, ..., d. Instead of using clipping, we use tanh-activated outputs and scale them to fit in the above ranges, which empirically improves the performance. Also, instead of using potential-based reward shaping used in AIRL (Fu et al., 2018), we update the moving mean of intermediate reward values and update the value network with mean-subtracted rewards—so that the value network gets approximately mean-zero reward—to stabilize the RL part of RAIRL. Note that this is motivated by Lemma 2 from which we can guarantee that any constant shift of reward functions does not change optimality.

heading: J.5 HYPERPARAMETERS
text: Table 2, Table 3 and Table 4 list the parameters used in our Bandit, Bermuda World, and MuJoCo experiments, respectively.

Review summaries:
This work considers a regularized IRL setup, where instead of the entropy regularization used in maximum entropy IRL, an arbitrary convex regularizer $\Omega$ is used. The work presents a number of theoretical results for this general setting, and it is shown that when $\Omega$ is Tsallis entropy, the $RL \cdot IRL$ is equivalent to minimizing a Bregman divergence defined based on the Tsallis entropy and the expert state-action distribution. A practical algorithm is presented for IRL with the Tsallis entropy. A number of experiments are performed to obtain understanding of various components.

This paper proposes a new method for regularized inverse RL. The paper builds upon work by Geist et al. who studied regularized MDPs with convex policy regularizers. The Shannon entropy is a special case of such a policy regularizer. The paper extends the analysis of Geist et al. for regularized IRL and devises tractable solutions to regularized IRL that only depend on the analytic knowledge of the regularizer. The paper further proposes regularized adversarial IRL (RAIRL), an extension of AIRL by Fu et al., as an algorithm for IRL in regularized MDPs. The algorithm is validated on a number of domains.

This paper shows a formulation of regularized Markov Decision Processes (MDPs), which is slightly different from that of Geist et al. (2019). Then, the authors propose a novel inverse reinforcement learning under regularized MDPs. One of the contributions is that policy regularization considered here is more general than that of Yang et al. (2019).