Robust Imitation via Mirror Descent Inverse Reinforcement Learning
Authors: Dong-Sig Han
Hyunseo Kim
Hyundo Lee
JeHwan Ryu
Byoung-Tak Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: inverse reinforcement learning
regularized Markov decision processes
imitation learning
learning theory

Abstract: Recently, adversarial imitation learning has shown a scalable reward acquisition method for inverse reinforcement learning (IRL) problems. However, estimated reward signals often become uncertain and fail to train a reliable statistical model since the existing methods tend to solve hard optimization problems directly. Inspired by a first-order optimization method called mirror descent, this paper proposes to predict a sequence of reward functions, which are iterative solutions for a constrained convex problem. IRL solutions derived by mirror descent are tolerant to the uncertainty incurred by target density estimation since the amount of reward learning is regulated with respect to local geometric constraints. We prove that the proposed mirror descent update rule ensures robust minimization of a Bregman divergence in terms of a rigorous regret bound of $\mathcal{O}(1/T)$ for step sizes $\{\eta_t\}_{t=1}^{T}$. Our IRL method was applied on top of an adversarial framework, and it outperformed existing adversarial methods in an extensive suite of benchmarks.

Robust Imitation via Mirror Descent Inverse Reinforcement Learning
Dong-Sig Han, Hyunseo Kim, Hyundo Lee, Je-Hwan Ryu, Byoung-Tak Zhang Artificial Intelligence Institute, Seoul National University {dshan, hskim, hdlee, jhryu, btzhang) smu.
Abstract
Recently, adversarial imitation learning has shown scalable reward acquisition method for inverse reinforcement learning (IRL) problems. However, estimated reward signals often become uncertain and fail to train reliable statistical model since the existing methods tend solve hard optimization problems directly. Inspired by a first-order optimization method called mirror descent, this paper proposes to predict sequence of reward functions, which are iterative solutions for constrained convex problem. IRL. solutions derived by mirror descent are tolerant to the uncertainty incurred by target density estimation since the amount of reward learning is regulated with respect to local geometric constraints. We prove that the proposed mirror descent update rule ensures robust minimization Bregman divergence in terms of a rigorous regret bound of O11/T) for step sizes {n H Our IRL method was applied on top of an adversarial framework, and it outperformed existing adversarial methods in extensive suite of benchmarks.
1 Introduction
One crucial requirement of practical imitation learning methods is robustness, often described as learning expert behavior for a finite number of demonstrations, overcoming various realistic challenges [I]. In real-world problems such motor control tasks, the demonstration size can be insufficient to create a precise model of an expert (2). and even in some cases, demonstrations
be noisy suboptimal solve the problem 3 For such challenging scenarios, imitation learning algorithms inevitably struggle with unreliable statistical models; thus, the way of handling the uncertainty of estimated cost functions dramatically affects imitation performance. Therefore, thorough analysis of addressing these issues is required to construct a robust algorithm.
Inverse reinforcement learning (IRL) is algorithm for learning ground -ruth rewards from expert demonstrations where the expert acts optimally with respect to an unknown reward function 54. Traditional IRL studies solve the imitation problem based iterative algorithms 66 77. alternating between the reward estimation process and a reinforcement learning (RL) [ algorithm. In contrast, newer studies of adversarial imitation learning (AIL.) 9 TO rather suggest learning reward functions of certain form "directly," by using adversarial learning objectives [1]] and nonlinear discriminative neural networks [10]. Compared to classical approaches, the AIL methods have shown great success on control benchmarks in terms of scalability for challenging control tasks [12).
Technically, it is well known that AIL formulates a divergence minimization problem with its discriminative signals, which incorporates fine-tuned estimations of the target densities [13]. Through the lens of differential geometries, the limitation of AIL naturally comes from the implication that minimizing the divergence does not guarantee unbiased progression due to constraints of the underlying space [14]. In order to ensure further stability, argue that an IRL algorithm's 's progress needs to be regulated, yielding gradual updates with respect to local geometries of policy distributions.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

We claim that there are two issues leading to unconstrained policy updates: ! statistical divergence often cannot be accurately obtained for challenging problems, and 2 an immediate divergence between agent and expert densities does not guarantee unbiased learning directions. Our approach is connected to a collection of optimization processes called mirror descent (MD) [15]. For a sequence of parameters {w and a convex function S2, an MD update for cost function F; is derived as
v = V - 7h V.F (w:).
(1)
In the equation, the gradient VΩ(.) creates a transformation that links a parametric space to its dual space. Theoretically, MD is first-order method for solving constrained problems, which enjoys rigorous regret bounds for various geometries [16. 17 including probability spaces. Thus, applying MD to the reward estimation process can be efficient in terms of the number of learning phases.
In this paper, we derive an MD update rule in IRL upon a postulate of nonstationary estimations of the expert density, resulting in convergent reward acquisition even for challenging problems. Compared to MD algorithms in optimization studies, our methodology draws a sequence of functions on an alternative space induced by reward operator VA (Definition [ To this end, we propose an AlL algorithm called mirror descent adversarial inverse reinforcement learning (MD-AIRL Our empirical evidence showed that MD-AIRL outperforms the regularized adversarial IRL. (RAIRL) 18 methods. For example, MD-AIRL showed higher performance in 30 distinct cases among 32 different configurations in challenging MuloCo [9] benchmarks, and it also clearly showed higher tolerance to suboptimal data. All of these results are strongly aligned with our theoretical analyses.
Table 1: A technical overview. Traditional IRL methods lack scalability, and RAIRL does not guarantee convergence of its solution for realistic cases. MD-AIRL combines desirable properties.
Method
Reference
Scalability
Rewards
Bregman divergence
Iterative solutions
Convergence analyses
BC 1991
20
MM-IRL 2004
GAIL 20/66
/
RAIRL 2021
18
/
/
MD-AIRL (ours)
Our contributions. Our work is complementary to previous IRL studies; the theoretical and technical contributions are built upon novel perspective of considering iterative RL and IRL. algorithms as a combined optimization process with dual aspects. Comparing MD-AIRL and RAIRL. both are highly generalized algorithms in terms of a variety of choices of divergence functions. Tab. [ shows that MD-AIRL brings beneficial results in realistic situations of limited time and data, since our approach is more aligned with earlier theoretical IRL studies providing formalized reward learning schemes and convergence guarantees. In summary, we list our main contributions below:
Instead of a monolithic estimation process of a global solution in AIL, we derive a sequence of reward functions that provides iterative local objectives (Section
We formally prove that rewards derived by an MD update rule guarantee the robust performance of divergence minimization along with rigorous regret bound (Section
We propose a novel adversarial algorithm that is motivated by mirror descent, which is tolerant of unreliable discriminative signals of the AIL framework (Sections and 7..
2 Related Works
Mirror descent. We are interested in a family of statistical divergences called the Bregman divergence (21). The divergence generalizes constrained optimization problems such as least squares 23] and also has been applied in various subfields of machine learning (24 25). In differential geometries, the Bregman divergence is first-order approximation for metric tensor and satisfies metric-like properties 14 26). MD is also closely related to optimization methods regarding non- Euclidean geometries with discretization of steps such as natural gradients [27 28 In the primal space, training with the infinitesimal limit of MD steps corresponds to a Riemannian gradient flow B0). In the RL domain, MD has been recently studied for policy optimization B3]. In this paper, we focus on learning with suboptimal representations of policy, and our distinct goal is to draw robust reward learning scheme based on MD for the IRL problem.
Imitation learning. As a statistical model for the information geometry 34 energy-based policies (i.c., Boltzmann distributions) appeared in carly IRL studies, such as Bayesian IRL, natural gradient

---

IRL, and maximum likelihood IRL 35 37 for modeling expert distribution to parameterized func- tions. Notably, MaxEnt IRL 7 38 is one of the representative classical IRL algorithms based on an information-tie theorettcc perspective toward IRI. solutions. Also, discriminators of All. are trained by logistic regression; thus, the logit score of the discriminator defines an energy function that approximates the truth data density for the expert distribution [39]. Other statistical entropies have also been applied to AIL, such as the Tsallis entropy [40]. On the one hand, our approach is closely related to RAIRL 18), which defined its AIL objective using the Bregman divergence. On the other hand, this work further employs the Bregman divergence to derive iterative MD updates for reward functions, resulting in theoretically pleasing properties while retaining the scalability of AIL.
Learning theory. There have been considerable achievements in dealing with temporal costs { often referred to as online learning 41. The most ordinary approach is stochastic gradient particular, SGD is a desirable algorithm when the parameter Wy resides in the Euclidean space since it ensures unbiased minimization of the expected cost. Apparently, policies appear in geometries of probabilities; thus, an incurred gradient may not be the direction of the steepest descent due to geometric constraints 227 [4]. An online form of MD in Eq. I is analogous to SGD for non-Euclidean spaces, where each local metric is specified by Bregman divergence [30]. Our theoretical findings and proofs follow the results of online mirror
descent (SGD): w/++1 = W) - 7) V (wxx).
descent (OMD) that appeared in previous literature for general aspects [15] 16 42 28 17 30). Our analyses extend existing theoretical results to IRL; at the same time, they are also highly general to cover various online imitation learning problems which require making decisions sequentially.
3 Background
For sets X and Y. let y be a set of functions from X to Y and Ax (A be a set of (conditional) probabilities over X (conditioned on Y). We consider an MDP defined as a tuple (S, A, 7, with the state space S, the action space A. the Markovian transition kernel P A the reward function E R25A and the discount factor 7 E [0, 1). Let function Q: - be strongly convex. Using Ω the Bregman divergence is defined as
Da A*) = f((=') (***)= (VN(*"), X²
where म² and f* denote arbitrary policies for a given state s. For a representative divergence, one can consider the popular Kullback-Leibler (KI.) divergence. The KI. divergence is Bregman divergence
when f2 is specified as the negative Shannon entropy: 2(TT*) = Σ. T² (a(a) In (a).
Regularization of the policy distribution with respect to convex Ω brings distinct properties to the learning agent [43] 44). The objective of regularized RL is to find π E ll that maximizes the expected value of discounted cumulative returns along with causal convex regularizer Ω2 i.e..
Ja( r) = ET Σ == 7 [T(s,, a.) Q(((( (
(2)
where the subscript on the expectation indicates that each action sampled by (-Iss,) for the given MDP. In this setup, regularized RL. algorithm finds a unique solution in subset of the conditional probability space denoted as II= [II' A constrained by the parameterization of a policy.
The objective of IRL is to find a function ri that rationalizes the behavior of an expert policy Te. For an inner product (,, .)), consider S the Legchele-Fenchel transform (convex conjugate) of Ω:
E Ω2 (a²) T'AA
.max (T", 4²)a- 2(x*).
where qº and nh denote the shorthand notation of g(s.-) and (-1s). Differentiating both sides with respect to g", the gradient of conjugate VΩ2 maps g to a policy distribution. One fundamental property in regularized IRL [43 is that Tye is the maximizing argument S2" for Gn. where In is the regularized state-action value function (x, a) Eie
a) -)) ao=a). Note that the problem is ill- pol-pd and every fgg that makes its value function g satisfy oe = VΩ (4%) Vs S is valid solution. Addressing this issue. Jeon et al. [18 proposed reward operator VT - providing a unique IRL solution by (.(T)))
Definition 1 (Regularized reward operators). Define the regularized reward operator V. as (x, a) = ('(s a; л) (T', VI(TT)),+ + 2(TT). ). for (s, (s.- n)) = VD(T*) = [VpD(p)) p=)())))))))"
The reward function the == (m)) replaces its state-action value function, since the sum of composite Bregman divergences derived from Eq. (2) allows reward learning in greedy manner [18].
3

---

4 RL-IRL as a Proximal Method
Policy Space
Reward Space
I1
Associated reward functions. We consider the RL-IRL processes as a sequential algorithm with local constraints and define sequences { and { that denote policies and associated reward functions, respectively. The associated reward func- tions are in a space V. (II), which is an alternative space of the dual space, defined by the regular- ized reward operator En Formally, we provide LemmaA] which shows bijective relation between the operators VI2" and V, in the set II. The proof in Appendix [
Wi
PP
++++1
VI"
Figure I: A schematic illustration. MD is lo- cally constrained by divergence (gray area), i.e., De( [[Te). An MD update is performed for the reward function th in an associated reward space of defined by Vn. and T++1 is achieved in the desired space of II by applying V for the function 1 The gray dashed lines provide another interpretation of MD with #441 and the projection operator Pa.
Lemma (Natural isomorphism). Ler E (II) for V/2 (X)
vb a)) (s) Vs E S A. TE X J. Then, (vi) is unique and for every π (b
II.
Fig. illustrates that there a unique 4 for He in every time step. Note that V/2 (m.) is different from (I((T(): it is shifted by vector 1c with constant = (2(m;²) (xī (3)(+())+ Since the underlying space is a probability simplex. the operator V reconstructs the original point for both V, and VΩ, as the distributivity
43 Ω (y + 1c) (2" (v) + holds (so V (y 1c) V (v)).
An alternative interpretation is of considering a projection (gray dashed line in Fig. D. Suppose that a policy w, is updated to Æ E RSSA The Bregman projection operator Pa is applied that locates the subsequent update T4+1 to the "feasible" region, i.e., Pa (# == argmin mefl Da( [+*]|
Consequently, one can consider an updated reward function t/++1 associated with an alternative parameterization of II. For instance, the parameters of t/ can construct softmax policy for discrete space, or a Gaussian policy for a continuous space. Using the reward function 1/41,, an arbitrary regularized RL process maximizing Eq.
projected target of MD
(2) at the f-th step [88
Ja( 11+1) -x E D' (-(((((1s)))
( (e+1(-18)))
becomes finding the next iteration a++1 = V12"(1144 by maximizing the expected cumulative return. The equation shows that regularized RL algorithm with the regularizer Ω forms cumulative sum of Bregman divergences; thus, the policy T++1 is uniquely achieved by the property of divergence.
Online imitation learning. Our setup starts from the apparent yet vital premise that an imitation learning algorithm does not retain the global target Te during training. That is, it fundamentally uncertain to model global objectives (such as Jn(R, th))). which are not attainable for both RL and IRL. Instead, we hypothesize on the existence of a random process ( ) where each estimation Ne.t resides in a closed, convex neighborhood of Tg. generated by an arbitrary estimation algorithm. Substituting V/ to tite. = Mn (Fg,), the nonstationary objective Jn(TT, vas..) forms a temporal cost:
(T( Ex Σ ==
7" DD((T( [s:) Te.e( -s:.))
(5)
For the sake of better understanding, we considered an actual experiment depicted in Fig. 2 Suppose that the policies of the learning agent and the expert follow multivariate Gaussian distributions at
iyy T Te ofg
b
1.0
C
Shannon regularizer
Tsallis regularizer
Bregman div.
10
1-100 (4X)
20
-2
0
[-10
(4X)
Timo Stop
Time Step
Figure 2: (a) policy W. learns from MD updates for temporal costs Do( ) (b) The updates of Te vary by n) and the distance between and ne can be closer than the distance between मæ. and when is sufficiently large and the r) effectively low. (c) Two plots show DI( || (Te) associated with entropic regularizers for four different (10 trials), with the red baselines Dp (Tg
Tg). (

---

[io of. I) and (1s, st, Eg) with Eg Let (suboptimal) reference policy TH.t be independently fitted with maximum likelihood estimator with a relatively high learning rate, starting from u.. 1
The policy Te was trained by cost function D (Te,)))e,e)) using the MD update rule in Eq. ( In Fig. 2 we first observed that choosing a high step size constant 1j accelerated the training speed mainly in
the early phase. The results also showed that the performance of MD (Da (T. (gg) outperformed
that of referenced maximum likelihood estimation (D fg. ([Ts) by choosing an effectively low step size. This empirical evidence suggests that there are clear advantages in formalization of the training steps and scheduling the step sizes, especially for unreliable statistical model Tg.t.
MD update rules. As result of these findings, we formulate subsequent MD steps with a regularized reward function. Let wt be a parameter on a set W and Fe : W + R be convex cost function from a class of functions F at the t-th step. Replacing the L2 proximity term of proximal gradient descent with Bregman divergence, the proximal form of the MD update for Eq.
written as 45
minimize (VF((w)), w w 0z Dol w wt), wEW
(6)
where az == 1/n, denotes an inverse of the current step size b)t [46]. Plugging each divergence of the cumulative cost Fi to Eq. (6) the MD-IRL update for the subsequent reward function d/++1 Ma (T)+1) is derived by solving a problem
minimize DD m FE x² mi)a + 0a Da( x mi) el
v[(*)) ()(
minimize Da( 1 Dee (T=)|===])) + De nh Dn ( (1- Da( mf)
I-III"
minimize TCIF"
Es S,
(7)
estimated expert
Jearning agent
where the gradient of Dq is taken with respect to its first argument x;-- Note that solving the optimization Eq. 7 requires interaction between n. and the dynamics of the given environment in order to minimize F;; thus, the corresponding RL process plays an essential role in sequential learning by the induced the value measures. At glance, the objective is analogous to finding an interpolation at each iteration where the point is controlled 1/. 3 shows that the uncertainty of m (blue region) gets minimal regardless of persisting uncertainty of Tg. (red region).
Glound-uut cost
Graund-uuh nuth cost
TE
7
1
1r
2
with f),
MD
Figure 3: Illustrations of MD at the (a) 1-th iteration and (b) (t 1)-th iteration where 7) > 1-+-- ( 100 is an arbitrary estimation process attained from neighborhood of mg with respect to a norm. The MD update is taken inside the interval of and āe,t using Eq. (7p.
5 Convergence Analyses
In this section, we present our theoretical results. The main goals of the following arguments are to address ① the convergence of MD updates for various cases and @ the necessity of scheduling the amount of learning. Suppose that state instances of e T2 cover the entire S by executing the policy t, in an infinite horizon. From this assumption, we define temporal cost function at the time step L:
f(m,, m)=== - 7 Da(iTe( -Io(")]
(8)
that involves Ty, and additionally a trajectory Te as inputs. We refer to the global objective as finding a unique fixed point n. E II that minimizes a total cost F(xr) == E[f(TT Tt),, where the expectation is taken over trajectories of entire steps, i.e., lim, - I. r)). Taking the (stepwise) gradient for (S((EE.((-ss)) = 0 when t - oc; (e (+1s)) Introducing the optimal policy n. allows not only
each x (-Is), an optimal policy ग. is found by ([ hence, ([(m. = lim, the specific situation when ① ग. = II and the estimation algorithm of Næ., is actually convergent with + oo, but also more general situations where @ Fe & II or the estimated expert policy N.,( does not converge; the algorithm finds convergence fixed point by scheduling updates.
5

---

We state two conditions of {m h to guarantee convergence justified in Theorems and
Convergent sequence & divergent series:
lim 7i = and Σt= 1/1 - 20.
(9)
Divergent series & convergent series of squared terms:
Et=1 1/: DC and Et=1 1/ 00.
(10)
Let us assume Lipschitz continuity of VΩ and boundedness of Do in Banach space. In some Q these two assumptions do not necessarily hold for extreme cases in A e.g., a distribution that [alss) 0 for some entries. Nevertheless, these outliers can be left out if the parametrization is constrained to satisfy the assumptions. For example, one can either ! prevent a policy from having non-zero entries of probabilities for discrete policy or @ prevent policy from having too low entropy for a continuous policy, by enforcing certain constraints on its parametric representations.
Theorem argues that the sequence {n J shall diverge for its series; therefore, Eq. ( is satisfied. Theorem (Stepsize considerationss. Let S be strongly convex, VΩ be Lipschitz continuous, and the associated Bregman divergence Da is bounded. Assume a general condition of the problem that inf [T TT)] >0. Then we gel lim Brr E. Dz(T= (a((=((-10))) (-16)) (-1x)) if and only if
Eq. (9) is satisfied.
(a) If o0 Th = 0, then TEN, n <T, and c>> 0 exist such that BruTl fI(TT, Tr)] 2 T-a-
(b) the step size is in the form of 1p- [+I' then [ Da( (-1:))]]]] ([s))]==
Next, we present Theorem 2 which addresses the convergence in a specific case when Ty resides in II. Additionally, the theorem addresses the bounds of the performance for fixed size update 1/1 = 7/1 Theorem 2 (Optimal cases). Let Ω be strongly convex, VΩ be Lipschitz continuous. and the associated Bregman divergences be bounded. Assume T1 & Te and inf Bff (TT,TT) ]] 0. Then, [ ] = 0 if and only Ei= de co. If 7n, then there exist C2 (0, 1) such that T-1 A1 S Ar S c T-1, .A, A, = supses E.... (n(T=](=)))
Lastly, Proposition provides the sufficient condition for the almost certain convergence of the algorithm by imposing the stronger condition of step size in Eq. 10 The proofs are in Appendix [ Proposition 1 (General cases). Assume that Te &f II, hence inf nn Bif(T, r.)) > If the step sizes satisfies Eq. 10 then oim E 3' >'Dn(TT (1 (-Is:)] (e ( converges to 0 almost surely.
Regrets. For sequence of state trajectories (Ti heN, let us define a regret at the t-th iteration as EE T2) inf (t r5)].
(11)
In the optimal case of infremt E[ Tx)] = O, the cost f inherits the property of Bregman divergence so that the infimum is achieved by 0 at Te. In this case, the regret is bounded to (1/ by the theorems. By Proposition the MD updates converge for the case of infzem [T Tx) > 0 when the step sizes abide by Eq. 10 Thus, the regret is bounded to (1/TT) even for the general case.
6 Algorithm: MD-IRL on an Adversarial Framework
In this section, we propose MD-AIRL, novel AIL algorithm which trains a parameterized reward function with adversarial learning and the MD update rule. Neural network parameters 0, d, and v newly presented representing agent policy, reward, and expert policy functions respectively.
Dual discriminators. In order bridge the gap between theory and practice, we propose a novel discriminative architecture, motivated by GAN studies regarding multiple discriminators [47] 48 Basically, the proposed discriminators separate two concepts All.: matching overall state densities and imitating specific behavior. Given a learning agent policy no, an estimation policy nv, and discriminative neural network for states dg S + R, the two discriminators are defined as
Ds(s, a; 9, E) = alog T. (als)/mm(als))
(als)} de())) and De(+) = a (d)(s))) YsES, a a A,
where (-)) denotes the sigmoid function. The discriminators are trained using binary logistic regression losses with respect to mini-batch adversarial samples:
maximize Ts+ = Exg [lo [s)] + E,, [log(1 - De(s))],
(12)
maximize Jn.. = Er log (s, a)] + E., [log(1 Dr(s,
(13)

---

Algorithm Mirror Descent Adversarial Inverse Reinforcement Learning.
1: Input: trajectories (Ti an agent Te. a reference policy nu. neural network 1)
Se:S + R. a regularized reward function the E (c.(L an,ar, and A. 2: for + 1 to I' do
OV
3: ar 4 a1 + (f-1)(er=as/--)) and then + 1/au-
4: Optimize de and Tr via binary logistic regression for Dę and D..
01
5: Optimize the with the objective in Eq. (14) using both Ti and Ts.
6: Train Ne via RL to maximize vig. (s,a) a) with regularizer A.((-).
Figure 4: A sequence example of (as, me).
7:
Output: na. 3/3
where de and To not trained for learning D.. Let Px As denote the state visitation density of π, which is defined as (s) == (1-7) En [i of T(s; =s}] where Tf-] is indicator function. The convergence of functions in an ideal case is found at TT and ==
(s) Pr, (s)}-
Learning with MD-based rewards. Based on the MD solution for a regularized reward function, we focus on developing an MD-based learning objective. Let the E (11) denote a parameterized regularized reward function, and Te denotes corresponding policy from Φ.. Note that the transforma- tion between & and Te. can be performed with shared & without the additional computational costs under specific parameterizations [18]. Using a step size T7s, the RL agent ne and the estimated expert policy TV, we define the objective of á as a direct interpretation of the update rule of Eq. (
minimize Ly Eg [n Do(mT( Iss |s)]| (-1s)) + (1- Da( (T (s) TT( (1))]] (14)
where the trajectory Ft denotes sample states using both agent and expert trajectories. As shown in Fig. 4 1/t is adjusted by linearly increasing 04, which originated from the analyses in Section
Another important consideration is the way of handling covariate shifts [49] since it is likely that state densities between the expert and the agent are misaligned. Thus, we define the IRL reward function as linear combinations of ve and the state density discriminative signal:
(s, = A d a) + dg(s),
(15)
with a coefficient E RR. Utilizing an arbitrary regularized RL algorithm with a regularizer AS((-) the reward learning regarding agent policy no is decomposed into the following:
Exo ui AN(TT( [s)]] = AE [Vo a) ((TT( [s)]] DKL (proll Pme)
= -AE, ( [s][[TT( [| (-[s)] - -KLL (Prol Pme)
Minimizing the first term of Ez, [Pr[[DDT represents learning with the MD formulation. Mini- mizing the second term DKL ilpre) plays an auxiliary role in facilitating the supports of state visitation densities be correctly matched. With the hyperparameter A, we report that learning the second term is helpful when the state densities are heavily misaligned in certain benchmarks. Algorithm summarizes the entire procedure. We defer additional details to Appendices B and C
7 Experimental Results
The aim of our experiments was to identify whether MD-AIRL facilitates robustness for various Ω while retaining the scalability of AIL. The comparative method was RAIRL with density-based models (RAIRL-DBM) which contained comparable expressiveness as MD-AIRL. For RL, we used
RAC < which is generalization of the SAC algorithm (50). We considered a class of regularizers
(p)) - Ez-ppppe [==x])((p(x)) with ① Shannon (f((z) = log(z)), @ Tsallis (<0(2; 4) = --1 -2 - 1),
q 2 by default), A exp (1((x) =e-e==e- cos (fx(x)) = cos({ x)), and 3 sin (f((r)) = 1- sin( x))
7.1 Large scale multiarmed bandits
To measure the performance of IRL, we first considered multiarmed bandit problems, where the cardi- nality of action spaces varies largely. Learning the optimal distribution of Te becomes challenging as the cardinality of the space [A] increases, because the frequency of each sample becomes sparse due to the curse of dimensionality. The stateless expert distribution nhe was generated by the parameters of softmax distribution The (a) exp(sa)/2 exp(=,), where the logits z; were randomly initialized. We set the action size to [A] 10?, 103, 10t and restricted each sample size of 16.
7

---

Table 2: The training results of A Dn (ee) with five types of regularization (five runs with different seeds).
grog
Mathid
RAIRI
MDDAIRL
RAIRI.
MI>AIRL
RAIRI
MD.AIRI
Steae
2,50 4: 1.20
2.28 8: 1.30
140.1
57.5
1 61
Figure 5: The cost (T (iee) on the log-
Tsalis
0.21 a 0.33
0.11 1: 0.04
0,55
2 0.30
0.24 0:03
4.95 2.3
4.21 0.2.2
0.27
0.337
0.13
1 0.06
0,35
1 0.12
0:231:0:03
5,05
24
4.97
a 0.7
scale at |A] = 10 The shade represents 95% confidence interval.
ood
0.00
0.02
a 0.01
0.03
0/02
0011:0001
0.21 06
0.05 0.1
0.31 a 0.26
0.12 0.04
3.82
8 3.40
1.07::0.33
8.12 58
7.50 1.10
Tab. 2 shows that MD-AIRL achieved overall lower Bregman divergence on average when three different cardinalities and five regularizers were considered. Fig. shows that the Bregman divergence was large for MD-AIRI. at the early training phase, because we chose the initial step size ?)1 to be greater than 1 (a1 0.5). MD-AIRL exceeded the discriminative performance of RAIRL after certain steps, while the progression of RAIRL mostly stopped at local minima. MD-AIRL outperformed RAIRL in four cases by choosing effectively low step size at 1/r to be less than 1 (a, = 2). These results match the properties of MD algorithms and our convergence analyses. Therefore, argue that constrained update rule with appropriate step sizes is necessary for robust reward acquisition and imitation for situations when the total number of data samples is limited.
b
Shannon regularizer
Tsallis regularizer
Reaching multigoals
ND Seae
Goll entroy
ND Maree
MOVAL
ontroples
Step 5K
Step 300K
Step 5K
Step 300K
Figure 6: (a) Visualization of trajectories trained by MD-AIRL, and the ground-truth reward surface. (b) The entropies for the probabilities of achieving four goals. The x-axis indicates the q value from the Tsallis regularizers (the Shannon regularizer considered by q == (51)) (c) The top and middle of each column show induced reward surfaces. The bottom shows the agent policy.
7.2 A continuous multigoal environment
We then considered a multigoal environment. In this environment, agent is a two-dimensional point mass initialized at the origin, and the four goals are located in the four cardinal directions. The objective of imitation learning to go toward each direction evenly as possible where the expert model was trained by the SAC algorithm. To draw informative reward surfaces regarding stochastic actions, we considered the multivariate Gaussian distribution policies parameterized with full covariance matrices instead of conventional diagonal Gaussian policies (see Appendices and and C
Fig. (a) shows trajectories generated by the trained agent. (b) shows that MD-AIRI achieved higher entropy for reaching the multiple goals. Fig. [c)c) shows reward surfaces with regularizers, which were calculated by Va (s, a) + (())((ass)) for each point of a E A and (5. -1). During the training, the MD reward was similar to the estimated ground truth using adversarial training. However, the surface of MD-AIRL became flatter than the ground-truth estimation when mi was sufficiently close to the expert behavior, As result, we claim that drastic change in the target distribution, which is one of the typical characteristics of adversarial frameworks, is prevented. We argue that these characteristics mitigate overfitting caused by unreliable discriminative signals.
7.3 A continuous control benchmark: MuJoCo
Lastly, we validated MD-AIRL on the MuJoCo continuous control benchmark suite. We assumed full covariance Gaussian policies for both learner's 's policy π and expert policy Tg. We used the hyperbolized environment assumption [18] where the action constraint is incorporated into the dynamics as a part of the environment using hyperbolic tangent activation.
Sample efficiency. For each task, we considered two different numbers of episodes collected by an expert policy. In Fig. [7] the performance of MD-AIRI, RAIRL, and behavior cloning (bc) algorithms (20) is shown with the expert and random agent performance. MD-AIRL was able
8

---

Hopper-v3
2aiker2d-v3
HallCheeiah--0
Ant-33
2003
400c
2400
5000
500e
2000
2000
200
1000
1000
10:0
Thp
Time stap
Tiro 1kp
Time sR3
ospert
MD-AIRL (Shanron)
MD MD-AIRL (Teollt)
RAIRL (Shammon)
-.. RAIRL (Tsolls)
random
Figure 7: Average scores for 5 runs with two different regularizers (Shannon and Tsallis regularizer). The agent and IRL reward functions were trained with 4 episodes of expert demonstrations.
Hogper
Wakkerd
He HelCheethh
Art
Hopper
HallCheanh
04
Figure 8: Scores the last 10a steps in total of 32 different settings. The a-axis indicates the value of the Tsallis regularizers. The scores are rescaled by considering the expert performance as I, and the error bars represent standard deviations. Left: 4 demonstrations. Right: 100 demonstrations.
to achieve consistent performance throughout the tasks and demonstration size. On the training curves, MD-AIRL showed high tolerance to the scarcity of data compared to RAIRL for expert demonstrations. The plots in Fig. 8 indicate that MD-AIRL showed higher average scores with lower variance compared to RAIRL, across 30 distinct cases among 32 configurations we have tested. MD-AIRL. inherits the scalability of All,, and it is highly stable with respect to limited sample sizes.
Table 3: Scores on noisy demonstrations. The values of E represents scales of the Gaussian noises.
Method
0.01
5 0.5
Metbod
0.01
0.5
Hopper
RAIRL (Sharzon)
3836.03 391.00
3573.74 508
RAIRL
4354.15 03.83
4216.90 661.17
MD-AIRL (Sharron)
3669 25 177 78
3653.31 01 + 267 87
MDDAIRL (Shasron)
4373 =: 68.12
4337 18 #: 106 40
RAIRL (Talla)
3871. 322.32
3576.17 515. 75
RAIRL
4364,13 68.09
4216.67 J. 248.08
MD-AIRI (Tolllo
3780.14 63.09
205.68
MID-AIRI.CTatte)
4388 87 73.19
4247
266.
Walker22d
RAIRL (Sharron)
2856.56 + 939.9
2451 00 1392.6
RAIRI (Summen)
4493.74 + 383.04
3717. 77 5055.8
MD AD-LIRL (Starran)
3386 38
953.9... 59
3252 65
1395.7
MD-AIRL (Shaszon)
4658.29 4: 201.37
4284 38 329.79 799.9
RAIRL (Dalla)
2731.84 841.84411 1058.7
2435 10 1585.2
RAIRL (Taalio)
4359.62 :: 1688.66
3680. 22 d: 508.54
MD-AIRL (Tuilis)
3624 00
09999920 63
3093.54 1: 963.96
MD-AIRL (Tialia)
4705.25 1: 130.53
4127 37 1: 457.25
Noisy demonstrations. Tab. shoowss the results of imitation learning experiments for 100 expert demonstrations with two levels of Gaussian additive noises, resulting in suboptimal demonstrations. MD-AIRL is highly tolerant noisy data, consistently achieving higher performance. The experiment is closely related to the general case in the theory: the results suggest that the characteristics of MD- AIRL are in alignment with our analyses of the MD reward learning scheme.
Shannon regularizer
Tsallis regularizer
We present a detailed analysis of the noisy demonstra- tion experiments (Fig. 9.. Let the Bregman divergence between agent and ground-truth expert policies be the error, and measured these errors by increasing the given noise level for the expert trajectories. Fig. Issows general tendency that MD-AIRL has lower errors than RAIRL. With Tab. and Fig. 9 we were able to find the evident correlation between average Bregman divergence and performance since imitation learning convergence when the divergence is zero. Thus, this is another piece of empirical evidence that verifies our theoretical claims.
Hald
351
Hoppor
Wirowld
Figure 9: Divergences after imitation
learning. Top: e 0.011 Bottom: e =055.
8 Conclusions and Discussion
In this paper, we presented MD-AIRL, practical AIL algorithm designed to solve the imitation learning problem in the real world. We proved that the proposed method has clear advantages over previous AlL. methods in terms of robustness. We verified MD-AIRI. a variety of situations, including high- -igh-dimensional spaces, limited samples, and imperfect demonstrations. The empirical

---

evidence showed that MD-AIRL outperforms previous methods on various benchmarks. We conclude that the rich foundation of optimization theories shows promising direction for AIL studies.
Considering RI. and IRI. with geometric perspectives is vital for solving real-world problems. Although our work covers various imitation learning problems with the Bregman divergence, this does not include some other problems when the proximity term is of other statistical divergence families, such as the f-divergence [52]. If the relationship between these classes of divergences is studied in more detail, it is expected to proceed with applications to various subfields of machine learning. The assumptions on Q in our analyses are usually justified by enforcing a specific policy space, but some outliers might have substantial meaning for certain tasks. Therefore, extensive analyses on these parameterizations remain as future works. The "impurity" of the MD-AIRL reward function compared to V. (II) can be regarded as a limitation. To fully resolve this problem, all data must be treated as on-policy samples, which might require a sophisticated sampling mechanism.
Societal impacts. The evolution of imitation learning algorithms is expected to bring a structural shift in the labor market. The negative impact could be mitigated by diversification, unification, and redefinition of routine and manual jobs. The results of our work can be abused as tool for analyzing individual data. Therefore, we stress that certain acts should be carefully regulated, such as collecting a substantial amount of individuals' data and aggressively tracking personal identity.
Acknowledgments
The authors would like to thank the anonymous reviewers, Woosuk Choi, Jaein Kim, and Min Whoo Lee for their helpful discussion and comments. This work was partly supported by the 10-01100.t.ar.abbb// 2019-0-01371-BBB --1---yayyMinnn Baad/Mind/109% grant funded by the
ITTP (2022-0-099511-BB//255 2022-0-0055--PCCA225%, 02068-ATHub/ 10% 1-11-00033--SAA///I Korean government, and the CARAI (UD1 (UD190031RI//1%% grant funded by the DAPA and ADD.
2021-0-
References
[1] Ziyu Wang, Josh S Mercl, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess, Robust imi- tation of diverse behaviors. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, 2017.
[2] Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, JAąnos KramAar, Raia Hadsell, Nando de Freitas, and Nicolas Heess. Reinforcement and imitation learning for diverse visuomotor skills. In Proceedings of Robotics: Science and Systems, 2018
[3] Beomjoon Kim, Amir-massoud Farahmand, Joelle Pineau, and Doina Precup. Learning from limited demonstrations. In Advances in Neural Information Processing Systems, volume 26, 2013.
[4] Andrew Y Ng and Stuart J Russell, Algorithms for inverse reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning, pages 663-670. Morgan Kaufmann Publishers Inc.. 2000.
[5] Robert Dadashi, Léonard Hussenot, Matthiuu Geist, and Olivier Pictquin. Primal wasserstein imitation lcarning. In 9th International Conference on Learning Representations, 2021.
[6] Pieter Abheel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Procedd- ings of the 21 International Conference on Machine Learning, page 1. ACM, 2004.
[7] Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of 23rd AAAI Conference on Artificial Intelligence, volume pages 1433-1438, 2008.
[8] Richard Sutton and Andrew Barto. Reinforcement Learning: An Intooducion.. The MIT Press, second edition, 2018
[9] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pages 4565-4573, 2016.
[10] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv: 1710.1114 2017.
10

---

[11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672-2680, 2014.
[12] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, Jonathan Tompson. Discriminatoo--ccto--riii Addressing sample inefficiency and reward bias in adversarial imitation learning. In 7th International Conference on Learning Representations, 2019.
[13] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. divergence minimization perspective on imitation learning methods, In Conference on Robot Learning, pages 1259-1277. PMLR, 2020.
[14] Dan Butnariu and Elena Resmerita. Bregman distances, totally convex functions, and method for solving operator equations in banach spaces. Abstract and Applied Analysis. volume 2006, 2006.
[15] Arkadil Semenovich Nemirovsky and David Borisovich Yudin. Problem Complexity and Method Efficiency in Optimization. A Wiley- Elley-nnterscience publication. Wiley, 1983, ISBN 9780471 103455.
[16] Nati Srebro, Karthik Sridharan, and Ambuj Tewari. On the universality of online mirror descent. In Advances in Neural Information Processing Systems, pages 2645-2653, 2011.
[17] Yunwen Lei and Ding-Xuan Zhou. Convergence of online mirror descent. Applied and Computational Harmonic Analysis, 48(1 (331)33333 373, 2020. ISSN 1063-5203,
[18] Wonseok Jeon, Chen Yang Su. Paul Barde, Thang Doan. Derek Nowrouzezahrai, and Joelle Pineau. Regularized inverse reinforcement lcarning. In 9th International Conference Learning Representations, 2021.
[19] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[20] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neurul computation, 3(1):88-97, 1991.
[21] Lev M. Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems convex programming. USSR computational mathematics and mathematical physics, 703)2000-117 1967.
[22] Stephen P. Boyd and Lieven Vandenberghee Convex Optimization. Cambridge University Press, 2014. ISBN 978-0-521-88 978-0-521-83378-2
[23] Jean-Baptiste Hiriart- Urruty and Claude Lemaréchal. Fundamentals of convex analysis. Springer Science & Business Media, 2004.
Hiriart-Urruty
[24] Arindam Banerjee, Srujana Merugu, Inderjit Dhillon, and Joydeep Ghosh. Clustering with bregman divergences. Journal of Machine Learning Research, ((58):1705-1744 2005.
[25] Akash Srivastava, Kristjan H. Greenewald, and Farzaneh Mirzazadeh. Bregmn: scaled-bregman generative modeling networks. CoRR, abs/1 /t//1906003133 2019.
[26] Sreangsu Acharyya, Arindam Banerjee, and Daniel Boley. Bregman divergences and triangle inequality. In Proceedings the 13th International Conference Data Mining, pages 476-484, 2013.
[27] Shun-ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251-276. 1998.
[28] G. Raskutti and S. Mukherjec. The information geometry of mirror descent. JEEE Transactions on Information Theory, 61(3):155-11457, 2015.
[29] Manfredo P Do Carmo. Differential geometry of curves and surfaces: revised and updated second edition. Courier Dover Publications, 2016.
[30] Suriya Gunasekar, Blake E. Woodworth, and Nathan Srebro. Mirrorless mirror descent: A more natural discretization of riemannian gradient flow. CoRR, abs/2004.01025, 2020.
[31] Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. CoRR, abs/2102.00115, 2021.

---

[32] Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D. Lee, and Yuejie Chi. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. CoRR, abs/2105.1 1066, 2021.
[33] Manan Tomar, I. Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. arXiv preprint arXiv:2005.09814, 2020.
[34] Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
[35] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In Proceedings of International Joint Conference on Artificial Intelligence, pages 2586-2591, 2007.
[36] Gergely Neu and Csaba Szepesvári. Apprenticeship learning using inverse reinforcement learning and gradient methods. In Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence, pages 295-302, 2007.
[37] Monica Babes- Vroman, Vukosi Marivate, Kaushik Subramanian, and Michael Littman. Apprenticeship learning about multiple intentions. In Proceedings of the 28th International Conference on International Conference on Machine Learning. page 897904, 2011. ISBN 9781450306195.
[38] Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pages 1255-1262, 2010.
[39] Junbo Jake Zhao, Michaël Mathicu, and Yann LoCun. Energy- based generative adversarial networks. In 5th International Conference on Learning Representations, 2017.
[40] Kyungjae Lee, Sungjoon Choi, and Songhwai Oh. Maximum causal tsallis entropy imitation learning. In Advances in Neural Information Processing Systems, pages 4403-4413, 2018.
[41] Amos Fiat and Gerhard Woeginger. Online algorithms: The state of the art, volume 1442. Springer, 1998.
[42] Amir Beck and Mare Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167-175, 2003.
[43] Matthicu Geist, Bruno Scherrer, and Olivier Pictquin. theory of regularized Markov decision processes. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2160-2169. PMLR, 2019.
[44] Wenhao Yang, Xiang Li, and Zhihua Zhang. A regularized approach sparse optimal policy in reinforce- ment learning. In Advances in Neural Information Processing Systems, pages 5938-5948 2019.
[45] Amir Beck and Mare Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167-175, 2003. ISSN 0167-6377.
[46] David H Gutman and Javier F Peña. A unified framework for bregman proximal methods: subgradient, gradient, and accelerated gradient schemes. arXiv, pages arXiv-1812, 2018
[47] Tu Nguyen, Trung Le, Hung Vu, and Dinh Phung. Dual discriminator generative adversarial nets. In
I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 30, 2017.
[48] LI Chongxuan, Taufik Xu, Jun Zhu, and Bo Zhang. Triple generative adversarial nets. In Advances in Neural Information Processing Systems, pages 4088-4098. 2017.
[49] Jonathan Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. In Advances in Neural Information Processing Systems, volume 34, pages 965 975,9979240 2021.
[50] Tuomas Haarnoja. Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor: In International Conference on Machine Learning, pages 1856-1865, 2018.
[51] Constantino Tsallis. Possible generalization of boltzmann-iibbs statistics, Journal of statistical physics, 52(1-2):479-45) 1988.
[52] Tianwei Ni, Harshit S. Sikchi, Yufei Wang. Tejus Gupta, Lisa Lee, and Ben Eysenbach. F-IRL; inverse reinforcement learning via state marginal matching. In Conference on Robot Learning, volume 155, pages 529-551. PMLR, 2020.
12

---

[53] Frank Nielsen and Richard Nock. On Rényi and Tsallis entropies and divergences for exponential families. arXiv preprint arXiv: 125.399,,,,,99 2011.
[54] Mohsen Pourahmadi. Joint mean-covariance models with applications to longitudinal data: parameterisation. Biometrika, 86(3):677-660, 1999.
Unconstrained
[55] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/1812.05905, 2018.
[56] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section X
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
See
Section 3
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 5
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] Our empirical studies be reproduced by from the detailed information in Appendices and C
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section 7
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] In experiments, each algorithm was executed CPU (a single thread).
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] The MuloCo simulator used in our experiments is freely available to everyone. See the site (https:/muuoco.oorg
(c) Did you include any assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l wou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "jqnc-rarpx-", "reviewer": "Reviewer_4DdJ", "summary": "The paper derives MD-AIRL, a new algorithm for inverse reinforcement learning as an instance of online mirror descent and proves convergence and bounded regret.\nThe derived algorithm is closely related to existing methods in the field of adversarial imitation learning, by using a structured discriminator akin to AIRL to estimate the expert densities (but additionally training a state-only discriminatoris to more directy match the state marginals). Furthermore, similar to RAIRL, the algorithm is tested for different Bregman divergences and policy regularizers and uses a density based model for the discriminator. The main algorithmical difference compared to prior methods, is that the discriminator does not directly specify the reward function for the given iteration, but instead is used to estimate the expert policy (which changes in every iteration, motivating the online MD formulation). The update in Eq. 14, hence, corresponds to an additional step that computes the next reward based on the expert policy estimate.\nMD-AIRL is compared to RAIRL on a multi-armed bandit problem, a continous point-mass environment and Mujoco environments, and shows slightly improved performance across all settings.", "questions": "I only have two questions, but it would be important to answer them thoroughly in the rebuttal:\n\n1. How crucial is the state reward (Eq. 15) for the performance of MD-IRL, how does it relate to the theory? \n2. In  Eq. 14, do I understand it correctly, that the objective is minimized with respect to $\\pi_{\\phi}$, and the optimal policy is then mapped to a reward function using the regularized reward operator? How does this relate to the paths in Fig. 1? If we know $\\pi_{t+1}$ already (because it is the policy that we used to compute $\\psi$), why do we need to perform reinforcement learning (cf. Eq.4 and the equation after Eq. 15) if we know the (valid) distribution for the current reward? I understand that we need to mitigate covariance shift and take into account the auxiliary state-reward, but how does this relate to the theory?  ", "limitations": "The paper only discussed the limitation that some divergences are not covered by this formulation.\n\nThe paper would be much stronger, if it would thoroughly discuss the *mismatch between theory and algorithm* and clearly state that the *effects of estimation errors* have substantial impact in practice, but are not considered in the current work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n## Originality\n__Strength__\n- The mirror descent formulation seems novel and leads to a slightly different algorithm that can make use of the broad theory around mirror descent.\n\n__Weakness__\n- Algorithmically, the changes compared to prior work are not overwhelming.  \n\n## Quality\n__Strength__\n+ The derivations seem sound and the main claims are proven with sufficient rigor.\n\n__Weakness__\n- Proofs on convergence do not consider estimation errors (for estimating log-ratios and for policy optimization) \n- I think some aspect of the algorithm are not fully consistent with theory and not sufficiently well motivated:\n1. The second (state-only) discriminator is motivated based on [45, 46], where the GAN setting is considered. I don't see how these paper motivate the particular use of a state-discriminator. \n2. As far as I understood, the regularized reward function (Eq. 15) deviates from the theory. While I understand the motivation behind it (biasing the agent to vist expert states), it is not clear how this additional term affect the theoretical results on convergence and regret bound. I think the algorithm should also be evaluated without the state-reward\n\n## Clarity\n__Strength__\n- Overall, the paper is well-structured and well-written.\n\n__Weakness__\n- Some parts are a bit hard to understand:\n1. In Eq.14, it is not clear how $\\pi_{\\phi}$ is computed from $\\psi_{\\phi}$. I assume the optimization is actually performed w.r.t. a parameterized policy and then mapped to the reward function (or the policy is directly expressed in terms of $\\phi$), however this should be made more explicit. Also, it is not clear how the reinforcement learning relate to the derivations; the MD update in Eq. 7 does not consider time series data. \n2. I don't understand how the main change to prior work (Eq. 14) follows from the MD formulation. After the estimation of the expert densities, couldn't we directly perform the MD update by optimizing the policy with respect to Eq. 7 (using expectations over trajectories)?\n\n## Significance\n__Strength__\n- The work is a solid contribution to the field by providing a mirror descent formulation for inverse reinforcement learning, along with the corresponding algorithm, which preforms reasonably well.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "icz7sDf18xZ", "reviewer": "Reviewer_mFTz", "summary": "Inspired by mirror descent, the paper proposes MD-IRL, which considers the reward function as an iterative sequence in a proximal method. The paper proves that such a method ensures robust minimization of a Bregman divergence under the mirror descent framework. \n", "questions": "See weaknesses. ", "limitations": "Yes. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - It is novel to consider using MD in the reward update. Rigorous regret upper bound is also provided to show that the Bregman divergence is minimized to a local optimum. In the meanwhile, stepsize considerations are also provided, which is good. \n - Thorough empirical studies are provided (though limited, see weaknesses). \n\nWeaknesses:\n - The paper is closely related to RAIRL. I think the authors should make a thorough comparison with RAIRL. \n - In the empirical study, the paper only compares with RAIRL, which is slightly limited. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FJKnNBsE06o", "reviewer": "Reviewer_Ptuv", "summary": "This paper cast the imitation learning problem as an iterative RL-IRL process and proposes a new adversarial imitation learning (AIL) algorithm based on mirror descent. The authors prove the convergence of the learning policy to an optimal policy in $\\mathcal{O}(\\frac{1}{T})$. The experiments show that the proposed method, MD-AIRL, consistently outperforms RAIRL.", "questions": "\n(1) The current theoretic analyses show the convergence to the optimal policy, which seems to come from the nature of MD. Is it possible to establish some results on the learned reward function? Or is it possible to give some analysis (experimental or theoretic) on the learned reward function?\n\n(2) I noticed the BC results in the MuJoCo tasks are very low, while BC can be a very strong baseline on MuJoCo tasks with only 1 demonstration (please see Figure 4 in [1]). Can you clarify the implementation details of BC?\n\n\n**Minor:**\nIn Thm. 2 (Line 220), what is the difference between $A_t$ and $A_T$?\n\n\n[1] Ziniu, et al., \"Rethinking ValueDice - Does It Really Improve Performance?\", ICLR Blog Track, 2022.", "limitations": "The authors proposed an appealing approach to make the AIL process more robust. The theoretical result on the convergence rate is of great importance and the experiments show MD-AIRL is consistently better than RAIRL. MD-AIRL learns the reward function, which is also important for the IRL problem, while the authors dig little into the learned reward. Besides, comparisons with other recent IL and IRL algorithms will be helpful.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\nMotivated by some practical observations, e.g., the global solution in AIL is hard to obtain by a monolithic estimation process, the authors convert the original AIL formulation to a sequence of policies and associated reward functions, and derive an MD update rule that enjoys a regret bound of $\\mathcal{O}(\\frac{1}{T})$. This approach is appealing and seems sound, which also provides a linear convergence rate to the optimal policy. The experiments demonstrate MD-AIRL consistently outperforms RAIRL and is more robust to the noise and the number of trajectories. \n\nOverall, this paper is clearly written and easy to follow.\n\n**Weaknesses**:\n\nWhile the analysis part contributes a lot, the compared baseline only consists of RAIRL and BC. Some recent work, e.g., f-IRL (in terms of IRL) [1] and ValueDice (in terms of imitation learning) [2] can also achieve a more robust learning process than AIL and AIRL, which are missing in the literature or as baselines. Besides, MD-AIRL extends the AIRL framework, thus the learned reward function may be used in the downstream task as done in f-IRL. However, the functionality of the learned reward function has not been revealed. \n\n[1] Tianwei Ni, Harshit S. Sikchi, Yufei Wang, et al. \"f-IRL: Inverse Reinforcement Learning via State Marginal Matching.\" CoRL 2020.\n[2] Kostrikov, Ilya, Ofir Nachum, and Jonathan Tompson. \"Imitation learning via off-policy distribution matching.\" ICLR 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CidadnfoeNt", "reviewer": "Reviewer_Mged", "summary": "The main idea of this paper is to modify adversarial inverse RL to perform mirror descent policy updates. In practice, this means that the regular RL actor objective is modified by minimizing the divergence between policy iterates. The paper proves that this method converges under reasonable assumptions (e.g., Robbins Monro learning rate), and shows that the method does work roughly on par with one prior methods (RAIRL), for a wide range of Bregman divergences.", "questions": "1. To what extent are the results limited to inverse RL, versus any algorithm that performs policy updates? It seems like the idea of doing regularized policy improvement could (and has been?) studied in a wide range of contexts, beyond imitation learning. To what extent are prior results *already* applicable to this setting, and to what extent do the proposed results extend to other settings (e.g., are they stronger than the results in [1]?)?\n2. While it seems like the theoretical results are correct, I'm unsure if they motivate the use of mirror descent. What would similar convergence results looks like for (say) standard/Euclidean gradient descent?\n\n\n\n[1] https://proceedings.mlr.press/v97/geist19a.html", "limitations": "The limitations section is OK. The discussion of extending the analysis to f-divergences is a good point, but not really addressing the limitations of the proposed method. The discussion about \"unsafe behavior\" and \"automation of labor\" seem overly broad. I would recommend making more specific limitations about the work (e.g., arbitrary Bregman divergences might not always be useful, and the choice of divergence measure adds an additional hyperparameter for the user to tune).", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The proposed method is applicable to a wide range of Bregman divergences.\n* Experiments study robustness to varying noise in the dataset and choices of hyperparameters.\n\nWeaknesses\n* Poor writing made the paper very difficult to understand. For example, when introducing the theoretical results, it'd be good to explain what question these results are answering before stating the results themselves. Similarly, when introducing the proposed method, it'd be good to explain the problem before introducing the solution (e.g., the dual discriminator is introduced without motivation).\n* I'm unsure about some of the claims in the introduction and related works section (see below).\n* Limited baselines for the continuous control experiments. E.g., for Fig 7/8/9, I'd recommend comparing to a method like DAC [1], or a more recent+competitive method.\n\n**Summary**: Overall, it seems likely that mirror-descent is the right way to do imitation learning, and that it can outperform vanilla gradient descent in many settings. However, I'm not convinced that this paper effectively makes this point, partially because the writing is very hard to follow, and partially because the experiments do not convincingly show that other Bregman divergences consistently outperform L2 distance (which is effectively what gradient descent uses). So, I think the paper is on the right track, and will eventually make for a strong resubmission after significant revisions to the writing and experiments.\n\n\nAdditional, less important questions/comments:\n* I'd recommend running a spelling + grammar checker on the paper.\n* L23 -- L30 -- It seems that this paragraph is really about imitation learning, not inverse RL. Indeed, adversarial imitation learning methods do not actually recover the expert reward function, only the expert policy. So, I'm not sure that the claim that these methods \"estimate the ... reward function directly\" is true.\n* Table 1 -- The column about \"rewards\" seems misleading, for two reasons. First, it's unclear of learning the reward function is actually useful, if the user only cares about acquiring the optimal agent. Second, methods like GAIL do not actually learn the expert's reward function. The column about Bregman divergences seems misleading, too, because BC can be implemented with different Bregman divergences (e.g., the standard BC corresponds to a forward KL divergence). BC also enjoys iterative solutions, and also inherits convergence guarantees from standard supervised learning theory.\n* L73 \"Pythagorean theorem\" -- Is it true that all Bregman divergences obey the pythagoream theorem? I don't think it's true for the KL divergence. The citation [13] doesn't mention Pythagorean theorem.\n* Fig 2 is very hard to read. I'd recommend making all subplots a consistent size (e.g., remove the tiny subplots in B) and making the font sizes all larger.\n* Eq 14 -- I'd recommend adding a paragraph of explanation before this, explaining the intuition before stating the final objective.\n* Eq 15 -- Why is the $\\lambda$ parameter needed? From a probabilistic perspective, it seems like $\\lambda = 1$ should be optimal\n* L263 -- L265 -- What is the motivation for introducing these different Bregman divergences? The proposed method is general, in that it can use any Bregman divergence. But making the claim that this generality is useful requires evidence that different Bregman divergences are useful in different situations.\n* Sec 7.1 -- I didn't understand the motivation for this experiment. It seems like the optimal policy is given by the empirical action distribution, which is trivial to compute with gradient descent or iterative updates.\n* Fig 5 -- Does the RAIL baseline also decay the learning rate? If the proposed method uses a learning rate schedule, it'd be good to equip the baselines with this schedule, too.\n* Fig 6, \"goal entropy\" -- Why is this a reasonable metric? It seems like some measure of divergence from the expert policy would be more meaningful.\n\n[1] https://arxiv.org/abs/1809.02925", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept