Distributionally Robust Imitation Learning
Authors: Mohammad Ali Bashiri
Brian D Ziebart
Xinhua Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Imitation Learning
Inverse Reinforcement Learning
Robust Learning

Abstract: We consider the imitation learning problem of learning a policy in a Markov Decision Process (MDP) setting where the reward function is not given, but demonstrations from experts are available. Although the goal of imitation learning is to learn a policy that produces behaviors nearly as good as the experts’ for a desired task, assumptions of consistent optimality for demonstrated behaviors are often violated in practice. Finding a policy that is distributionally robust against noisy demonstrations based on an adversarial construction potentially solves this problem by avoiding optimistic generalizations of the demonstrated data. 
This paper studies Distributionally Robust Imitation Learning (DRoIL) and establishes a close connection between DRoIL and Maximum Entropy Inverse Reinforcement Learning. We show that DRoIL can be seen as a framework that maximizes a generalized concept of entropy. We develop a novel approach to transform the objective function into a convex optimization problem over a polynomial number of variables for a class of loss functions that are additive over state and action spaces. Our approach lets us optimize both stationary and non-stationary policies and, unlike prevalent previous methods, it does not require repeatedly solving an inner reinforcement learning problem. We experimentally show the significant benefits of DRoIL’s new optimization method on synthetic data and a highway driving environment.

Distributionally Robust Imitation Learning
Mohammad Ali Bashiri
Brian D. Ziebart Department of Computer Science University of Illinois at Chicago
Xinhua Zhang
Chicago, II. 60607
(mbashi4, zhangx, bziebart/@uic.ed
Abstract
We consider the imitation learning problem of learning policy in Markov Decision Process (MDP) setting where the reward function is not given, but demon- strations from experts are available. Although the goal of imitation learning is to learn policy that produces behaviors nearly as good as the experts' for de- sired task, assumptions of consistent optimality for demonstrated behaviors often violated in practice. Finding policy that is distributionally robust against noisy demonstrations based problem by avoiding optimistic generalizations of the demonstrated data. This paper studies Distributionally Robust Imitation Learning (DRoIL) and establishes a close connection between DROIL and Maximum Entropy Inverse Reinforcement Learning. We show that DROIL can be seen framework that maximizes a generalized concept of entropy. We develop novel approach to transform the objective function into convex optimization problem over a polynomial number of variables for a class of loss functions that are additive over state and action spaces. Our approach lets us optimize both stationary and non-stationary policies and, unlike prevalent previous methods, it does not require repeatedly solving an inner reinforcement learning problem. We experimentally show the significant benefits of DRoIL's new optimization method on synthetic data and a highway driving environment.
adversarial construction potentially solves this
Introduction
We consider the imitation learning setting of learning to perform task based only demonstrations that are provided by experts. There are two main approaches often considered for this learning problem: Behavioral Cloning [19] and Inverse Reinforcement Learning [18] [I] 24] [14]. In behavioral cloning, the learner attempts to learn a policy in a supervised learning manner, in which a direct mapping from states to actions is estimated from the demonstrated trajectories. Behavioral cloning, while simple, often generalize poorly when attempting to predict goal-directed sequential decisions due compounding
caused by covariate shift and only tends succeed when given large amounts of data [22] [23]. Alternatively, inverse reinforcement learning (IRL) rationalizes demon- strated trajectories by estimating reward function that makes the expert's policy optimal. The problem of determining the reward function is inherently ill-posed, since a single policy can be optimal for multiple reward functions.
To obtain a unique solution IRL, many methods have been proposed, with the maximum entropy principle [31 32 and margin maximization [20] being two widely employed methods among researchers. Maximum (causal) entropy IRI. (32], in particular, seeks an entropy-maximizing distribution sequences that matches expected feature counts [I] with those observed from demonstrations. For imitation learning in very high dimensional and continuous spaces, where function approximations such deep neural networks are often used, IRL methods have generally
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

been less efficient than behavioral cloning methods 13 since they require reinforcement learning as however, seem more likely be effective.
an inner loop. Recent adversarial IRL methods [T] 9 25
common assumption in imitation learning is that all expert behaviors have the same level of trustworthiness and are optimal/near-optimal [30.. However, it is common for noisy expert behaviors to violate this optimality assumption in practice. Thus, relying heavily on the optimality of the expert's 's behavior may degrade an imitation learner, making it prone to failure [30]. Existing methods have proposed to inject noise into the expert's 's policy demonstrations to obtain a more robust policy [13.
or to train a discriminator that distinguishes between expert trajectories and generated trajectories
that learns reward functions that are robust to changes in dynamics
An alternative approach to obtain robust policy, which we adopt in this work, is search for policy that is distributionally robust given the training data. For this purpose, the learner's policy is obtained by solving a game between a learner and an adversary [5], where the learner seeks minimize loss defined between them, and the adversary seeks to maximize this loss by choosing a distribution over policies subject
set of constraints that match statistics from the training data. This approach leverages two uncertainty sets as opposed to typical Distributionally Robust Optimization (DRO) where the uncertainty set is only defined over the adversary (demonsttatoo--ssiimaoor and the learner's policy is assumed to have a specific parametric form. Previous work (5) proposed to solve imitation learning under an adversarial formulation using the Double Oracle method [17] to solve the corresponding optimization. However, this method may take up to exponential time in the size of state-action space and decision horizon.
methods 4 6
In this paper, we connect Distributionally Robust Imitation Learning (DROIL) and Maximum Entropy Inverse Reinforcement Learning (MaxEnt IRL) by showing that MaxEnt IRL. is special case of DROIL when a certain loss function and policy description is used. We then show that DRoIL is a framework for maximizing a general entropy function that is defined based on a particular loss of interest. We then cast DROIL's is objective function into a convex optimization problem over a polynomial number of variables, which is simpler understand and implement and also significantly improves the training time. We extend the formulation to stationary policies, which enables us to experimentally show the benefits of learning a robust policy in highway driving simulation.
2 Preliminaries
We model sequential decision making problems using discrete Markov Decision Processes (MDPs).
A MDP M is specified by tuple (S,A, T, R, Po) where: S and A are state and action (control input) spaces (assumed to be finite, (S), | < oo); Po is the initial state distribution; I represents the transition probabilities, the state distribution upon taking action a in state 8, P(s la, s); reward function R : S x A + IR; and discount factor 7 E (0, 1]. We assume feature vectors & ; S x A + [0, 1] over state- action pairs that capture the most salient properties distinguishing preferred and nonpreferred trajectories where R can be written as (linear) function of these feature vectors given a reward vector w: R(s, a) w (08.00 a).
A policy π E II is the probability of taking action a in state 8, π T(a, s) P(als) and II represents the set of all possible stochastic policies. Demonstrations by an expert are given as a set of trajectories D = {71 horizon T. From an optimization perspective, Imitation Learning with a General Loss, finds a policy f that minimizes the difference between learner behavior and expert behavior Tg:
TTh J. A trajectory is a sequence of state-action pairs T (so, "..... ST, uT) over
A E argmin L(TT, TT)
where L measures the dissimilarity between two policies' behaviors.
Having access to only sample demonstrations of a policy for training, it is essential to quantify the behavior of policy. Commonly used in behavioral cloning, one approach is to measure the marginal distribution of states and actions, P(s, a) induced by policy. Note that in the case of infinite-horizon MDPs, expected (discounted) number of visits to state-action pairs is used. A more desirable measure for long horizons is the expectation of trajectory features (1] This approach is often used in inverse reinforeement learning, where reward function of the features is learned. The expected (discounted
We overload L for any type of loss in different contexts (e.g.. ... loss between deterministic policies states).
2

---

if y E (0, 1)) feature of a policy x (similar to ① is defined as:
p E
r (sr, ai) ai) RR.
3 Robust Imitation Learning and Maximum Entropy
Foundational work 27 [2] has developed a close relationship between robust Bayes decisions. maximum entropy, and minimizing worst-case expected loss also known adversarial learning. Several works have followed this framework of robustness for different supervised learning problems, such as cost-sensitive classification (2], multivariate loss prediction [29]. ordinal regression [7]. and
graphical models [ For the imitation learning setting, [5] has applied the adversarial learning framework on inverse reinforcement learning problem for restricted class of loss functions and proposed double oracle algorithm 17 to solve the corresponding optimization problem.
In the following, define general framework of distributional robustness for imitation learning obtain the policy that robustly minimizes an imitative loss:
Definition 1. Given an imitative loss function L that measures the distance of two policies' behavior, Distributionally Robust Imitation Learning (DRoIL.) defined two-player zero-sum game between the learner and the demonstrator, in which each player chooses a distribution over control policies constructing a stochastic policy, ft and E E, then the players receive the loss between the behaviors L(ft, #) as their payoff. The minimax strategy for the learner is given by:
min max C(ă, #); RE
where E is set of constraints churacterized by the demonstrated data.
Generally, E can be in the form of moment matching in ( that is commonly used in inverse reinforcement learning:
T
# E
E
Σ (oss a)) = E 1=0
a IE. IT.
where T represents the policy that demonstrated trajectories are generated from.
This formulation assumes that except for certain properties of the limited samples of available demonstrated behavior, the demonstrator's 's policy is the worst-case possible for the learner. This approach avoids generalizing from available demonstrations in optimistic manner that may be unrealistic and lead to a policy that does not work well in practice, especially in situations where there is noise in the demonstrated behaviors.
The minimax formulation in Definition D closely related to the principle of maximum entropy, which is used in Maximum Entropy Inverse Reinforeement Learning (MaxEnt IRL) [31. MaxEnt IRL provides a probabilistic approach under the constraint of matching the reward value of demonstrated behavior to resolve the ambiguity in choosing distribution over decisions. Under this model, trajectories with equivalent rewards have equal probabilities, and trajectories with higher rewards are exponentially more preferred according the following:
P(C (W)
Oc.
(3)
Z(w)
where Ço and ΦG.. represent trajectory and its corresponding sum of features.
Note that distribution over trajectories (paths) is alternative policy description that provides a stochastic policy where the probability of an action is weighted by the expected exponentiated rewards of all paths that begin with that action:
P(alw)
Σ P(C/w).
C:RES

---

To show the connection between DROIL and MaxEnt IRL, we employ tool from the general theory of exponential families [ that shows for certain classes of two-player zero-sum games, there exists a parametric distribution for the minimax strategy, as shown in Lemma]] [
Lemma 1 (Barddrff-Nielsen (... Let p E E be a probability distribution over space X, where: E =p {p: (T())) = C} describes mean-value constraint; T(X) represents a vector value statistic; and C is a constant. Let q be also a distribution chosen from the set of all probability mass functions. For the maximum entropy distribution, max miny E[-log (XX)), p' = 9" exists as parameterized function of the form p" exp(a0 + T(X) with parameters og and a.
Equipped with Lemma] T
we develop a connection between DROIL and MaxEnt IRL in Theorem
Theorem 1. The stochastic policy P(G/ () obtained from MaxEnt IRL in Equation is obtained from DRoIL minimax strategy in Definition 7 when the logarithmic loss is used.
Proof. We construct stochastic policy in Definition Das probability distribution over all possible trajectories {(I,,.,, SMr.. Let मे = P()) and म = Q{) be the probability distributions of the learner and the demonstrator, respectively. The expected feature of demonstrator now can be written as P = Σ. (()))ç which resembles the mean value constraint in Lemma | therefore, for logarithmic loss, (G) = (G)) = exp(wo + w TT
We can generalize beyond the logarithmic loss and resulting maximum entropy policy (Theorem to other losses of interest by using Generalized Entropy functions, H(P) == infq ()...
Proposition ! describes the relation between DRoIL and generalized maximum entropy.
Proposition 1. For any policy descriptions and loss functions that DROIL in Definition / has Nash equilibrium, से is robust action and is the maximizer of a generalized entropy function.
Proposition ! provides a general approach resolve the ambiguity of matching constraints-wher many policies lead to the same feature counts- by choosing either policy स that does not exhibit any additional preferences beyond matching feature expectations with respect to a loss of interest L (maximum generalized entropy) or policy A that minimizes the worst-case expected loss. This can be seen as Maximum Generalized Entropy Inverse Reinforcement Learning, where the choice of loss function is not restricted to the logarithmic loss function.
We extend the following lemma that describes class of loss functions and policies for which the solution of the game in Definition [ exists, thus Proposition [ be applied.
Lemma 2. For the game in Definition D where learner āt and demonstrator ă simultaneously choose a stochastic policy, if the loss function is additive and the payoff can be written as a bilinear function: Σ Eø=1 j the game has a solution and minimax (maximin) strategy producing the solution as long as and are finite.
Proof. In his generalized minmax theorem, Von Neumann arbitrary nonempty closed, bounded convex sets of actions have saddle points as long m and n are
28 shows that bilinear games with
finite. Assuming the demonstrated data are generated from a policy (optimality not necessary) the
convex moment matching constrained in Definition keeps E closed, bounded, and non-empty. O
One illustrating example let each player's pure strategy to be a deterministic policy 5 and define the loss as the expectation over a given loss between each individual deterministic policies. A mixed strategy now represents stochastic policy n, which is probability distribution over the set of all deterministic policies and Equation (2) can be written as a bilinear game with where lij is L(),5)).
Example 1.
payoff Em (ey
Lemma [ describes a restricted class of loss functions (e.g., that the logarithmic loss does not belong to) for which the resulting policy is the maximizer of more flexible concept of entropy. A class of loss functions of interest are loss functions that are additive over state and action spaces. Similar to Example
[] previous work [5] constructs policy using distribution over deterministic policies and for additive loss function by writing Equation (2) as a bilinear game and solving the corresponding optimization. We also focus on this class of loss functions. However, we provide an efficient optimization algorithm for this class of loss functions that avoids the exponential worst-case time
complexity of the previous double oracle [17 approach.
4

---

Learning and Inference
To solve the optimization problem in Equation (2). one needs to specify the policy description and the distance measure between two policies' behaviors L. The choice π and can result in different algorithmic approaches.
4.1 Double Oracle
31
82
8,
In [5], the authors construct the learner
and the demonstrator stochastic poli- cies mixtures of deterministic non- stationary policies. Concretely, let Y {ō1. J..... be the set of all possible non- stationary deterministic policies, then A == P(fo E T) and F P(f6 E )) Note that ['] E O(I which exponential in size of state space and time horizon, S). T. Assuming the loss function additive over state-action pairs, [5] shows that Equation (2) can be written as bilinear game:
82
[(81. 81)
[(8,, 82) +v((22)
((
+ (81)
+8((8,)
[(02, 61) +w(6))
[(d2. 82)
((62.0g)
82
+v((22)
+w(6,)
[ 81) (x)()))
[(oi, 52) (62)
C 8,) (1,)
8,
Table I: Payoff matrix of G(# A tT ww with loss function L. deterministic policies 5 and Lagrangian potentials sb.
G(fa.Halw)
min min max 5A
ΣΣ p())) ((0)))(6)) + Ep(5)) n
p(55) w B[Hlö,] w fi. ((,,))
This will result in matrix game of exponential size in the number of states S and time horizon, as shown in [ To obtain the Lagrange variables w, this matrix game needs to be repeatedly solved to compute the gradient with respect to w. This requires solving a linear program with (([A))S variables with simplex constraint, which is impractical for even modestly sized problems. To mitigate this problem, they employed the double oracle method [17 in an attempt to construct smaller sub-portion of the matrix by gradually adding pure actions through solving a time-varying control problem. However, there is no guarantee that the support set of Nash-equilibrium of the defined game is small and the algorithm may need solve up to exponential number of time- varying optimal control problems.
4.2 State-Action Distribution
We propose an alternative way to transform the optimization in Equation (2) into a convex problem. Our approach is based on using state-action marginals to construct the stochastic policies of learner and the demonstrator, where the number of required variables is linear in [S] and [A], and it can also be extended to stationary policies. the following, we first look at the non-stationary case and then extend our method to stationary policies.
policy π induces a probability distribution Pt(s) over the states of an MDP M at each time step. State-action marginals are similarly defined as Pr(s, a) = (a)ss) A valid state-action marginal is set of simplices corresponding
each time-step that satisfy the Bellman flow con- straints for given MDP M. Let Pa a) E A be the probability of state-action pairs (a, b) at time t; valid marginal distribution P E satisfies the following affine constraints: for all s', Σ,, (s, P) ((s'|ss a) = Σ. (s' a), where P(s'(s,a) E I' and Ω represents the set of all valid marginal distribution for given MDP M.
Employing state-actis action marginals allows to write the objective function in Equation problem of ([A[ variables as shown in Theorem 2 in a vectorized form.
convex
Theorem 2. For additive loss function over slutes and actions L, Solving DROIL optimization in Equation (2) is equivalent to solving the following convex minimax problem over marginal state-action probabilities of the learner P (Pi.... PT) and demonstrator Q (q1...., 4T)
5

---

parameterized with Lagrange multipliers w:
T Σ t=
min max min w QER PER
P. Lq. + To qi w p.
(4)
where Pt (similarly q:) is a vector of size |A|||] storing marginal probability of state-action pairs at time t: (s,s, a); is the general loss function that is defined over state-action pairs L [LA|[S] X [A|]S). á is a x [||SS] matrix storing the feature function for each state-action pair. We denote system dynamics with I', which is a matrix of size [A||S] [8] storing transition probabilities P(s'|s, a).
Intuitively, each player in the above formulation searches over valid state-action distribution to reach an equilibrium. One of the benefits of constructing policy using marginals is that the feature expectation can written as an inner product of the state-action distribution and the state-action feature vector. Thus, the demonstrator's expected feature is ((Q) Ei= Lqq, and the feature
matching constraint is realized by minimizing w (Et= Φq. ja) over dual variables Writing the objective function Equation (2) in terms of state-action marginal probabilities reduces the number of variables needed to represent the equilibrium from O A (S(T') to O(IA[|))T). The unconstrained optimization over dual variables w can be solved using any gradient descent method where the gradient is given by Σ 2qï current wy:
jt and Q" is the solution of the following game for the
T Cq: + w. T 9:
G(w,)
max min
Σ
Pi
(5)
Qef PES
4.3 Stationary Policy
Our approach extends with some modifications to the stationary policy setting. Stationary policies are desirable because they are simpler to describe, and are more natural and intuitive in terms of the behavior that they prescribe. Similar to state-action marginals, we utilize an occupancy measure Par S x A + U characterize stationary policy æ. is defined as the expected (discounted) number of visits to atate-action pair (s, a), when following policy π and can be written as a feasible set of affine constraints:
G = (p p p0:
Σ pls' po(s') [7P(61ssc))p( EsE)),
8,0.
where P(s'ss, a) E I' and po is the distribution of starting states.
For a given additive loss L. with the use of an occupancy measure, we write the expected loss B[[(T1, T2)] as PT, CpT, and the expected discounted feature μ, 4 Ps. Theorem 3 shows how we can write Equation (2 as a convex optimization using occupancy measure:
Theorem 3. For additive loss function
state and actions f, solving DRoIL optimization in Equation (2p is equivalent to solving the following convex minimax problem over occupancy measures of the learner P and demonstrator Q:
min max min P T LQ +w+wl. w - W
DEG PEG
The convex optimization in Equation 7 can also be solved using any gradient-based method where the gradient obtained by solving constrained game with O(AA]|S)) variables.
4.4 Inferred Policy
In the non-stationary case, after obtaining w", one can use either Q' or P as the produced non- pe(a.s)
stationary Markovian stochastic policy by computing π (a(sss)) (r(a)s) Σ. (c(.s) Q corre- sponds to the policy that maximizes the generalized entropy and P corresponds to the policy that has minimized the worst-case expected loss.
For the stationary case, [26] has proved that there a one-to-one mapping between & and II, in P(a.s) is the only policy that results in P.. sense that for an occupancy measure P E G, (a]s) ع Σ. P(a.s)
Therefore, one similarly use P Q as the produced stationary stochastic policy.

---

For either case, assuming the reward function can be written R(s, a) w o(s, a), then w" plays the role of the reward weight vector that rationalizes the demonstrated behaviors. Consequently, we can use w' Φ to obtain the optimal reward function R. and use an MDP solver to obtain the (deterministic) corresponding policy.
5 Optimization
In both stationary and non-stationary cases, any gradient descent method can be used to optimize over dual variables w. To compute the gradient, one needs to compute Q" for non-stationary and Q* However, since w is unconstrained, by adding a norm w with hyperparameter A, one can directly solve and replace it in the objective. Since the optimization algorithms similar in both cases, we only mention the non non-stationary case:
A //wl
min max min QE PEn
Σ P. Lqc + w t q:
2
and setting: w= /
Li q
we have:
T
max QEΩ
91
min
p Lq.
(9)
2X
PEB
1=0
which is constrained quadratic optimization in Q. Using Danskin's 's theorem, the gradient of qu is given by
1 10 T - Σ T
where p" be found using linear programming (linear objective with affine constraints) effi- ciently using standard linear programming toolbox. An alternative approach to solve the dual of optimization over Pt and maximize it along with Q.
Algorithm Distributionally Robust Imitation Learning (DRoIL)
Input: D {r', ... , T,Po
Initialize Q" compute ja using D, and set i 0
repeat
Compute v.f(Q') in Equation
i= 1= Using V (Q') calculate with Q++1
if QQ+H E Ω then
+++ Q++1
else
+++1_ project (Q++1) where projection function is defined in Equation (10
end if until convergence
Projection Step
At each iteration of the algorithm, we need to project Q convex domain maintain a valid state-action distribution given I' and the initial state distribution. Essentially,
min Q 2
Σ
|ل
s.t. Q Ω
(10)
from which, by using Lagrangian method and strong duality, we obtain:
max min QEA
g qill2 (q._,- q. Z)ve,
(11)
where Z operator computes the state distribution: q.- zZ= (s, a). To compute the gradient V. a quadratic program over Q with probability simplex constraints needs to be solved. This can be analytically determined by sorting each qr, which takes [[A (o)()))|AA)) time.
7

---

Experimental Results
In our experiments, we compare DROIL with prior methods several imitation learning tasks. We investigate: 1) How our convex optimization improves the training time compared to the double oracle method; 2) How the choice of loss function affects the performance of DRoIL; and 3) How accurately DROIL predicts actions compared to other IRL. methods.
6.1 Training Time
To compare the training time of our pro- posed convex optimization with the double oracle approach [5], we adopt their experi- mental setup in GridWorld. In this experi- ment, trajectories
DROIL
DROIL
DO
Do
Size
Time
Cost
Time
Cost
128
1.6
-7.39
30.1
7.39
collected from simu- lated navigation across a discrete 2D grid where the agent starts from random start- ing point, and navigates through the grid to reach specified target location. Taking step in the grid has a cost and the agent's goal to reach the target location while minimizing the accumulated navigation cost (maximizing the reward). This problem can be formulated as an optimal sequential decision-making problem in a finite Markov decision process where the optimal policy non-stationary. We consider linear cost function C(s) w ((s)) + ((s). where feature function ( a) and weight vector w are drawn from U(0, 1)d, and UO, 1). Transition function is non-deterministic with parameter Pm E (0, 1] which navigates the agent to randomly choose neighbors with probability (1 Pm).
432
1.8
-21.61
42.1
211.74
1024
6.0
-20.0
141.1
- 20.0
2000
98.6
-30.1
608.1
30.1
3500
496
39.2
2020
39.1
5500
881
-58.7
4970
- 58.9
Table 2: Elapsed time in second until convergence with 10-3 tolerance.
The loss function for this experiment is set to E
(Y, -
x,)
Y,)² where
(X, Y) represents the grid position the agent. We generate trajectories from the optimal policy that obtained by solving the true reward function and train DRoIL. and DO until convergence. Along with expected loss, we report the elapsed time that it takes to converge for our proposed method and double oracle (DO) in Table 2 averaged
eight repetitions of the experiment for this result and our later results. As Table 2 shows, our proposed method requires significantly less training time for the same performance and scales very well with the size of state space.
6.2 Loss Function Effect
0.5
0.45
action-loss
For the second question, we show that different choices of loss function result in different performance for the learned policy in the imitation learning setting. There- fore, the choice of loss function provides extra tool to incorporate certain domain knowledge, and design problem-specific loss that potentially results in better produced policies. For the purpose of comparison of different losses, revisit the GridWord environment, however, we train several stationary policies with differ- losses. We consider 0-1 loss, which equally penal- izes any mismatch between state-action pairs; action- loss, which incurs loss only when action differs from another pair in the same state; random loss, which drawn from a uniform distribution; and finally Eu- clidean distance between two positions in the grid. It clear from Figure [ that the policy produced from Fuclidean loss outperform other policies from other losses, which shows the benefit of using a task-specific loss function.
random
0.4
auclidean
0.35
misprediction rate
0.3
0.25
0.2
0.15
0.1
0.05
0
8
16
32
64
examples
Figure/ Performance of DROIL. when different loss functions are used.

---

6.3 Highway Driving
To evaluate DRoIL in complex environment with more realistic behavior, we compared it with several other IRL methods in a highway driving simulator with non- non-linear reward function: MMP [0)] the projection algorithm of Abbeel and Ng M, and LEARCH (2]]
Following the setting in Levine et al. [66. the task is to drive a car on three-lane highway which the agent switch lanes and drive up four times the speed of traffic while all other vehicles move at a constant speed. The set of features includes the distance to the nearest vehicle each lane (in front and behind), current speed, and current lane. We also evaluate each method on the original environment and on four additional random environments, denoted as "transfer". We set a uniformly random loss for DRoIL and train all methods using examples sampled from the stochastic MaxEnt IRL policy which can intuitively be viewed as noisy samples of an underlying optimal policy. To evaluate the performance of each method, we use the misprediction rate, which is defined as the ratio of incorrect actions compared to the optimal policy, and the expected value difference, which measures the suboptimality of the learned policy under the true reward. Since DROIL. able to produce stochastic policy, with the same argument from [16], we could evaluate the optimal stochastic policies. However, this would unfairly penalize other methods. Therefor, we first obtain the reward weight vector and find the optimal deterministic policy the corresponding reward function. Then, we measure its expected sum discounted rewards under the true reward function, and subtract this quantity from the expected sum of discounted rewards of the optimal policy.
original
transfer
Abbeel & Ng
Abbear & Ng
MMP
MMP
200
LEARCH
200
LEARCH
expected value difference
DRoiL
expected value difffrence
DRoiL
150
150
100
100
50
50
8
16
32
64
2
4
16
32
64
examples
examples
Figure 2: Expected value differences for 64-car-length highways with varying example counts. Lower values are better.
As shown in Figure 2 and Figure 3 DROIL performs very well in terms of the obtained reward and the accuracy the produced policy in both the original and transfer environments. contrast, we find that MMP and Abbeel & Ng's approach degrade as the number of examples increase. This matches theory since the suboptimality of the demonstrations becomes more apparent as the number of examples increases. This indicates that under noisy demonstrations (samples from a stochastic policy), a robust approach has the potential to outperform alternative approaches.

---

original
transfer
0.45
Abbaol Ng
Abbeel Ng
0.4
MMP
0.45
LEARGH
LEARCH
0.4
0.35
DiRett.
DESOr
misprediction rate
0.3
misprediction rate
0.35
0.3
0.25
0.2
0.2
0.15
0.15
0.1
0.1
0.05
0
2
4
8
16
32
64
2
4
16
32
64
examples
examples
Figure 3: Misprediction rate results for 64-car-length highways with varying example counts. Lower values are better.
7 Discussion & Conclusion
We demonstrated connection between DROIL, which accepts any loss of interest, and Maximum Entropy Inverse Reinforcement Learning- of the widely used IRI. approaches-wwiel robustly minimizes logarithmic loss. We showed that MaxEnt is a special case of DRoII. framework when logarithmic loss function used and showed that DRoIL can seen as the maximizer of generalized concept of entropy. We provided novel approach to cast DRoIL's objective into convex optimization over a polynomial number of variables and experimentally showed our proposed algorithm provides faster training time. DRoIL is naturally designed to perform robustly against noisy demonstrations. Our experiment in the highway driving task showed that when demonstrations noisy. it robustly learns appropriate policy.
Improvements imitation learning have the potential for both societal benefits and harms. For example, better imitating surgeons could scale their abilities to a broader populations that are medically under-served. Enabling robots that better imitate effective soldiers could cause great harm if used inappropriately. Like all general purpose tools, avoiding intentional harms while still providing benefits is an unsolved challenge. We take the position that providing methods that are more robust to noise will help to avoid unintentional harms-the the application of methods in well-intentioned manner that fail to maximize their benefits and may instead produce harms through their fragility.
Our presented experiments are restricted to discrete/low-dimensionn dimensional decision processes, For future work, we are interested in finding a connection between DRoIL and the Generalized Exponential family and applying DRoIL on very high dimensional state and action spaces that need function approximators such as deep neural networks.
10

---

Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant Nos. 1652530 and 1910146.
References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the International Conference Machine Learning, pages 1-8. ACM, 2004.
[2] Kaiser Asif, Wei Xing, Sima Behpour, and Brian D. Ziebart. Adversarial cost-sensitive classifi- cation. In Proceedings of the Conference Uncertainty in Artificial Intelligence, 2015.
[3] Ole E Bamdorff-Nielsen Information and exponential families. 1978.
[4] Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2)341-357, 2013.
[5] Xiangli Chen, Mathew Monfort, Brian D Ziebart, and Peter Carr. Adversarial inverse optimal control for general imitation learning losses and embodiment transfer. In Proceedings of the Conference Uncertainly in Artificial Intelligence, 2016.
[6] Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research, 58(3):595-612, 2010.
[7] Rizal Fathony, Mohammad Ali Bashiri, and Brian Ziebart. Adversarial surrogate losses for ordinal regression. In Advances in Neural Information Processing Systems, pages 563-573,
2017.
[8] Rizal Fathony, Ashkan Rezaei, Mohammad Ali Bashiri, Xinhua Zhang, and Brian Ziebart. Distributionally robust graphical models. In Advances in Neural Information Processing Systems, pages 3344-8355, 2018.
[9] Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprini arXiv: 1.03852, 2016.
[10] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the International Conference on Machine Learning, pages 49-58, 2016.
[11] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710. 11248 2017.
[12] Peter D. Grünwald and Phillip Dawid. Game theory, maximum entropy, minimum discrep- ancy, and robust Bayesian decision theory. Annals of Statistics, 32:1367-1433 2004.
[13] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pages 4565-4573, 2016.
[14] Rust John. Maximum likelihood estimation discrete control processes. SIAM Journal on Control and Optimization,
1988.
[15) Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg, Dart: Noise injection for robust imitation learning. arXiv preprint arXiv: 00337,,9 2017.
[16] Sergey Levine, Zoran Popovic, and Vladien Koltun. Nonlinear inverse reinforcement learning with Gaussian processes. Advances in Neural Information Processing Systems, 24:19-27, 201
[17] Brendan McMahan, Geoffrey J Gordon, and Avrim Blum. Planning the presence of cost functions controlled by an adversary. In Proceedings of the International Conference on Machine Learning, pages 536-543, 2003.
[18] Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Proceed- ings of the International Conference on Machine Learning, pages 663-670, 2000.
[19] Dean A Pomerleau. ALVINN: An autonomous land vehicle in neural network. In Advances in Neural Information Processing Systems, pages 305-313, 1989.
[20] Nathan Ratliff, J Andrew Bagnell, and Martin Zinkevich. Maximum margin planning. In Proceedings of the International Conference on Machine Learning, pages 729-736, 2006.
11

---

[21] Nathan D Ratliff, David Silver, and J Andrew Bagnell. Learning to search: Functional gradient techniques for imitation learning. Autonomous Robots, 27(1):25-53, 2009.
[22] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 661 661-668, 2010.
[23] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings the International Conference on Artificial Intelligence and Statistics, pages 627-635, 2011.
[24] Stuart J Russell. Learning agents for uncertain environments. COLT, volume 98, pages 101-103, 1998.
[25] Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In Proceedings of the International Conference on Machine Learning, pages 10022-10032, 2021.
[26] Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear programming. In Proceedings of the International Conference Machine Learning, pages 1032-1039, 2008.
[27] Flemming Topsøe. Information theoretical optimization techniques. Kybernetika, 15(1):8-27, 1979.
[28] John Von Neumann. Uber ein okonomsiches glcichungssystem und eine verallgemcinering des browerschen fixpunktsatzes. In Erge. Math. Kolloq. volume 8, pages 73-83, 1937.
[29] Hong Wang, Wei Xing, Kaiser Asif, and Brian Ziebart. Adversarial prediction games for multivariate losses. Advances in Neural Information Processing Systems, pages 2710-2718, 2015.
[30] Jiangchuan Zheng, Siyuan Liu, and Lionel M Ni. Robust Bayesian inverse reinforcement learning with sparse behavior noise. In Proceedings of the AAAI Conference on Artificial Intelligence, 2014.
[31] Brian D Ziebart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, pages 1133-1438, 2008.
[32] Brian Ziebart, Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In Proceedings of the International Conference on Machine Learning, 2010.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Theoretical claims are supported by theorems and experimental claims supported empirically.
(b) Did you describe the limitations of your work? [Yes] See Section 7
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We discussed them in each Theorem
(b) Did you include complete proofs of all theoretical results? [Yes] We include these in Supplementary materials.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We provide these in the supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We describe these details in the supplementary materials.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
See the Supplementary Materials.
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13

---

A Proof of Theorem 2 & 3
Suppose the loss function C is decomposable over state action pairs, then we can write, Equation as following:
8, A, = -
min max
(12)
8.4
T
subject to: Σ
E
= -
[=1 8,0
By introducing dual variables w for the feature expectation constraints, the Lagrangian function of Equation 12) is given by:
min max min
Σ
P(S, = 8, = = A, = = =
(13)
8.0
8,0
T
Σ 8,0
P( = 8, = ())(8 i4)
w
The optimization in Equation 13 is over ग and #. However, the objective function Equation (3)
decomposes over the state-action distribution induced by policies Å and #. We directly optimize over the marginals:
min
max
min
p((5.0)((8....0))0.....
(14)
(ps "r.PP.....PP) (4s (T)
R.
3,
T
+ w
Σ 3
qe a))(3, ūi)
where (s, à) = P(š. = 8, A, = a) and g: (3. () P(š = A -
=ū) a). This optimization needs to be over valid state-action marginals (marginals induced by policy). So the following constrained need to be satisfied:
Σ (8)18)))))188
Emi(3'
1,8'
8.0
Similarly for
Σ (33 (3) = En ()(',aa) V t,
Since Equation
14
is convex in all variables Pt. gt. and w, the order of optimization can be changed:
min max min QEN Pen
Σ
Pa(s, 5, 8, t.w. Σ
(.( w j.
8,8
where P = (pI, (p.,P2,...,PT' and Q = (qi, QT)
The proof for stationary case is similar.
14


Reviews:
[{"review_id": "tbScFkjqzbg", "reviewer": "Reviewer_iosT", "summary": "This paper studies Distributionally Robust Imitation Learning (DIRL) and reveals that Maximum Entropy (MaxEnt) can be a special case of DIRL. In the case of an \"additive\" loss function over state and actions, it also proposes a new algorithm to solve the adversarial training problem by using formulating the problem as a convex optimization problem over the space of marginal state-action distributions. Experiments show that the proposed algorithm can learn a good policy with a similar performance to the baseline, but much faster. Another set of experiments show that the proposed algorithm can find a better reward weight vector than the baseline algorithms, \n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed algorithm is interesting but is limited to additive loss functions, discrete state/action spaces and requires a known transition model and a feature extractor. The connection between DIRL and MaxEnd seems to be quite simple and intuitive. The experiments seem good but I expect more baselines. The writing can be improved a lot. Notations are not consistent and sometimes not defined. \n\nPreliminaries: Do you assume $\\phi$ is known and $\\textbf{w}$ is unknown? \nDefinition of $\\mu(\\pi)$: What is $k$? Is it $d$? \n\nEq.1: What's the expectation over? The same question goes to $H(P)$ in L.136. \n\nLemma.1: What are $\\mathcal{T}, C, \\tilde\\Xi, \\alpha_0, \\alpha, p^*, q^*$? Does $q$ really mean the set of all mass probability functions? Are $\\alpha_0, \\alpha$ scalars or vectors? Do $\\alpha$ or $\\alpha_0$ depend on $C$ to guarantee that $p^* \\in \\Xi$ (or do we need it)? \n\nTheorem.1: What does \"logarithmic loss\" mean? Is it the cross-entropy loss like in Lemma.1? Please give an explicit form, especially if it's not too complicated. \nTheorem.1 proof: What's $\\textbf{w}_0$? Is it a scalar or a vector? If it's a scalar, please use a consistent font style to avoid confusion. \n\nLemma.2: Please put the assumptions into the lemma description instead of the proof. \n\nL.158: asa => as a\nL.159: $\\mathcal{L}(\\hat \\delta_i, \\check \\delta_j)$. \n\nSec 4.2: What is an additive loss function? Is the logarithmic loss function additive? \n\nEq.3, 4, 7, 8: Please use $\\mathbf{\\Omega}$ or $\\Omega$ consistently. \n\nSec 5: Is the transition matrix $\\Gamma$ known? \n\nExperiments: I'm looking for more baselines. The most recent baseline used has been published for more than 10 years. I'm not an expert in this area so I don't know what's the recent progress and it might be possible that no directly comparable algorithms were developed in the last 10 years. \n1. As this paper uses an environment from \"Levine et. al [16]\", can the method in that paper also be compared? \n2. How does MaxEnt perform? \n3. How does Double Oracle perform? \nIt's a little bit weird if we study the relationship between DRIL and MaxEnt but don't compare them in experiments. \n\n---\n\nAfter reading the response from the authors, I increase my score from 4 to 5. But I think both the theoretical contribution is minor as the technical difficulty is not too much, and the experiments are limited due to strong assumptions. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I feel like the assumption that the transition model is known is very strong. I didn't see an obvious way to run the proposed algorithm in continuous state/action space, either. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "rjPGzrBiUHf", "reviewer": "Reviewer_xMat", "summary": "For discrete low-dimensional Markov decision processes, the paper proposed a distributionally robust imitation learning framework to learn a policy with respect to the worst-case demonstrator (subject to feature matching constraints from data). The maximum generalized entropy imitation learning framework naturally includes the predominant maximum entropy IRL. Compared to previous double oracle approach, the contribution of this paper is then transforming the original minimax problem over policies (in Definition 1) to optimization over occupancy measures (state-action distributions), such that only a polynomial number of variables need to be solved instead of exponential. The policies are later obtained based on the link between occupancy measures and stochastic policies. Besides, the flexibility of generalized entropy allows us to incorporate more domain knowledge into the design of loss function, although this point can be better demonstrated with experiments.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is trying to address an important problem in imitation learning, that expert demonstrations may not be consistently optimal (suboptimal demonstrations). The proposed approach is based on finding a policy that matches the worst-case demonstrator, thus avoiding optimistic generalization. However, since the worst-case demonstrator is still under the feature matching constraint, it seems if the expert demonstrations is not near optimal, the solution will still be quite suboptimal, even with the distributionally robust framework. Some recent works attempts to address this problem by assuming access to extra information such as ranking/scoring associated with the demos, which seems to be more plausible to address this challenging scenario.\n\nThat being said, I think incorporating the notion of generalized entropy from Grünwald P D & Dawid A P.'s paper to formulate the robust imitation learning problem is interesting. It is natural that the maximum entropy IRL framework is a special case of DRIL since maximum entropy is a special case of generalized entropy (e.g. in Grünwald P D & Dawid A P.'s paper). From this perspective, I also think the traditional maximum generalized entropy notion (Grünwald P D & Dawid A P.'s paper) should be discussed in more details as preliminaries to provide background and allow readers understand the contributions properly.\n\nBelow are some detailed comments/questions:\n- When introducing state, action spaces in preliminaries, please explicitly say these are (low-dimensional) discrete spaces. discrete MDP is unclear (might be discrete/continuous time).\n- In definition 1, eq 1, what distributions is the expectation taken over? Just based on the definition, I don't see any other random variables and the loss already takes both learner and demonstrator as input.\n- In distributionally robust optimization, we usually find an adversary within some lp-ball around some data, hence tolerating some small error/differences around demos. Do you think this notion is necessary/can be implemented: in the minimax problem, the adversary (worst-case) demonstrator need to match the feature statistics of the data up to some small error (rather than perfect feature matching).\n- I think Eq 2 is true only for deterministic dynamics.\n- In lemma 1, q is not \"the set of all mass probability functions\", but a single mass function?\n- In line 135, need to add ref to the generalized entropy function.\n- Section 6.2 is trying to demonstrate that by smartly choosing loss functions, we can incorporate proper domain knowledge to the definition of generalized entropy, which is also an important question to think about when using generalized entropy in practice. However in the provided example, the best euclidean distance seems to be already the default loss choices, which is not a surprising example. Any other examples where using a smart choice of loss function leads to better performance than an obvious/default choice? Also, the baselines such as random or zero-one seem to be too weak, and the popular logarithm loss in MaxEnt IRL is missing?\n- Some experimental details may be better discussed perhaps in the Appendix. For example, I'm not clear why in the setting of Fig 3, more examples lead to worse performance for the baselines.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The negative societal impact is properly discussed. For comments on technical contents, see main review section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "qV8dMhP2IBw", "reviewer": "Reviewer_8Tiy", "summary": "The paper tackles the problem of imitation learning in an environment with a discrete set of states and actions, known dynamics, and an (unknown) reward function that is linear with respect to known state-action features. The key idea is to formulate imitation as a two-player zero-sum game between the agent and demonstrator, and to efficiently compute the agent’s minimax strategy by solving a convex optimization problem. This approach, called Distributionally Robust Imitation Learning (DRIL), generalizes maximum-entropy inverse reinforcement learning to non-logarithmic losses. Simulation experiments in simple grid-world navigation and highway driving environments show that DRIL has a lower running time than comparable prior methods, can be used with a variety of different additive loss functions, and learns more effectively from noisy demonstrations than prior methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I enjoyed reading this paper. I look forward to the future work on extending DRIL to high-dimensional state and action spaces discussed in lines 388-391!\n\nOne issue is that the experiments do not compare DRIL to recent imitation learning methods that aren't derived from MaxEnt IRL or adversarial games, but still attempt to perform state distribution matching:\n - RCE https://arxiv.org/abs/2103.12656\n - RED https://arxiv.org/abs/1905.06750\n - DRIL https://openreview.net/pdf?id=rkgbYyHtwB \n - VDI https://arxiv.org/abs/2002.06473\n - IQ-Learn https://arxiv.org/abs/2106.12142 (this paper was released after the NeurIPS submission deadline, so I don't expect the authors to have somehow compared to it in advance, but I want to mention it anyway because it's relevant)\n\nIt would be nice to add comparisons to some subset of these methods -- not necessarily to show that the proposed DRIL method outperforms the prior methods, but primarily to show that the reason that the proposed DRIL method performs better than MMP, LEARCH, and the projection algorithm of Abbeel and Ng is, in fact, distributional robustness.\n\nA few minor issues, typos, and questions:\n - The name “DRIL” conflicts with another DRIL from prior work: disagreement-regularized imitation learning https://openreview.net/pdf?id=rkgbYyHtwB\n - Line 173: do you mean exponential in the size of the state space?\n - Line 209: “construing” → constructing\n - Equation 5: should “p_0(s)” be p_0(s’)?\n - Line 241: why are the dual variables w also the reward weights w?\n - Line 294: “We run the both algorithms until convergences,” → “We run both algorithms until convergence.”\n - Why should the loss L ever be non-diagonal? It seems like you would only ever want to measure the difference between the agent’s occupancy of (s, a) and the demonstrator’s occupancy of the same state-action pair (s, a), rather than the agent’s occupancy of (s, a) and the demonstrator’s occupancy of some different state-action pair (s’, a’).\n\nUpdate after reading author response\n-----\nThank you for your response. I will keep my original score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Section 6 adequately addresses limitations and societal impacts. See the main review for suggestions on how to improve the writing.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "2TlEzsvDQ98", "reviewer": "Reviewer_w9mr", "summary": "The paper extends on a previous method [5] for distributional robust inverse reinforcement learning. Both works learn a reward function, the agent's policy, and an adversarial policy that aims to incur maximum loss (w.r.t. a given loss function) on the agent while being constrained to matching given features in expectation.\n\nThe contributions of the submission are\n* showing that MaxEnt-IRL is a special case of that formulation (using the cross-entropy between adversary and agent as loss)\n* presenting a more efficient optimization procedure compared to the previous (\"double oracle\"-approach)", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n---------------\nI think that both above-listed contributions are novel and significant enough to justify publication.\n\nQuality\n-----------\nThe derivations seem correct, but I did not check them thoroughly. Using distributional robustness in an IRL setting makes a lot of sense and the approach seems sound.\n\nClarity\n----------\nThe presentation is mostly clear. \n\n* Reading the reference [5] helped me a lot, because it made it a bit more clear that the weight vector corresponds to the Lagrangian of the feature matching constraint (as in MaxEnt). It would be helpful to add this somewhere before Theorem 1\n \n* It was not immediately clear to me that the expectation of the loss was computed w.r.t. p, e.g. for the log-loss in line 124. Its easy to guess, but it would be better to indicate it in the equations.\n\n* The benefit of changing to marginal distribution also needs to be made clearer. The paper mainly talks about how linear and nice everything becomes, but from what I understand the constraint that the marginal distributions must be realizable does not make the problem easier. For example, the optimization for Eq. 4 still requires the inner reinforcement learning. The need for inner RL only seems to be removed in Section 5, when by expressing the weight in terms of the distributions, the order of optimizing the policy and the reward get switched. I think at this point the paper should draw the connection to GAIL which also achieves faster imitation learning (without explicitly learning the rewards) by moving from $\\text{IRL} \\circ \\text{RL}$ to $\\text{RL} \\circ \\text{IRL}$.\n\n* Line 336 states \"We also evaluated each method on the original environment and on 4 additional random environments\" on a transfer setting. I assume you evaluated the methods by evaluating the _policies_ that have been learned on the source domain on additional random environments? The paper should be more explicit here.\n\nSignificance\n-----------------\nI think that the new insights might be valuable for some researchers, in particular, because I think that distributional robustness is a logical and sound approach to deal with the non-optimality of the expert and the ill-posedness of IRL. The current work is somewhat limited (which is also mentioned in the paper) because it is restricted to simple problems, such as Gridworlds. Correspondingly, the comparisons with LEARCH and MMP seem a bit outdated. I do not think that this a major problems, since the current contribution can still proof useful in the future, but it diminishes the significance nevertheless.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact was sufficiently addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see ethical concerns regarding the presented work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is somewhat borderline. The paper shows an interesting connection between distributional robustness and MaxEnt IRL, showing that MaxEnt IRL is a special case of distributionally robust IL. Using DRO in IRL is interesting & underexplored. The paper also has a minor contribution of making distributionally robust IL more efficient. The main downsides of the paper are as follows. First, the approach is limited to discrete state and action spaces, severely limiting its practical relevance. As a result, the experimental comparisons are also outdated. Second, the connection between DRO and MaxEnt IRL has limited technical difficulty since it simply invokes a lemma about exponential families (though simplicity can also be viewed as a good thing).

Overall, this is a scenario where the paper is correct and generally well-executed but has potentially limited expected impact and limited technical difficulty. Given that impact is subjective and at least half of the reviewers found the ideas to be interesting, it seems like others in the community would also get value out of the paper.
Decision: Accept (Poster)