Distributionally Robust Imitation Learning
Authors: Mohammad Ali Bashiri
Brian D Ziebart
Xinhua Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Imitation Learning
Inverse Reinforcement Learning
Robust Learning

Abstract: We consider the imitation learning problem of learning a policy in a Markov Decision Process (MDP) setting where the reward function is not given, but demonstrations from experts are available. Although the goal of imitation learning is to learn a policy that produces behaviors nearly as good as the experts’ for a desired task, assumptions of consistent optimality for demonstrated behaviors are often violated in practice. Finding a policy that is distributionally robust against noisy demonstrations based on an adversarial construction potentially solves this problem by avoiding optimistic generalizations of the demonstrated data. 
This paper studies Distributionally Robust Imitation Learning (DRoIL) and establishes a close connection between DRoIL and Maximum Entropy Inverse Reinforcement Learning. We show that DRoIL can be seen as a framework that maximizes a generalized concept of entropy. We develop a novel approach to transform the objective function into a convex optimization problem over a polynomial number of variables for a class of loss functions that are additive over state and action spaces. Our approach lets us optimize both stationary and non-stationary policies and, unlike prevalent previous methods, it does not require repeatedly solving an inner reinforcement learning problem. We experimentally show the significant benefits of DRoIL’s new optimization method on synthetic data and a highway driving environment.

Distributionally Robust Imitation Learning
Mohammad Ali Bashiri
Brian D. Ziebart Department of Computer Science University of Illinois at Chicago Chicago, II. 60607
Xinhua Zhang
(mbashi4, zhangx, bziebart/@uic.ed
Abstract
We consider the imitation learning problem of learning policy in Markov Decision Process (MDP) setting where the reward function is not given, but demon- strations from experts are available. Although the goal of imitation learning is to learn policy that produces behaviors nearly as good as the experts' for de- sired task, assumptions of consistent optimality for demonstrated behaviors often violated in practice. Finding policy that is distributionally robust against noisy demonstrations based problem by avoiding optimistic generalizations of the demonstrated data. This paper studies Distributionally Robust Imitation Learning (DRoIL) and establishes a close connection between DROIL and Maximum Entropy Inverse Reinforcement Learning. We show that DROIL can be seen framework that maximizes a generalized concept of entropy. We develop novel approach to transform the objective function into convex optimization problem over a polynomial number of variables for a class of loss functions that are additive over state and action spaces. Our approach lets us optimize both stationary and non-stationary policies and, unlike prevalent previous methods, it does not require repeatedly solving an inner reinforcement learning problem. We experimentally show the significant benefits of DRoIL's new optimization method on synthetic data and a highway driving environment.
adversarial construction potentially solves this
Introduction
We consider the imitation learning setting of learning to perform task based only demonstrations that are provided by experts. There are two main approaches often considered for this learning problem: Behavioral Cloning [19] and Inverse Reinforcement Learning [18] [I] 24] [14]. In behavioral cloning, the learner attempts to learn a policy in a supervised learning manner, in which a direct mapping from states to actions is estimated from the demonstrated trajectories. Behavioral cloning, while simple, often generalize poorly when attempting to predict goal-directed sequential decisions due compounding
caused by covariate shift and only tends succeed when given large amounts of data [22] [23]. Alternatively, inverse reinforcement learning (IRL) rationalizes demon- strated trajectories by estimating reward function that makes the expert's policy optimal. The problem of determining the reward function is inherently ill-posed, since a single policy can be optimal for multiple reward functions.
To obtain a unique solution IRL, many methods have been proposed, with the maximum entropy principle [31 32 and margin maximization [20] being two widely employed methods among researchers. Maximum (causal) entropy IRI. (32], in particular, seeks an entropy-maximizing distribution sequences that matches expected feature counts [I] with those observed from demonstrations. For imitation learning in very high dimensional and continuous spaces, where function approximations such deep neural networks are often used, IRL methods have generally
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

been less efficient than behavioral cloning methods 13 since they require reinforcement learning as however, seem more likely be effective.
an inner loop. Recent adversarial IRL methods [T] 9 25
common assumption in imitation learning is that all expert behaviors have the same level of trustworthiness and are optimal/near-optimal [30.. However, it is common for noisy expert behaviors to violate this optimality assumption in practice. Thus, relying heavily on the optimality of the expert's 's behavior may degrade an imitation learner, making it prone to failure [30]. Existing methods have proposed to inject noise into the expert's 's policy demonstrations to obtain a more robust policy [13.
or to train a discriminator that distinguishes between expert trajectories and generated trajectories
that learns reward functions that are robust to changes in dynamics
An alternative approach to obtain robust policy, which we adopt in this work, is search for policy that is distributionally robust given the training data. For this purpose, the learner's policy is obtained by solving a game between a learner and an adversary [5], where the learner seeks minimize loss defined between them, and the adversary seeks to maximize this loss by choosing a distribution over policies subject
set of constraints that match statistics from the training data. This approach leverages two uncertainty sets as opposed to typical Distributionally Robust Optimization (DRO) where the uncertainty set is only defined over the adversary (demonsttatoo--ssiimaoor and the learner's policy is assumed to have a specific parametric form. Previous work (5) proposed to solve imitation learning under an adversarial formulation using the Double Oracle method [17] to solve the corresponding optimization. However, this method may take up to exponential time in the size of state-action space and decision horizon.
methods 4 6
In this paper, we connect Distributionally Robust Imitation Learning (DROIL) and Maximum Entropy Inverse Reinforcement Learning (MaxEnt IRL) by showing that MaxEnt IRL. is special case of DROIL when a certain loss function and policy description is used. We then show that DRoIL is a framework for maximizing a general entropy function that is defined based on a particular loss of interest. We then cast DROIL's is objective function into a convex optimization problem over a polynomial number of variables, which is simpler understand and implement and also significantly improves the training time. We extend the formulation to stationary policies, which enables us to experimentally show the benefits of learning a robust policy in highway driving simulation.
2 Preliminaries
We model sequential decision making problems using discrete Markov Decision Processes (MDPs).
A MDP M is specified by tuple (S,A, T, R, Po) where: S and A are state and action (control input) spaces (assumed to be finite, (S), | < oo); Po is the initial state distribution; I represents the transition probabilities, the state distribution upon taking action a in state 8, P(s la, s); reward function R : S x A + IR; and discount factor 7 E (0, 1]. We assume feature vectors & ; S x A + [0, 1] over state- action pairs that capture the most salient properties distinguishing preferred and nonpreferred trajectories where R can be written as (linear) function of these feature vectors given a reward vector w: R(s, a) w (08.00 a).
A policy π E II is the probability of taking action a in state 8, π T(a, s) P(als) and II represents the set of all possible stochastic policies. Demonstrations by an expert are given as a set of trajectories D = {71 horizon T. From an optimization perspective, Imitation Learning with a General Loss, finds a policy f that minimizes the difference between learner behavior and expert behavior Tg:
TTh J. A trajectory is a sequence of state-action pairs T (so, "..... ST, uT) over
A E argmin L(TT, TT)
where L measures the dissimilarity between two policies' behaviors.
Having access to only sample demonstrations of a policy for training, it is essential to quantify the behavior of policy. Commonly used in behavioral cloning, one approach is to measure the marginal distribution of states and actions, P(s, a) induced by policy. Note that in the case of infinite-horizon MDPs, expected (discounted) number of visits to state-action pairs is used. A more desirable measure for long horizons is the expectation of trajectory features (1] This approach is often used in inverse reinforeement learning, where reward function of the features is learned. The expected (discounted
We overload L for any type of loss in different contexts (e.g.. ... loss between deterministic policies states).
2

---

if y E (0, 1)) feature of a policy x (similar to ① is defined as:
p E
r (sr, ai) ai) RR.
3 Robust Imitation Learning and Maximum Entropy
Foundational work 27 [2] has developed a close relationship between robust Bayes decisions. maximum entropy, and minimizing worst-case expected loss also known adversarial learning. Several works have followed this framework of robustness for different supervised learning problems, such as cost-sensitive classification (2], multivariate loss prediction [29]. ordinal regression [7]. and
graphical models [ For the imitation learning setting, [5] has applied the adversarial learning framework on inverse reinforcement learning problem for restricted class of loss functions and proposed double oracle algorithm 17 to solve the corresponding optimization problem.
In the following, define general framework of distributional robustness for imitation learning obtain the policy that robustly minimizes an imitative loss:
Definition 1. Given an imitative loss function L that measures the distance of two policies' behavior, Distributionally Robust Imitation Learning (DRoIL.) defined two-player zero-sum game between the learner and the demonstrator, in which each player chooses a distribution over control policies constructing a stochastic policy, ft and E E, then the players receive the loss between the behaviors L(ft, #) as their payoff. The minimax strategy for the learner is given by:
min max C(ă, #); RE
where E is set of constraints churacterized by the demonstrated data.
Generally, E can be in the form of moment matching in ( that is commonly used in inverse reinforcement learning:
T
# E
E
Σ (oss a)) = E 1=0
a IE. IT.
where T represents the policy that demonstrated trajectories are generated from.
This formulation assumes that except for certain properties of the limited samples of available demonstrated behavior, the demonstrator's 's policy is the worst-case possible for the learner. This approach avoids generalizing from available demonstrations in optimistic manner that may be unrealistic and lead to a policy that does not work well in practice, especially in situations where there is noise in the demonstrated behaviors.
The minimax formulation in Definition D closely related to the principle of maximum entropy, which is used in Maximum Entropy Inverse Reinforeement Learning (MaxEnt IRL) [31. MaxEnt IRL provides a probabilistic approach under the constraint of matching the reward value of demonstrated behavior to resolve the ambiguity in choosing distribution over decisions. Under this model, trajectories with equivalent rewards have equal probabilities, and trajectories with higher rewards are exponentially more preferred according the following:
P(C (W)
Oc.
(3)
Z(w)
where Ço and ΦG.. represent trajectory and its corresponding sum of features.
Note that distribution over trajectories (paths) is alternative policy description that provides a stochastic policy where the probability of an action is weighted by the expected exponentiated rewards of all paths that begin with that action:
P(alw)
Σ P(C/w).
C:RES

---

To show the connection between DROIL and MaxEnt IRL, we employ tool from the general theory of exponential families [ that shows for certain classes of two-player zero-sum games, there exists a parametric distribution for the minimax strategy, as shown in Lemma]] [
Lemma 1 (Barddrff-Nielsen (... Let p E E be a probability distribution over space X, where: E =p {p: (T())) = C} describes mean-value constraint; T(X) represents a vector value statistic; and C is a constant. Let q be also a distribution chosen from the set of all probability mass functions. For the maximum entropy distribution, max miny E[-log (XX)), p' = 9" exists as parameterized function of the form p" exp(a0 + T(X) with parameters og and a.
Equipped with Lemma] T
we develop a connection between DROIL and MaxEnt IRL in Theorem
Theorem 1. The stochastic policy P(G/ () obtained from MaxEnt IRL in Equation is obtained from DRoIL minimax strategy in Definition 7 when the logarithmic loss is used.
Proof. We construct stochastic policy in Definition Ia probability distribution over all possible trajectories {(I,,.,, SMr.. Let मे = P()) and म = Q{) be the probability distributions of the learner and the demonstrator, respectively. The expected feature of demonstrator now can be written
as P = Σ. (()))ç which resembles the mean value constraint in Lemma | therefore, for logarithmic loss, (G) = (G)) = exp(wo + w TT
We can generalize beyond the logarithmic loss and resulting maximum entropy policy (Theorem to other losses of interest by using Generalized Entropy functions, H(P) == infq ()...
Proposition ! describes the relation between DRoIL and generalized maximum entropy.
Proposition 1. For any policy descriptions and loss functions that DROIL in Definition / has Nash equilibrium, से is robust action and is the maximizer of a generalized entropy function.
Proposition ! provides a general approach resolve the ambiguity of matching constraints-wher many policies lead to the same feature counts- by choosing either policy स that does not exhibit any additional preferences beyond matching feature expectations with respect to a loss of interest L (maximum generalized entropy) or policy A that minimizes the worst-case expected loss. This can be seen as Maximum Generalized Entropy Inverse Reinforcement Learning, where the choice of loss function is not restricted to the logarithmic loss function.
We extend the following lemma that describes class of loss functions and policies for which the solution of the game in Definition [ exists, thus Proposition [ be applied.
Lemma 2. For the game in Definition D where learner āt and demonstrator ă simultaneously choose a stochastic policy, if the loss function is additive and the payoff can be written as a bilinear function: Σ Eø=1 j the game has a solution and minimax (maximin) strategy producing the solution as long as and are finite.
Proof. In his generalized minmax theorem, Von Neumann arbitrary nonempty closed, bounded convex sets of actions have saddle points as long m and n are
28 shows that bilinear games with
finite. Assuming the demonstrated data are generated from a policy (optimality not necessary) the
convex moment matching constrained in Definition keeps E closed, bounded, and non-empty. O
One illustrating example let each player's pure strategy to be a deterministic policy 5 and define the loss as the expectation over a given loss between each individual deterministic policies. A mixed strategy now represents stochastic policy n, which is probability distribution over the set of all deterministic policies and Equation (2) can be written as a bilinear game with where lij is L(),5)).
Example 1.
payoff Em (ey
Lemma [ describes a restricted class of loss functions (e.g., that the logarithmic loss does not belong to) for which the resulting policy is the maximizer of more flexible concept of entropy. A class of loss functions of interest are loss functions that are additive over state and action spaces. Similar to Example
[] previous work [5] constructs policy using distribution over deterministic policies and for additive loss function by writing Equation (2) as a bilinear game and solving the corresponding optimization. We also focus on this class of loss functions. However, we provide an efficient optimization algorithm for this class of loss functions that avoids the exponential worst-case time
complexity of the previous double oracle [17 approach.
4

---

Learning and Inference
To solve the optimization problem in Equation (2). one needs to specify the policy description and the distance measure between two policies' behaviors L. The choice of π and can result in different algorithmic approaches.
4.1 Double Oracle
31
82
8,
In [5], the authors construct the learner
and the demonstrator stochastic poli-
82
[(81. 81)
[(8,, 82) +v((22)
((
cies mixtures of deterministic non- stationary policies. Concretely, let Y {ō1. J..... be the set of all possible non- stationary deterministic policies, then A == P(fo E T) and F P(f6 E )) Note that ['] E O(I which exponential in size of state space and time horizon, S). T. Assuming the loss function additive over state-action pairs, [5] shows that Equation (2) can be written as bilinear game:
+ (81)
+8((8,)
[(02, 61) +w(6))
[(d2. 82)
((62.0g)
82
+v((22)
+w(6,)
[ 81) (x)()))
[(oi, 52) (62)
C 8,) (1,)
8,
Table I: Payoff matrix of G(# A tT ww with loss function L. deterministic policies 5 and Lagrangian potentials sb.
G(fa.Halw)
min min max 5A
ΣΣ p())) ((0)))(6)) + Ep(5)) n
p(55) w B[Hlö,] w fi. ((,,))
This will result in matrix game of exponential size in the number of states S and time horizon, as shown in [ To obtain the Lagrange variables w, this matrix game needs to be repeatedly solved to compute the gradient with respect to w. This requires solving a linear program with (([A))S variables with simplex constraint, which is impractical for even modestly sized problems. To mitigate this problem, they employed the double oracle method [17 in an attempt to construct smaller sub-portion of the matrix by gradually adding pure actions through solving a time-varying control problem. However, there is no guarantee that the support set of Nash-equilibrium of the defined game is small and the algorithm may need solve up to exponential number of time- varying optimal control problems.
4.2 State-Action Distribution
We propose an alternative way to transform the optimization in Equation (2) into a convex problem. Our approach is based on using state-action marginals to construct the stochastic policies of learner and the demonstrator, where the number of required variables is linear in [S] and [A], and it can also be extended to stationary policies. the following, we first look at the non-stationary case and then extend our method to stationary policies.
policy π induces a probability distribution Pt(s) over the states of an MDP M at each time step. State-action marginals are similarly defined as Pr(s, a) = (a)ss) A valid state-action marginal is set of simplices corresponding
each time-step that satisfy the Bellman flow con- straints for given MDP M. Let Pa a) E A be the probability of state-action pairs (a, b) at time t; valid marginal distribution P E satisfies the following affine constraints: for all s', Σ,, (s, P) ((s'|ss a) = Σ. (s' a), where P(s'(s,a) E I' and Ω represents the set of all valid marginal distribution for given MDP M.
Employing state-actis action marginals allows to write the objective function in Equation problem of ([A[ variables as shown in Theorem 2 in a vectorized form.
convex
Theorem 2. For additive loss function over slutes and actions L, Solving DROIL optimization in Equation (2) is equivalent to solving the following convex minimax problem over marginal state-action probabilities of the learner P (Pi.... PT) and demonstrator Q (q1...., 4T)
5

---

parameterized with Lagrange multipliers w:
T Σ t=
min max min w QER PER
P. Lq. + To qi w p.
(4)
where Pt (similarly q:) is a vector of size |A|||] storing marginal probability of state-action pairs at time t: (s,s, a); is the general loss function that is defined over state-action pairs L [LA|[S] X [A|]S). á is a x [||SS] matrix storing the feature function for each state-action pair. We denote system dynamics with I', which is a matrix of size [A||S] [8] storing transition probabilities P(s'|s, a).
Intuitively, each player in the above formulation searches over valid state-action distribution to reach an equilibrium. One of the benefits of constructing policy using marginals is that the feature expectation can written as an inner product of the state-action distribution and the state-action feature vector. Thus, the demonstrator's expected feature is ((Q) Ei= Lqq, and the feature
matching constraint is realized by minimizing w (Et= Φq. ja) over dual variables Writing the objective function Equation (2) in terms of state-action marginal probabilities reduces the number of variables needed to represent the equilibrium from O A (S(T') to O(IA[|))T). The unconstrained optimization over dual variables w can be solved using any gradient descent method where the gradient is given by Σ 2qï current wy:
jt and Q" is the solution of the following game for the
T Cq: + w. T 9:
G(w,)
max min
Σ
Pi
(5)
Qef PES
4.3 Stationary Policy
Our approach extends with some modifications to the stationary policy setting. Stationary policies are desirable because they are simpler to describe, and are more natural and intuitive in terms of the behavior that they prescribe. Similar to state-action marginals, we utilize an occupancy measure Par S x A + U characterize stationary policy æ. is defined as the expected (discounted) number of visits to state-action pair (s, a), when following policy π and can be written as a feasible set of affine constraints:
G = (p p p0:
Σ pls' po(s') [7P(61ssc))p( EsE)),
8,0.
where P(s'ss, a) E I' and po is the distribution of starting states.
For a given additive loss L. with the use of an occupancy measure, we write the expected loss B[[(T1, T2)] as PT, CpT, and the expected discounted feature μ, 4 Ps. Theorem 3 shows how we can write Equation (2 as a convex optimization using occupancy measure:
Theorem 3. For additive loss function
state and actions f, solving DRoIL optimization in Equation (2p is equivalent to solving the following convex minimax problem over occupancy measures of the learner P and demonstrator Q:
min max min P T LQ +w+wl. w - W
DEG PEG
The convex optimization in Equation 7 can also be solved using any gradient-based method where the gradient obtained by solving constrained game with O(AA]|S)) variables.
4.4 Inferred Policy
In the non-stationary case, after obtaining w", one can use either Q' or P as the produced non- pe(a.s)
stationary Markovian stochastic policy by computing π (a(sss)) (r(a)s) Σ. (a(.s) Q corre- sponds to the policy that maximizes the generalized entropy and P corresponds to the policy that has minimized the worst-case expected loss.
For the stationary case, [26] has proved that there a one-to-one mapping between & and II, in P(a.s) is the only policy that results in P.. sense that for an occupancy measure P E G, (a]s) ع Σ. P(a.s)
Therefore, one similarly use P Q as the produced stationary stochastic policy.

---

For either case, assuming the reward function can be written R(s, a) w o(s, a), then w" plays the role of the reward weight vector that rationalizes the demonstrated behaviors. Consequently, we can use w' Φ to obtain the optimal reward function R. and use an MDP solver to obtain the (deterministic) corresponding policy.
5 Optimization
In both stationary and non-stationary cases, any gradient descent method can be used to optimize over dual variables w. To compute the gradient, one needs to compute Q" for non-stationary and Q* However, since w is unconstrained, by adding a norm w with hyperparameter A, one can directly solve and replace it in the objective. Since the optimization algorithms similar in both cases, we only mention the non non-stationary case:
A //wl
min max min QE PEn
Σ P. Lqc + w t q:
2
and setting: w= /
Li q
we have:
T
max QEΩ
91
min
p Lq.
(9)
2X
PEB
1=0
which is constrained quadratic optimization in Q. Using Danskin's 's theorem, the gradient of qu is given by
1 T - Σ T
where p" be found using linear programming (linear objective with affine constraints) effi- ciently using standard linear programming toolbox. An alternative approach to solve the dual of optimization over Pt and maximize it along with Q.
Algorithm Distributionally Robust Imitation Learning (DRoIL)
Input: D {r', ... , T,Po
Initialize Q" compute ja using D, and set i 0
repeat
Compute v.f(Q') in Equation
i= 1= Using V (Q') calculate with Q++1
if QQ+H E Ω then
+++ Q++1
else
i++1_ project (Q++1) where projection function is defined in Equation (10
end if until convergence
Projection Step
At each iteration of the algorithm, we need to project Q convex domain maintain a valid state-action distribution given I' and the initial state distribution. Essentially,
min Q 2
Σ
|ل
s.t. Q Ω
(10)
from which, by using Lagrangian method and strong duality, we obtain:
max min QEA
q qill2 (q._,- q. Z)ve,
(11)
where Z operator computes the state distribution: q.- === (s, a). To compute the gradient V. a quadratic program over Q with probability simplex constraints needs to be solved. This can be analytically determined by sorting each qr, which takes [[A (o)()))|AA)) time.
7

---

Experimental Results
In our experiments, we compare DROIL with prior methods several imitation learning tasks. We investigate: 1) How our convex optimization improves the training time compared to the double oracle method; 2) How the choice of loss function affects the performance of DRoIL; and 3) How accurately DROIL predicts actions compared to other IRL. methods.
6.1 Training Time
To compare the training time of our pro- posed convex optimization with the double oracle approach [5], we adopt their experi- mental setup in GridWorld. In this experi- ment, trajectories
DROIL
DROIL
DO
Do
Size
Time
Cost
Time
Cost
128
1.6
-7.39
30.1
7.39
collected from simu- lated navigation across a discrete 2D grid where the agent starts from random start- ing point, and navigates through the grid to reach specified target location. Taking step in the grid has a cost and the agent's goal to reach the target location while minimizing the accumulated navigation cost (maximizing the reward). This problem can be formulated as an optimal sequential decision-making problem in a finite Markov decision process where the optimal policy non-stationary. We consider linear cost function C(s) w ((s)) + ((s). where feature function ( a) and weight vector w are drawn from U(0, 1)d, and UO, 1). Transition function is non-deterministic with parameter Pm E (0, 1] which navigates the agent to randomly choose neighbors with probability (1 Pm).
432
1.8
-21.61
42.1
211.74
1024
6.0
-20.0
141.1
- 20.0
2000
98.6
-30.1
608.1
30.1
3500
496
39.2
2020
39.1
5500
881
-58.7
4970
- 58.9
Table 2: Elapsed time in second until convergence with 10-3 tolerance.
The loss function for this experiment is set to E
x,) - (Y,
Y,)² where
(X, Y) represents the grid position the agent. We generate trajectories from the optimal policy that obtained by solving the true reward function and train DRoIL. and DO until convergence. Along with expected loss, we report the elapsed time that it takes to converge for our proposed method and double oracle (DO) in Table 2 averaged
eight repetitions of the experiment for this result and our later results. As Table 2 shows, our proposed method requires significantly less training time for the same performance and scales very well with the size of state space.
6.2 Loss Function Effect
0.5
0.45
action-loss
For the second question, we show that different choices of loss function result in different performance for the learned policy in the imitation learning setting. There- fore, the choice of loss function provides extra tool to incorporate certain domain knowledge, and design problem-specific loss that potentially results in better produced policies. For the purpose of comparison of different losses, revisit the GridWord environment, however, we train several stationary policies with differ- losses. We consider 0-1 loss, which equally penal- izes any mismatch between state-action pairs; action- loss, which incurs loss only when action differs from another pair in the same state; random loss, which drawn from a uniform distribution; and finally Eu- clidean distance between two positions in the grid. It clear from Figure [ that the policy produced from Fuclidean loss outperform other policies from other losses, which shows the benefit of using a task-specific loss function.
random
0.4
auclidean
0.35
misprediction rate
0.3
0.25
0.2
0.15
0.1
0.05
0
8
16
32
64
examples
Figure/ Performance of DROIL. when different loss functions are used.

---

6.3 Highway Driving
To evaluate DRoIL in complex environment with more realistic behavior, we compared it with several other IRL methods in a highway driving simulator with non non-linear reward function: MMP [0)] the projection algorithm of Abbeel and Ng M, and LEARCH (2]]
Following the setting in Levine et al. [66. the task is to drive a car on three-lane highway which the agent switch lanes and drive up four times the speed of traffic while all other vehicles move at a constant speed. The set of features includes the distance to the nearest vehicle each lane (in front and behind), current speed, and current lane. We also evaluate each method on the original environment and on four additional random environments, denoted as "transfer". We set a uniformly random loss for DRoIL and train all methods using examples sampled from the stochastic MaxEnt IRL policy which can intuitively be viewed as noisy samples of an underlying optimal policy. To evaluate the performance of each method, we use the misprediction rate, which is defined as the ratio of incorrect actions compared to the optimal policy, and the expected value difference, which measures the suboptimality of the learned policy under the true reward. Since DROIL. able to produce stochastic policy, with the same argument from [16], we could evaluate the optimal stochastic policies. However, this would unfairly penalize other methods. Therefor, we first obtain the reward weight vector and find the optimal deterministic policy the corresponding reward function. Then, we measure its expected sum discounted rewards under the true reward function, and subtract this quantity from the expected sum of discounted rewards of the optimal policy.
original
transfer
Abbeel & Ng
Abbear & Ng
MMP
MMP
200
LEARCH
200
LEARCH
expected value difference
DRoiL
expected value difffrencc
DRoiL
150
150
100
100
50
50
8
16
32
64
2
4
16
32
64
examples
examples
Figure 2: Expected value differences for 64-car-length highways with varying example counts. Lower values are better.
As shown in Figure 2 and Figure 3 DROIL performs very well in terms of the obtained reward and the accuracy the produced policy in both the original and transfer environments. contrast, we find that MMP and Abbeel & Ng's approach degrade as the number of examples increase. This matches theory since the suboptimality of the demonstrations becomes more apparent as the number of examples increases. This indicates that under noisy demonstrations (samples from a stochastic policy), a robust approach has the potential to outperform alternative approaches.

---

original
transfer
0.45
Abbaol Ng
Abbeel Ng
0.4
MMP
0.45
LEARGH
LEARCH
0.4
0.35
DiRett.
DESOr
misprediction rate
0.3
misprediction rate
0.35
0.3
0.25
0.2
0.2
0.15
0.15
0.1
0.1
0.05
0
2
4
8
16
32
64
2
4
16
32
64
examples
examples
Figure 3: Misprediction rate results for 64-car-length highways with varying example counts. Lower values are better.
7 Discussion & Conclusion
We demonstrated connection between DROIL, which accepts any loss of interest, and Maximum Entropy Inverse Reinforcement Learning- of the widely used IRI. approaches-wwiel robustly minimizes logarithmic loss. We showed that MaxEnt is a special case of DRoII. framework when logarithmic loss function used and showed that DRoIL can seen as the maximizer of generalized concept of entropy. We provided novel approach to cast DRoIL's objective into convex optimization over a polynomial number of variables and experimentally showed our proposed algorithm provides faster training time. DRoIL is naturally designed to perform robustly against noisy demonstrations. Our experiment in the highway driving task showed that when demonstrations noisy. it robustly learns appropriate policy.
Improvements imitation learning have the potential for both societal benefits and harms. For example, better imitating surgeons could scale their abilities to a broader populations that are medically under-served. Enabling robots that better imitate effective soldiers could cause great harm if used inappropriately. Like all general purpose tools, avoiding intentional harms while still providing benefits is an unsolved challenge. We take the position that providing methods that are more robust to noise will help to avoid unintentional harms-the the application of methods in well-intentioned manner that fail to maximize their benefits and may instead produce harms through their fragility.
Our presented experiments are restricted to discrete/low-dimensionn dimensional decision processes, For future work, we are interested in finding a connection between DRoIL and the Generalized Exponential family and applying DRoIL on very high dimensional state and action spaces that need function approximators such as deep neural networks.
10

---

Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant Nos. 1652530 and 1910146.
References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the International Conference Machine Learning, pages 1-8. ACM, 2004.
[2] Kaiser Asif, Wei Xing, Sima Behpour, and Brian D. Ziebart. Adversarial cost-sensitive classifi- cation. In Proceedings of the Conference Uncertainty in Artificial Intelligence, 2015.
[3] Ole E Bamdorff-Nielsen Information and exponential families. 1978.
[4] Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2)341-357, 2013.
[5] Xiangli Chen, Mathew Monfort, Brian D Ziebart, and Peter Carr. Adversarial inverse optimal control for general imitation learning losses and embodiment transfer. In Proceedings of the Conference Uncertainly in Artificial Intelligence, 2016.
[6] Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research, 58(3):595-612, 2010.
[7] Rizal Fathony, Mohammad Ali Bashiri, and Brian Ziebart. Adversarial surrogate losses for ordinal regression. In Advances in Neural Information Processing Systems, pages 563-573,
2017.
[8] Rizal Fathony, Ashkan Rezaei, Mohammad Ali Bashiri, Xinhua Zhang, and Brian Ziebart. Distributionally robust graphical models. In Advances in Neural Information Processing Systems, pages 3344-8355, 2018.
[9] Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprini arXiv: 1.03852, 2016.
[10] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the International Conference on Machine Learning, pages 49-58, 2016.
[11] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710. 11248 2017.
[12] Peter D. Grünwald and Phillip Dawid. Game theory, maximum entropy, minimum discrep- ancy, and robust Bayesian decision theory. Annals of Statistics, 32:1367-1433 2004.
[13] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pages 4565-4573, 2016.
[14] Rust John. Maximum likelihood estimation discrete control processes. SIAM Journal on Control and Optimization,
1988.
[15) Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg, Dart: Noise injection for robust imitation learning. arXiv preprint arXiv: 00337,,9 2017.
[16] Sergey Levine, Zoran Popovic, and Vladien Koltun. Nonlinear inverse reinforcement learning with Gaussian processes. Advances in Neural Information Processing Systems, 24:19-27, 201
[17] Brendan McMahan, Geoffrey J Gordon, and Avrim Blum. Planning the presence of cost functions controlled by an adversary. In Proceedings of the International Conference on Machine Learning, pages 536-543, 2003.
[18] Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Proceed- ings of the International Conference on Machine Learning, pages 663-670, 2000.
[19] Dean A Pomerleau. ALVINN: An autonomous land vehicle in neural network. In Advances in Neural Information Processing Systems, pages 305-313, 1989.
[20] Nathan Ratliff, J Andrew Bagnell, and Martin Zinkevich. Maximum margin planning. In Proceedings of the International Conference on Machine Learning, pages 729-736, 2006.
11

---

[21] Nathan D Ratliff, David Silver, and J Andrew Bagnell. Learning to search: Functional gradient techniques for imitation learning. Autonomous Robots, 27(1):25-53, 2009.
[22] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 661 661-668, 2010.
[23] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings the International Conference on Artificial Intelligence and Statistics, pages 627-635, 2011.
[24] Stuart J Russell. Learning agents for uncertain environments. COLT, volume 98, pages 101-103, 1998.
[25] Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In Proceedings of the International Conference on Machine Learning, pages 10022-10032, 2021.
[26] Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear programming. In Proceedings of the International Conference Machine Learning, pages 1032-1039, 2008.
[27] Flemming Topsøe. Information theoretical optimization techniques. Kybernetika, 15(1):8-27, 1979.
[28] John Von Neumann. Uber ein okonomsiches glcichungssystem und eine verallgemcinering des browerschen fixpunktsatzes. In Erge. Math. Kolloq. volume 8, pages 73-83, 1937.
[29] Hong Wang, Wei Xing, Kaiser Asif, and Brian Ziebart. Adversarial prediction games for multivariate losses. Advances in Neural Information Processing Systems, pages 2710-2718, 2015.
[30] Jiangchuan Zheng, Siyuan Liu, and Lionel M Ni. Robust Bayesian inverse reinforcement learning with sparse behavior noise. In Proceedings of the AAAI Conference on Artificial Intelligence, 2014.
[31] Brian D Ziebart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, pages 1133-1438, 2008.
[32] Brian Ziebart, Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In Proceedings of the International Conference on Machine Learning, 2010.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Theoretical claims are supported by theorems and experimental claims supported empirically.
(b) Did you describe the limitations of your work? [Yes] See Section 7
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We discussed them in each Theorem
(b) Did you include complete proofs of all theoretical results? [Yes] We include these in Supplementary materials.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We provide these in the supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We describe these details in the supplementary materials.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
See the Supplementary Materials.
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13

---

A Proof of Theorem 2 & 3
Suppose the loss function C is decomposable over state action pairs, then we can write, Equation as following:
8, A, = -
min max
(12)
8.4
T
subject to: Σ
E
= -
[=1 8,0
By introducing dual variables w for the feature expectation constraints, the Lagrangian function of Equation 12) is given by:
min max min
Σ
P(S, = 8, = = s,A, = = =
(13)
8.0
8,0
T
Σ 8,0
P( = 8, = ())(8 i4)
w
The optimization in Equation 13 is over ग and #. However, the objective function Equation (3)
decomposes over the state-action distribution induced by policies Å and #. We directly optimize over the marginals:
min
max
min
p((5.0)((8....0))0.....
(14)
(ps "r.PP.....PP) (4s (T)
R.
3,
T
+ w
Σ 3
qe a))(3, ūi)
where (s, à) = P(š. = 8, A, = a) and g: (3. () P(š = A -
=ū) a). This optimization needs to be over valid state-action marginals (marginals induced by policy). So the following constrained need to be satisfied:
Σ (8)18)))))188
Emi(3'
1,8'
8.0
Similarly for
Σ (33 (3) = En ()(',a)) V t,
Since Equation
14
is convex in all variables Pt. gt. and w, the order of optimization can be changed:
min max min QEN Pen
Σ
Pa(s, 5, 8, t.w. Σ
(.( w j.
8,8
where P = (pI, (p.,P2,...,PT' and Q = (qi, QT)
The proof for stationary case is similar.
14


Reviews:
[{"review_id": "srvuehhiCM", "reviewer": "Reviewer_4wnP", "summary": "This paper studied distributionally robust imitation learning, and connects the formulation with maximum entropy inverse reinforcement learning.  Based on the objective, the authors transform the optimization problem into a convex optimization problem and it can be efficiently solved by using standar linear programming toolbox. Empirical results show the performance gain using the new formulation compared with prior works. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studied distributionally robust imitation learning, where the learners and the demonstrators play a zero-sum game, thus the new learned policy can be robust to distribution shift and noisy demonstrations. \n\nOriginality: The formulation is new to me, and indeed the original idea of distributionally robust optimization can solve the distribution shift problem, and can obtain a model that are robust to noisy samples, which would be a good direction to explore for imitation learning. \n\nQuality and clarity: The paper is well written, and the content is well organized, from motivation to problem formulation, and then to the optimization solutions. Finally empirical results show the performance gain of the proposed paper. \n\nSignificance: The paper introduces a new perspective for robust imitation learning. while there is a major concern about the scalability of the proposed method, which may limit its general usage in general imitation learning application scenarios. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "My major concern is the scalability and empirical evaluation of the proposed method. From the paper we can see that the proposed method can not be scalable to complex function approximators, such as neural networks. Besides, if we can many samples (millions of samples) for learning, how will the method perform?\nFurther, the evaluation domain is too simple, there is even no simple mujoco evaluation experiments.", "review_text": "", "extra_scores": {"ethical_concerns": "There is no ethical concerns. ", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "UIehFsd9KEq", "reviewer": "Reviewer_rEEu", "summary": "This work aims to address the problem of imitation learning where the expert demonstrations may not be optimal and instead near-optimal and somewhat noisy. This problem is approached through the perspective of distributionally robust optimization. This work extends the work of Chen et al. 2016, by reformulating the optimization problem in a manner that results in a significantly more efficient optimization process.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:**\n\nThis work operate under the formulation of Chen et al. 2016, so the problem formulation is not the novelty of this work. The key novelty of this work is in how the optimization problem is reformulated in order to obtain a more efficient optimization process, which is an important contribution.\n\n**Quality:**\n\nSoundness: Methodology and claims seem valid and sound.\n\nClaims Supported:\n- The improvements in runtime are clearly demonstrated in section 6.1\n- The performance of DRIL when compared to baseline methods is presented in section 6.3.\n\nMajor Comments:\n- Lines 363-369: I am not sure I understand why exactly your method is doing better here? Your method is still trying to match the features of the sub-optimal policy, so what in particular about your approach is leading to the improvements? It would be very valuable to include a discussion.\n- In section 4.4, you discuss how you can obtain 3 distinct policies from the optimization process. Q: max-ent, P: min worst-case, and lastly a deterministic policy obtained by optimizing the MDP whose reward is given by $\\mathcal{R}(s,a) = w^*\\cdot\\phi(s,a)$. First question: does $w$ define the reward for which Q is optimal, or P is optimal? Second comment: It would be nice if you could provide intuitions for when to use which one (between Q and P and deterministic from $w$). It would be great also if there were experiments that could verify your intuitions.\n- It would be nice to also include the max-ent and stochastic worst-case results (policies from Q and P) in section 6.3 as well.\n\n**Clarity:**\n\nThis work is generally well-written.\n\nMajor Comments:\n- Line 339: I'm not sure I understand how you can train your method using a uniform random loss in DRIL? Doesn't the loss need to convey some meaningful notion of similarity?\n- Line 338: Please describe what the transfer domains are more accurately instead of \"random environments\".\n- Line 344: Could elaborate 1-2 sentences in addition to saying \"with the same argument as [16]\".\n\nMore Minor Comments:\n- Lines 108-111: I'm not sure I understand what this sentence is trying to say \"This approach avoids....\". Could you clarify? Do you mean that it avoids generalizing (which requires optimism), and this pessimism is unrealistic and leads to a policy that does not work well in practice? Also, can you provide support/evidence for this claim?\n- In section 6.3, in the description of the highway environment, it would be useful to also describe how the states and actions are discretized. It would be useful for a reader such as myself not familiar with the MDP.\n- Lemma 1: Could you clarify the notation? You have not defined T and C. Also, is the order of max and min not the reverse of the order in equation 1?\n- Example 1: What is a pure strategy?\n\n**Significance:**\n\nI think the reformulation of the optimization problem of Chen et al. 2016 in the current form is a significant contribution. First, due to the significant speed-up, and second because this work's formulation appears to be more amenable to eventual application with continuous spaces and neural network function approximators.\n\nComments:\n- Even though the method is not concerned with continuous spaces and function approximators, as a discussion section I think it would be interesting to motivate why this work would be valuable to those settings. Namely, if we manage to transfer this solution to neural networks, would we be able to obtain a method that is importantly different from current adversarial inverse RL literatures (GAIL [Ho et al.], AIRL [Fu et al.], state-marginal matching [Ghasemipour et al.], etc.)?\n- One question I have is about author's intentions with section 6.2? I would like to make sure that I am not misunderstanding something. The ability to use differing loss functions does not appear be unique to your methodology. It seems that the same loss functions could have been used by Chen et al. 2016 as well? Am I understanding correctly or is the ability to use varying losses due to the contributions of your work?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe any negative impacts would be due to the application domain of the method and not inherently due to the work presented here. I do not have any immediate concerns.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not have any immediate concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "Lb5roj5ZnFX", "reviewer": "Reviewer_KGfa", "summary": "This paper considers the problem of robust imitation learning when demonstrations are noisy. The authors propose an adversarial imitation learning optimization problem that results in improved training time compared to a prior double oracle approach. Results show that DRIL outperforms several older imitation learni", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "How is Theorem 2 related to Syed et al. [25]? They also use the state-action occupancies to construct the bellman flow constraints to create a more efficient convex optimization problem. What is different about the current approach?\n\nThe paper only compares runtime against the Double Oracle method. It would also be good to compare against a method such as [25] which seems very similar as it also does not require an MDP solver in the inner loop.\n\nFurthermore, the later results only compare against MMP, Abbeel and Ng, and LEARCH. These are all very old algorithms and it's fine to compare against them but they are not considered state of the art. Better baselines including maximum entropy IRL, LPAL [25], and BROIL (Brown et al. 2020) mentioned below would strengthen the results.\n\nLines 363-369 do not seem correct. They state that the suboptimality becomes more apparent with more demos. However, the demos are from a stochastic max ent policy so optimal actions will always be taken more frequently than suboptimal ones so more data will be good since it will allow the truly optimal actions to be sampled more.\n\nThe paper states that DRIL performs robustly to noisy demonstrations. But it is not compared with Max Ent IRL or Bayesian IRL (Ramachandran et al. 2007) both of which are known to be robust to some noise in the demonstrations. These baselines should be added. Another missing related work/baseline is Brown et al. \"Bayesian robust optimization for imitation learning.\" NeurIPS 2020.\nIt is very similar in motivation as it seeks an imitation learning policy that is distributionally robust. It would be good to compare as a baseline and contrast with the current approach.\n\nClarity:\n\nLines 108-110: I don't know what is mean here. Especially the \"optimistic manner\" part. A little more intuition would be helpful.\n\nLine 339: what is meant by a uniformly random loss? How do you achieve good performance with a random loss function?\n\nFigures 3 have legends that are behind the actual lines making them hard to read.\n\nTypos:\n\nLine 264 *tasks\nTable 2 *seconds\nLine 345 *Therefore\n\n---\nPost Rebuttal:\nThank you for the author response. I think this is an interesting area of work. Unfortunately, I do not think this paper is ready for NeurIPS. Adding more baselines and better comparing to previous works mentioned above is needed. Also adding a more complex domain would significantly strengthen the paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Ar0cCnRE5hI", "reviewer": "Reviewer_4tjM", "summary": "The paper considers distributionally robust imitation learning (DRIL) which aims to find an imitating policy that is robust against changes in distribution of the expert policy within a feature expectation constraint. Solving this problem gives an imitating policy that minimizes the worst-case performance given a fixed feature map and a loss function. The paper proposes a computationally efficient algorithm to solve DRIL where the main idea is to optimize for state-action distributions instead of policies. The algorithm is empirically evaluated on synthetic and highway driving tasks. \n\nContributions :\n\n1. A theoretical analysis that gives two insights: 1) MaxEnt-IRL (Zeibart et al., 2008) solves the DRIL problem with the logarithm loss. 2) The solution of DRIL is guaranteed to exist for a class of bilinear loss functions.\n\n2. A computationally efficient algorithm that solves DRIL for MDPs with discrete state-action spaces. The algorithm improves upon the existing algorithm (Chen et al., 2016) in terms of computational efficiency by solving for state-action distributions instead of policies.  \n", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Distributionally robust learning is a promising approach for robust imitation learning. This paper presents a novel theoretical result and a novel algorithm for this. The writing is quite clear and the contents are well structured, but there are issues regarding mathematical clarity. The algorithm is derived using a standard optimization technique, i.e., projection gradient, which leads to a simple and computationally efficient algorithm.\n\nHowever, the significance of the paper is quite low due to two major issues: 1) lack of comparison against MaxEnt-IRL, and 2) practical limitations of the algorithm. I recommend rejection.\n\n1) The paper firstly claims that MaxEnt-IRL is equivalent to performing DRIL with a logarithmic loss. Then, the paper proposes the algorithm to perform DRIL with different losses. However, it does not explain issues of using logarithm loss for DRIL, i.e., issues of MaxEnt-IRL. Moreover, the proposed algorithm is not empirically compared against MaxEnt-IRL. \n\n2) The algorithm is applicable only to problems with a discrete state-action space. In addition, it also requires a transition probability matrix $\\Gamma$ and a fixed feature map $\\phi$. These are severe practical limitations when compared to recent researches on imitation learning that are applicable to both discrete and continuous state-action space and does not require a transition probability matrix and a fixed feature map, e.g., AIRL (Fu et al., 2017) which performs model-free MaxEnt-IRL with deep networks. \n\nMinor comments:\n\n- The expectation notation $\\mathbb{E}[\\cdot]$ is ambigious in the paper since the corresponding distribution is not defined. In particular, I do not understand the expectation in equation 1: $\\mathbb{E} [ \\mathcal{L} (\\hat{\\pi}, \\check{\\pi}) ]$. What is the random variable here and what is the distribution of the random variable for this expectation? I also find it strange that a similar objective in lines 78-79 does not have an expectation. \n\n- The solution $w$ in line 251 is missing a denominator $\\lambda$.\n\n- Algorithm 1 refers to $\\nabla_i f(Q^i)$ in equation 8, but $f(Q^i)$ is not defined anywhere. \n\n- The experiments are conducted with two simple tasks. I suggest including a large-scale task to make the paper more practically convincing. \n\n- Besides study the effect of loss function, it would be beneficial to also study the effect of feature map. That is, how does changing the feature map change the robustness. In addition, it would be interesting to study the effect of changing the constraint set $\\Xi$. For example, $\\Xi$ could be a norm-ball around the expert policy. \n\n## Update after rebuttal\n\nI read the rebuttal and other reviews. I thank the authors for the clarification in the response. However, the response does not change my opinion and I still rate the paper as a rejection.\n\nI understand that the paper focus on a particular setting of model-based distribution robust IL in discrete MDPs. However, I think this setting is too limited given the current research frontier in IRL/IL. In addition, evaluation of MaxEnt-IRL should be in the scope of this work since the paper relates MaxEnt-IRL to DRIL.\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "Around 4 hours "}, "extra_text": {}}]

Author rebuttal:

Meta-review: The rebuttal did not overcome the reviewers' objections. Notably, they are still concerned about the scalability of the proposed approach and the unconvincing experiments (notably the lack of comparison to maxEntIRL, given its relation to the proposed approach). Additional feedback is provided in some reviews (last section of the review).

There is a consensus that this paper is not ready for publication at Neurips, I therefore recommend rejection.
Decision: Accept (Poster)