A Ranking Game for Imitation Learning
Authors: Harshit Sikchi
Akanksha Saran
Wonjoon Goo
Scott Niekum
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Imitation Learning
Inverse Reinforcement Learning
Deep Reinforcement Learning
Robotics
Robot Learning

Abstract: We propose a new framework for imitation learning---treating imitation as a two-player ranking-based game between a policy and a reward. In this game, the reward agent learns to satisfy pairwise performance rankings between behaviors, while the policy agent learns to maximize this reward. In imitation learning, near-optimal expert data can be difficult to obtain, and even in the limit of infinite data cannot imply a total ordering over trajectories as preferences can. On the other hand, learning from preferences alone is challenging as a large number of preferences are required to infer a high-dimensional reward function, though preference data is typically much easier to collect than expert demonstrations. The classical inverse reinforcement learning (IRL) formulation learns from expert demonstrations but provides no mechanism to incorporate learning from offline preferences and vice versa. We instantiate the proposed ranking-game framework with a novel ranking loss giving an algorithm that can simultaneously learn from expert demonstrations and preferences, gaining the advantages of both modalities. Our experiments show that the proposed method achieves state-of-the-art sample efficiency and can solve previously unsolvable tasks in the Learning from Observation (LfO) setting.

34 tation as a ranking game between two agents: reward agent and a policy agent-the A Ranking Game for Imitation Learning
Anonymous Author(s)
Affiliation Address email
Abstract
We propose a new framework for imitation learning Saaming-treatingg imitation as a two- player ranking-basel game between a policy and a reward. In this game, the reward agent learns to satisfy pairwise performance rankings between behaviors, while the policy agent learns to maximize this reward. In imitation learning, near-optimal expert data can be difficult to obtain, and even in the limit of infinite data cannot imply total ordering over trajectories as preferences can. On the other hand, learning from preferences alone is challenging as a large number of preferences are required to infer a high-dimensiin typically much easier to collect than expert demonstrations. The classical inverse reinforcement learning (IRL) formulation learns from expert demonstrations but provides mechanism to incorporate learning from offline preferences and vice versa. We instantiate the proposed ranking-gam -ame framework with novel ranking loss giving an algorithm that can simultaneously learn from expert demonstrations and preferences, gaining the advantages of both modalities. Our experiments show that the proposed method achieves state-of-the-an sample efficiency and can solve previously unsolvable tasks the Learning from Observation (LfO) setting.
reward function, though preference data is
10
14
15
16
Introduction
Reinforcement learning relies environmental reward feedback learn meaningful behaviors. Reward specification is a hard problem [39], thus motivating imitation learning (IL) as technique to bypass reward specification and learn from expert data, often via Inverse Reinforcement Learning (IRL) techniques. Learning from expert observations (imitation learning) alone can require efficient exploration when the expert actions are unavailable as in LfO [36]. Incorporating preferences over potentially suboptimal trajectories for reward learning can help reduce exploration burden by regularizing the reward function and providing effective guidance for policy optimization. Previous literature in learning from preferences either assumes no environment interaction [10, 9] assumes active query framework with restricted reward class [47]. The classical IRL formulation suffers from two issues: (1) Learning from expert demonstrations and learning from preferencess/ankingg provide complementary advantages for increasing learning efficiency [30, 47]; however, existing
20
21
22
23
24
25
27
28
29 IRL methods that learn from expert demonstrations provide no mechanisms to incorporate offline
30 preferences and vice versa. (2) Optimization is difficult, making learning sample incfficient [5, 28]
31
due to the adversarial min-max game.
Our primary contribution is an algorithmic framework casting imitation learning as rank- 33 ing game which addresses both of the above issues in IRI. This framework treats imi-
35 reward agent learns to satisfy pairwise performance rankings between different behaviors
Submitted to 36th Conference Neural Information Processing Systems (NeurIPS 2022). Do distribute.

---

36
formance under the learned reward function. and is specified by three components: (1) The dataset of pairwise behavior rankings, (2) A ranking loss function, and (3) An optimization strategy. This game encompasses large subset of both inverse reinforcement learning (IRL) methods and methods which learn from subop- timal offline preferences, Popular IRL methods such as GAIL, AIRL, f-MAX [28, 22, 34] are instantiations of this ranking game which rankings are given only between the learning agent and the expert, and gradient descent ascent (GDA) optimization strategy is used with a ranking loss that maximizes the performance represented as state-action or state visitations,
while the policy agent maximizes its per- The ranking game is detailed Figure
37
38
39
Leaderffolower
max/or
Follower/Leader
40
Palicy agent
Reward
42
43
Policy agent rankings (venilia)
Satisfy rankings min Ly (D)
44
Dataset
45
proret p
46
Pf p py pt!
Aj x of Pi
47
Automatically generated (ouso)
plo, 4 pln Wh
pl-i 4 på
Offllne annotated
50
Rankings from human or offline dateset (pref)
51 gap between the behavior rankings.
52 The ranking loss used by the prior IRL. approaches is specific to the comparison of optimal (expert) vs. suboptimal (agent) data, 55 and precludes incorporation of comparisons among suboptimal behaviors. In this work, we instantiate the ranking game by proposing a new ranking loss (Iı) that facilitates incorporation 59 of rankings over suboptimal trajectories for
Figure 1: rank-game The Policy agent maximizes the reward function by interacting with the environment. The Reward agent satisfies a set of behavior rankings obtained from various sources: generated by the policy agent (vanilla), automatically generated (auto), or offline annotated rankings obtained from human offline dataset (pref). Treating this game in the Stackelberg framework leads to either Policy being a leader and Reward being a follower, or vice-versa.
53
58
57
58
60
reward learning. Our theoretical analysis reveals that the proposed ranking loss results in a bounded performance gap with the expert that depends on a controllable heperarrameter. Our ranking loss can also ease policy optimization by supporting data augmentation make the reward landscape smooth and allowing control over the learned reward scale. Finally, viewing our ranking game in the Stackelberg game framework (see Section an efficient setup for solving general-sum games-we obtain two algorithms with complementary benefits in non-stationary environments depending on which agent is set to be the leader.
61
63
64
65
66
In summary, this paper formulates new framework rank-game for imitation learning that allows 68 us to view learning from preferences and demonstrations under unified perspective. We instantiate
69 the framework with a principled ranking loss that can naturally incorporate rankings provided by di- or offline-our method: 71 (a) outperforms state-of of-the-art methods for imitation learning in several MuJoCo simulated domains
70
verse sources. Finally, by incorporating additional
72 by a significant margin and (b) solves complex tasks like imitating to reorient pen with dextrous ma-
73
nipulation using only a few observation trajectories that none of the previous LfO baselines can solve.
Related Work
74
75
Imitation learning methods are broadly divided into two categories: Behavioral cloning [48, 54] and Inverse Reinforcement Learning (IRL) [44, 1, 72, 18, 20, 28, 22]. Our work focuses on developing a new framework in the setting of IRL through the lens of ranking. Table shows a comparison of the proposed rank -amme method to prior works.
76
77
78
79
Classical Imitation Game for IRL: The classical imitation game for IRL aims to solve the adversarial min-max problem of finding a policy that minimizes the worst-case performance gap between the agent and the expert. A number of previous works [22, 60, 34] have focused on analyzing the properties of this min-max game and its relation to divergence minimization. Under some additional regularization, this min-max objective can be understood as minimizing a certain f-divergence [28, 22, 34] between the agent and expert state-action visitation. More recently, [60] 85 showed that all forms of imitation learning (BC and IRL) can be understood performing moment matching under differing assumptions. In this work, we present perspective imitation
80
81
82
83
84
86

---

Offlin
Expert Data
Ranking Loss
Reward Function
Active Human Query
IL Method
Preferences
MaxEndRLL, AARLLGANN-CLL GAIL,/-MAX, AIRL BCO,GATIO, DACIO, OPOLO.f-IRL TREX, DREX BREX
LID
supremum
non-linear
x
X
LFO
supremmum
non-lincar
x
X
Bradlcy-Terry
non-lincar
Bradley-Terry
linear
DemPref
LIOLD
Bradley-Terry
linear
Ibarz et al[30]
LID
Bradicy-Temy
non-linear
rank-game
LIOILID
L*
non-linear
Table I: summary of IL methods demonstrating the data modalities they can handle (expert data and/or preferences), the ranking-loss functions they use, the assumptions they make on reward function, and whether they require availability of an external agent to provide preferences during training. We highlight whether a method enables LID, LfO, or both when it is able to incorporate expert data.
87 in which the reward function is learned using dataset of behavior comparisons, generalizing
88 previous IRL. methods that learn from expert demonstrations and additionally giving the flexibility
a9 to incorporate rankings over suboptimal behaviors.
Learning from Preferences and Suboptimal Data: Learning from preferences and suboptimal data important when expert data limited or hard to obtain. Preferences [3, 65, 55, 14, 47] have the advantage of providing guidance in situations expert might not get into, and in the limit provides full ordering over trajectories which expert data cannot. A previous line of work [10, 11, 9, 13] has studied this setting and demonstrated that offline rankings over suboptimal behaviors can be effectively leveraged to learn a reward function. [14, 47, 30] studied the question of learning from preferences in the setting when a human is available to provide online preferences' (active queries), while [47] additionally assumed the reward to be linear in known features. Our work makes no such assumptions and allows for integrating offline preferences and expert demonstrations under a common framework.
90
91
92
93
94
95
96
97
98
99
L Learning from Observation (LfO): LfO is the problem setting of learning from expert observations. This is typically more challenging than the traditional learning from demonstration setting (LiD), because actions taken by the expert are unavailable. LIO is broadly formulated using two objectives: state-next state marginal matching [63, 71, 58] and direct state marginal matching [45, 43). Some
100
101
102
103
prior works [61, 67, 16) approach L.O by inferring expert actions through a learned inverse dynamics model. These methods assume injective dynamics and suffer from compounding errors when the policy deployed. A recently proposed method OPOLO [71] derives an upper bound for the LfO objective which enables it to utilize off off-policy data and increase sample efficiency. Our method outperforms baselines including OPOLO, by significant margin.
104
105
107
3 Background
109 We consider a learning agent in a Markov Decision Process (MDP) [49, 59] which can be defined
110 as a tuple: .M- = (S, P, R2 7, Po), where S and A are the state and action spaces; P is the state transition probability function, with P(s'|s, a) indicating the probability of transitioning from s to s' 112 when taking action a; R: S A + IR is the reward function bounded in [o, Rmaz]; We consider
111
113 MDPs with infinite horizon, with the discount factor y E [0, 1]. though our results extend finite
114 horizons as well; Po is the initial state distribution. We use IT and R denote the space of policies and reward functions respectively. A reinforcement learning agent aims to find policy π S A that maximizes its expected return, JRR; T) I-y (s.a) [R(s, a)), where p² (s, a) is the stationary state-action distribution induced by π. In imitation learning, we are provided with samples from the state-action visitation of the expert pT (s, a) but the reward function of the expert is unknown. We will use (f(s, a) as a shorthand for pT* (F((s,)) a).
117
118
119
Classical Imitation Learning: The goal of imitation learning
close the imitation gap J(R: x J (R; T) defined with respect to the unknown expert reward function R. Several prior works [28, 60. 38, 45] tackle this problem by minimizing the imitation gap on all possible reward hypotheses. This
120
122
We will use preferences and ranking interchangebly
3

---

123 leads to a zero-sum (min- max) game formulation of imitation learning in which policy is optimized with respect to the reward function that induces the largest imitation gap:
124
imi t (a) ) = arg min sup (p.( [f a)] - -Ep (a.a) (f [f a)).
(1)
* SER
125
Here, the imitation gap is upper bounded as follows (VT): J(R; - - ((;; S sup
I fER
Pp"(ts,) - (8
(2)
Note that, when the performance gap maximized between the expert E and the agent π, we can observe that the worst-case reward function f= induces ranking between policy behaviors based on their performance: ,f > p² := (x. (f (s, a)] 2 (e,,)))) [fm (8, a)], VT. Therefore, we can regard the above loss function that maximizes the performance gap (Eq. 2) as an instantiation of the ranking-loss. We will refer to the implicit ranking between agent and the expert p > p² as vanilla rankings and this variant of the ranking- loss function as the supremum-loss
126
129
130
131
132
Stackelberg Games: A Stackelberg game is general-sum game between two agents where one agent is set to be the leader and the other follower. The leader in this game optimizes its objective under the assumption that the follower will choose the best response for its own optimization objective. More concretely, assume there are two players A and B with parameters f.. B3 and corresponding losses ). A Stackelberg game solves the following bi-level optimization when A is the leader and B is the follower: ming, C (0 A, Oz (0)) s.t 0% (0x) = arg ming CB (OA. 0).
135
136
L A A, BB) and (f 0B).
137
138
[51] showed that casting model-based RL as an approximate Stackelberg game [6] leads to
139
performance benefits and reduces training instability in comparison to the commonly used GDA [56] and Best Reponse (BR) [12] methods. [17, 69] prove convergence of Stackelberg games under smooth player cost functions and show that they reduce the cycling behavior to find an equilibrium and allow for better convergence.
140
142
4 A Ranking Game for Imitation Learning
In this section, we first formalize the notion of the proposed two-player general-sum ranking game for imitation learning. We then propose a practical instantiation of the ranking game through a novel ranking-loss (Lx). The proposed ranking game gives us the flexibility to incorporate additional rankings- botkingg-bothhhoo auto-generated (a form of data augmentation mentioned as 'auto' in Fig. 1) and offline ('pref' in Fig. 1)-which improves learning efficiency. Finally, we discuss the Stackelberg formulation for the two-player ranking game and discuss two algorithms that naturally arise depending on which player is designated as the leader.
145
146
147
148
149
150
4.1
The Two-Player Ranking Game Formulation
152
We present a new framework, rank-game, for imitation learning which casts it as a general-sum ranking game between two players - a reward and a policy.
153
argmax (1t; argmin neR LDD; R)
Policy Agent
Reward Agent
154
In this formulation, the policy agent maximizes the reward by interacting with the environment, and the reward agent attempts find reward function that satisfies in the given dataset DP; reward function satisfies these rankings if 1E pm [R(s, a)] p [R(ssa)],
155
of pairwise behavior rankings
= DP, where p² . can be state-action or state vistitations.
158
The dataset of pairwise behavior rankings DP can be comprised of the implicit 'vanilla' rankings
between the learning agent and the expert's 's policy behaviors (p" - p"), giving us the classical
159
160
IRL methods when specific ranking loss function - supremum-loss rankings are provided between trajectories, they can be reduced the equivalent ranking between the corresponding actton/cioo//tte visitations. In the case when DP comprises purely of offline trajectory performance rankings then, under specific ranking loss function "Luce-sheparde the ranking game
used [28, 22, 34]. If
161
162

---

Algorithm 1 Meta algorithm: rank-gare grae (vanilla) for imitation
ف Initialize policy =g. reward funtion Rg. empty dataset D empirical expert data
for = 1..7 iterations do
Collect empirical visitation data ~a with mi in the environment. Set D
{(0= 3 E0)} [
Train reward Rs to satisty rankings in D using ranking loss L3 in equation 3.
Optimize policy under the reward function: sf+1
Tg
end for
reduces to prior reward inference methods like T-REX [10, 11, 9, 13]. Thus, the ranking game affords us broader perspective of imitation learning, going beyond only using expert demonstrations. Ranking Loss Lş for the Reward Agent
165
4.2
We use ranking-loss to train the reward function-an objective that minimizes the distortion [31] between the ground truth ranking for a pair of entities {x, y) and rankings induced by a parameterized function R + R for a pair of scalars { {R(x), R(y)}. One type of such a ranking-loss is the supremum-loss in the classical imitation learning setup.
188
169
170
We propose class of ranking-loss functions L that attempt induce performance gap of k: for all behavior preferences in the dataset. Formally, this can be implemented with the regression loss:
La (DP; RR) E (ort .px JNDD
E Baaappi [R[ss a) 0) + EE aaapp [R((s.a) - k) (3)
where DP contains behavior pairs (p" p=)) px 3p"
174
The proposed ranking loss allows for learning bounded rewards with user-defined scale k in the agent and the expert visitations as opposed to prior works Adversarial Imitation Learning [28, 20, 22]. Reward scaling has been known improve learning efficiency in deep RI; large reward scale can make the optimization landscape less smooth [27, 24] and a small scale might make the action-gap small and increase susceptibility extrapolation errors [7]. In contrast the supremum loss, Lx can also naturally incorporate rankings provided by additional sources by learning reward function satisfying all specified pairwise preferences. The following theorem characterizes the equilibrium of the rank Lank-gare game for imitation learning when LE is used as the ranking-loss.
175
176
177
180
181
182
Theorem 4.1. (Performance of the rank-gams equilibrium pair) Consider an equilibrium of the imitation rank -game (f, Rt,, such that the ranking loss Lg generalization error is bounded by 2 Er and the policy near-optimal with JRR; ft) 2 JR; TT) Er. Yn, then at this equilibrium pair under the expert's unknown reward function Rit bounded in [0, R
183
184
185
4RE
1)- 4Rmax
[R -
[R (RRg),5))
#)1
k
(4)
188
If reward is state-only function and only expert observations are available, the same bound applies to the LfO setting.
187
188
Proof. We defer the proof to Appendix A.
189
Theoretical properties: now discuss some theoretical properties of T.g- Theorem shows that rank-game has an equilibrium with bounded performance gap with the expert. An optimization step by the policy player, under reward function optimized by the reward player, is equiva- lent to minimizing an f-divergence with the expert. Equiv- alently, at iteration in Algorithm 1: max,* E,.' [[]]]
GAIL
30
rank-qame
190
191
12
194
195
Figure 2: Figure shows Iearned reward function when agent and expert has a visitation shown by pink and black markers respectively. rank-game (auto) results smooth reward functions more amenable to gradient-based policy optimization compared to GAII.
p= [R]]] = ming+ (p |10==) We elaborate on the regret of this idcalized algorithm Appendix A. Theorem 1 suggests that large values of i can guarantee the agent's performance is close the expert. In practice, observe intermediate values of also preserve imitation equilib- rium optimality with benefit of promoting sample efficient learning (as effect of reward scaling
195
197
198
199
5

---

202
described earlier). We discuss this observation further in Appendix D.9. rank-game naturally ex- tends to the LfO regime under a state-only reward function where Theorem 4.1 results in a divergence bound between state-visitations of the expert and the agent. A state-only reward function is also a suf- ficient and necessary condition to ensure that we learn a dynamies--disentaggeed reward function [20].
203
204
205
Lx can incorporate additional preferences that can help learn a reggularized/shaped reward function that provides better guidance for policy optimization, reducing the exploration burden and increasing sample efficiency for IRL. A better-guided policy optimization is also expected to incur a lower
207
208
209
T However, augmenting the ranking dataset can lead to decrease in the intended performance gap (fefs= < k) between the agent and the expert (Appendix A). This can loosen the bound in Eq and lead to non- optimal imitation learning. We hypothesize that given informative preferences, decreased 6r compensate potentially decreased intended performance gap f f ensure near optimal imitation. In experiments, we observe this hypothesis holds true; we enjoy sample efficiency benefits without losing any asymptotic performance. To leverage these benefits, we present two methods for augmenting the ranking dataset below and defer the implementation details to Appendix B.
210
211
212
215
216
4.2.1
Generating the Ranking Dataset
Reward loss w/ automatically generated rankings (auto): In this method, we assume access to the behavior generating trajectories in the ranking dataset. For each pairwise comparison p; = Pj present in the dataset, La sets the regression targets for states in P. to be 0 and for states visited by PJ to be k. Equivalently, we can rewrite minimizing L& as regressing input of trajectory T; to vector 0, and T to vector k1 where Ti, Tj are trajectories that generate the behavior Pi; Pj respectively, We use the comparison Ps I Pj to generate additional behavior rankings Pj I PAL,i I PAa.áj.. I PAr.ss I Pj 223 where 0- < Å1 < A < < Ap < 1. The behavior Pxy,éj is obtained by independently sampling
217
218
219
222
224 the trajectories that generate the behaviors Ps: Pj and taking convex combinations i.e TAp.ij = Ap + (1 - TT and their corresponding reward regressions targets are given by X, + (1 k.
225
This form of data augmentation can be interpreted as mixup [68] regularization in the trajectory space. Mixup has been shown to improve generalization and adversarial robustness [25, 68] by regularizing the first and second order gradients of the parameterized function. Following the general principle of using a smoothed objective with respect to inputs to obtain effective gradient signals, explicit smoothing in the trajectory-space can also help reduce the policy optimization error en. A didactic example showing rewards learned using this method shown in Figure 2. In special case when the expert's 's unknown reward function is linear in observations, these rankings reflect the true underlying rankings of behaviors.
228
229
230
231
232
234
Reward loss w/ offline annotated rankings (pref): Another way of increasing learning efficiency is augmenting the ranking dataset containing the vanilla ranking (p" = p with offline annotated rankings. These rankings may be provided by a human observer or obtained using an offline dataset of behaviors with annotated reward information, similar the datasets used in offline RL [19, 41]. We combine offline rankings by using a weighted loss between T. for satisfying vanilla rankings (0" < pP)) and offline rankings, grounded by an expert. Providing offline rankings alone that are sufficient to explain the reward function of the expert [10] is often difficult task and the number of offline preferences required depends on the complexity of the environment. In the LfO setting, learning from an expert's state visitation alone can be a hard problem due to exploration requirements [36]. This ranking-loss combines the benefits of using preferences to shape the reward function and guide policy improvement while using the expert to guarantee near-optimal performance.
235
236
237
238
241
242
243
244
245
4.3
Optimizing the Two-Player General-Sum Ranking Game as a Stackelberg Game
246
Solving the ranking-game in the Stackelberg setup allows us to propose two different algorithms depending on which agent is set to be the leader and utilize the learning stability and efficiency afforded by the formulation as studied in [51, 69, 17].
248
249
Policy as leader (PAL): Choosing policy as the leader implies the following optimization:
max [f(:tT) s.t. R = arg min L(D", R))
(5)

---

250
Reward as leader (RAL): Choosing reward as the leader implies the following optimization:
min L(D"; R) s.t π = arg max J (Rt; T)
(6)
A
x
251
We follow the first order gradient approximation for leader's update from previous work [51] to de- velop practical algorithms. This strategy has been proven to be effective and avoids the computational complexity of calculating the implicit Jacobian term (d0 / A). PAL updates the reward to near con- vergence on dataset D (D* contains rankings generated using the current policy agent only π 3 x²) and takes few policy steps. Note that even after the first-order approximation, this optimization strategy differs from GDA often only few iterations are used for training the reward even in hyper- parameter studies like [46]. RAL updates the reward conservatively. This is achieved through aggregat- ing the dataset of implicit rankings from all previous policies obtained during training. PAL's strategy of using on-policy data D for reward training resembles that of methods including GAIL [28, 62), f-MAX [22], and f-IRL [45]. RAL uses the entire history of agent visitation to update the reward
252
253
254
257
258
259
260
261
function and resembles methods such as apprenticeship learning and DAC [1. 37]. PAL and RAL
282
bring together two seemingly different algorithm classes under unified Stackelberg game viewpoint.
Experimental Results
263
284
We compare rank- bank-game game against state-of-the-art LfO and LfD approaches on MuJoCo benchmarks having continuous state and action spaces. The LfO setting is more challenging since no actions are available, and a crucial imitation learning problem that can be used in cases where action modalities differ between the expert and the agent, such as in robot learning. We focus on the LfO setting in this section and defer the L.fD experiments to Appendix D.2. We denote the imitation learning algorithms that use the proposed ranking-loss La from Section 4.2 as RANK-[PAL RAL). We refer to the rank-game variants which use automatically generated rankings and offline preferences as (auto) and (pref) respectively following Section 4.2. In all our methods, we rely on an off-policy model-free algorithm, Soft Actor-Critic (SAC) [26], for updating the policy agent.
285
266
267
270
271
272
273
We design experiments answer the following questions:
1. Asymptotic Performance and Sample Efficiency: Is our method able to achieve near-expert performance given limited number (1) of expert observations? Can our method learn using fewer environment interactions than prior -tate-of-the-art imitation learning (LfO) methods?
275
278
277
Utility of preferences for imitation learning: Current Lf0 methods struggle to solve a number of complex manipulation tasks with sparse success signals. Can we leverage offline annotated preferences through rank-game in such environments achieve near-expert performance?
278
278
280
3.
Choosing between PAL and RAL methods: Can we characterize the benefits and pitfalls of each method, and determine when one method is preferable over the other?
281
282
Ablations for the method components: Can we establish the importance of hyperparameters and design decisions in our experiments?
Baselines:
We compare RANK-PAL and RANK-RAL against 6 representative Lf0 approaches that covers spectrum of on-policy and off- oollcolicy model-free methods from prior work: GAIfO [62, 28], DACTO [37], BCO [61], f-RL [45] and recently proposed OPOLO [71] and IQLearn [21]. We do
287
not assume access to expert actions in this setting. Our LID experiments compare to the IQLearn [21], DAC [37] and BC baselines. Detailed description for baselines can be found in Appendix D.2.
288
5.1
Asymptotic Performance and Sample Efficiency
to baselines on set of MuJoCo locomotion tasks of varying complexities: Swimmer-ve - Hopper-v2, Hal Salffheetah--a Walker hallke2dd-2, Ant-v2 and numanoid-ves In this experiment, we provide one expert trajec- tory for methods and do not assume access to any offline annotated rankings.
In this section, we compare RANK-PAL(auto) and
291
282
293
294
Asymptotic Performance: Table 2 shows that both rank-game methods are able to reach near- expert asymptotic performance with single expert trajectory. BCO shows poor performance which
7

---

Eny
Hopper
HallCheetah
Walker
Ant
Humanoid
BCO
20.10:2.11
5.1223.82
4.00=1.25
12.80=1.22
3.90=1.22
GalFO
81.131 9.99
13.5417.24
83.8312.55
20.10.124.41
3.9111881
DACIO
94.73+3.63
85.0345.009
54.70=44.64
86,45++1.77
+9.31
[ -IRL.
97.45+ 0.61
96.06-+-4.3
101.16+1..25
71.18+ 19.80
77.93+6..72
OPOLO
89,5625.46
88,9243.20
79.19=24.35
93.37: 3.78
24.87=11.004
RANK-PAL(OUN)
87.141 16.14
94.05.13.59
10.82
1..9
96.84
KANK-RAL(OUs)
99,24
101.14:7.455
93.24
93.21=+2.9%
94.45441113
Expert
100.00:1
100.00:1 O
100.00:1:
100.001 O
100.00::
[A)
(11.3)
(17.6)
(17.6)
(111,8)
(376, 17)
Table 2: Asymptotic normalized performance of L.fO methods at 2 million timesteps on MuloCo locomotion
tasks. The standard deviation is calculated with 5 different runs each averaging over 10 trajectory returns. For
unnormalized score and more details, check Appendix We omit IQlearn due to poor performance.
298 can be attributed the compounding error problem arising from its behavior cloning strategy, GAIfO and DAC1O use GDA for optimization with a supremum loss and show high variance in their asymptotic performance whereas rank-game methods are more stable and low-variance
Hopper-v2
Walker-v2
Humanoid-v2
4000
Average Reward
6000
4000
2003
2000
2000
RANK-PAL (ours) RANK-RAL (ours) OPOLO
10-eam
FIRL
Figure 3: Comparison of performance on OpenAl gym benchmark tasks. The shaded region represents standard
deviation across random runs. RANK-PAL and RANK-RAL substantially outperform the baselines sample efficiency. Complete set of results can be found in Appendix D.I
299
Sample Efficiency: Figure 3 shows that RANK-RAL and RANK-PAL are among the most sample efficient methods for the LfO setting, outperforming the recent state-of-the-an art method OPOLO [71] by a significant margin. We notice that IQL.ear fails to learn in the Lf0 setting. This experiment demonstrates the benefit of the combined improvements of the proposed ranking-loss with automat- ically generated rankings. Our method is also simpler to implement than OPOLO, as require fewer lines of code changes on top of SAC and need to maintain fewer parameterized networks compared to OPOLO which requires an additional inverse action model to regularize learning.
300
301
302
305
308
Door-vo
Pen-v0
307
5.2
Utility of Preferences in Imitation
Averanee Rewerd
308
Our experiments on complex manipulation environments-door opening with a parallel- jaw gripper [70] and pen manipulation with dexterous adroit hand [50]- reveal that none
2X1
309
310
25
28
3.
Timasiops
Timestaps
312
of the prior LfO methods are able to imitate the expert even under increasing amounts of
RANK-PAL (pen))
(auns)
OPOLO
RANKRALL (grof)
RANK RAL (bul))
313
314
expert data. This failure of LfO methods can
Figure 4: Offline annotated preferences can help solve LfO tasks in the complex manipulation environments Pen-vO and Door, whereas prior LfO methods fail. Black dotted line shows asymptotic performance RANK- PAL (auto) method.
315
be potentially attributed to the exploration requirements of LfO compared to LiD [36],
316
317
coupled with the sparse successes encountered
in these tasks, leading to poorly guided policy gradients. In these experiments, we show that rank- game can incorporate additional information in the form of offline annotated rankings to guide the agent in solving such tasks. These offline rankings are obtained by uniformly sampling a small set of trajectories (10) from the replay buffer of SAC [26) labeled with a ground truth reward function. We use a weighted ranking loss (pref) from Section 4.2.
320
321
322
323
Figure 4 shows that RANK-PAL/RAL method leveraging offline ranking is the only method that can solve these tasks, whereas prior LfO methods and PAL/RAL(auto) with automatically
8

---

325
generated rankings struggle even after a large amount of training. We also point out that T-REX, a method that learns using the preferences alone is unable to achieve near-expert performance, thereby highlighting the benefits of learning from expert demonstrations alongside a set of offline preferences.
326
327
328
5.3
Comparing PAL and RAL
329
PAL uses the agent's current visitation for re- ward learning, whereas RAL learns reward consistent with all rankings arising from the his-
Intent-Adaptation
Jynamics-Adaptann
Average Reward
4855
2000
332
tory of the agent's s visitation. These properties can present certain benefits depending on the task setting. To test the potential benefits of PAL
1000
2000
333
334
335
and RAI., we consider two non-stationary imita- tion learning problems, similar [50] -00--00 one in which the expert changes it's intent and the other where dynamics of the environment change dur- ing training in the 12 locomotion task. For changing intent, we present a new set of demonstrations where the hopper agent hops backwards rather than forward. For changing environment dynamics, we increase the mass of the hopper agent by a factor of 1.2. Changes are introduced le5 time steps during training at which point notice a sudden performance drop. In Figure 5 (left), we notice that PAL adapts faster to intent changes, whereas RAL needs to unlearn the rankings obtained from the agent's 's history and takes longer to adapt. Figure 5 (right) shows that RAL adapts faster to the changing dynamics of the system, as it has already learned a good global reward function in the LIO setting, whereas PAL only has a local understanding of reward as a result of using ranking obtained only from the agent's current visitation.
Timestops
Timestaps
338
RANK-PAL
RANK-RAL
Figure 5: We compare the relative strengths of PAL and RAL. Left plot shows a comparison when the goal is changed, and right plot shows a comparison when dynamics of the environment is changed. These changes occur at le5 timesteps into training. PAI. adapts faster changing intent and RAL adapts faster changing dynamics.
339
340
341
312
343
344
346
347
348
349
notion of the dynamics-disentlee
350
Ablation of Method Components: Appendix D contains eight additional experiments to study the importance of hyperparameters and design decisions. Our ablations validate the importance of using automatically generated rankings, the benefit of ranking loss over supremum loss, and sensitivity to hyprrparameters like the intended performance gap k. policy iterations, and the reward regularizer.
352
353
354
Conclusion
355
356
In this work, we present a new framework for imitation learning that treats imitation as a two-player ranking-game between a policy and a reward function. Unlike prior works in imitation learning, the ranking game allows incorporation of rankings over suboptimal behaviors aid policy learning. We instantiate the ranking game by proposing a novel ranking loss which guarantees agent's performance to be close to expert for imitation learning. Our experiments on simulated MuJoCo tasks reveal that utilizing additional ranking through our proposed ranking loss leads to improved sample efficiency for imitation learning, outperforming prior methods by a significant margin and solving some tasks which were unsolvable by previous LfO methods.
357
358
359
382
363
364
Limitations and Negative Societal Impacts: Preferences obtained in real world are usually noisy [40, 32, 8) and one limitation of rank rank-game is that it does not suggest way handle noisy preferences. Second, rank -ane proposes modifications to learn reward function amenable to policy optimization but these hyperparameters are set manually. Future work can explore methods to automate learning such reward functions. Third, despite learning effective policies we observed that
365
367
368
369
we do not learn reusable robust reward functions [45]. Negative Societal Impact: Imitation learning
370
can cause harm if given demonstrations of harmful behaviors, either accidentally or purposefully.
371
Furthermore, even when given high-quality demonstrations of desirable behaviors, our algorithm does
372
not provide guarantees of performance, and thus could cause harm if used in high-stakes domains
without sufficient safety checks on learned behaviors.

---

References
374
375
[1]
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty- first international conference on Machine learning, page 1, 2004.
378
377
[2] Joshua Achiam. Spinning Up Deep Reinforcement Learning. 2018.
[3] Riad Akrour, Marc Schoenauer, and Michele Sebag. Preference-based policy learning. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pages 12-27. Springer, 2011.
381
[4] Syed Mumtaz Ali and Samuel D Silvey. A general class coefficients of divergence one distribution from another. Journal of the Royal Statistical Society: Series (Methodological), 28(1):131-142, 1966.
382
383
[5] Oleg Arenz and Gerhard Neumann. Non-adversarial imitation learning and its connections to adversarial methods. arXiv preprint arXiv:2008.033: 2020.
386
66]
Tamer Başar and Geert Jan Olsder. Dynamic noncooperative game theory. SIAM, 1998.
387
Mare G Bellemare, Georg Ostrovski, Arthur Guez, Philip Thomas, and Rémi Munos. Increas- ing the action gap: New operators for reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.
[8] Erdem BIyIk, Dylan P Losey, Malayandi Palan. Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, page 0278364921 1041652, 2021.
391
392
383
[9] Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Nickum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference Machine Learning, pages 1 1165-1177 117.. PMLR, 2020.
397
[10] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating be- yond suboptimal demonstrations via inverse reinforcement learning from observations. ArXiv, abs/1904.0633 2019.
398
399
[11] Daniel Brown, Wonjoon Goo, and Scott Niekum. senter-than-demonstere imitation learning via automatically-ranke demonstrations. In Conference on robot learning, pages 330-359, PMLR, 2020.
402
403
[12] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
404
[13] Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. arXiv preprint arXiv:2010.117233 2020.
407
[14] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. arXiv preprint arXiv: (tt:::7006.3355 03741, 2017.
408
[15] Imre Csiszár. Information-type measures of difference of probability distributions and indirect observation. studia scientiarum Mathematicarum Hungarica, 2:229-318, 1967.
411
[16] Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. In International Conference on Machine Learning, pages 1763. PMLR, 2019.
412
415
[17] Tanner Fiez, Benjamin Chasnov, and Lillian J Ratliff. Convergence of learning dynamics in stackelberg games. arXiv preprint arXiv:1906.012/71 2019.
10

---

416
[18] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
417
418
419
[19]
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
420
421
[20] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse
422
reinforcement learning. arXiv preprint arXiv:1710.11244 2017.
423
[21] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34, 2021.
424
425
426
[22] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence mini- mization perspective on imitation learning methods. In Conference on Robot Learning, pages 1259-1277. PMLR, 2020.
427
429
[23] Gustavo Gilardoni. On the minimum f-divergence for given total variation. Comptes Rendus Mathemalique- C MATH, 343:763-766, 12 2006.
430
431
[24] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
[25] Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regularization. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 33, pages 3714-3771 2019.
436
[26] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
440
[27] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
441
442
443
[28] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29:4565-4573 2016.
444
445
[29] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. In The collected works of Wassily Hoeffding, pages 409-426. Springer, 1994.
446
447
[30] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. arXiv preprint arXiv:1811.06521 2018.
448
449
450
[31] Rishabh Iyer and Jeff Bilmes. The submodular bregman and lovász-bregman divergences with applications: Extended version. Citeseer, 2012.
451
452
[32] Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances Neural Information Processing Systems, 33:4415- 4426, 2020.
453
454
455
[33]
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
456
457
[34] Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srini- vasa. Imitation learning as f-divergence minimization. In Algorithmic Foundations of Robotics XIV: Proceedings of the Fourteenth Workshop on the Algorithmic Foundations of Robotics 14, pages 313-329. Springer International Publishing, 2021.
458

---

481
[35] Michael Kearns and Satinder Singh. Finite-sample convergence rates for q-learning and indirect algorithms. Advances in neural information processing systems, 11, 1998.
462
463
[36] Rahul Kidambi, Jonathan Chang, and Wen Sun. Mobile: Model-based imitation learning from observation alone. Advances Neural Information Processing Systems, 34, 2021.
464
465
[37] Ila Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. peerimiinator-ccor-eriiee Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv: 1809.02299 2018.
467
488
[38] Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policc distribu- tion matching. arXiv preprint arXiv: 1912.0502, 2019.
489
470
[39] Victoria Krakovna. Specification gaming examples ai. Available at vkrakovna. wordpress. com, 2018.
471
472
[40]
Minae Kwon, Erdem Biyik, Aditi Talati, Karan Bhasin, Dylan P Losey, and Dorsa Sadigh, When humans aren't 't optimal: Robots that collaborate with risk-aware humans. In 2020 15th ACM/IEEE International Conference Human-Robot Interaction (IIRI), pages 43-52. IEEE, 2020.
475
476
[41] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
477
478
[42] Friedrich Liese and Igor Vajda. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10):439-44112 2006.
479
[43] Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment- bassd imitation learning. arXiv preprint 1rXiv:1911.10947 2019.
482
[44] Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, page 2, 2000.
483
484
[45] Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Benjamin Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. arXiv preprint 1,04709, 2020.
485
486
[46] Manu Orsini, Anton Raichuk, Léonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, and Marcin Andrychowiez. What matters for adversarial imitation learning? Advances in Neural Information Processing Systems,
490
34, 2021.
491
[47] Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward func- tions by integrating human demonstrations and preferences. arXiv preprint arXiv: 1906609925 2019.
482
493
[88]
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural compulation, 3(1):88-97, 1991.
[49] Martin L Puterman. Markov decision processes: discrete stochastic dynamic progranming. John Wiley & Sons, 2014.
437
498
[50] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint 170/v:1709.10087, 2017.
499
500
[51] Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. game theoretic framework for model based reinforcement learning, In ICML, 2020.
12

---

503
[52]
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv: / 108, 2019.
504
505
[53] Alfréd Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, pages 547-561. University of California Press, 1961.
508
507
508
[54] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
509
510
511
512
[55] Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learning of reward functions. 2017.
513
514
[56] Florian Schäfer and Anima Anandkumar. Competitive gradient descent. arXiv preprint arXiv: /905. 12/03,1022220 2019.
515
518
[57] Shai Shalev-Shwartz and Shai Ben-David Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
517
(58) Wen Sun, Anirudh Vemula, Byron Boots, and Drew Bagnell. Provably efficient imitation learning from observation alone. In International conference on machine learning, pages 1036-6045. PMLR, 2019.
519
620
521
[59] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
522
523
[60]
Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Steven Wu. Of moments and match- ing: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pages 10022-10032. PMLR, 2021.
624
525
526
[61] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint 1rXiv:180..01955 2018.
527
662] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion. arXiv preprint arXi:: 2018.
529
[63] Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from observation. arXiv preprint arXiv: /905. 13566,, 2019.
664] Igor Vajda. Note discrimination information and variation (corresp.). IEEE Transactions Information Theory, 16(6):771-773 1970.
534
665]
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. Advances in neural information processing systems, 25:1133- 1141, 2012.
666] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
539
[67] Chao Yang, Xiaojian Ma, Wenbing Huang. Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagree- ment. arXiv preprini
2019.
542
[68] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
544
[69] Liyuan Zheng, Tanner Fiez, Zane Alumbaugh, Benjamin Chasnov, and Lillian Ratliff. Stackelberg actor-critic: Game-theoretic reinforcement learning algorithms. arXiv preprint arXiv:2109.12286, 2021.
545
13

---

547
[70]
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. /2293,
548
549
2020.
[71] Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. Advances in Neural Information Processing Systems, 33, 2020.
551
[72] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433 1438. Chicago, IL, USA, 2008.
14


Reviews:
[{"review_id": "SgK1L2cdV2Z", "reviewer": "Reviewer_ZWmg", "summary": "The paper proposes a ranking-game setting for imitation learning by using a ranking loss for the reward agent, and incorporating automatically generated rankings data augmentation as well as offline annotated rankings. The proposed method show improved performance and data efficiencies compare to state-of-the-art baselines on several benchmark tasks.", "questions": "May the author give some more explanation on the automatically generated rankings. Why would a linear combination of trajectories give a linear combined ranking?", "limitations": "The major limitations are described at the end of the paper, which I feel is acceptable.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The relationship between the proposed setting with the existing IRL frameworks is well described.\n1. The author gives a performance bound for the proposed ranking loss at equilibrium.\n2. The experiments for the proposed method without offline preference show clear sample efficiency improvement compared to other methods.\n3. The experiments for using offline preference give a significant performance improvement. While the improvement is expected but is still impressive.\n4. The ablation study is thorough.\n\nWeaknesses:\n1. I am not quite convinced about the automatically generated rankings. Why would a linear combination of trajectories give a linear combined ranking?\n2. The other limitations are described at the end of the paper, which I feel is acceptable.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PUPh_wvCh-n", "reviewer": "Reviewer_48YC", "summary": "In this paper, the imitation learning (IL) problem, including learning from observation (LfO) and learning from demonstration (LfD), is studied. The paper proposes a new solution to IL which include three key aspects: (1) a new ranking loss is proposed for IL, which is capable to learn whenever the rankings of the trajectories are given or not; (2) a mixup-like data augmentation strategy is introduced to smooth the loss landscape; (3) the IL process is formulated into Stackelberg game, allowing the studied of the update frequencies of reward and policy functions. The proposed method is tested under benchmark continuous control tasks, showing performance improvement over existing IL approaches.\n\n", "questions": "1. Can you provide some explanations on why the proposed ranking loss is better than classical ranking losses, under the IL setting?", "limitations": "The limitations and negative societal impact are properly discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes several interesting ideas for IL. In my view, the most significant ones are two-fold. One is the importance of smoothing the loss landscape, which can be achieved by the proposed loss and the mixup-like data augmentation. Another is the Stackelberg game formulation. The experiment of task changes in Section 5.3 is very interesting to me.\n\n2. The overall experimental performance is impressive. \n\nWeaknesses:\n1. The proposed approach is somehow complicated. There are several components included, making it hard to identify their contributions to the performance improvement. Even though a number of ablation study results are reported, they only show that none of the components is useless. Their precise contributions, as well as underlying mechanisms, remain unclear.\n\n2. It still remains unclear to me why the proposed ranking loss could outperform other classical ranking losses. As discussed in the paper, the loss is somehow like the one proposed in [1]. While in [1], since the algorithm is Q-learning, the choice of the reward function seems to have a less impact on the optimization objective. \n\n3. I think conducting deeper studies following the direction in Section 5.3 would make the paper more interesting. The results in Section 5.3 seem to indicate that the PAL training is robust to changes in reward functions, while the RAL robust to changes in dynamics. I think these results show that one of the major advantages of formulating IL into Stackelberg games is the convenience of dealing with task changes. \n\nOverall, I think the paper is more on proposing several useful techniques for IL without exploring the underlying mechanism deeply, which can be of good practical usefulness. \n\n[1] SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards, ICLR'20.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O0hdJSbCt7Q", "reviewer": "Reviewer_MNRH", "summary": "The authors create a new method for learning a policy through IRL. The method contains two different agents: (i) a policy agent; and (ii) a reward agent. The authors perform a set of experiments with the OpenAI Gym + MuJoCo environments and show that they achieve better results than prior work.", "questions": "- How much time does it take to run each experiment in each of these methods? I think that a reason that the authors used so few trajectories in their experiments could be a hint that even though RANK is sample efficient, it does not seem to be time efficient. Clarification on time efficiency would be very important here.", "limitations": "- I think the authors addressed some of the concerns I had during my review in their limitation section. The ones they have not addressed are pointed out in the weakness section of this review.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths:\n    - The authors create a consistent work with a good explanation of why they created the method, and how the method works (theoretically and practically).\n    - They perform extensive ablations on their method and how each part of the method reacts when deviating from the proposed method in the main material.\n    - The work is well presented, with good writing and a couple of minor misspellings, nothing major that would damage the reading comprehension and experience.\n\n- Weaknesses\n    - I think that the major weakness of this work is the lack of experimentation with sub-optimal expert samples. With each year, we have more IL methods being developed and one of the main difficulties for a lot of them is to be resilient to sub-optimal samples. I think this should have been addressed in this work. Yes, the method is very efficient but what is the trade-off that controls this efficiency.\n    - Even though RANK achieves good results with only one trajectory, I think there should be a study on how well it performs with more samples. We cannot forget that RANK has access to the environment and online samples have a higher cost than offline samples to collect and learn. I think this work would benefit from drawing this distinction and showing how much it gains when using different numbers of trajectories. Even more so when we think that there are other methods that were not used in these baselines (such as MobILE [1] and ILPO [2]) that already use a lower number of trajectories (10) while not having this iterative nature of RANK. Moreover, using a single trajectory was a limitation that RANK imposed to itself.\n    - Table 2 only shows the Performance metric, and does not display the expert rewards, which hurts the readability of the results.\n    - Another major concern that I have is the efficiency study. The authors say that their method achieves higher reward thresholds in 2M timesteps with only one trajectory (that can vary in sample size). However, this is not entirely true. If we consider that offline samples are more efficient to learn, we could argue that BCO, and all future work based on it, could be more efficient by using 10 trajectories and only needing a fraction of the timesteps in this experiment, for example.\n    - Finally, regarding the experiments, 5 trajectories seem a lower number of trajectories to have an accurate statistical result. OPOLO used 10 times more samples to draw its average and standard deviation, while GaIFO used 100 trajectories for each experiment.\n    - The authors’ overuse of math notations hurts the readability of their work quite extensively. Most of the time the iterator changed without proper explanation and it takes the reader quite some time to understand what the authors are iterating over.  I think that rewriting the equations and standardizing the notation would benefit this work.\n    - If the policy agents interact with the environment to reduce the reward function, is it really IL? Why training an imitation learning agent if, in the end, we will have an agent trained by the reward function of the environment (which theoretically is the most optimal thing we could have in an MDP problem)?\n    \n    [1] Kidambi, Rahul, Jonathan Chang, and Wen Sun. \"MobILE: Model-Based Imitation Learning From Observation Alone.\" *Advances in Neural Information Processing Systems*\n     34 (2021): 28598-28611.\n    \n    [2] Edwards, Ashley, et al. \"Imitating latent policies from observation.\" *International conference on machine learning*. PMLR, 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NSqjUFrRrBi", "reviewer": "Reviewer_NadH", "summary": "The paper considers the standard imitation learning problem by treating imitation as a two-player ranking-based game between a policy and a reward. A novel ranking loss is proposed which has the ability to learn from both expert demonstrations and preferences while keeping the learned reward bounded. The authors introduce two optimization methods based on Stackelberg games with different leaders. Experiments show that the two proposed algorithms work well in LfO settings and outperforms the baselines. In addition, by incorporating additional offline annotated rankings, the proposed method can work in more complex environments where other baselines fail. Finally, the pros and cons of the proposed two algorithms are also discussed.", "questions": "The questions are mainly discussed in the Weaknesses. Here are the selected questions that I think are most important:\n\n- The intuition behind the proposed ranking loss (Eq 3).\n\n- How to generate the ranking dataset in the setting of reward loss w/ automatically generated rankings? Will this cause some problem when combining the automatically generated rankings with the vanilla rankings?\n\n- Can you provide some insights on the reason why the additional annotated is so useful?", "limitations": "The limitations and potential negative societal impact mentioned in the paper are adequately addressed. One suggestion: the authors can consider how the framework will work in the setting of learning from suboptimal demonstrations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe idea of treating imitation as a two-player ranking-based game is novel, which encompasses both a kind of IRL methods and ranking-based methods. The proposed ranking loss learns a smoother reward function (Figure 2) compare with the IRL methods. The experimental results are strong, showing the efficacy of the proposed methods and the benefit of the additional annotated rankings. The comparison of the two proposed method is interesting. The writing of the paper is generally good. \n\n**Weaknesses**\n\n- The ranking loss Eq (3) is novel, but it may lack some intuition. The loss of GAIL is based on occupancy measure matching, and the loss of AIRL makes it possible to learn the ground-truth reward under some assumptions, and the loss of TREX is based on Luce-Shepard rule. However, the proposed loss lack some theoretical background and I would appreciate if the authors can provide additional intuitions behind this loss. \n\n- Line 215: Here the behavior generating trajectories in the ranking dataset is assumed to be accessible. How do you generate the ranking dataset? Also, if you use auto magically generated rankings, does it mean that the demonstrations are not fully optimal? Will this cause some problem when combining the automatically generated rankings with the vanilla rankings, e.g., the learned policy may behave better than the low-ranked trajectories in the demonstrations, leading to some noisy ranking data?\n\n- In Section 5.2, the results in Door-v0 show that RANK-PAL (pref) is the only method that can solve the problem. Can you provide some insight why the additional annotated rankings are better than the automatically generated rankings, and why the additional annotated rankings can reduce the exploration requirements of LfO (line 315) that much?\n\n- In Section 5.2, the authors only show the performance of RANK-PAL. How does RANK-RAL work in these scenarios?\n\n- For the writing:\n  - Line 21: [Leaning from expert data (imitation learning) alone]: \"data\" here is a bit unclear here since actions are also data.\n  - Line 25: [assumes no environment interaction]: The definition of \"environment interaction\" is unclear. I think you mean the environment interaction while learning the reward function, since DREX needs to collect interaction data while collecting noisy trajectories. In addition, are you suggesting no environment interaction is a bad thing here? Why is this a bad thing?\n  - Line 57: [suboptimal rankings]: This is vague. What do you mean by saying rankings are \"suboptimal\"?\n  - Line 135: a right parenthesis is missed.\n  - Line 182: What is $\\epsilon_r$?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Kws7IQp6vjl", "reviewer": "Reviewer_Q6bL", "summary": "The paper proposes to model imitation learning as a two-player general sum game, where one player, the \"reward player\", aims to rank a set of behaviors correctly, where the \"policy player\" aims to maximize the reward the \"reward player\" uses for ranking. This formulation unifies multiple prior methods for LfD and LfO. Then, the authors propose a novel way to solve this two player game as a Stackelberg game which leads to a novel imitation learning algorithm that outperforms prior methods in a series of simulated MuJoCo experiments. The proposed algorithm learns from demonstrations and preferences simultaneously which allows it to combine the benefit of both.", "questions": "What is the intuitive motivation for choosing the proposed ranking loss?\n\nDo you consider any of the theoretical results apart from Thm. 4.1 a key contribution of your paper?\n\nDoes you assume all comparisons in the dataset are noise-free?\n\nCould the proposed approach be extended to active learning by modifying what the reward agent does (i.e., it also chooses queries to make to a human)?\n\n\n**Minor suggestions:**\n\n- Capitalization:\n   - \"bradley-terry\" -> \"Bradley-Terry\" (Table 1)\n   - \"jacobian\" -> \"Jacobian\" (line 253)\n   - In the references: \"stackelberg\" -> \"Stackelberg\", \"Iq-learn\" -> \"IQ-learn\", \"soft-q\" -> \"soft-Q\", etc.\n\n- Also, make sure named paragraphs are consistently capitalized. For example \"Ablation of method components\" (line 350), but \"Limitations and Negative Societal Impacts\" (line 363)\n\n", "limitations": "If the paper indeed assumes noise-free preferences, this should be made clearer when setting up the problem. It would also be good to include as a dimension in Table 1, to provide a fair comparison.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The two-player game framework is a conceptually elegant way of describing and unifying existing methods for IL. It is neat how the choice of different ranking loss functions yields different algorithms.\n- The proposed algorithm is quite natural, and it is analyzed theoretically and empirically.\n- The empirical evaluation is particularly thorough. The authors compare to various natural baselines on different tasks. They provide ablations of various components of their algorithm, providing the reader with a clear understanding of the performance of their algorithm in MuJoCo environments.\n\n**Weaknesses**\n- The paper does not provide a good intuition for the proposed ranking loss. After reading Sec. 4.2 I understood the choice that was made, but did not have a good understanding of why this is the right choice among many possible ranking losses.\n- Parts of the paper could be written more clearly. For example, in Sec. 4.2 it would be natural to split it up in subsections discussing the ranking loss, the theoretical properties, and then the ways to generate the ranking dataset.\n- I found it hard to understand the theoretical results following Theorem 4.1 without reading the Appendix.\n- If I understand correctly, the method assumes all preferences that are provided to be noise-free, which limits the applicability of the method significantly.\n- The experimental evaluation is limited to MuJoCo locomotion tasks, which can sometimes be too simple to draw general conclusions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject