A Ranking Game for Imitation Learning
Authors: Harshit Sikchi
Akanksha Saran
Wonjoon Goo
Scott Niekum
Venue: Accepted by TMLR ()

Abstract: We propose a new framework for imitation learning---treating imitation as a two-player ranking-based game between a policy and a reward. In this game, the reward agent learns to satisfy pairwise performance rankings between behaviors, while the policy agent learns to maximize this reward. In imitation learning, near-optimal expert data can be difficult to obtain, and even in the limit of infinite data cannot imply a total ordering over trajectories as preferences can. On the other hand, learning from preferences alone is challenging as a large number of preferences are required to infer a high-dimensional reward function, though preference data is typically much easier to collect than expert demonstrations. The classical inverse reinforcement learning (IRL) formulation learns from expert demonstrations but provides no mechanism to incorporate learning from offline preferences and vice versa. We instantiate the proposed ranking-game framework with a novel ranking loss giving an algorithm that can simultaneously learn from expert demonstrations and preferences, gaining the advantages of both modalities. Our experiments show that the proposed method achieves state-of-the-art sample efficiency and can solve previously unsolvable tasks in the Learning from Observation (LfO) setting.

Harshit Sikchi Published in Transactions on Machine Learning Research (01/2023)
A Ranking Game for Imitation Learning
hsikchi edu
Deportment of Computer Science, The University of Texas at Austin
Akanksha Saran
Microsoft Research NYC
Wonjoon Goo Deportment of Computer Science, The University of Tezas at Austin
wonjoonn@es.uleras edu
Scott Niekum
miekummcss wmwss. edu
Department of Computer Science, University of Massachusetts Amherst
Reviewed on OpenReview: https::/ opwnneviee.......... net/ forun? id= d3rHR44VA0
Abstract
We propose new framework for imitation learning treating imitation as two-player ranking-bused game between policy and reward. In this game, the reward agent learns lo satisly pairwise performance rankings between behaviors, while the policy agent learns to maximize this reward. In imitation learning, near-optimal expert data can be difficult to obtain, and even in the limit of infinite data cannot imply a total ordering over trajectories as preferences can. On the other hand, learning from preferences alone is challenging as a large number of preferences are required to infer a high-dimensional reward function, though preference data is typically much easier to collect than expert demonstrations, The classical inverse reinforcement learning (IRL) formulation learns from expert demonstrations but provides no mechanism to incorporate learning from offline preferences and vice versa. We instantiate the proposed ranking-gane framework with novel ranking loss giving an algorithm that can simultaneously learn from expert demonstrations and preferences, gaining the advantages of both modalities. Our experiments show that the proposed method achieves saate-of-the-are sample efficiency and can solve previously unsolvable tasks the Learning from Obscrvation (LfO) setting. Project vidco and code can be found at this URL
1 Introduction
Reinforcement learning relies environmental reward feedback to learn meaningful behaviors. Reward specification is hard problem (Krakovna] 2018), thus motivating imitation learning (TI.) as technique to bypass reward specification and learn from expert data, often via Inverse Reinforcement Learning (IRL) techniques. Imitation learning typically deals with the setting of Learning from Demonstrations (LfD), where expert states and actions are provided the learning agent. more practical problem imitation learning is Learning from Observations (LfO), where the learning agent has access only the expert observations. This setting common when access to expert actions are unavailable such as when learning from accessible observation sources like videos or learning to imitate across different agent morphologies. We note that LfD and LfO settings differ from the setting where the agent has access to the environment reward function along with expert transitions, referred to as Reinforcement Learning from Demonstrations (RLID) (Jing et al. 2020
Zhang et al. 2020 Brys el al. 2015)-
Learning imitate using expert observations alone can require efficient exploration when the expert actions are unavailable as in LfO (Kidambi et al.) 2021). Incorporating preferences over potentially suboptimal trajectories for reward learning can help reduce the exploration burden by regularizing the reward function

---

Published in Transactions on Machine Learning Research (01/2023)
and providing effective guidance for policy optimization. Previous literature in learning from preferences either assumes no environment interaction (Brown et al. 2019 2020a) or assumes an active query framework with a restricted reward class (Palan et al. (011)). The classical IRL formulation suffers from two issues: (1) Learning from expert demonstrations and learning from wweferenccs/rankkinn
provide complementary advantages for increasing learning efficiency (Ibarz et al., 2018 Palan et al. 2019 however, existing IRL
methods that learn from expert demonstrations provide no mechanisms incorporate offline preferences and vice versa.
(2) Optimization is difficuut, making the learning sample inefficient (Arenz & Neumann Ho & Ermon
2020
2016) due to the adversarial min-max game.
Our primary contribution is an algorithmic framework casting imitation learning as ranking game which addresses both of the above issues in IRL. This framework treats imitation as a ranking game between two agents: a reward agent and a policy agent- the reward agent learns to satisfy pairwise performance rankings between different behaviors represented as state-action or state visitations, while the policy agent maximizes performance under the learned reward function. The ranking game detailed in Figure | and is specified by three components: (1) The dataset of pairwise behavior rankings, (2) A ranking loss function, and (3) An optimization strategy. This game encompasses a large subset of both inverse reinforcement learning (IRL) methods and methods which learn from suboptimal offlinc preferences. Popular IRL methods such as GAIL, AIRL, f-MAX (Ho & Ermon 2016 Ghasemipour et al., 2020 Ke et al. 2021) are instantiations of this ranking game in which rankings are given only between the learning agent and the expert, and a gradient descent ascent (GDA) optimization strategy is used with a ranking loss that maximizes the performance gap between the behavior rankings.
The ranking loss used by the prior IRL approaches is specific to the comparison of optimal (expert) vs. suboptimal (agent) data, and precludes incorporation of comparisons among suboptimal behaviors. In this work, we instantiate the ranking game by proposing a new ranking loss (Lx) that facilitates in- corporation of rankings over suboptimal trajectorics for reward learning. Our theoretical analysis reveals that the proposed ranking loss results in a bounded performance gap with the expert that depends controllable hyperparameter. Our ranking loss can also ease policy optimization by supporting data augmentation to make the reward landscape smooth and allowing control over the learned reward scale. Finally, viewing our ranking game in the Stackelberg game framework (see Section 3) an efficient setup for solving general-sum games we obtain two algorithms with complementary benefits in non-stationary environments depending on which agent is set to be the leader.
max/(r; R)
Leader/Follower
Follower/Leader
Policy agent
Reward Agent
Policy agent generates rankings (vanillo)
Satisfy rankings min (D)
p poont < p
pant & pi P1 & pl
at 4 pi it x
Automatically generated rankings (outo)
obnt pl ph & p
Ph-1 < pē
Offline annotated Rankings from human or offline dataset (prof)
In summary, this paper formulates a new framework rank-game for imitation learning that allows us to view learning from preferences and demonstrations under unified perspective. We instantiate the framework with principled ranking loss that can naturally incorporate rankings provided by diverse sources. Finally, by incorporating additional rankings-auto-gnnnnn anto-generated or offline our method: (a) outperforms state-of-the-art methods for imita- tion learning in several MuJoCo simulated domains by a significant margin and (b) solves complex tasks like imitating to reorient pen with dextrous manipulation using only few observation trajectories that none of the previous LIO baselines can solve.
Figure 1: rank-game: The Policy agent maximizes the re- ward function by interacting with the environment. The Re- ward agent satisfies set of behavior rankings obtained from various sources: generated by the policy agent (vanilla), au- tomatically generated (auto), offline annotated rankings obtained from a human or offline dataset (pref). Treat- ing this game in the Stackelberg framework leads either Policy being a leader and Reward being follower, or vice- versa.
2

---

Published in Transactions on Machine Learning Research (01/2023)
Offline
Expert Data
Ranking Loss
Reward Function
Active Human Query
IL Method
Preferences
MaxEntIRL, GANGGTG G-T,, AIRL
X
LfD
supremum
non-linear
x
BCO,O DACIO, OPOLO,f-IRI TREX, DREX BREX DemPref
LfO
supremum
non-linear
X
Bradley-Terry
non-linear
V
X
Bradley- Terry
linear
X
LIO/LID
Bradley-Terry
linear
Ibarz et al. 2018
LID LIO/LID
Bradley-Terry
non-linear
Bank-gaae
L
non-linear
Table 1: A summary of TI. methods demonstrating the data modalities they can handle (expert data and /or preferences), the ranking-lowd loss functions they use, the assumptions they make on reward function, and whether they require availability of an external agent to provide preferences during training. We highlight whether method enables LfD, LfO, or both when able to incorporate expert data.
2 Related Work
Imitation learning methods are broadly divided into two categories: Behavioral cloning (Pomerleau 1991
Ross et al. 2011) and Inverse Reinforcement Learning (IRL) (Ng et al. 2000; Abbeel & Ng. 2004 Ziebart
et al. 2008 Finn et al., 2016 Fu et al.) 2017 Ho & Ermon, 2016 Ghasemipour et al. 2020). Our work
focuses on developing a new framework in the sctting of IRL through the lons of ranking. Table
shows a
comparison of the proposed rank-game method to prior works.
Classical Imitation Game for IRL: The classical imitation game for IRL aims to solve the adversarial min-maz problem of finding a policy that minimizes the worst-case performance gap between the agent and the expert. A number of previous works (Ghasemipour et. al.) 2020 Swamy el al. 2021 Ke et al. 2021 have focused on analyzing the properties of this min-mar game and its relation to divergence minimization. Under some additional regularization, this min-maz objective can be understood as minimizing a certain f-divergence (Ho & Ermon 2016 Ghasemipour et al. 2020 Ke et al.) 2021) between the agent and expert state-action visitation. More recently, (waamy al. (2021 showed that all forms of imitation learning (BC and IRL) can be understood as performing moment matching under differing assumptions. In this work, we present a new perspective imitation in which the reward function is learned using dataset of behavior comparisons, generalizing previous IRL methods that learn from expert demonstrations and additionally giving the ficxibility to incorporate rankings over suboptimal behaviors.
Learning from preferences and suboptimal data is important when expert data is limited or hard to obtain. Preferences (Akrour et al. 2011 Wilson et al. 2012
Learning from Preferences and Suboptimal Data:
Sadigh et al.] 2017 Christiano al. 2017 Palan et al.) 2019 Cui al. 2021 have the advantage of providing guidance in situations expert might not get into, and in the limit provides full ordering over trajectories which expert data cannot. A previous line of work (Brown al.
2019
2020b
Chen et al. 2020) has studied this
setting and demonstrated that offline rankings over suboptimal behaviors can be effectively leveraged to learn a
reward function. Christiano et al. (217)); Palan et al. (2019] [barz et al. [0018) studied the question of lcarning from preferences in the setting when a human available to provide online preference (active queries), while Palan et al. (2019) additionally assumed the reward to be lincar known features. Our work makes no such as- sunptions and allows for integrating offline preferences and expert demonstrations under a common framework.
Learning from Observation (LfO): LfO the problem setting of learning from expert observations. This is typically more challenging than the traditional learning from demonstration setting (LfD), because actions taken by the expert are unavailable. LIO is broadly formulated using two objectives: state-next state marginal
matching (Toaabi et al. 2019 Zhu et al.) 2020b Sun et al., 2019 and direct state marginal matching (Ni ct al. 2020: Liu et al. 2019). Some prior works (Toaaii et al. 2018a Yang et al.] 2019 Edwards et al. 2019
approach LfO by inferring expert actions through a learned inverse dynamics model. These methods assume
Wee will preferences and ranking interchangebly
3

---

Published in Transactions on Machine Learning Research (01/2023)
injective dynamics and suffer from compounding errors when the policy is deployed. A recently proposed method OPOLO (Zhu et al. (020b) derives an upper bound for the LfO objective which enables it to utilize oof-py data and increase sample efficiency. Our method outperforms baselines including OPOLO, by a significant margin.
3 Background
We consider a learning agent in a Markov Decision Process (MDP) Puteeraa 2014 Sutton & Barto 2018
which can be defined as a tuple: M P, 7, where S and A are the state and action spaces; P is the state transition probability function, with P(s'|s, a) indicating the probability of transitioning from to s' when taking action a; : x A + R is the reward function bounded in [o, Rmazl; We consider MDPs with infinite horizon, with the discount factor y E [o. 1], though our results extend to finite horizons well; Po is the initial state distribution. We use IT and R to denote the space of policies and reward functions respectively. A reinforcement learning agent aims to find a policy π: : - A that maximizes its expected return, x) T-, (s,)(()))===ss [R(s, a)], where p=(ss a) is the stationary state-action distribution induced by π. In imitation learning, we are provided with samples from the state-action visitation of the expert p p= a) but the reward function of the expert, denotod by Rgt, unknown. We will use p=(ss a) as a shorthand for (s)
Classical Imitation Learning: The goal of imitation learning is to close the imitation gap J J at n J( defined with respect to the unknown expert reward function Rgt. Several prior works (Ho & Ermon 2016: Swamy et al. 2021 Kostrikov et al. 2019 Ni et al., 2020) tackle this problem by minimizing the imitation gap on all possible reward hypotheses. This leads to a zero-sum (min-max) game formulation of
imitation learning in which a policy is optimized with respect to the reward function that induces the largest imitation gap:
imit-game(r) arg min sup (s a)) Ex (s))())))))) *ell feR
Here, assuming realizability (Rgt E R), the imitation gap is upper bounded follows (Vr):
J(Rgr; x J( x) S sup JER 1 - (x)=(sa))f(ssss Ep (f
(2)
Note that, when the performance gap is maximized between the expert F and the agent т, we can observe that the worst-case reward function fr induces a ranking between policy behaviors based on their performance: F - =p= == F (e.a)) a)] 2 (= (s, (s. a)). Vr. Therefore, we can regard the above loss function that maximizes the performance gap (Eq. 2 as an instantiation of the ranking-loss, We will refer the implicit ranking between agent and the expert pd p= as vanilla rankings and this variant of the ranking-loss function as the supremamm-ooss.
Stackelberg Games: A Stackelberg game (Başar & Olsder 1998) is a general-sum game between two agents where one agent is set to be the leader and the other a follower. The leader in this game optimizes its objective under the assumption that the follower will choose the best response for its own optimization objective. More concretely, assume there are two players A and B with parameters 0,,, 0B and corresponding losses (A(OA, On) and (0, OR). A Stackelberg game solves the following bi-level optimization when A is the leader and B is the follower: ming, CO og (Oa)) s.t (0A)) arg ming (B 0). Raaeswaran ct al. (200)) showed that casting model-based RL as an approximate Stackelberg game leads to performance
benefits and reduces training instability in comparison to the commonly used GDA (Schäfer & Anandkumar
2019) and Best. Reponse (BR) "Jsa-BBianchi & Lugosi 2006) methods. Fiez et al. (2019): 2heng et. al. (2021
prove convergence of Stackelberg games under smooth player cost functions and show that they reduce the cycling behavior to find an equilibrium and allow for better convergence.
4 A Ranking Game for Imitation Learning
In this section, we first formalize the notion of the proposed two-player general-sum ranking game for imitation learning. We then propose practical instantiation of the ranking game through novel ranking-loss (T&).
4

---

Published in Transactions on Machine Learning Research (01/2023)
Algorithm Meta algorithm: rank-game (vanilla) for imitation
Initialize policy T8, reward funtion Rø, empty dataset D= empirical expert data
p
2: for t 1.T iterations do
31 Collect empirical visitation data gTd with To in the environment. Set D
{00 3 (P)
4: Train reward Rg satisfy rankings in D using ranking loss L1 in equation 3
5: Optimize policy under the reward function: m <- argmax,..
6.
end for
The proposed ranking game gives us the flexibility to incorporate additional rankings both auto-gencrated (a form of data augmentation mentioned 'auto' in Fig. D and ofllin ('pref" in Fig. D which improves learning efficiency. Finally, discuss the Stackelberg formulation for the two-player ranking game and discuss two algorithms that naturally arise depending on which player is designated as the leader.
4.1 The Two-Player Ranking Game Formulation
We present new framework, rank-game, for imitation learning which casts it as general-sum ranking game belween two players reward and a policy.
argmax xE ((R
argmin RERL angmingeRLL(0
Policy Agent
Reward Agent
In this formulation, the policy agent maximizes the reward by interacting with the enviromment, and the reward agent attempts to find reward function that satisfies set of pairwise behavior rankings in the given dataset DP; a reward function satisfies these rankings if E B, [R(s, a)] E p [R(s,a)], Vp" 3 p² E Dr, where p"i, .px" can be state-action or state vistitations.
The dalaset of pairwise behavior rankings DP can be comprised of the implicit 'vanilla' rankings between the learning agent and the expert's policy behaviors (p" 3 p²), giving us the classical IRL methods when a al.) 2020 Ke
specific ranking loss function supremum-loss is used (Ho & Ermon 2016 Ghasemipour et al.) 2021). If rankings are provided between trajectories, they can be reduced to the equivalent ranking between the corresponding state-action/state visitations. In the case when DD comprises purely of offline trajectory performance rankings then, under a specific ranking loss function (Luce-shepart), the ranking game reduces lo prior reward inference methods like T-REX (Brown et al. 2019 2020b a Chen et al. 2020).
Thus, the ranking game affords us a broader perspective imitation learning, going beyond only using expert demonstrations.
4.2 Ranking Loss LL for the Reward Agent
We use a ranking-lose to train the reward function an objective that minimizes the distortion [Iyer & Bilmes 2012) between the ground truth ranking for a pair of entities {æ, y} and rankings induced by a paramcterizod function R: X' + R for pair of scalars (R(x), One type of such ranking-loss is the supremum--osss in the classical imitation learning setup.
We propose a class of ranking-loss functions La that attempts to induce performance gap of k E [0, Rmaz] for all behavior preferences in the dataset. Formally, this can be implemented with the regression loss:
L+ (or . )o [Bna-prr (R((s, - 0) E samppr [R(ss, a) k)
[
where DP contains behavior pairs (0* , with the prespecified ranking p= 3 x=
The proposed ranking loss allows for learning bounded rewards with user-defined scale in the agent and
the expert visitations as opposed to prior works in Adversarial Imitation Learning (Ho & Ermon 2016 Fu et al. 2017 Ghasemipour al.) 2020). Reward scaling has been known to improve learning efficiency in deep RL; large reward scale can make the optimization landscape less smooth (Henderson al. 2018 Glorot &

---

Published in Transactions on Machine Learning Research (01/2023)
Bengio 2010) and small scale might make the action-gap small and increase susceptibility to extrapolation errors Bellemare et al. 2016). In contrast to the supremum loss, La can also naturally incorporate rankings provided by additional sources by learning a reward function satisfying all specified pairwise preferences. The following thcorem characterizes the equilibrium of the rank game for imitation learning when L is used as the ranking-losse
Theorem 4.1. (Performance of the rank- equdlibrium pair) Consider on equilibrium of the imitation rank ramk- (ft R), such that the ranking loss La generalization error is bounded by 2 2 and the policy is near-optimal with J [RR;; R) J(;; En Va, then at this equilibrium pair under the expert's 's unknown reward function Ryt bounded in [0, Rmazl:
[[ n ) [Rgt, R)1 S
4R
manx
&
(4)
If reward is a state-only function and only expert observations are available, the same bound applies the LfO setting.
Proof. We defer the proof to Appendix
A
Comments on Theorem 4.1 The ranking loss trains the reward function with finite samples using supervised learning. We can quantify 6,, the finite sample generalization error for the reward function, using standard concentration bounds (Shalev- Shwartz Ben-David 2014 Hoeffding 1994) with high prob- ability. We use E, to denote the policy optimization error from solving the reinforcement lcarning problem. In Deep Rein- forcement Learning, this error can stem as result of function approximation, biases in value function update, and finite sam- ples. Accounting for this error allows us to bring our analysis closer to the real setting. Note that the performance gap be- tween agent policy and expert policy depends on the scale of the expert reward function R This behavior expected as the performance gap arising as a result of differences in be- Daviios/visitations of agent policy and expert policy, can be amplified by the expert's unknown reward scale. We assume realizability i.e the expert reward function lies in the agent reward function class, which ensures that R < Rmaz- The performance bound in Theorem 4.1 is derived in Appendix by first proving an intermediate result that demonstrates p* and p=* are close in a specific f-divergence at the equilibrium, bound that does not depend the unknown expert reward scale RE
GAIL
rank-game
30
2
1
18
0
12
-2
Figure 2: Figure shows learned reward function when agent and expert has visitation shown by pink and black markers respectively. rank-game (auto) results in smooth reward functions more amenable to gradient-based policy optimization compared GAIL.
Thcoretical propertics: We now discuss some theoretical properties of LĄ. Thcorem 4.1 shows that rank- rank-game game has an equilibriun with bounded performance gap with the expert. Second, our derivation for Theorem 4.1 also shows that - an optimization step by the policy player, under a reward function optimized by the reward player, is equivalent to minimizing an f-divergence with the expert. Equivalently, at iteration in Algorithm max,i E, [R]] - -E,-. p= [R;] ming' Dy(p" [|p==). We characterize and elaborate on the regret of this idealized algorithm in Appendix A Theorem 4.1 suggests that large values of k. upto Rmazz can guarantce the agent's performance close the expert. In practice, we observe intermediate values of k also preserve imitation equilibrium optimality with a benefit of promoting sample efficient learning. We attribute this observation to the effect of reward scaling described earlier. We validate this observation further in Appendix D.9 rank-game -aae naturally extends to the LfO regime under a state-only reward function where Theorem results in a divergence bound between stale-visitations of the expert and the agent. A state-only reward function is also a sufficient and necessary condition to ensure that we learn a dynamics-disentangle reward function (Fu et al. 2017
LA can incorporate additional preferences that can help learn a vides better guidance for policy optimization, reducing the exploration burden and increasing sample efficiency
reward function that pro-
6

---

Published in Transactions on Machine Learning Research (01/2023)
for IRL. A better-guided policy optimization is also expected to incur lower Cy. However, augmenting the rank- ing dataset can lead to decrease in the intended performance gap (keeff < k) between the agent and the expert (Appendix Aj. This can loosen the bound in Eq | and lead to sub-optimal imitation learning, We hypothesize that given informative preferences, decreased 6 can compensate potentially decreased intended performance gap kess 5 to ensure near optimal imitation. In our experiments, we observe this hypothesis holds true; we enjoy sample efficiency benefits without losing any asymptotic performance. To leverage these benefits, we present two methods for augmenting the ranking dataset below and defer the implementation details to Appendix B
4.2.1 Generating the Ranking Dataset
Reward loss w/ automatically generated rankings (auto): In this method, we assume access to the belavior-generating trajectories in the ranking dataset. A trajectory T is a sequence of states (LfO) given by (80, 88) state-action pairs (LID) given by [8o, (0,,0n dn 81, 01,01,811, an] For each pairwise comparison Pi 3 Pj present in the dataset, La sets the regression targets for states in pj to be 0 and for states visited by Pj to be k. Equivalently, we can rewrite minimizing L; as regressing an input of trajectory т; to vector 0, and Tj to vector k1 where Ti, Ty are trajectories that generate the behavior Pi;Pi respectively. We use the comparison P: Pj to generate additional behavior rankings p; = Pxa.i I Py I Pxa.ij.. I Pxr.ij 3 Pj = PXP+1.5 where = Ao < A < A2 <... ... < Ap < =1 Ap+1. The behavior PAp,á is obtained by independently sampling the trajectories that generate the behaviors Pi. Py and taking convex combinations i.c. TAp.sf >pT + (1 Ap)TT and their corresponding reward regressions targets are given by kp 2p0 100+0 (1 Ap)k1. The loss function takes the following form:
P+1
(Dx Ep
1
Σ P=0
E
Bs (5,00) (R(s,a)) - kpp
P+2
This form of data augmentation can be interpreted mixup [hang et al. 2017 regularization in the trajectory space. Mixup has been shown to improve generalization and adversarial robustnoss (Guo et al. 2019 Zhang et al. 2017) by regularizing the first and second order gradients of the parameterized function. Following the general principle of using a smoothed objective with respect to inputs to obtain effective gradient signals, explicit smoothing in the trajectory-space can also help reduce the policy optimization error Eg. A didactic example showing rewards learned using this method is shown in Figure 2 In a special case when the expert's un- known reward function is linear in observations, these rankings reflect. the true underlying rankings of behaviors.
Reward loss w/ offline annotated rankings (pref): Another way of increasing learning efficiency is augmenting the ranking dataset containing the vanilla ranking (fp" 3 p²}} = D²) with offline annotated rankings (Doffline). These rankings may be provided by a human observer or obtained using an offline dataset of behaviors with annotated reward information, similar to the datasets used in offline RL (Fu et al.] 2020 Levine et al. 2020).. We combine oflline rankings by using a weighted loss between La for satisfying vanilla rankings (p" 3 p²) and offline rankings, grounded by an expert. Providing offline rankings alone that are sufficient to explain the reward function of the expert (Brown et al. 2019) is often difficult. task and the number of offline preferences required depends on the complexity of the environment. In the LIO setting, learning from an expert's 's state visitation alone can be a hard problem due to exploration requirements (Kidambi et al.) 2021). This ranking- losis combines the benefits of using preferences to shape the reward function and guide policy improvement while using the expert to guarantee ncar-optimal performance. The weighted loss function for this setting takes the following form:
(D Doffline, R) = a + (1 aa) a) LD R)
(6)
4.3 Optimizing the Two-Playee General-Sum Ranking Game as Stackelberg Game
Solving the ranking-gams game in the Stackelberg setup allows us to propose two different algorithms depending on which agent is set to be the leader and utilize the learning stability and efficiency afforded by the formulation studied in Rajeswaran et al. (2020); Zheng et al. (2021): Fiez et. al. (2019).
7

---

Published in Transactions on Machine Learning Research (01/2023)
Policy leader (PAL): Choosing policy the leader implics the following optimization:
max { [[(::)) s.l. R = arg min L(D"; R)
(7)
n
Reward as leader (RAL): Choosing reward as the leader implies the following optimization:
min (DD" s.t -= arg max J (ft
(8)
We follow the first order gradient approximation for leader's update from previous work (Rajeswaran et al. 2020) to develop practical algorithms. This strategy has been proven to be effective and avoids the computational complexity of calculating the implicit Jacobian term (d0)//0A)) PAL updates the reward to near convergence on dataset D (D" contains rankings generated using the current policy agent only 3 xB) and takes few policy steps. Note that even after the first-order approximation, this optimization strategy differs from GDA as often only few iterations are used for training the reward even in hyperparameter studies like Orsini et al. (2021). RAL updates the reward conservatively. This is achieved through aggregating the dataset of implicit rankings from all previous policies obtained during training. PAL's strategy of using on-policy data D" for reward training resembles that methods including GAIL (Ho Ermon 2016 Torabi et al., 2018b). f-MAX (Ghasemipour et al. 2020), and f-IRL (Ni al.) 2020). RAL uses the entire history of agent visitation and DAC (Abbeel & Ng. 20044 Kostrikov al. 2018). PAL and RAL bring together two scemingly diffcrent. algorithm classes under unified Stackelberg game viewpoint.
update the reward function and resembles methods such as apprenticeship learning
5 Experimental Results
We compare rank-game against state-of-the-arl LIO and LID approaches on MuJoCo benchmarks having continuous state and action spaces. The LfO setting is more challenging since no actions are available, and is a crucial imitation learning problem that can be used in cases where action modalities differ between the expert and the agent, such as in robot learning. We focus on the LfO setting in this section and defer the LiD experiments to Appendix D.2 We denote the imitation learning algorithms that use the proposed ranking-loss La from Section 4.2 RANK-(PAL, RALJ. We refer to the rank-game variants which use automatically generated rankings and offline preferences as (aulo) and (pref) respectively following Section 4.2 In all our methods, rely updating the policy agent (in step 5 of Algorithm D.
off-policy model-free algorithm, Soft Actor-Critee (SAC) (Haarnoja et al.
2018 for
We design experiments to answer the following questions:
1. Asymplolic Performance and Sample Efficiency: Is our method able to achieve near-expert performance given a limited number (one) of expert observations? Can our method learn using fewer environment interactions than prior statc-of-tho-art imitation learning (LfO) mothods?
2. Utility of preferences for imitation learning: Current LfO methods struggle to solve number of complex manipulation tasks with sparse success signals. Can we leverage offline annotated preferences through rank-gam in such environments to achieve near-expert performance?
3. Choosing belween PAL and RAL methods: Can we characterize the benefits and pitfalls of each method, and determine when one method is preferable over the other?
4. Ablations for the method components: Can we establish the importance of hyperparameters and design decisions in our experiments?
We compare RANK-PAI PAL and RANK-RAL against 6 representative LfO approaches that covers a spectrum of on-policy and off-policy model-free methods from prior work: GAIIO (Toabi et al. 2018h
Baselines:
Ho & Ermon 20166). DACfO (Kostrikov al. 2018), BCO (Torabi al. 2018a). f-IRL (Ni al. 2020 and recently proposed OPOLO (Zhu al. 2020b) and TQLearn (Garg al.] 2021 We do not assume
8

---

Published in Transactions on Machine Learning Research (01/2023)
access to expert actions in this setting. Our LfD experiments compare to the IQLearn Garg et al. 2021 DAC (Kostrikov et al. 2018) and BC baselines. Detailed description about the environments and baselines can be found in Appendix
5.1 Asymptotic Performance and Sample Efficiency
In this section, we compare RANK-PAL(auto) and RANK-RAL(auto) to baselines set of MuJoCo locomotion lasks of varying complexities: Swimmer-v2, Hopper-v2, HalfCheetah-v2, Walker2d- v2, Ant-v2 and Humanoton d-v2. In this experiment, we provide one expert trajectory for all methods and do not assume access to any offline annotated rankings.
Env
Hopper
HalfCheetah
Walker
Ant
Humanoid
BCO
20. 20.10-2.11
5.12+3.82
4.00+1.22
12.80+1.26
3.90+1.24
GalFO
81.131 9.99
13.5417.24
83.8312.55
20.10:24.44
3.93=1881
DACIO
94.73+33.3
85.03+5.009
54.70-444.61
86.45+11.67
19.31+332.1
-IRL
97.45 0.61
96.0614.63
101.16=1.25
71.18#19.80
77.93:16377
OPOLO
89.5615.46
88.9213.20
79.19122.35
93.374 3.78
21.87:17.001
RANK-
87.14 16.14
94.05=33.99
93.88=0072
98.93:1.88
96.84+3.22
PAL(our))
RANK-
99.34.10.00
101.14.7.45
93.2411.22
93.2112.98
44.45:14.13
RAL(OUT))
Expert
100.00:1 0
100.00:1 0
100.00:: O
100.00 0
100.00:: 0
(1S), (A])
(11,3)
(17,6) 6)
(17,
(111,8)
(376, 17)
Table 2: Asymptotic normalized performance of LfO methods at 2 million timesteps on MuJoCo locomotion tasks. The standard deviation calculated with diffferent runs each averaging 10 trajectory returns. For unnormalized score and more details, check Appendix D We omit IQlearn due to poor performance.
Swimmer-v2
Hopper-v2
HalfCheetah-v2
4000
Average Reward
12500
160
100
2000
7500
5000
1000
2900
-60
166
1c5
166
Walker-v2
Ant-v2
Humanoid-v2
Average Reward
6000
4000
4000
2000
2000
Timesteps
Timesteps
Timesteps
RANK-PAL (ours)
RANK-RAL (ours)
OPOLO
IQ-learn
F-
DACIO
GAIIO
BCO
Figure 3: Comparison of performance on OpenAl gym benchmark tasks. The shaded region represents standard deviation across 5 random runs. RANK-PAL and RANK-RAY RAL substantially outperform the baselines sample efficiency. Complete set of results can be found in Appendix D.I
Asymptotic Performance: Table shows that both game methods are able reach near-expert asymptotic performance with single expert trajectory. BCO shows poor performance which can be attributed

---

Published in Transactions on Machine Learning Research (01/2023)
to the compounding error problem arising from its behavior cloning strategy. GAIfO and DACIO use GDA for optimization with supremum loss and show high variance in their asymptotic performance whereas rank-game methods are more stable and low-variance.
Sample Efficiency: Figure B shows that RANK-RAL and RANK-PAL are among the most sample efficient methods for the LiO setting, outperforming the recent state-of-the-art method OPOLO (Zhu et al. 2020b) by significant margin. We notice that IQLearn fails learn in the LfO setting. This experiment demonstrates the benefit of the combined improvements of the proposed ranking- with automatically generated rankings. require fewer lines of code changes on top of
Our method is also simpler to implement than OPOLO, SAC and need to maintain fower parameterized networks compared to OPOLO which requires an additional inverse action model to regularize learning.
5.2 Utility of Preferences in Imitation
Our experiments on complex manipulation environments door opening with a parallel- jaw gripper (Zhu et al. 2020a) and pen manipulation with a dexterous adroit hand (Rajeswaran el al.,
Door-v0
Pen-v0
Average Reward
100
2000
2017] reveal that none of the prior LfO methods are able to imitate the expert even under increasing
200
2000
1000
amounts of expert data. This failure of LfO methods
can be potentially attributed to the exploration requirements of LfO compared to LiD) (Kidambi
0/
455
Timesteps
15
20
25
3.0
00
1.2 Timesteps
2.0
RANK-PAL (praf)
RANK-PAL (auto)
T-REX
OPOLO
DACIO
ct al., 2021), coupled with the sparse successes encountered in these Lasks, leading to poorly guided policy gradients.
RANK-RAL (preO)
RANK-RAL (auto)
Offline annotated preferences can help solve LfO tasks in the complex manipulation environments Pen-v0 and Door, whereas prior LIO methods fail. Black dotted line shows asymptotic performance of RANK-PAL (auto) method.
Figure 4:
In these experiments, we show that rank-game can incorporate additional information in the form of offline annotated rankings to guide the agent in solving such lasks. These ofline rankings are obtained by uniformly sampling a small set of trajectories (10) from the replay buffer of SAC (Haarnoja et al. 2018 labeled with a ground truth reward function. We use a weighted ranking loss (pref) from Section 4.2
Figure 4 shows that RANK-PAAL/RALLpa solve these tasks, whereas prior LiO methods and SANNK-PALRAA((ME
method leveraging offline ranking is the only method that can
with automatically generated rankings struggle even after large amount of training. We also point out that T-REX, an offline method that learns using the preferences grounded by expert is unable to achieve near-expert performance, thereby highlighting the benefits of learning online with expert demonstrations alongside a set of offline preferences.
5.3 Comparing PAL and RAL
PAL uses the agent's 's current visitation for reward learning, whereas RAL learns a reward consistent with all rankings arising from the history of the agent's 's visitation. These properties can present certain benefits depending on the task setting.
To test the potential benefits of PAL and RAL, we consider two non-stationary imitation Icarning problems, similar to (Rajeswaran et al. 2017) one in which the expert changes it's intent and the other where dynamics of the environment change during training in the Hopper-v2 locomotion task. For changing intent, present a new set of demonstrations where the hopper agent hops backwards rather than forward. For changing environment dynamics, we increase the mass of the hopper agent by a factor of 1.2. Changes are introduced at le5 time steps during training at which point we notice a sudden performance drop.
In Figure (left), we notice that PAL adapts faster to intent changes, whereas RAL needs to unlearn the rankings obtained from the agent's history and takes longer to adapt. Figure 5 (right) shows that RAL. adapts faster to the changing dynamics of the system, it has already learned good global
10

---

Published in Transactions on Machine Learning Research (01/2023)
notion of the dynamics--isennaangled reward function in the LfO setting, whereas PAL only has local understanding of reward as a result of using ranking obtained only from the agent's 's current visitation.
Intent-Adaplation
Dynamics-Adaptation
Ablation of Method Components: Appendix
Average Reward
contains eight additional experiments to study the importance of hyperparameters and design decisions. Our ablations validate the importance of using auto- matically generated rankings, the benefit of ranking loss over supremum loss, and sensitivity to hyperpa- rameters like the intended performance gap k, policy iterations, and the reward regularizer. We find that key improvements in learning efficiency are driven by using the proposed ranking loss, controlling the reward range, and the reward/ policy update frequency in the Stackelberg framework. In Figure ( we also analyze changed, and right plot shows comparison when dynamics the performance of rank-gen -ame with a varying num- of the environment. is changed. These changes occur at ber of expert trajectories and its robustness to noisy 1e5 timesteps into training. PAT, adapts faster changing offine--nnooated preferences. We find rank-game to intent and RAL adapts faster to changing dynamics.
2003
2000
1000
1000
-1000
02
05
1.
1.5
3.0
0.2
0.5
1.0
1.5
3.0
Timesteps
Timesteps
RANK-PAL
RANK-RAL
Figure 5: We compare the relative strengths of PAL and RAL. Left plot shows a comparison when the goal is
consistently outperform baselines with a varying num-
ber of expert trajectories. On Door manipulation task rank rank-game is robust to 60 percent noise in the ofline--nnotated preferences. Experiments more environments and additional details can be found in Appendix D
Hol #alrCheetah RAME-RLL Jactod a dioto
Lumanoid-v2 a100-9a1 (ouno) #
SPOLO
Door-vo
DAZEO
secso
Average Reward
4200
Rotuen
4300
2309
1500
I-erpeort
Z-ropert
Betapertt
2 Pragpert
eb Sborreere
-100
Fnergest
Z-orgere
Srngert
10-epeort
Timesteps
1099H12 R000m0/2
(b)
Figure 6: (a) RANK-PAL outperforms other methods with varying number of expert trajectories. Error bars denote standard deviation. (b) On Door-v0 enviroument, RANK- (ALppe)) robust to at least 60 percent noisy preferences.
6 Conclusion
In this work, we present a new framework for imitation learning that treats imitation as a two-player ranking-game between a policy and a reward function. Unlike prior works in imitation learning, the ranking game allows incorporation of rankings over suboptimal behaviors to aid policy learning. We instantiate the ranking game by proposing a novel ranking loss which guarantees agent's performance to be close to expert for imitation learning. Our experiments on simulated MuJoCo tasks reveal that utilizing additional ranking through our proposed ranking loss leads to improved sample efficiency for imitation learning, outperforming prior methods by a significant margin and solving some tasks which were unsolvable by previous LfO methods.
Limitations and Negative Societal Impacts: Preferences obtained in the real world are usually noisy (Kwon al. 2020: Jeon et al.) 2020 Biyik et al. 2021) and one limitation of rank-game is that it does not suggest way to handle noisy preferences. Second, lank-game proposes modifications to learn reward function amenable to policy optimization but these hyperparameters are set manually. Future work can explore methods to automate lcarning such reward functions. Third, despite learning effective policies we observed that we do not learn reusable robust. reward functions (Ni el al. 2020). Negalive Societal Impact: Imitation learning can cause harm if given demonstrations of harmful behaviors, either accidentally or purposefully. Furthermore, even when given high-quality demonstrations of desirable behaviors, our algorithm does not provide guarantees of performance, and thus could cause harm if used in high-stakes domains without sufficient safety checks on learned behaviors.
11

---

Published in Transactions on Machine Learning Research (01/2023)
Acknowledgements
We thank Jordan Schneider, Wenxuan Zhou, Caleb Chuck, and Amy Zhang for valuable feedback on the paper. This work has taken place in the Personal Autonomous Robotics Lab (PeARL) at The University of Texas at Austin. PeARL research is supported in part by the NSF (TIS-1724157, TTS- 1541749,,4, TTS-1925082) AFOSR
and ARO (78372-CS). This research was also sponsored by the Army Research Office under Cooperative Agreement Number W911NF-19-2-088.. The views and conclusions contained in this document are those of the authors and should not be interpreted representing the official policies, either expressed implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
12

---

Published in Transactions on Machine Learning Research (01/2023)
References
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Riad Akrour, Marc Schoenauer, and Michole Scbag. Preference-basee policy learning. In Joint Europeon Conference on Machine Learning and Knowledge Discovery in Dalabases, pp. 12 27. Springer, 2011.
Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistienl Sociely: Series (Methodologicul), 28(1):131- 1966.
Oleg Arenz and Gerhard Neumann. Non-adversarial imitation learning and its connections to adversarial methods. Xrv preprint ar Xrxi:20008. 03525, 2020.
Tamer Başar and Geert Jan Olsder. Dynamic noncooperative game theory. SIAM, 1998.
Marc G Bellemare, Georg Ostrovski, Arthur Guez, Philip Thomas, and Rémi Munos. Increasing the action gap: New operators for reinforcement learning, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Erdem Biyik, Dylan Losey, Malayandi Palan, Nicholas C Laaddlli, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, pp. 02113649211041652, 2021.
Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Nickum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning, pp. 1165 1177.
PMLR, 2020a.
Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott. Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. ArXiv, abs/1194.00637, 2019.
Daniel Brown, Wonjoon Goo, and Scott Nickum. Bettor-than-demonstraato imitation learning via automaticalll-raakee demonstrations. In Conference on robot learning, pp. 330 359. PMLR, 2020b.
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthow E Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In Twenly--ourth international joinl conference on artificial intelligence, 2015.
Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self- supervisod reward regression. arXie preprint arXiv:2010.11 111222 2020.
Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. arXiv preprint arXiv:1706. 1xX1:11066.0844,, 2017.
Imre Csiszár. Information-type measures of difference of probability distributions and indirect observation. studia scientiarum Mathematicarum Hungarica, 2:229-318, 1967.
Yuchen Cui, Bo Liu, Akanksha Saran, Stephen Giguere, Peter Stone, and Scott. Niekum. Aux-airl: End-Lo-end self-supervised reward learning for extrapolating beyond suboptimal demonstrations. In Proceedings of the ICML Workshop on Self-Supel Self-Superoised Learning for Reasoning and Perception, 2021.
Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. In International Conference on Machine Leurning, pp. 1755 1763. PMLR, 2019.
Tanner Fiez, Benjamin Chasnov, and Lillian Ratliff. Convergence of learning dynamics in stackelberg
games. XiXiv preprint arXiu:1966.012f 2019.
13

---

Published in Transactions on Machine Learning Research (01/2023)
Chelseaa Finn, Sergey Levine, and Picter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference machine learning, pp. 49-58. PMLR, 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXi::170.111 11288, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint ar xrX::220440072 2020.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34, 2021.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective imitation learning methods, In Conference Robot Learning, pp. 1259- 1277. PMLR, 2020.
Gustavo Gilardoni. On the minimum f-divergence for given total variation. Comptes Rendus Mathematique C MATH, 343:763-766, 12 2006. doi:
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-2566 JMLR Workshop and Conference Proceedings, 2010.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regularization. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3714 3722, 2019.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAT conference on artificial intelligence, volume 32, 2018.
Jonathan Ho and Stelano Ermon. Generative adversarial imitation learning. Advonces in neural information processing systems, 454555-4377, 2016.
Wassily Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding, pp. 409-426. Springer, 1994.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. arXiv preprint 1xXi:1811.0683 2018.
Rishabh Iyer and Jeff Bilmes. The submodular bregman and breggan divergences with applications: Extended version. Cileseer, 2012.
Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33:4415 4426, 2020.
Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Chao Yang, Bin Fang, and Huaping Liu. Reinforcement learning from imperfect demonstrations under soft expert guidance. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5109-5116, 2020.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. In Algorithmic Foundations of Robotics XIV: Proceedings of the Fourteenth Workshop on the Algorithmic Foundations of Robotics 14, pp. 313-320 Springer International Publishing, 2021.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Michael Kearns and Satinder Singh. Finite-sample convergence rates for q-learning and indirect algorithms. Advances in neural informalion processing systems, 11, 1998.
Rahul Kidambi, Jonathan Chang, and Wen Sun. Mobile: Model-based imitation learning from observation alone. Advances in Neurol Information Processing Systems, 34, 2021.
lyya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Disei-iintor--atooocttti:::: Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprinl arv: 1809, 0020:9999, 2018.
llya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. arXiv preprinl ar 1rx:i:1111 05032, 2019.
Victoria Krakovna. Specification gaming examples ai. Available af vkrakoena. wordpress. com, 2018.
Minae Kwon, Erdem Biyik, Aditi Talati, Karan Bhasin, Dylan P Losey, and Dorsa Sadigh. When humans aren't optimal: Robots that collaborate with risk-aware humans. In 2020 15th ACM/IEEE International Conference on Human-Robot Interaction (HRT), pp. 43-52. IEEE, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 2rXiv:2005.014 2020.
Friedrich Liese and Igor Vajda. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10):4394 4412, 2006.
Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment-based imitation learning, arXiv preprint Xre: 19t1::911.10999 2019.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEFE international conference on computer vision, pp. 2794 2802, 2017.
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, pp. 2, 2000.
Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Benjamin Eysenbach. f-ill: Inverse reinforcement learning via state marginal matching. arXiv preprint arXiv:2011.04700, 2020.
Manu Orsini, Anton Raichuk, Léonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, and Marcin Andrychowicz. What matters for adversarial imitation learning? Advances in Newral Information Processing Systems, 34, 2021.
Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions by integrating human demonstrations and preferences. arXiv preprint arXiv:1906.00 08928, 2019.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Newral computation, 3(1):88-97, 1991.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic progromming. John Wiley & Sons, 2014.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint orxiv: 70t.1779.1100 10087, 2017.
Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. A game theoretic framework for model based reinforeement learning. In ICML, 2020.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. Xiv preprint X9m: 11108, 2019.
15

---

Published in Transactions on Machine Learning Research (01/2023)
Alfréd Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium Mathemalical Statislics and Probability, Volume 1: Contrimutions to the Theory of Stolislics, pp- 547-561. University of California Press, 1961.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and slatislics, pp. 627 635. JMLR Workshop and Conference Proceedings, 2011.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learning of reward functions. 2017.
Florian Schäfer and Anima Anandkumar. Competitive gradient descent. Xiv preprint or 2019.
Shai Shalev-Shwartz and Shai Ben-David Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Wen Sun, Anirudh Vemula, Byron Boots, and Drew Bagnell. Provably efficient imitation learning from observation alone. In International conference on machine learning, pp. 6036 6045. PMLR, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Gokul Swamy, Sanjiban Choudhury, Andrew Bagnell, and Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pp. 10022 10022-10032. PMLR, 2021.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint MrXii:1805.0100 2018a.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXie:1807.06155 2018b.
Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from observation. arxiv preprint arXiv:1903 10oc199051366,, 2019.
Igor Vajda. Note on discrimination information and variation (corresp.). IEEE Transactions on Information Theory, 16(6))771-773, 1970.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. Adnances in neural information processing systems, 25:1133-1 1141, 2012.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. arXiv preprint arXiv:1910004417, 2019.
Haoran Zhang, Chenkun Yin, Yanxin Zhang, and Shangtai Jin. Self-guided actor-critic: Reinforcement learning from adaptive expert demonstrations. In 2020 59th. IEFE Conference on Decision and Control (CDC), pp. 572 577. IEEE, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez- Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 0tt:::171.0004. 09410. 2017.
Liyuan Zheng, Tanner Ficz, Zane Alumbaugh, Benjamin Chasnov, and Lillian J Ratliff. Stackelberg actor-critic: Game-theoretic reinforcement learning algorithms. arXiv preprint arXiv:2109. 1228662099.1286, 2021.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint or Xiv:2000. 12293.2293. 2020a.
16

---

Published in Transactions on Machine Learning Research (01/2023)
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. Aduances in Newral Information Processing Systems, 33, 2020b.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433 1438. Chicago, II., USA, 2008
17

---

Published in Transactions on Machine Learning Research (01/2023)
A
Theory
We aim to show that rank-game has an equilibrium that bounds the f-divergence between the agent and the expert (Theorem A.1] in the imitation learning setting. For imitation learning, we have the vanilla implicit ranking pagert N pp between the behavior of agent and the expert. Later, we show that, the bounded f-divergence can be used to bound the performance gap with the expert under the expert's unknown reward function using a solution to Vajda's 's light lower bound (Corollary [A.1.]]. Our proof proceeds by first showing that minimizing the empirical ranking loss produces a reward function that is close to the reward function obtained by the true ranking loss. Then, show that even under the presence of policy optimization errors maximizing the obtained reward function will lead to bounded f-divergence with the expert.
Theorem A.1. (Performance of the rank-game equilibrium pair) Consider an equilibrium of the iniiaation rank- game (t, R), such that R ménimizes the empirical ranking-loss loosn for dataset D* ((p=,,=))> and the ranking-lor loss generalization error is bounded by c, = 2Rmex' and the policy &t has bounded suboptimality with J (ft; +) 2 JBe; x²) ex VT, then we have that at this equilibrium pair:
D, (o (s, (s)))))(()))))) S (1-7)er + RRmax V 2E,
k
where Dj is an f- divvergeecc with the generator function f(x)
1-x
Réngd 1961:
Ali 8 Silvey 1966
1+x
Csiszár 1967 Liese Vajda 2006).
Proof Previous works (Xu et al. 2021 Swamy et al. 2021 characterize the equilibrium in imitation learning based the supremum ranking loss/ min-max adversarial sctting under error assumption. In this section, we consider the ranking loss function La and derive the equilibrium for the rank-game in presence of reward learning and policy optimization errors. Lx attempts to explain the rankings between the agent and the expert using their state-action visitations D= = {o (s, pF (s, a)} respectively, by attempting to induce a performance gap of k. With this dataset D=, La regresses the return of state or state-action pairs in the expert's 's visitation to a scalar k and the agent's visitation to a value of Thus, we have:
(D R) (s, [ [(((s,a) R) + (s [R(s, a) =
(10)
The above ranking loss is minimized (V = 0) pointwise when
((pF(s.))
a² p=((sa) + pT
(11)
In practice, we have finile samples from both the expert visilation distribution and the agent distribution we minimize the following empirical ranking loss (D; R):
LD; R) Ex [[(ss a) k) Enneor [(((s, a) - -
(12)
where o and ~r are empirical stale-action visitations respectively.
From empirical loss function to reward optimality: Since the reward function is trained with supervised
learning, we can quantify the sample error in minimizing the empirical loss using concentration bounds (Shaloy- Shwartz & Ben-David 2014) up to a constant with high probability. Since < R(s, a) < Rmax With high
probability,
VR, (D: (a(:)) ( 2
(13)
where er is the statistical estimation error that can be bounded using concentration bounds such as Hoeffding's. ss Let R' belong to the optimal solution for (D; R) and R¹ belong to the optimal minimizing solution for La (D; R). Therefore, we have that,
VR, Ln,D;; RC)) < (D; R)
(14)
Using Eq 13 and Eq 14
we have
VR, LA(D <
(15)
$ LD(D;; R) + 2 2R2
(16)
< LD( R²) 2122 2 Er
(17)
18

---

Published in Transactions on Machine Learning Research (01/2023)
and similarly
VR, Lx(D;R") S RR)
(18)
S (D; R)) + 22
(19)
< (D; ft") + 22
(20)
Eq
17
and Eq
20
implies that (D; and (D( ft*) are bounded with high probability. i.e (D; R²* - (D; 1e) 2
(21)
We will use Eq
21 show that indeed R has a bounded loss compared to R
(D; LD; 2 C LD(D; R*) 2R Rmaa (D(;; LD; Rt) 4R Cm
(22)
R* E S 2R Ey
(23)
(24)
We consider the labular MDP setting and overload R to denole a vector of reward values for the entire state-action space of size IS x Al. Using the Taylor series expansion for loss function Lb, we have:
Lx(D;tt) LA (D) S 4R2 R
(25)
(D) + (V (D; ft - -R²) +0.5(Rt T Rt) Tx R²) & 4R fr
(26)
RR) R*) ) 1 8R Cr
(27)
where H denotes the hessian for the loss function w.r.t R and is given by p + apd
where pe is a
matrix of size IS x A|x x Al with a vector of visitations as its diagonal and zero elsewhere.
(fr RR) T H(A - R³) I &Rmare (s)
(28)
Ea-p' [fft (s,aa) R2 (8, ()) + Eoupe (10))) 18 a))² < 8R
(29)
Since both terms in the LHS are positive we have Braanp (f' (s) S 8 & Cr
and Examp' (ff - R² as) s 8R Applying Jensen's inequality, we further have (Be,arp* (s, (s. j && and (Eg (s, R* (s) < 8R Hence,
R²" E BeaxpE (s, -R*(s, a) Rmar V8,,
Ess -
Rmar and
(30)
(31)
At this point we have bounded the expected difference between the reward functions obtained from the empirical ranking loss and the true ranking loss. We will now characterize the equilibrium obtained by learning policy on the reward function ft* that is optimal under the empirical ranking loss. Under a policy optimization error of e we have:
J ;R 2 J(P( VT E
(32)
where J(R; R) denotes the performance of policy & under reward function R. Taking T' == we can reduce the above expression as follows:
(1Ki, J siP a) Sea
(33)
1 7 E (x.() [s. - E (a) R [s,00] sc
(34)
19

---

Published in Transactions on Machine Learning Research (01/2023)
Using Eq 30 and Eq 31 we can lower bound (s,(s R (s,
Ep (x,a) Ar (s, (s. a)] as follows:
E (s,a))))
Pe" (x, a)
2 E ( ( 110 a)] Rmaa Scr
(35)
E (s.a)
R¹ (s)
3 Exp (s.a) [ a) + Rmav V Sc,
(36)
where (s( a) is given by Equation
11
Subtracting Equation 36 from Equation
35
we have
E (s.() At* (s, a) (x.)) Ar (s.
a) (H (s, (1 a)l - 2Rmaz /8r
(37)
Plugging in the lower bound from Equation 37 in Equation 34 we have:
1
[ (a (s. as) - E p (s.0) R (s,a)) 2 22 |C,] S CT
(38)
1-7
Replacing R" using Equation we get,
(spE(s,a) pE (s) + (=)(s)
kspB(s,a))
1- E (Ma))
- E Ex.
2R v v86r
S c
(39)
(5((s)) + p= (s)
ksp"(ss
E (s.aa)
(e.a)
$ (1 - 7)e + 22Raz Ser
(40)
B + (8,
(8, + (8,a) (s (s,a))
(1 - 7)xx + 22maa Voe,
(x,a))
(41)
(5, p" (8,
k
The convex function f(x) 1-z in + defines an f-divergence. Under this generator function, the LHS of Equation 41 defines an f-divergence between the state-visitations of the agent p= a) and the expert p as. Hence, we have that
1+z
(s, (1 7)7 + 4Rmax v 22,
(42)
k
This bound shows that the equilibrium of the ranking game a near-optimal imitation learning solution when ranking loss target trades off effectively with the policy optimization error e, and reward generalization error 6y. We note that, since k S Rmar we can get the tightest bound when k Rmax- Now, in imitation learning both l and Rmar are tunable hyperparameters. We vary k while keeping k Rmer and observe in appendix D.9 that this hyperparameter can significantly affect learning performance.
Corollary A.1.1. (From f f-divergence to performance gap) For the equilibrium of the rank rank- (#, R) described in Theorem A.11 have that the performance gop of the expert policy with ă is bounded under the unknown expert's 's reward function (rge) bounded in [0, R follows:
ERE Raz
2), + +4Rmum 2r,
J( .rge) (r9)) r(
(43)
Proof. In Theorem A.1 we show that the cquilibrium of rank-game ensures that.the f-divergence of expert visitation and agent visitation bounded with the generator [unction / I-z First we allempl to find a tight lower bound of our f-divergence in terms of the total variational distance between the two distributions, Such bound has been discussed in previous literature for the usual f-divergences like KI, Hellinger and T-
x². This problem of finding a tight lower bound in terms of variational distance for general f-divergence was introduced in Vajda (1970) and referred to as Vajda's tight lower bound and solution for arbitrary f-divergences was proposed in Gilardoni (0006). The f-divergence with generator function f = I+e that f()) ( + (1) 1) and hence the total variational bound for this f divergence takes the form
I-r satisfies
20

---

Published in Transactions on Machine Learning Research (01/2023)
Dj 2- Drv f(2 ((2DDx)) Drx) (1) Plugging in the function definition f 1- the inequality simplifies to:
(s (s) 2 (s.l
(44)
We also note an upper bound for this f-divergence in TV distance, sandwiching this particular f-divergence with TV bounds:
(s(.a)
(s(,,
Dy(0 (s, (x.))
(s,0)
(45)
( pT
(P pm (s,a)
( (3( + p* (s,a)
()( -60)-pp0 ((8.0))-"(ss.0))
(46)
s,aESxA
(s..(("(s
(47)
So,
(s,
(rv(o"ss.a)ll (s(0" >
(48)
4
Therefore from Eq 42 we have that,
(1 7)67 4mmax /e,
Dpv (s, (s,a))ll"(sss s 2/
(49)
For any policy n, and experts unknown reward function ritt [TT, r) = T a (r(s, Therefore,
[TT", rgx) [ rgr)
E p (s
B Ba Io
VT
(50)
Σ 1 - -
(51)
8,0ESxA
RE
max
10B - -p")
(52)
1
s,0ESxA
2RE max
DrV
(53)
(54)
where R is the upper bound for the expert's reward function. Under a worst case expert reward function which assigns finite reward values to the expert's visitation and -oo outside the visitation, even small mistake (visiting any state outside the expert's visitation) by the policy result in infinite performance gap between expert and the agent. Thus, this parameter is decided by the expert and is not in control of the learning agent.
From Eq 49 and Eq 53 we have
4RE E max
(1 ---) (ele)++fftt +4llma 2er
J(m rrtt) - J(ft,rg()s
(55)
Lemma A.2. (Regret bound under finile data assumptions) Let Mr denote the approzimate transition model under the collected dataset of transitions until iteration t. Assume that the ground truth model and the reward function are realizable. Under these assumptions the regret of rank-game fth iteration:
M 5 27, Rmax 4Rmaz
2estat + 4.maa
vx M
Dy(oTE 1177)
(56)
(1 (1-7)2 1-
k
21

---

Published in Transactions on Machine Learning Research (01/2023)
where Vī denotes the performance of policy under transition dynamics M, En is expected model error under policy x² 's visilation, PM is the visitation of policy in transilion dynamics M and stat is the stalistical error due to finite expert samples.
Proof. We use M to denote the ground truth model and M, to denote the approximate transition model with collected data until the 9dh iteration of rank-game. We are interested in solving the following optimization problem under finite data assumptions:
max E, a Ma [f(e.aa] Esaapelftts,
s.t. ft arg min
(57)
10
where o is the empirical distribution generated from finite expert samples and M, (1 M. PR)). Using standard concentration bounds such Hoeffding's (Hoeffding 1994), we can bound the empirical estimate with true estimate Vx with high probability:
Exaepe (fr [facpelfi(s,a]
E, Ex- (s, a)l
Catat
(58)
Using the concentration bounds and the fact that x' the solution that maximizes the optimization problem Eq 57
t-iteration,
(8,
E Baappp [f)(
>E
(59)
saapps
1
Mi,
2 E sampps (fio((,))) - B, Bepanp [fia (5,
Estat
(60)
Mr;
fr. is the reward function that minimizes the empirical ranking loss L.p. Let fa be the solution to the true ranking loss Lg. As shown previously in Eq 30 and Eq 31 we can bound the expected values of these two quantities with high probability under agent or expert distribution.
We also have from concentration bound:
c. (s. Enngps (
E
samppt [f2(
B Baaa f (s.
Extat
(61)
Therefore, combining Eq
61 and Eq 59
E s.appp 23, (8))( E Egga (s) 2 E
[ E Beanps (fis(s,a)) -Estaat
sampp Mr.
(62)
The LHS of the Eq. can be further upper bounded as follows:
E eaappp M.
(5, E, Egg (s,)(
B < saapptt (+,(),) - B Beanng (b, + 2R /Ber
(63)
Mi
kpTt (s,a)
E s.app
p (s, + P (8,
kp (s,a)
E sampp
22 VSc,
(64)
Th (s, + prx (8,a)
P (s,
P3r (s,a) a)
= EE
+ 22
(65)
Laupp
P (s, pxt (s, a)
= =DkD N3, 11px50 + 22Rma
(
a /ep
(66)
22

---

Published in Transactions on Machine Learning Research (01/2023)
Similarly the RHS of Eq 62 can be further lower bounded follows:
E spampr ( ai] - E sapppt [ (3, Nr
ststat
(67)
2 E spapp [5 (s, a)] - -Bsnpy [s,
2Eceat - 22maxx v SE,
(68)
P Mr (s, PM (s, a)
kE s.aruprr
-2istat -2Rmax v Sc,
(69)
PM (8, + PM (s.a)
= = y F.
Mr l1
2E - 22 V Se,
(70)
Plugging the relations obtained (Eq 70 and 66 back in Eq 62 we see that the f-divergence between the agent visitation in the learned MDP and the expert visitation the ground truth MDP is bounded by the f-divergence of the expert policy's s visitation on the learned vs. ground truth environment. We expect this term to be low if the dynamics are accurately learned at the transitions encountered in visitation of expert.
Dy(0%, (1pT)) S (y(p%), (PMM) 2Eseat + 4Rmaa V&c,
(71)
We can use the vatalivariation lower bound for this f-divergence to later obtain a performance bound between the policy in learned MDP and expert in ground-truth MDP.
M, (PM)) + 2estat + 4Rmaa /6,
Drv (P llpTf) s 2 Dy
M, (P())) S 2 Dy(P,,
(72)
k
Similar to Corollary A.I.I we can further get a performance bound:
2Rmas
Pmma
26 + 4 Roma v /Ee
V
V Is
1-y Drv Il I
1-y V Di l1%h))
(73)
k Let the local model error in the visitation of nt be bounded by exi i.e Beawor m. Using simulation lemma for local models (Kearns & Singh 1998
a))]
-
Kakade Langford 2002). we have:
27 Rmar
V
V
(74)
(1-7)²
We are interested in bounding the performance of the policy xf in ground-truth MDP rather than the learned MDP.
V + 4Rmax
De 4Rmaz v /8,
V
VT
V Di(PT,, (P38))
(75)
M
1 1-y
k
2 Rimax
4 4Rmax
1 1-7 v Dy (1p3))) 2estat + 4 ARmax Ber
+
(76)
7)
The regret of algorithm with ranking-loss depends on the accuracy of the approximate transition model at the visitation of the output policy иe and the expected accuracy of the approximate transition model at the transitions encountered in the visitation of expert. Using an exploratory policy optimization procedure, the regret grows sublinearly as shown in (Kidambi et al. 2021). Kidambi et al. (2021) uses an exploration bonus and shows that the RHS in the above regret simplifies to be information gain and for a number of MDP families the growth rate of information gain is mild.
23

---

Published in Transactions on Machine Learning Research (01/2023)
Potential imitation suboptimality with additional rankings
In this section, we consider how additional rankings can affect the intended performance gap as discussed
in 4.2 Consider a tabular MDP setting in which we are given a set of rankings p" 3 p² 3 3 pM 3 pf In such a case, we regress the state-action pairs from their respective visitations to [0, k1, k2, .kk,, k] where l < l2... < kn < k. We will discuss Appendix [B..J] how this regression generalizes La and make it computationally more efficient. For this regression, the optimal reward function that minimizes the ranking loss pointwise is given by:
D kap (8,a) + (s, Rd (s) p² (3 Σ. p" (s,0) + (s,
(77)
We consider a surrogate ranking loss with regression target kess that achieves the same optimal reward when only p= p ranking given. Therefore:
Ei kapp a) kp
kessp
(78)
p= (8, + ΣE pl (8, + (F(,,) (0( + p
keff can be upper bounded as follows:
p ( p-
Ei=1 kip (s, kkp
kefs
(79)
(8,00)
(=)(s,)) + Σi=1 (0, (s) + (s)
p (s, ( + p" (s,,a)
E kip (s. + (s,aa)
(80)
(8
(= + (5)
(s,a)
k +
(81)
R (s.a)
feff can be lower bounded by:
pd (s, a) + p (s,a)
EL hip (8, + k (s,
keff
(82)
(5(,,
1² (8) Ei= pr (8, (s,a)
1 (s) + p² (s,
kp a)
(83)
(s,00)
(=(s + 2im pr (s, a) + p (s,
(84)
1 prt(sa) p² (p.a)
Thus, kess can increase or decrease compared to after augmenting the ranking datasct. We discuss the consequences of decreased & in Section 4.2
B
Algorithm Details
B.1 Ranking Loss for the Reward Agent
3 p))} wherein for of - i denotes Consider a dataset of behavior rankings D = ((p) Z pi), (rb. 12),--(phh ( the comparison index within pair of policies, j denotes the pair number, and pf 1 př denotes that of is preferable in comparison p and in turn implies that of has a higher return. Each pair of behavior comparisons in the dataset between the state-action state visitations. We will restrict our attention to specific instantiation of the ranking loss (a regression loss) that attempts to explain the rankings between each pair of policies present in the dataset by a performance gap of at least k i.c. E [R(s, a)] < E, a)] - k.
Formally, the ranking loss is defined as follows:
min (D; R)- = min Exp "0)) [Ba (a.0) [R(s a) - 0)²] + Ea- [R(s, k) R
(85)
R
When ls is set to (le 1). this loss function resembles the loss function used for SQIL RRedyy et al. 2019 if fixed rewards were used instead of learned. Thus, SQIL can be understood as special case. We also note
24

---

Published in Transactions on Machine Learning Research (01/2023)
that similar ranking loss function has been previously used for training generative adversarial networks in LS-GAN (Mao et al. 2017).
Our work explores the setting of imitation learning given samples from state or state-action visitation مf of the expert T² We will use uh to denote the mh update of the agent in Algorithm [ The updated agent generates a new visitation the environment which is stored in an empty dataset Donisne given by Donline
fp mo
B.1.1 Reward loss with automatically generated rankings (auto)
The ranking dataset DP contains pairwise comparison between behaviors p: 3 Pj. First, we assume access the trajectories that generate the behaviors, i.e p² = {ri: ri....TA) and p¹ = [ Tg....hh) propose to automatically generate additional rankings using the following procedure: (a) Sample trajectory 7' p' and 7 ~ p². Both trajectories are equal length because of our use of absorbing states (see Appendix Cp.
} In this method we
(b) Generate an interpolation Tú between trajectories depending on parameter Ap- A trajectory is a matrix of dimensions x ([S)+ where H the horizon length of all the trajectories.
R To (1 Ap)TT
(86)
These intermediate interpolated trajectories lead to ranking that matches the ranking under the expert reward function if the reward function is indeed linear in state features. We further note that t can also be a trajectory of features rather than state-action pairs.
4 the regression Next, we generate regression targets for the interpolated trajectories. For a trajectory XXp larget is given by a vector 0 + (1 - Ap),1, where vectors 0, 1 are given by [0,0...0] and [1,1,...1]] of length И respectively. This procedure can be regarded as a form of mixup (Zhang et al. 2017) in trajectory space. The set of obtained x after expending the sampling budget forms our behavior PX
A generalized and computationally efficient interpolation strategy for rank-game
Once we have generated P interpolated rankings, effectively have O(P2) rankings that we use to augment our ranking dataset. Using them all naively would incur a high memory burden. Thus, we present another method for achieving the same objective of using automatically generated rankings in a more efficient. and generalized way. For each pairwise ranking p² I p² in the dataset DV, we have the following new set of rankings p² 3 PX, .. 3 P 3 p². Using the O(P2) rankings in the ranking loss LR, the ranking loss can be simplified to the following using basic algebraic manipulation:
(P + ()) [R(s, a) - k)]] (e.))))))) [(((s,)) k) +.. (a) (R(s,a) k)]] [R((s,)) - 0)²] (87)
+(P + (),) (R(s,a) 0)² + ( (R(s,a) - 0)3] +.. +1)E (L)
The reward function that minimizes the above loss pointwise is given by:
Rd (s, k((P+ +1p +PP + - (P- 1)p2p + p&11
(88)
(P+1)(r) + Pxxp + +.. P.. +p²)
19
klpp + P P + P+1 P Px +.. 1
(89)
(p3 + Px + + PX + p²)
We consider a modification to the ranking loss objective (Equation 85 that increases flexibility in regression
targets for ranking as well as reducing the computational burdon from dealing with O(P2) rankings pairs to O(P). In this modification we regress the current agent, the expert, and each of the intermediate interpolants (p', Px,"" .... ) to a fixed scalar return (ko, k ..,,.. kp41) where ko S k1 S S Skpt = k. The optimal reward function for this loss function is given by:
as) = kpti0 (8, a) + hpp ij (v, + kpp 1P (8, + .. + Kip (s) +
(90)
(pt + ( +.. Px a) + +p (s,
25

---

Published in Transactions on Machine Learning Research (01/2023)
This modified loss function generalizes Eq 88 and recovers it exactly when [Ko, ki.,, kp41 is set to be k [0, p4, A... k.. We will call this reward loss function generalized ranking loss.
Shaping the ranking loss: The generalized ranking loss contains a set of regression targets (Ko, ki, kp+1) i) which needs to be decided apriori. We propose two strategies for deciding these regression targets. We consider two families of parameterized mappings: (1) linear a (ka = a k) and (2) rate of increase exponential in a dBa ed9,, where the temperature parameter and denote this family by exp-8. We also set ka-0 = 0 (in agent's visitation) and K,-1 = k (in expert's visitation) under the reward function that is bounded in [0, Rmaz]. The shaped ranking regression loss, denoted by SLx (D; R), that induces performance gap between + consccutive rankings (p'
return
do
.. Pxp. =PX PXr+.) is given by:
P+1
SL,(D;
E
s (s,a) [R(s,, a) - k:))
(91)
+ 2 1=0
10
8
4
0.0
0.2
0.4
0.6
0.8
1.0
a -nterpooaaion
Inear
exp exp
exp
exp -1
exp
exp -5
Figure 7: This figure shows the assignment of value k. (intended return value) corresponding to values a (degree of time-conditional interpolation between the visitation distribution of the agent and the expert). When the rate of increase is exponential with positive slope, we have higher performance gap over comparisons closer to the expert and when the rate of increase negative, the performance gap higher for comparisons closer to the agent.
Figure 7 above shows the flexibility in reward shaping afforded by the two families of parameterized functions. The temperature parameter 3 > 0 encourages the initial preferences to have smaller performance gap than the latter preferences. Conversely, < encourages the initial preferences to have a larger performance gap compared to the latter preferences. We ablate these choices of parameteric functions Appendix D.5
B.1.2 Reward loss with offline annotated rankings (pref)
Automatically generated rankings are generated without any additional supervision and can be understood as form of data augmentation. By contrast, with offline annotated rankings, we are given a fixed dataset of comparisons which is form of additional supervision for the reward function. Automatically generated help
rankings only help by making the reward landscape easier to optimize, but offline rankings reduce the exploration burden by informing the agent about counterfactuals that it had no information about. This can, for instance, help the agent avoid unnecessary exploration by providing dense improvement signal. The offline rankings are either provided by a human or extracted from a set of trajectories for which ground truth reward known. our work, we extract offline preferences by uniformly sampling p trajectorics from an offline dataset obtained from training run of an RL method (SAC) (Haarnoja al., 2018)) with ground truth reward.
26

---

Published in Transactions on Machine Learning Research (01/2023)
For imitation learning with offline annotated rankings, at every iteration m of Algorithm have new dataset of rankings given by Donline = {pggent 3 و²] along with fixed offline dataset containing rankings of the form (Doffline_ = {p² I pp)). We always ground the offline preferences by expert's visitation in our experiments, ic pp I p². We incorporate the offline rankings as a soft constraint in reward learning by combining the ranking loss L between the policy agent and the expert, with a ranking loss Lg or shaped
ranking loss ST. (Equation 91 over offline trajectories:
o flime (Donline Dofe, R) a (Donbnn; R) + - a)- I (D R)
(92)
Here, instead of the consecutive rankings being interpolants, they are offline rankings. The videos attached in the supplementary show the benefit using preferences imitation learning. The policy learned without preferences in the pen environment drops the pen frequently and in the door environment is unable to successfully open the door.
B.2 Stackelberg Game Instantiation
A Stackelberg game view of optimizing the two-player game with a dataset of behavior rankings leads to two methods: PAL (Policy as Leader) and RAL (Reward as Leader) (refer Section 4.3). PAL uses a fast reward update step and we simulate this step by training the reward function until convergence (using validation set) on the dataset of rankings. We simulate slow update step of the policy by using few iterations of the SAC (Haarnoja et al., 2018) update for the policy. RAL uses slow reward update which approximate by dataset aggregation aggregating all the datasets of rankings generated by the agent in each previous iteration enforces the reward function to update slowly. A fast policy update is simulated by using more iterations of SAC. Since SAC does not perform well with high update to environment step ratio, more iterations of SAC would imply more environment steps under a fixed reward function. This was observed to lead to reduced learning efficiency, and an intermediate value of SAC updates was observed to perform best, (Table 5
B.2.1 Policy as Leader
Algorithm 2 presents psuedocode for a practical instantiation of the PAL methods RANK-PAL (vanilla), RANK-PAL (auto) and RANK-PAL (pref) that we use in our work. Recall that (vanilla) variant uses no additional rankings, whereas (auto) uses automatically generated rankings and (pref) uses offline annotated ranking.
B.2.2 Reward as Leader
Algorithm 3 presents psuedocode for practical instantiation of the RAL methods RANK-RAL (vanilla), RANK-RAT. (auto).
C Implementation and Experiment Details
Figure 8 shows some of the environments we use in this work. For benchmarking we use 6 MuJoCo (licensed under CC BY 4.0) locomotion environments, We also test our method on manipulation environments Door opening environment from Robosuite (Zhu et al.) (020a) (licensed under MIT License) and the Pen-vO environment from mjrl (Rajeswaran et al. 2017) (licensed under Apache License 2.0).
Environments:
Expert data: For all environments, obtain expert data by policy trained until convergence using SAC (Haarnja et al. 2018) with ground truth rewards.
Baselines: We compare our proposed methods against 6 representative LIO approaches that cover a spectrum
of on-policy and off-policy, model-free methods from prior work: GAIfO Torabi et al. 2018b Ho & Ermon
2016). DAC1O (Kostrikov et al. 2018), BCO (ooabi et al. 2018a), f-IRL (Ni et al. 2020 OPOLO (Zhuj et al. 2020b) and IQ-Learn Garg et al. (2021). GAIfO [orabi et al., 2018b) a modification of the vevesrriial GAIL, method (Ho & Ermon 2016), in which the discriminator trained to distinguish between
27

---

Published in Transactions on Machine Learning Research (01/2023)
Algorithm Policy As Leader (PAL) practical instantiation
Initialize: Policy network ne, reward network RS; replay buller R
Hyperparameters: Common: Policy update steps Tpoll Reward update steps nrew; Performance gap k, empty ranking dataset Doniime, RANK-PAL (auto); number of interpolations P, TRANK-PAL Offiine annotated rankings Doife
3: for m = 1, 1, do
4: Collect transitions in the environment and add to replay buffer R. Run policy update step: "g" Soft Actor-Critic( ;,m with transitions relabelled with reward obtained from R // call Npol times
Add absorbing state/stato-actions to all early-terminated trajectories collected in the current Npol policy update steps to make them full horizon and collect in Doniine, Doniine Donisne (discard old data).
6: (for RANK- (AANK-)AL(auto)) Generate interpolations for rankings in the dataset Donline and collect in Donline
Reward Update step: // call Nrew times
min L2 (Donline; RT-1);
RANK-PAL (vanilla) (Equation 85
R
min SLy (Danléne Rg
RANK- PAL (auto) (Equation 91
RANK-PAL (pref) (Equation 92 min of flne (Donline Dos Aofs, R),,
end for
Algorithm Reward As Leader (RAL) practical instantiation
1: Initialize: Policy network ne, reward network Rş, replay bulfer R, trajectory buffer D
2: Hyperparameters: Common: Policy update steps Repol Reward update steps Nrews Performance gap k, empty ranking dataset Donline, RANK-PAL (auto): number of interpolations P, RANK-PAL (pref): Offine annotated rankings fline
for 2, do
Collect transitions in the environment and add to replay buffer R. Run policy update step: T Soft Actor-Critic(F"" ": times
( with transitions relabelled with reward obtained from R // call Apol
5: Add absorbing state/state-aations to all early-terminated trajectories collected the current npot policy update steps to make them full horizon and collect D Donline. Aggregate data in Donline
Donline Dontine
(for RANK-RAL(auto)) Generate interpolations for rankings in the dataset Dontine and collect in Dontine
7: Reward Update step: // call Brew times
min LA (Donline, 1m--1), 1), 6 min ST. (Dontine R Rmm-1).
RANK-RAL (vanilla) (Equation 85
R
RANK-RAI((atto) (Equation 91
end for
state-distribattios rather than state-action distributions. DAC-fO (Kostrikov et al. 2018 is an off- off-policy modification of GAIfO (Torabi et al. 2018b), in which the discriminator distinguishes the expert states with respect to the entire replay buffer of the agent's previously visited states, with additional implementation details such as added absorbing states to early-terminated trajectories. BCO Torabi et al. 2018a) learns an inverse dynamics model, iteratively using the state-action-nes state visitation in the environment and using predict the actions that generate the expert state trajectory. OPOLO (Zhu al. 2020b) is a recent method which presents a principled off-policy approach for imitation learning by minimizing an upper-bound of the state marginal matching objective. TQ-Learn (Garg et al. 2021 proposes to make
28

---

Published in Transactions on Machine Learning Research (01/2023)
Env
Swimmer
Hopper
HalfCheetah
Walker
Ant
Humanoid
BCO
102.76=0.90
20.10=2.15
5.12=3.82
4.00=1.25
12.80 .26
3.90+1.22
GaiFO
99.0441.61
81.131 9.99
13.5417.24
83.83.1..55
20. 20.10:124.41
3.93.11.81
DACIO
95.09-6.14
94.73.1.663
8..03.15.09
54..001144.64
86.45.11.77
19.31 19
f -IRL
103.89-2.37
97.45+ 0.61
96.06+4.66
101.16+1.28
71.18+19.80
77.93+6.372
OPOLO
98.6410.11
89.5625.45
88.9213.20
79. 29.19124.35
93.374 3.78
24.87417.04
IMIT-
105.93+3.10
86.47+ 7.66
90.65+15..11
75.60+1.90
82.40++9.05
94.49++3.21
PAL
(ours)
IMIT- 100.35=3.6
92.34=8.63
96.80=2.45
94.41+2.994
78.06:4..2
91.27+9.33
RAL
(ours)
RANK- 98.83+00.09
87.14+ 16.14
94.05++3.59
93.88+0.72
98.93+1.88
96.84+3.28
PAL
(ours)
RANK- 19.31
99.3410.20
101.1417.45
93.2411.25
93.2112..8
94.45:1.13
RAL
(ours)
Expert 100.00
100.00:::
100.00::
100.00
100.00
100.00 (376, 17)
[[S),A] (A)
(8,2)
(11, 3)
(17,6)
(17.6) 6)
(111,8)
Table 3: Asymptotic normalized performance of LIO methods at 2 million timesteps on MuJoCo locomotion tasks. The results in this Table also include evaluations for the IMIT- (PAL, RAL} methods.
Env
Swimmer
Hopper
HalfCheetah
Walker
Ant.
Humanoid
RCO
210.22+3.45
721.92 89.89
410.83 238.02
224.58+71.42
704.88 13.49
324.94+-44336
GATRO
202. 4.887
2871.47 46511777666777
1532.37:+693.72
4666.31+148..
1141.66 1000.11
326.69++133 13.26
DACIO
194.65+14:08
1350.55+:41.66
11057,5444000
3045.21424485.3
5112.15+:38001
1865.40:11867 61
-IRL
612.50=6643
3416.334:35.0
12527.21 214344.95
5630.32:11.30
4200 18:1241711
4362.46-
OPOLO
210.81=1.31
3168. 206.2
11576.12:1580
1337.00:1336.33
5529.
161191191
1468.902
1011.83 853
IMIT-PAL (ours)
216.6417.95
3039. 43:.283.85
11806.47:
1750,24
4208, 171:07.411
1572.30+-180.23
5265.60 (00.0002287.4
IMIT-RAL (ours)
205.33-8.92
3266. 288 2818-188.00
12626.
184454.71
1254.544165.11
4612.8 192.06
5089. 88 +-61 .07
RANK-PAL (ours)
202.2441..8
3082, 5822,99
12259.06 06:t 206.82
1725.49-442..00
5862.42 42.1-42-777766
5393.45-: 291.16
RANK-RAL (ours)
203 203.20:44.65
3512.67=21..09 21.09
49-04449
5189 177.221
5520. 1100144:1100 116.77
5262, 96-7-633-344 .44
Expert
201.6 = 0
3535, 58 # O
13051.46 =
5456.91 +
5926.17 +:
5565.53 = 0
(15), A))
(8,2)
(11,3)
(17.6)
(17.6)
(111,8)
(376, 17)
Table 4: Asymptotic performance of LIO methods at 2 million timesteps on MuJoCo locomotion tasks. The results in this Table also include evaluations for the IPIT, RAI.) methods.
imitation learning non-adversarial by directly optimizing the Q-function and removing the need to learn a reward as a subproblem. All the approaches only have access expert state-trajectories.
We use the author's open-source implementations of baselines OPOLO, DACIO, GAIOO, BCO available We use the author-provided hyperparameters (similar to those used in (Zhu et al. 2020bb) for all MuJoCo locomotion environments. For f-IRL, we use the author implementation available at htttp:://githut http://ggiituuu..ooomoootttta
at
and use the author provided hyperparameters. IQ-Learn was tested on our expert dataset by following authors implementation found here: http:///gtttub...oo////9/9/////9
We tested two TQ-Learn loss variants: 'v0' and 'value" as found their hyperparametter configurations and took the best out of the two runs.
Policy Optimization: We implement RANK-PAL and RANK-RAL with policy ing using SAC (Haarnoja et al. 2018). We build upon the SAC code (Achiam without changing any hyperparameters.
learn-
2018
(https//gitthub.com) http:///w..oooooooaa//oooa/aa///
Reward Learning: For reward learning, we use an MLP parameterized by two hidden layers of 64 dimensions cach. Furthermore, we clip the outputs of the reward network between [-10, 10) range to keep the range of rewards bounded while also adding an L2 regularization of 0.01. We add absorbing states to early terminated agent trajectories following Kostrikov et al. (2019). For training the ranking loss until convergence in both
29

---

Published in Transactions on Machine Learning Research (01/2023)
Figure 8: We evaluate rank-game over environments including Hopper-v2, Ant-v2, Humanoid-v2, Door, and Pen-vO.
update strategies (PAL and RAL), we used evaluation on a holdout set that is 0.1 the total dataset size as a proxy for convergence.
Data sharing between players: We rely on data sharing between players to utilize the same collected transitions for both players' gradient updates. The reward learning objective in RANK-PAL and RANK-RAL requires rolling out the current policy. This makes using an off-policy routine for training the policy player quite inefficient, since off-policy model-free algorithms update a policy frequently even when executing a trajectory. To remedy this, we reuse the data collected with a mixture of policies obtained during the previous off off-policy policy learning step for training the reward player. This allows us to reuse the same data for policy learning as well as reward learning at cach itcration.
Ranking loss for reward shaping via offline annotated rankings: In practice for the (pref) setting (Section 4.2 to increase supervision and prevent overfitting, augment the offline dataset by regressing the snippets (length !) of cach offline trajectory r' for behavior of to k*l, in addition to regressing the rewards for each state to k. The snippets are generated as contiguous subsequence from the trajectory, similar to Brown
ct al. 2019
C.1 Hyperparameters
Hyperparameters for RAA (vanilla,auto and pref) methods are shown in Table 5 For RANK- PAL, found the following hyperparameters to give best results: npol = H and Nrew ('validation' or H H/b), where H is the environment horizon (usually set to 1000 for MuJoCo locomotion tasks) and b is the batch size used for the reward update. For LANK-RAL, we found Root = H and Prew = ('validation' or DD/b), where [D indicates the cumulative size of the ranking dataset. We found that scaling reward updates proportionally to the size of the dataset also performs well and a computationally effective alternative to training the reward until convergence (see Section D.7..
Hyperparameter
Value
Policy updates Nool
Reward batch size(b)
1024
Reward gradient updates nrew
val [DJ/1024
Reward learning rate
1e-3
Reward clamp range
[-10,10]
Reward 12 weight decay
0.0001
Number of interpolations [auto]
5 exp-[-1]
Reward shaping parameterization [auto]
Offline rankings loss weight (A) [pref] Snippet length [pref]
0.3
10
Table 5: Common hyperparameters for the RANK-GAME algorithms. Square brackets in the left column indicate which hyperparameters that specific to 'auto' and 'pref' methods.
30

---

Published in Transactions on Machine Learning Research (01/2023)
D Additional Experiments
D.1 Complete evaluation of LfO with rank-game(auto)
Figure shows a comparison of RANK-PAL(auto) and (ANK-RAL(auto) for the LIO setting on the Mujoco benchmark tasks: Swimmer-v2, Hopper-v2, HaLfCheetah-v2, Walker2d-v2, Ant-v2 and Humanoid-v2. This section provides complete results for Section 5.1 in the main paper.
Swimmer-v2
Hopper-v2
HalfCheetah-v2
4:00
Average Reward
12500
160
2000
7800
5000
1000
2300
-60
166
1c6
166
Walker-v2
Ant-v2
Humanoid-v2
Average Raward
6000
4000
4000
2000
2000
Timesteps
Timesteps
Timesteps
RANK-PAL (ours)
RANK-RAL (ours)
OPOLO
IQ-learn
AR
DACIO
GAIIO
BCO
Figure Comparison of performance on OpenAl gyı benchmark lasks.
The shaded region represents standard
deviation across 5 random runs. RANK-PAL and RANK-RAI substantially outperform the baselines in sample
efficiency. Dotted blue line shows the expert's performance.
D.2 Evaluation of LfD with rank-game(auto)
rank-game is a general framework for both LfD(with expert states and actions) and LfO (with only expert states states/obsce /observations................oo) We compare performance of rank-gam game compared to LfD baselines: IQ-Learn (Garg
et al. 2021 DAC (Kostrikov et al. 2018) and BC (Pomerleau 1991).
we observe that rank-game among the most sample efficient methods for learning from demonstrations. IQlearn shows poor learning performance on some tasks which we suspect is due the low number of expert trajectories we in our experiments compared the original work. DAC was tuned using the guidelines from Orsini et al. (2021) to ensure fair comparison.
In figure 10
D.3 Utility of automatically generated rankings in rank-game(auto)
We investigate the question of how much the automatically generated rankings actually help in this experiment. To do that, we keep all the hyperparameters same and compare RANK-GAME (vanill)) with RANK-GAME (auto). RANK-GAME, (vanilla) uses additional ranking information and Ly is used as the reward loss.
31

---

Published in Transactions on Machine Learning Research (01/2023)
Swimmer-v2
Hopper-v2
HalfCheetah-v2
Average Reward
12600
3000
10000
2000
130
5000
1030
2600
0.0
0.2
0.4
0.6
0.8
1.0
105
165
166
Walker-v2
Ant-v2
Humanoid-v2
6000
Average Reward
6000
5000
4000
4000
2000
3000
2000
1000
-2000
0.0
0.2
0.4
0.8
0.8
1.0
Timesteps
1e5
Timesteps
1e6
Timesteps
1e5
LFD-RANK-PAL (ours)
LFD-RANK-RAL (ours)
IQlearn
DAC
BC
Figure 10: Comparison of Pank-game methods with baselines in the LID setting (expert actions are available). RANK-(PAL.RAT) are competitive state of the art methods.
Walker-v2
Ant-v2
Average Reward
4000
4000
2000
Timesteps
1e6
Timesteps
1e6
RANK-PAL(auto)
RANK-PALLyyarilla)
Figure 11: RANK-PAL(vanilla) has high variance learning curves with lower sample efficiency compared to RANK- PAL(auto).
Figure 11 shows that in RANK-PAL (auto) has lower variance thronghout training (more stable) and is more sample efficient compared to RANK-PAA((vanlll).
D.4 Comparison of imit-game and rank-ganl game methods
Imitation learning algorithms, particularly adversarial methods, have number of implementation com- ponents that can aflect learning performance. In this experiment, we aim to further reduce any imple- mentation/hypeenn ptprroornttperrrreeeeerrr gap between adversarial imitation learning (AIL) methods that are based on the supremum-losss (described scction 3) function and rank-game to bring out the obtained algorithmic improvements. To achieve this, we swap out the ranking loss Lş based on regression with supremum-loss and call this method RMIT-(PAI,RAT.). This results in all the other hyperparameters such as batch size,
32

---

Published in Transactions on Machine Learning Research (01/2023)
reward clipping, policy and reward learning iterations, and optimizer iterations to be held constant across experiments.
We present a comparison of RANK-(PAL, RAL] and IMIT-(PAL, RAL} in terms of asymptotic performance in Table 3 and their sample efficiency in Figure 12 Note that Table g shows normalized returns that are mean-shifted and scaled between [0-100] using the performance of uniform random policy and the expert policy. The expert returns are given in Table 1 and we use the following performance values from random policies for normalization: { Hopper= 13.828, HalfCheetah= 2711.3, Walker= 1.53, Ant -62.01, Humanoid= 112.19). Table shows unnormalized asymptotic performance of the different methods.
In terms of sample efficiency, we notice IMIT-(PAL, RAL) methods compare favorably to other regularized supremum-loss counterparts like GAIL and DAC but are outperformed by RANK-(PAL, RAL] (aulo) methods, We hypothesize that better learning efficiency in Lg compared to supremum- is due to regression fixed targets being a simpler optimization than maximizing the expected performance gap under two distributions.
Swimmer-v2
Hopper-v2
HalfCheetah-v2
300
4000
Average Reward
200
7600
100
5000
1000
2500
0
0.0
0.2
0.4
0.6
0.8
1.0
1e5
1e5
168
Walker-v2
Ant-v2
Humanoid-v2
Average Reward
6000
6000
2000
2000
-2000
0.0
0.2
Timesteps
Timesteps
168
Timesteps
RANK-PAL (ours)
RANK-RAL (ours)
IMIT-PAL (ours)
IMIT-RAL (ours)
OPOLO
DACIO
GaiFO
Figure 12: Comparison of performance OpenAT gym benchmark tasks. Specifically, seek to compare RANK- (PAL, RAL) methods to IMIT-(PAL, RAL} methods and IMIT-(PAL, RAL} methods to their non-Stackelberg counterparts GAIfO and DACIO. The shaded region represents standard deviation across random runs. RANK-PAL and RANK-RAL substantially outperform the baselines in sample efficiency and IMIT-(PAL, RAL} is competitive to the strongest prior baseline OPOLO.
D.5 Effect of parameterized reward shaping in rank-game (auto)
We experiment with different ways of shaping the regression targets (Appendix B for automatically generated interpolations in RANK-GAME (auto) in Figure 13 In the two left-most plots for RANK-PAL (auto), we see that reward shaping instantiations (exponential with negative temperature) which lcarns a higher performance gap for pairs of interpolants closer to the agent lead to higher sample efficiency. We note that decreasing the temperature too much leads to a fall sample efficiency. The same behavior is observed in RANK-RAL (two right-most plots) methods but we find them be more robust to parameterized shaping than PAL methods. We use the following interpolation scheme: exponential with temperature= for our experiments in the main paper.
33

---

Published in Transactions on Machine Learning Research (01/2023)
Walker-v2
Ant-v2
HalfCheetah-v2
Walker-v2
Average Reward
Average Reward
4000
5000
8000
2003
00 0.2
04
04
00
62
05 Timesteps
25
15
Timesleps
Timesteps
Timesteps
linear
exp
exp -1
-2
exp -
Figure 13: The two left-most plots show the effect of reward shaping RANK-PAL (auto) methods using linear and exponential shaping functions. The two right-most plots show the same effect of reward shaping in RANK-RAL (auto) methods. Reward shaping instantiations which induce a higher performance gap between pairs of interpolants closer to the agent perform better and RAL is more robust to reward shaping variants than PAL.
D.6 On the rank preserving nature of SLμ
Increasing number of rankings
increasing number Of states
1.00
0.50
0.50
0.50
0.75
0.75
10 rantanga
160 clate:
50 randrngs
0.65
3n0 ctatec
100 carkings
3nd. stoles
500 in masange
765, rtaees
0.60
1000 miving
Increasing the state size of the domain increases the rank consistency afforded by ST and increasing the number of rankings decreases the rank consistency.
Figure 14:
The ranking loss SLk (Appendix B Eq 91 regresses the pt, p and each of the intermediate interpolants
1 3) "....... Pxp: pXe,1 to fixed scalar returns (ko, ki,... kp41) where ko S ki ... S kipt k. (p' = P2g Px, ...
The ranking loss SLk is given by:
P-1 Σ STa R) p+2 1=0
snp (e,a) [R((s, k;)
(93)
ST. provides a dense reward assignment for the reward agent but does not guarantee that minimizing SL, would lead to the performance ordering between rankings, i.e [f]] < [s][ < [f][f(s]] < Epr+1 [[))]] An ideal loss function for this task regresses the expected return under each behavior to scalar values indicative of ranking, but needs to solve complex credit assignment problem. Formally, we can write the idcal loss function for reward agent as follows
1 p 2+2
SLideal
(D;R)
E E (s.0) [R(s,)) a) - k,
(94)
We note that the SL* upper bounds SLideal using Jensen's inequality and thus is reasonable target for optimization. In this section we wish to further understand if SL has rank- of loss function for ranking that assigns a scalar reward value for each states of a particular state visitation corresponding to its ranking. Ideally, given a ranking between behaviors pd 3 pp 1 p²... 2.... 2ppt p we aim to learn reward function that satisfics [f)) < E,, [f()) < [f](((s)] <.. < E [f()). We empirically test the ability of the ranking loss function SL, to facilitate the desired behavior in performance ranking.
policy. SL, is a family
We consider finite state space S and number of rankings P. We uniformly sample P + possible state
34

---

Published in Transactions on Machine Learning Research (01/2023)
visitations and the intermediate regression targets (M s.t h2 S he41- To evaluate the rank- preservieg ability of our proposed loss function we study the fraction of comparisons the optimization solution that minimizes SL₃ is able to get correct. Note that P + sequential ranking induces P(P + 1)/2 comparisons.
Figure 14 shows that with large state spaces SL& is almost rank preserving and the rank preserving ability degrades with increasing number of rankings to be satisfied.
D.7 Stackelberg game design
We consider the sensitivity of the two-player game with respect to policy update iterations and reward update itcrations. Our results (Figure 15) draw analogous conclusions to Rajeswaran et al. (2020) where we find that using a validation loss for training reward function on on-policy and aggregale datasel PAL and RAL respectively works best. Despite its good performance, validation loss based training can be wall-clock inefficient. We found substitute method to perform similarly while giving improvements in wall-clock time- make number of iterations of reward learning scale proportionally to the dataset set size. A proportionality constant of (1/ l//bacch-siee) worked as well as validation loss in practice. Contrary to RRaeeswrran et al. (2020) where the policy is updated by obtaining policy visitation samples from the learned model, our ability to increase the policy update is hindered due to unavailability of learned model and requires costly real-enviromment interactions. We tune the policy iteration parameter (Figure 16 and observe the increasing the number policy updates can hinder learning performance.
Walker-v2
Ant-v2
Walker-v2
Ant-v2
Average Reward
Average Reward
5002
5000
1000
4000
2000
2000
Timesteps
160
Timesteps
1e0
Timestaps
Timesteps
166
val
row hr
2 rew
raw ltr
row
Figure 15: The left two plots use PAL strategy and the right two plats use RAL strategy. Reward learning using a validation loss on a holdout leads to improved learning performance compared to hand designed reward learning iterations.
Walker-v2
Ant-v2
Average Reward
6000
4000
4000
2000
2000
Timesteps
Timesteps
1e6
(auto)
1_sac (auto)
(vanilaa
(vanila)
Figure 16: Small number of policy updates useful for good learning performance in the PAL setting here.
D.8 Sensitivity of reward range for the ranking loss Tak
In Section 4.2, we discussed how the scale of learned reward function can have an effect on learning performance. We validate the hypothesis here, where we set Rmen = h and test the learning performance RANK-PAL. (auto) on various different values of Our results in figure D.9 show that the hyperparameter & has large effect on learning performance and intermediate values of k works well with A 10 performing the best.
35

---

Published in Transactions on Machine Learning Research (01/2023)
Walker-v2
Humanoid-v2
Average Reward
8000
5000
4000
4000
3000
2000
2000
0
0.0
0.2
0.4
1.0
Timesteps
Timesteps
1
k=10
k 100
k=1000
Figure 17: Intermediale values of k work best in practice.
D.9 Effect of regularizer for rank-game
rank-game(auto) incorporates automatically generated rankings which be understood form of regularization, particularly mixup Zhang et al. (017)) in trajectory space. In this experiment, work in the PAL setting with ranking loss L3 and compare the performances of other regularizers: Weight-decay (wd), Spectral normalization (sn), state-based mixup to (auto). Contrary to trajectory based mixup (auto) where we interpolate trajectories, in state- based mixup we sample states randomly from the behaviors which are pairwise ranked and interpolate ween them.
Walker-v2
Humanoid-v2
Average Reward
4000
4000
2000
2000
0.0
0.2
0.4
1,0
Timesteps
1e5
Timesteps
166
(auto)
state-mbuup
Figure 18: (auto) regularization outperforms other forms regularization in rank-game
Figure
18
shows learning with (auto) regularizer is more efficient and stable compared to other regularizers.
D.10 Ablation analysis summary
We have ablated the following components our method: Automaticcally-generaake rankings D.3 Ranking Stackelberg game d..
D.7 and range of the bounded reward D.9 Our analysis above (Figure I2 17 and 15 shows quantitatively that the key improvements over baselines are driven by using the proposed ranking loss, controlling the reward range and the reward/police update frequency in the Stackelberg framework. Parameterized reward shaping (best hyperparameter exp -1 compare to unshaped/ /linear shaping) and soaomnatically-gnertteeee rankings contribute to relatively small improvements. We note that a single hyperparameter combination (Table 5) works well across all tasks demonstrating robustness of the method to environment changes.
loss D.4. Parameterized reward shaping D.5
36

---

Published in Transactions on Machine Learning Research (01/2023)
D.11 Varying number of expert trajectories for imitation learning
In the main lext, we considered experiment settings where the agent is provided with only expert trajectory. In this section, test how methods performs compared to baselines we increase the number of available expert observation trajectories.
Door-v0
Average Reward
We note that these experiments are the LIO setting. Figure 20 shows that RANK-GAME compares favorably to other meth- ods for varying number of expert camoosttatioons/oseevvatit trajectories.
400
200
D.12 Robustness to noisy preferences
Timesteps
1e5
In this section, we investigate the effect of noisy preferences on imitation learning. We consider the sctting of Section 5.2 where
nosem0. nokwedd
rolumo
nolownd.6
rolsen
we attempt to solve hard exploration problems for LIO setting by
leveraging trajectory snippet comparisons. consider setting similar to Brown et al. (2019) where
this experiment,
Figure 19: We investigate learning from expert observation-offllee preferences where the of- inject varying level of noise, i. flip x% of trajectory snippet fiiee preferences are noisy. RANK-PAL shows considerable robustness noisy preferences.
random. Figure 19 shows that RANK-PAT(pre]]] is robust in learming near-expert behavior upto 60 percent noise in the Door environment. We hypothesize that this robustness to noise is possible because the preferences are only used to shape reward functions and does not change the optimality of expert.
D.13 Learning purely from offline rankings in manipulation environments
Door-v0
Pen-vO
Average Reward
400
2000
300
200
100
0.0
0.5
1.0
1.5
1.0
Timesteps
1e5
Timesteps
1e6
T-REX (10 pref)
T-REX (20 pref)
T-REX (50
Figure 21: Testing with 10, 20 and 50 suboptimal preferences uniformly sampled from replay buffer of SAC trained from pre-specified reward we see that TREX not able to solve these tasks. The black dotted line shows asymptotic performance of RANK-PAL (auto) method.
In section 5.2 we saw that offline annotated preferences can help solve complex manipulation tasks via imitation. Now, we compare with the ability of prior method TREX (Brown et al., 2019) that learns purely from suboptimal preferences-unde increasing numbers of preferences. We test on two manipulation tasks: Pen-v0 and Door-v0 given varying number of suboptimal preferences: 10, 20, 50. These preferences are uniformly sampled from replay buffer of SAC trained until convergence under pre-specified reward, obtained via D4RL (licensed under CC BY) We observe in Figure 21 that T-REX is unable to solve these tasks under any selected number of suboptimal preferences.
37

---

Published in Transactions on Machine Learning Research (01/2023)
Swinme E-V2
Hopper-v2
(auto)
35.00
Average Retuzn
Average Retuza
3000
150
2500
2000
103
1500
1000
53
1-aopoet
Z-oxpezt
E-espert
10-bapert
et ea--appxt
1-expect
2-expezt
I0-eapert
Gi-eapert
Hal L...-eeeeehvv22
Walker ker2kr22
PAI (auto)
OFOLC
(auso)
DAC.CO
6000
12k
Average Retuen
Average Return
50.00
8K
4000
4k.
1000
1-expert
2-orpert
E-expert
10-osport
6B expprrt
1-expext
2-expart
S-eeport
It-espurt
Ant-v2
Humanoid-v2
DANK PAE (suLo)
GEOSI
DACIO
BCD
RANK DAL (auce)
OPOLO
DAC.CO
BOO
6003
5000
Average Reburn
5000
Average Return
9000
8000
3000
3300
2000
1000
1000
-1003
1-eapert
2-eapert
S-expert
10-ospert t
S0-ospert
1-ospert
z-espert
S-eepert
C0-apertt
60-ospert
Figure 20: Performance analysis of different algorithms in the LfO setting with varying number of expert trajectories.
RANK-PAL (auto) compares favorably to other methods
38


Reviews:
[{"review_id": "dyXbO56sr4", "reviewer": "Reviewer_UMKz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* The paper demonstrates good empirical results on several datasets.\n* Ranking and incorporating references is a topic interesting to the deep reinforcement learning community.\n\n# Weaknesses\n* The paper has limited novelty. The method can be seen as an instantiation of GAILfO combined with the LSGAN loss function.\n* The writing needs to be improved. The paper is hard to follow. In particular, it is unclear whether the authors consider LfD as an imitation learning of a reinforcement learning setup. Several previous papers on learning from demonstrations (for example, DQfD and DDPGfD) considered the setting where the rewards are available.\n* From algorithm 1. It's not clear how step 5 is performed. Do you use directly SAC to learn a policy based on these rewards? Do you perform any modifications for SAC?\n* The paper doesn't describes datasets used for evaluation and information about the baseline implementation is missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a ranking framework for imitation learning. The frameworks consists of an agent that attempts to maximize rewards estimation via another ranking agent.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Update algorithm 1 to clarify the policy update step.\n* Update the paper with infomation about baselines and datasets.", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HtbwXOlLf0", "reviewer": "Reviewer_qjK9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths: \n\n* New perspective: The idea of using both the expert data and preferences together for increasing the efficiency of learning techniques seems reasonable and useful. The proposed unified perspective of understanding various Imitation Learning algorithms from a game theoretic perspective might be interesting and inspiring to the community. \n\n* Impressive results: The paper provides strong empirical studies, including hyper-parameter and ablation studies, that support the claims regarding the practical algorithm. The results show major improvements compared to existing methods. The authors have shown that for their setting, where both types of data modalities are present, the proposed approach is indeed promising (Sec 5.2). \n\n* I’m not an expert in Imitation Learning but it seems that the authors have conducted a proper discussion of the related work. \n\n### Weaknesses:\n\nThe biggest weakness of the work is the lack of clarity in Section 4.2.  While the motivation of the proposed loss is that it facilitates better incorporation of rankings over suboptimal trajectories, can you highlight why this is a principled loss?  I appreciate the theoretical result that tells the performance with the expert is bounded, but it seems like the upper bound is quite loose. The first term in the result is the maximum attainable return under the expert reward function $\\frac{R^E_{max}}{1-\\gamma}$, whereas the multiplicative term mostly depends on hyper-parameter $k$ as the other quantities that are not defined before Theorem 4.1. What is the generalization error and when will it be bounded by the given quantity in Theorem 4.1? It looks like the user should set $k \\rightarrow \\infty$ to reduce the gap, however, from the experiments (Fig. 17) it seems like intermediate $k$ works better than large $k$. Moreover, does the equilibrium always exist? Unfortunately, the theoretical result in the present form makes things rather confusing. \n\nI understand that lot of these details might already be present in Appendix A, however, Sec 4.2 should be able to convey these results in a standalone manner without expecting the reader to go through Appendix A and D. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\nThe primary contribution of the paper is the `rank-game` framework that merges different imitation learning settings (learning from expert data and learning from preferences) by formulating it as a game between two players: a policy agent and a reward agent. The policy agent is the typical RL policy optimization agent that aims to maximize return under some reward function. Whereas, the reward agent seeks to find a reward function that satisfies the relative pairwise rankings of trajectories (or state-action distributions) as provided in the dataset.  These rankings can be the relative w.r.t. the expert trajectories (falls into ‘auto’ methodology) or offline human-labelled trajectory rankings (falls into ‘pref’ approach). The secondary contribution is a new ranking loss, $L_k$ loss, that seems to work in this new setting of using data augmentation for efficient policy optimization. The final optimization is solved using the methodology of the Stackelberg game framework.  The combination of the above two ideas leads to impressive results on a variety of synthetic mujoco domains and complex dextrous tasks. \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Recommended changes:\n\n- Notational inconsistencies: Sometimes the same notation is used for multiple quantities. For instance, the reward function of the MDP $R$ is also used for an unknown expert reward function (in the classical imitation learning section), and later as a ranking function in Sec. 4.2.1. I believe addressing similar notational inconsistencies will make the paper clear. \n\n- Implications of Theorem 4.1: See my comment in the weakness section. In particular, when do the assumptions hold? How does this statement change when using other kinds of loss functions? This does not necessarily need to be studied analytically, but rather even empirical evidence on some toy domains would suffice. \n\n###  Clarification questions: \n- In Eq (2), should there be $\\frac{1}{1-\\gamma}$ on RHS based on definition of $J(\\pi, R)$?\n- Eq 4: why is $R_{max}$ in the square root term? Should it be $R^E_{max}$? Is this linked to the generalization error mentioned in Theorem 4.1's statement? \n- In Fig. 3, is the cost for generating the dataset for auto also included in the figure? The algorithm needs to interact with the environment to generate the dataset so that should be included in the sample efficiency. \n\n### Optional suggestions:\n- More motivation on the setting will make the paper more accessible to readers that are not experts in the Imitation learning domain. \n\n", "broader_impact_concerns": "The authors have acknowledged the limitations of their approach and the potential side effects stemming due to a lack of guarantees for safety and robustness.  ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PfhqZMzFDq", "reviewer": "Reviewer_ws4a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The proposed ranking viewpoint provides the advantage of enabling the use of suboptimal behaviors as well as expert data for IRL. \n2. The experimental results show that rank-game significantly outperforms baselines on a diverse set of standard benchmarks (but see the comment below about fair comparison to T-REX).\n3. The comparison between the policy-as-leader and reward-as-leader versions of rank-game on the non-stationary task are also helpful in showing the practical differences in both (but see the question below on RAL).\n\n\nWeaknesses\n\n\n1. The main apparent weakness, which appears to be a crucial flaw, is that the ranking loss (eq. 3) is confusing and inconsistent.\nFirstly, suppose that a pair $(s,a)$ is sampled under both $\\rho^{\\pi^i}$ and $\\rho^{\\pi^j}$ when $(\\rho^{\\pi^i}, \\rho^{\\pi^j})$ appear as a ranked pair. In that case, the loss function trains the reward $R$ to map $(s,a)$ to $0$ and to $k$. That is inconsistent with the very definition of a function. Secondly, suppose we have $\\pi^i \\preceq \\pi^j$ and $\\pi^k \\preceq \\pi^i$. Because $\\pi^i \\preceq \\pi^j$, samples $(s,a) \\sim \\pi^i$ should have reward value zero, but because $\\pi^k \\preceq \\pi^i$, then $(s,a) \\sim \\pi^i$ should now have reward value k. There is no reward function that can produce both value k and value 0 for the same $(s,a) \\sim \\pi^i$, unless $R$ is given information about the other policy in the pair in addition to the $(s,a)$---i.e., to map some arbitrary $(s,a)$ to either $k$ or $0$ correctly, $R$ needs to know whether it came from a distribution that is ranked lower or higher in the pair $(\\rho^i, \\rho^j)$ that was sampled. But since that information is not provided to $R$, it is hard to see how the loss function in (eq. 3) is a consistent evaluation of $R$.\n2. This paper claims that \"existing IRL methods that learn from expert demonstrations provide no mechanisms to incorporate offline preferences and vice versa.\" However, it appears there is previous work e.g. T-REX [1], that use preferences and can accommodate the use of expert demonstrations, simply by assigning highest preference to expert demonstrations. As such, the paper's characterization of T-REX [1] in Table 2, which says T-REX is neither LfD nor LfO, appears to be incorrect. Quoting from Brown et al., T-REX is a ``reward-learning-from-observation algorithm'' that can use a set of ranked demonstrations to learn a reward function. From Section 5.2, it appears that this paper did not provide expert demonstrations to T-REX, which appears to be an unfair comparison.\n3. In the second paragraph, the paper claims to address the issue of difficulty of solving an adversarial min-max game in a sample efficient way. Even though the proposed rank-game is presented as a Stackelberg game, the actual optimization procedure is still the same as in the case of a min-max game, whereby the policy and reward are optimized in an alternating and iterative manner. It is not clear, theoretically and conceptually, how the ranking game viewpoint leads to an easier optimization problem or more sample efficient learning (notwithstanding the fact that experiments show that rank-game is more sample efficient in practice).\n\n\n[1] Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. Brown et al. 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed to formulate imitation learning as a two-player ranking game (rank-game). The reward player aims to find a scalar-valued reward function that produces a separation of value $k$ between samples from any ordered pair $(\\rho^{\\pi^i}, \\rho^{\\pi^j})$ of behaviors. The policy player aims to maximize that reward. This viewpoint addresses the limitation of previous inverse reinforcement learning approaches by enabling the use of offline preferences and suboptimal behaviors in addition to expert demonstrations. The paper derives a bound on the sub-optimality of the rank-game equilibrium policy in comparison to the expert policy. Experiments in MuJoCo benchmarks show that rank-game outperforms baselines in nearly all tasks, and experiments on sparse-reward door/pen manipulation simulations show that only rank-game with the help of offline rankings can solve the tasks, while previous methods cannot.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(Critical) Address the main apparent weakness above.\n\n(Critical) The paper writes \"The ranking loss used by the prior IRL approaches is specific to the comparison of optimal (expert) vs. suboptimal (agent) data, and precludes incorporation of comparisons among suboptimal behaviors\". However, T-REX [1] explicitly allows the use of ranked suboptimal desmontrations. This paper's statements need to be revised, or it should be explained more convincingly why previous methods like T-REX have the limitations that this paper claims they do. Specifically, why can't T-REX simply assign highest preference to expert behavior.\n\n(Critical) In Table 2, f-IRL on Walker has the highest average reward and small variance. However, in Figure 3, f-IRL is shown to have mediocre average reward and extremely large variance. This is inconsistent.\n\n(Minor) In Section 4.2, the papers should define what it means for $\\rho^{\\pi} \\preceq \\rho^{\\mu}$. Previously, the notation $\\preceq$ was only define when there is a reward $f_{\\pi}$ that induced the ranking $\\rho^E \\succeq \\rho^{\\pi} := E_{\\rho^E}[f_{\\pi}(s,a)] \\geq E_{\\rho^{\\pi}}[f_{\\pi}(s,a)]$. But in the absence of a reward $f_{\\pi}$, it is not clear what $\\rho^{\\pi} \\preceq \\rho^{\\mu}$ means. Perhaps the paper means the rankings are predefined by the dataset D? This can easily be clarified.\n\n(Minor) The reference to Theorem 1 at the top of page 6 should be to Theorem 4.1?\n\n(Minor) In section 4.2.1, the paper should clarify what a convex combination of two trajectories $\\tau_i$ and $\\tau_j$ means, where $\\tau_i$ is a trajectory of states and actions. E.g. in discrete state-action spaces, what does a convex combination of two discrete objects mean?\n\n(Minor) There needs to be more motivation for the choice \"RAL updates the reward conservatively. This is achieved through aggregating the dataset of implicit rankings from all previous policies obtained during training\". By training on data from all previous policies, RAL of course cannot adapt as fast as PAL in non-stationary environments. Why should one not simply let RAL train on the rankings generated by the current policy?\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviews are largely in concensus for acceptance, as the authors also had satisfying rebuttals. 

Reviewer UMKz expressed "The paper has limited novelty. The method can be seen as an instantiation of GAILfO combined with the LSGAN loss function". While ranking loss is a simple common objective, LSGAN does not consitute a major conflict with this work as it deals with supervised learning. Therefore, this does not invalidate the novelty of the work. Focused discussion and experimentation of IL+ranking in sequential decision making setting still makes the work valuable for the audience.

Minor comments:
- improve writing clarity: Table 1, Section 4.1, Eq 1 etc: write down exact mathematical formula for each ranking loss in one place. 
- additional experiments: auto-generated preference orders through mixup likely do not scale to high-dimensional state / action (image / texts). Additional experiments to show that the method scales would be helpful.
Decision: Accept (poster)