Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment
Authors: Weichao Zhou
Wenchao Li
Venue: NeurIPS 2024 poster (2024)
Keywords: inverse reinforcement learning
imitation learning
reinforcement learning
Primary area: reinforcement_learning
TL;DR: Propose a new formulation for inverse reinforcement learning-based imitation learning to mitigate the task-reward misalignment.

Abstract: Many imitation learning (IL) algorithms use inverse reinforcement learning (IRL) to infer a reward function that aligns with the demonstration.
However, the inferred reward functions often fail to capture the underlying task objectives.
In this paper, we propose a novel framework for IRL-based IL that prioritizes task alignment over conventional data alignment. Our framework is a semi-supervised approach that leverages expert demonstrations as weak supervision to derive a set of candidate reward functions that align with the task rather than only with the data. It then adopts an adversarial mechanism to train a policy with this set of reward functions to gain a collective validation of the policy's ability to accomplish the task. We provide theoretical insights into this framework's ability to mitigate task-reward misalignment and present a practical implementation. Our experimental results show that our framework outperforms conventional IL baselines in complex and transfer learning scenarios.

Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment
Weichao Zhou Boston University Boston, MA 02215 zwc662@bu. edu
Wenchao Li Boston University Boston, MA 02215 wenchao@bu. edu
Abstract
Many imitation learning (IL) algorithms use inverse reinforcement learning (IRL) infer a reward function that aligns with the demonstrations. However, the inferred reward function often fails to capture the underlying task objective. In this paper, propose novel framework for IRL-based IL that prioritizes task alignment over conventional data alignment. Our framework is semi-supervised approach that leverages expert demonstrations as weak supervision signals derive a set of candidate reward functions that align with the task rather than only with the data. adopts an adversarial mechanism this set of reward functions to gain collective validation of the policy's 's ability accomplish the task. We provide theoretical insights into this framework's 's ability to mitigate task-reward misalignment and present practical implementation. Our experimental results show that our framework outperforms conventional baselines in complex and transfer learning scenarios. The complete code are available at
train a policy with
Introduction
Inverse reinforcement learning (IRL) Ng and Russell 2000), Finn et al. [2017] has become a popular method for imitation learning (IL). allowing policies be trained by learning reward functions from expert demonstrations Abbeel and Ng [2004], Ho and Ermon [2016]. Despite its widespread use, IRL-based IL faces significant challenges that often stem from overemphasizing data alignment rather than task alignment. For instance, reward ambiguity, where multiple reward functions can be consistent with the expert demonstrations, makes it difficult to identify the correct reward function. This problem persists even when there are infinite data Ng and Russell [2000). Cao et al.) (2021) further exacerbate Skalse et al. 2022abb b). Additionally, limited availability of demonstrations
this problem. as the data may not fully capture the nuances of the task. Misaligned reward functions lead to policies that optimize the wrong objectives, resulting in poor performance and
reward hacking Hadfild-Menell et al.] [2017). Amodei et al. (2016). Pan et al. [2022). phenomenon where the policy exploits loopholes in the inferred reward function. These challenges highlight the limitation of exclusively pursuing data aligmment in solving real-world tasks.
In light of these considerations, this paper advocates for paradigm shift from narrow focus data alignment to a broader emphasis on task alignment. Grounded a general formalism of task objectives, we propose identifying the task-aligned reward functions that more accurately reflect the underlying task objectives their policy utility spaces. Expanding on this concept, we explore the intrinsic relationship between the task objective, reward, and expert demonstrations. This relationship leads us to a novel perspective where expert demonstrations can serve as weak supervision signals for identifying set of candidate task-aligned reward functions. Under these reward functions, the expert
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

achieves high - but not necessarily optimal - performance. The rationale is that achieving high performance under task-aligned reward function often adequate for real-world applications.
Building on this premise, we leverage IRI. to derive the set of candidate task-aligned reward func- tions and propose Protagonist Antagonist Guided Adversarial Reward (PAGAR), a semi-supervised framework designed to mitigate task-reward misalignment by training a policy with this candidate reward set. PAGAR adopts an adversarial training mechanism between a protagonist policy and an adversarial reward searcher, iteratively improving the policy learner to attain high performance across the candidate reward set. This method moves beyond relying on deriving single reward function from data, enabling a collective validation of the policy's 's similarity to expert demonstrations in terms of effectiveness in accomplishing tasks. Experimental results show that our algorithm outperforms baselines on complex IL tasks with limited demonstrations and in challenging transfer environments. We summarize our contributions below.
Introduction of Task Alignment in IRL-based IL: We present a novel perspective that shifts the focus from data alignment to task alignment, addressing the root causes of reward misalignment in IRI-bbaed II..
Protagonist Antagonist Guided Adversarial Reward (PAGAR): We propose a new semi-supervised framework that leverages adversarial training to improve the robustness of the learned policy.
Practical Implementation: We present practical implementation of PAGAR, including the adversarial reward searching mechanism and the iterative policy-improving process. Experimental results demonstrate superior performances in complex and transfer learning environments.
2 Related Works
IRL-based IL circumvents many challenges of traditional II. such compounding error Ross and Bagnell 20100. Ross et al. (2011). Zhou et al. [2020) by learning reward function to interpret the expert behaviors Ng et [999]] Ng and Russell [2000] and then learning policy from the reward function via reinforcement learning (RL)Sutton and Barto (0018). However, the learned reward function may not always align with the underlying task, leading to reward misspecification Pan et al.
2022], Skalse and Abate [2022]. reward hacking Skalse et al. (2022b). and reward ambiguity Ng and
Russell [2000]. Cao et al. [2021] ]. The efforts on alleviating reward ambiguity include Max- Max-Entropy
IRL et al. [2008]. Max- Margin Abbeel and Ng (2004), Ratliff et al. (2006). and Bayesian
IRI. Ramachandran and Amir
2007 GAN-based methods Ho and Ermon [0166] Jeon et al. 2018).
Finn et al. [2016). PPng et al.
2019 Fu et al. (2018) use neural networks to learn reward functions However, these efforts that aim to address reward ambiguity fall short of mitigating the general impact of reward misalignment which can be caused by various reasons such as IRL making false assumptions about the relationship between expert policy and expert reward function [Skalse et al. [202aa] Hong al. 2023]. Other attempts to mitigate reward misalignment involve external information other than expert demonstrations Hejna and Sadigh 2023 J. Zhou and Li [2018] 2022a b). Our work adopts the generic setting of IRL-based IL without needing additional
from limited demonstrations.
information. The idea of considering a reward set instead of focusing on a single reward function is supported by Metelli al. [202T] and Lindner et al. [0022). However, these works target reward ambiguity instead of reward misalignment. Our protagonist and antagonist setup is inspired by the concept of unsupervised environment design (UED) Dennis et al. [2020). In this paper, we develop novel theories in the context of reward learning.
3 Preliminaries
Reinforcement Learning (RL) models the environment as a Markov Decision Process M = (S, A. P, d,) where S is the state space, A is the action space, P is the transition probability, do is the initial state distribution. A policy π {a))s) s) determines the probability of an RL agent performing an action a at state s. By successively performing actions for T steps from an initial state (()) ~ do a trajectory т = s g00) (!))
...))...s)) ((T) is produced. A state-action based reward function is mapping r S A R. The soft Q-value function of x is (s. = r(s,a) ? E [V'(s') where E [0,1] is a discount factor, V,, is the soft state- function of π (PP(()).
defined as (s)s) = E [s, a)l + H(((-11)), and H ((((-s)) is the entropy of i at state s.
a-)()11))
2

---

Policy Utility
aac
U
PAGAR
7+
II
Rg.
(a) Task-Reward Alignment
(b) PAGAR- ResARB II.
Figure 1: (a) The two bars respectively represent the policy utility spaces of a task-aligned reward function r* and task-misaligned reward function r. The white color indicates the utilities of acceptable policies, and the blue color indicates the unacceptable ones. Within the utility space of r+, the utilities of all acceptable policies are higher (2 e,++) than those of the unacceptable ones, and the policies with utilities higher than U,- have higher orders than those of utilities lower than ,,+, Within the utility space of r acceptable and unacceptable policies' utilities are mixed together, leading low U- and an even lower U. ,- (b) IRL-based relies solely on IRL's optimal reward
function r' which can be task- mask-mislignned and lead to an unacceptable policy E II [lace while PaseR- IL learns an acceptable policy x² E aa from set RE.5 of reward functions.
The soft advantage of performing action a at state and then following policy afterwards is Ax a) = 2T (s, a) - (s). The expected return of т under a reward function r is given as (T) = 15 7t r(s(t), ("))). With a slight abuse notations, we denote the entropy of Pm
E 12 ((((-1s(()) The standard RL learns optimal policy by maximizing (T). The entropy regularized RL learns soft-optimal policy by maximizing the objective function JRL(F;T) == (T) + H(T).
policy H(T))
Inverse Reinforcement Learning (IRL) assumes that a set E = (TI...., TN of expert demonstra- tions are sampled from the roll-outs of the expert's policy Te and aims to learn the expert reward func- tion rg. IRL and Russell [2000] assumes that Te is optimal under and learns rg by maximizing
margin (E) max (TT) while Maximum Entropy IRL (MaxEnt IRL) Ziebart et al. 20088
maximizes an entropy regularized objective function (TRI( (E) (max U.(TT) ())).
Generative Adversarial Imitation Learning (GAIL) Ho and Ermon 2016 draws connection
between IRL and Generative Adversarial Nets (GANs) as shown in Eq where a discriminator D S x A [0, 1] is trained by minimizing Eq so that D accurately identify any (s, a)
generated by the agent. Meanwhile, an agent policy æ is trained as generator to maximize Eq so that D cannot discriminate
from TE. Adversarial inverse reinforcement learning (AIRL) Fu
[Taas) al. 2018 uses neural-network reward function r to represent D(s, a) exp(r(s.a)) ((()s))
rewnites JIrL as minimizing E and proves that the optimal reward satisfies x" log π Ang. By training with r* until optimality, π will behave just like TE.
[log ((s, + (8,0)~т
[log(1 (ss
(s,a)er
4 Task- Ress-Reward Alignment
In this section, we formalize the concept of task-reward misalignment in IRL-based IL. We start by defining notion of task based on the framework from Abel et al. [202T] Definition 1 (Task). Given the policy hypothesis set IT, a task (TI, Stask, Tacc) is specified by partial order Stask over 11 and a non-empty of acceptable policies 1g C 11 such that Vi e Io and VTz & 1lace T2 Ztaek J always hold.
The notions of policy acceptance and order allow the definition of task to accommodate broad range of real-world taskş including the standard RL. tasks (learning the optimal policy from a reward function r): given a reward function and policy hypothesis set II, the standard RI task can be written as a tuple (II, Stask: Iace) where Itask satisfies VT 22 E II, _ Stack 42 + (TT)) S U, (172), and aa { VT E 11 Stask ) contains all the optimal policies.
Remark:
See examples in Abel 2021
3

---

Designing reward function(s) that align with the underlying task is essential in RL. Whether a designed reward aligns with the task hinges on how policies are ordered by the task and the utilities of the policies under the reward function. Therefore, we define the task-reward alignment by examining the utility spaces of the reward functions. If the acceptable policy set Naaee of the task given, we let U, = min (T) be the minimal utility achieved by any acceptable policy under r.
sellace
Definition 2 (Task-
Reward Functions). A reward function is a task-aligned reward function functionn (denoted as r+) if and only if क E ITaee U,- (T) < E (T.) Conversely, if this condition is not met, it is task-misaligned reward function (denoted )-
The definition suggests that under a task alikned reward function rt, all acceptable policies for the task yield higher utilities than unacceptable ones. It also suggests that a policy is deemed acceptable as long as its utility is greater than U, for some task-aligned reward function r+, even this policy is not optimal. We also examine whether high utility under a reward function r suggests higher order under Stask. We define U, := max (T( s.t. VTI, T2 E I1, UT(T)) < (+) S (T2
FEll
(m1 Stask T) A (T11 Stask TT2), which is the highest utility threshold such that any policy achieving higher utility than U, has a higher order than those achieving lower utilities than U,. In Figure (a)
we illustrate how U, and U, vary between task-aligned and misaligned reward functions.
Proposition 1. Given the policy order Stack of a task, for any two reward functions T2. if
{ (T) 2 } C { (T) 2 U.Z). ). then there must exist policies Tı E { Un (T) 2
1, T2 E [T] (72(T) 2 r such that Ur, (T2) S Ur, (T1) and T2 Stask Ti while Ur, (T2) 2 (T)).
This proposition implies that high threshold U, indicates that a high utility corresponds to a high order in terms of Stask- In particular, for any task-aligned reward function r+. {-T U,++ (T) Z
C = {Tr (T) 2 U,+ } (see proof in Appendix A.2). Thus, a small {x U (π) 2 3 leads to a large { (1)+(T) E [ U,,]]. Hence, a task-aligned reward function r is more likely to be aligned with the task if it has wide (U,+, -,-] and a narrow [Un+, max U,+ (T)]. sell
4.1 Mitigate Pask-Reward Misalignment in IRL-Based IL
In IRL-based II., a key challenge is that the underlying task is unknown, making it difficult assert if a learned policy is acceptable. We denote the optimal reward function learned from the demonstration set E as r", and the optimal policy under r" as TT.,, When ny, has a poor performance under TE. it is considered to have a high Regret(Tr. rE) which is defined in E If Regret(TT, rE) > max Urg (x') - Lr then T.. is unacceptable and r* is mis..id.
E'lll
Regret( r) == max U²(7') (T) e'llI
Several factors can lead to a high ( For instance, Viano et al. [202]] shows that when expert demonstrations are collected in an environment whose dynamical function differs from that of the learning environment, ((tegre((TT= rg) can be positively related to the discrepancy between those dynamical functions. Additionally, prove in Appendix A. that learning from only a few representative expert trajectories can also result in a large [Regret( (ay-; rE)l with a high probability,
Our insight for mitigating such potential fask-reward misalignment in IRL-based II. is to shift our focus from learning an optimal policy that maximizes the intrinsic TE to learning an acceptable policy x² that achieves a utility higher than U,+ under any task task-aligned reward function r+. Our approach is to treat the expert demonstrations as weak supervision signals based on the following.
Theorem 1. Let I be an indicator function. For any k 2 {min ΣtEn I(U (x) 2 Up (T.)
if satisfies (Erem (T),(T) 2 U+{+*)}} < [Maacll for all r E RE := fr| EVEN [TU,(=) Z (T()))} < k), then T² is an acceptable policy, ie., n²" E Nacc. Additionally, if l < I Nacel such an acceptable policy n² is guaranteed to exist.
The statement suggests that we can obtain an acceptable policy by training it to attain high performance across reward function set RÆ that includes all the reward functions where, for each reward function at most policies outperform the expert policy Tg. The minimal value of k determined by all the task- reward functions in the reward hypothesis set. Appendix A.2 provides the proof.

---

How to build Rg Building Rg involves setting the parameter k. rg is task-aligned reward function and F is optimal solely under TE, then the minimal rg. However, relying on a singleton Ræ,0 equates to applying vanilla IRI., which is susceptible to misalignment issues, as noted earlier. It is crucial to recognize that rE might not meet the task-aligned reward function criteria specified in Definition[] even though its optimal policy rg is acceptable. This situation necessitates a positive k. thereby expanding Rg beyond a single function and changing the role of expert demonstrations from strong supervision to weak supervision. Note that we suggest letting& k < [Maael instead of allowing k + oc because RE would then encompass all possible reward functions, and it is impractical across all reward functions. Letting k E |Ilace| guarantees there exists a feasible policy πτ,, e.g. Te itself. As the task aligmment of each reward function typically remains unknown in IRL settings, this paper proposes treating it as an adjustable parameter starting with a small l and adjusting based on empirical learning outcome, allowing for iterative refinement for alignment with task requirements.
0, and RE only contains
identify policy capable of achieving high performance
In practice, IT can be uncountable, e.g., a Gaussian policy. Hence, we adapt the concept of Rg to hyperparameter å < 8* := max (T),(r), leading to redefine R k as a 8-optimal reward
function set R3 {r| (TR 2 8}. This superlevel set includes all the reward functions under which the optimal policies outperform the expert by at most -8. If is appropriately selected such that RÆ includes task-aligned reward functions, we can mitigate reward misalignment by satisfying the conditions outlined in Definition 3 which are closely related to Definition 2 and Proposition [] Definition 3 (Mitigation of Rawk-d Misalignment), Assuming that the reward function set R contains task-aligned reward function r+'s, 's, the mitigation of task-reward misalignment in IRL-based IL is to learn policy x² such that (i) (Weak Acceptance) Vr+ E RE (T+(TT) 2 U,+, or (ii) (Strong Acceptance) Vr+ RRg,, U,+ (TT) U,+.
While condition (i) states that x" is acceptable for the task, i.e., n² E Iace. condition (ii) further states that π have a high order in terms of Stask- Hence, condition (i) is weaker than (ii) because a policy n" satisfying (ii) automatically satisfies (i) according to Definition 2 Given the uncertainty in identifying which reward function is aligned, our solution is to train a policy to achieve high utilities under all reward functions in Rg s to satisfy the conditions in Definition this approach in the following semi-supert cup-rvuer paradigm, PAGAR.
3
We explain
5 Protagonist Antagonist Guided Adversarial Reward (PAGAR)
PAGAR is an adversarial reward searching paradigm which iteratively searches for reward function to challenge a policy learner by incurring a high regret as defined in Eq[]] We refer to the policy to be learned as the protagonist policy and re-write it Tp. We then introduce second policy, dubbed antagonist policy π A. as a proxy of the arg max U²(T'') for Eq 2
For each reward function r, we call
x'ell the regret of xp under i.e., Regret(MTP.) r)
max AT
(,P)),, the Protagonist Antagonist
axEl Ell
Induced Regret. We then formally define PAGAR in Definition 4
Definition& 4 (Protagonist Antagonist Guided Adversarial Reward (PAGAR)). Given a candidate reward function set Rt and protagonist policy TT, PAGAR searches for a reward function within R to maximize the Protagonist Antagonist Induced Regret, i.e., max Regret(Tp,r). re
PAGAR-based IL learns a policy from Rg. by minimizing the worst-case Protagonist Antagonist Induced Regret via M inimozRegrett Rg. as defined in Eq where R can be any input reward function set and is set as R Rg in PAGAR- based IL.
HinimazReggett R) arg min max Regret(Tp,.) xrell rER
Our subsequent discussion will focus on identifying the sufficient conditions for PAGAR- IL to mitigate task-reward misalignment as described in Definition 3 In particular, we consider the case where (r) == (F) max (T(.). We use I.p to denote the Lipschitz constant of r(r), and
WE to denote the smallest Wasserstein 1-distance W, (H, F)) between 7 ~ क of any и and T ~ F, i... WE A min (T E). Then, we have Theorem
TEll
Theorem (Weak Acceptance). If following conditions (1) (2) hold for RE then the optimal protagonist policy Te := Minima Regret s) satisfies Vr E RE,,, Up+ (TP) 2 U,t.
5

---

(1) There exists r+ E Rg,5s, and
max rt Rp,
{max U,+ (π) }+}}
< - rt ERR,
min (U,+ - ---
TEl
(2) Vr+ E R Lxt WE max Upt (T) U and Vr- E RE.S, Ly- WE 8
mell
min rt ERE
-
This statement shows the conditions for PAGAR-based IL to attain the 'Weak Acceptance' goal described in Definition 3 The condition (1) states that the task-aligned reward functions R all have a high level of alignment in matching Stask within their high utility ranges. The condition (2) requires that for the policy x = arg min Wi E), the performance difference between E and Hell
n is small enough under all r E Rg.5. Since for each reward function r E Rg.5, the performance difference between E and the optimal policy under is bounded by 8, condition (2) implicitly requires that x" not only performs well under any task-aligned reward function y+ (thus being acceptable in the task) but also achieve relatively low regret under task-misaligned reward function r-. However, the larger the rage , .n,+ is across the task-aligned reward function r-, the less strict the requirement for low regret under r- becomes. The proof can be found in Appendix A.5 The following theorem further suggests that 6 close to its upper-bound d == max JIRL (r) can
help Minimar Regret(Rg, gain better chance of finding acceptable policy for the underlying task and attain the 'Strong Acceptance" goal described in Definition 3
Theorem 3 (Strong Acceptance). Assume that the condition (1) in Theorem 2 holds for Rg 6- for
any E RE L, WE - S min {max (TT(TT) - U,+ }, then the optimal protagonist policy
rleRg, well
TP Minimar Regret Rg,5) satisfies Vr+ E RE Up+ (TP) 2 U,+.
When do these assumptions hold? The condition (1) in Theorem rrquuiess all the task-aligned reward functions in Rg exhibit a high level of conformity with the policy order Stask- Being task task-aligned already sets strong premise for satisfying this condition. We further posit that this condition more easily satisfied when the task has binary outcome, such as in reach-avoid tasks so that the aligned and misaligned reward functions tend to have higher discrepancy than tasks with quantitative outcomes. In the experimental section, we validate this hypothesis by evaluating tasks of this kind. Regarding condition (2) of Theorem 2 and the assumptions of Theorem B which basically require the existence a policy with low regret across RE set, it is reasonable to assume that expert policy meets this criterion.
5.1 Comparing PAGAR-Based IL with IRL-Based IL
We illustrate the difference between IRL-based IL and PAGAR-based IRL in Fig l[b). While IRL- based IL aims to learn the optimal policy म.- under the IRL-optimal reward r", PAGAR- based IL learns a policy Λ² from the reward function set Rg Both PAGAR IL and IRL-based IL are zero-sum games between a policy learner and a reward learner. However, while IRL-based IL only aims to reach equilibrium at single reward function under strong assumptions, e.g., sufficient demonstrations, convex reward and policy spaces, etc., PAGAR-based II. can reach equilibrium with mixture of reward functions without those assumptions.
Proposition 2. Given arbitrary reward function set R. there exists constant c and a distribution R, over R such that MinimarRegret// yields the same pol- Regret(m.r2) --U,;(7) (R)
E [0 Regret(r.))) - (+]] where r arg U,(T) s.t. E arg max Regret(T,,"), It
icy as arg max nEll
(+2(T)+ + (-) c-0,(x)
r'Ell
detailed derivation can be found in Theorem 6 in Appendix A.4 a nutshell, R2 (x(r) is baseline r= [U][))) holds for all the r's that do not always perform
distribution over R such that (i) = E
worse than any other policy under R. (ii) among all the R2 's that satisfy the condition (i), we pick the one with the minimal c; and (iii) for any other policy RS uniformly concentrates on arg max U, (T)). Note that PAGAR-bare IL, where Rg,s is used in place of arbitrary R. R, F a distribution over RE and r is constrained to be within Re.s. Essentially, the mixed reward functions dynamically assign weights to ~ R2 and r depending If performs worse under

---

r7 than under many other reward functions (U,, (T) falls below c), higher weight will be allocated to using r to train x. Conversely, if æ performs better under r than under many other reward functions (c falls below Ur (æ)), )) a higher weight will be allocated to reward functions drawn from RR. Furthermore, we prove Appendix A.7 that the Minimar Regret objective function deñined in 3 is convex optimization w.r.t the protagonist policy pp-
We also prove in Appendix A.8 that when there is no misalignment issue, i.e., under the ideal conditions for IRL, PAGAR- based IL can either guarantee inducing the same results as IRL-based IL with 5 max JIRL (r), or guarantee inducing an acceptable TP by making max JIRL(r) no
r greater than max U (T) U++ for r+ E RE,6-
sell
6 A Practical Approach to Implementing PAGAR-based IL
We solve MinimaxRegrre Rg,,). s. by alternating between policy learning and reward search. Based on [ we introduce an on-and-off policy learning framework and an adversarial reward search objective. Moreover, we embed the constraint r E RR into the reward search objective using IRL, resulting in meta-algorithm compatible with various IRL methods.
6.1 Policy Optimization with On-and-Off Policy Samples
Given an intermediate learned reward function r. we use RI. to train To to minimize the regret min Regret(TP..) r) = min {max U, (TA)) - (TP) as indicated by Eq where n is trained to up
"H FA
serve as the optimal policy under r as noted in Section []] Since we have to sample trajectories with TA and Tr, we propose to combine off- off-policy and on-policy samples to optimize Tp that we can leverage the samples maximally. Off-Policy: We leverage the Theorem in Schulman et al. 2015 to derive a bound for the utility subtraction: U, (TT)) U, (TA) S
Σ PTTA (s) (Tp(as)) (s,a)
SEB
aEA
(-((()))ss,(())))))) (=)+s))² where PTA (8)= = - (rrb((s( = a|TA) is the discounted C Drv (TA((s),
visitation frequency of TA, Ara is the advantage function without considering the entropy. and C is some constant. Then we follow the derivation in Schulman al. [2017]. which is based on Theorem 1 in SnSScuulman et al. [2015). to derive from the inequality an importance sampling-base based objective function Jr. (
Ex-TA [min(((s, a) Ara (s, a), clép(E(s, a), 1 o, + σ)) Ar, (s, a)] where a is a clipping threshold, E(s, a) = xp(als) is an importance sampling rate. The details can be found in Appendix B.T. This objective function allows us to train Tpp by using the trajectories of πA- On-Policy: We also optimize mp with the standard RL objective function (= by using the trajectories of тP itself. As result, the objective function for optimizing πр is max (TP:T) + (T) As for TA we only use the standard RL objective function, i.c., Felll
<x(a]s]
max (TAT)) Although the computational complexity equals the sum of the complexities of RL mAell
update steps for T and Tp, these two RL update steps can be executed in parallel.
6.2 Regret Maxmization with On-and-Off Policy Samples
Given the intermediate learned protagonist and antagonist policy Tr and тA, according to W'nnmmrRRegre in 3 we need to optimize to maximize (,(TA) - U,(T)) In practice, we found that the subtraction between the estimated (T( and (-(T)) can have a high variance. To resolve this issue, we derive two reward improvement bounds to approximate this subtraction.
Theorem 4. Suppose policy π2 E II is the optimal solution for JRI (T;:)) (T) Then, the inequalities Eq and 5 hold any policy Ta E 11, where max Dry ( (-15), T2 (+1s)). = max |Ar (s, a)).
8.0
and A
E a~p
(A,,(,,0) - E
ami72
AA(s
207e
U,(TT) (T(TT2)
E 2(1)
5 (1 7)
(4)
20r + 1)c - 71-
U,(TT) UT(TT2)
E a(1)
AA(s(1)]
(5)
7

---

In both tasks, Algorithm An Meta-Algorithm for Imitation Learning with PAGAR
Expert demonstration set E, IRL objective function JInL, loss bound 8, parameter À 2 initial protagonist policy Tp, antagonist policy TA reward function r, maximum iteration number N. Output: TP
Input:
for iteration i 0, 1, ...., .N do
2: Sample trajectory sets D ~ Ta and Pp ~ TT
3: Optimize A estimate Jnz ( (HA;T) with a. update нA to maximize Jne (TAA;)
4: Optimize Tp: estimate (TI((T:;) with Rp; estimate J= (Tp;r) with AA; update T to maximize (TI(TP::)
Jx (Tp;r)
5: Optimize estimate APA with Dp and ID estimate (TRI(F) with DD a and
E: update r to minimize APA A (r)r(r)); then update À to maximize
8 (r)
6: end for
7:
return 7P
By letting Tp be I and a be T2, Theorem enables us to bound (T (TP) by using either only the samples of a or only those of R.. Following Fu et al. (2018). we let r be a proxy of
Ar, in Eq 4 and 5 Then we derive two loss functions JR. TP, TA) and JR,2(1; TP, Ra) for
as shown 6 and 7 where C₁ and C2 are constants proportional to the estimated maximum KL divergence between TA and TP (to bound [Schulman et al. (2015)). The objective function for then JPAGAR == JR.1 + JR.2. The complexity equals that of computing the reward along the trajectories sampled from AA and TP.
(T: = E
EE (E(s!0, ,a(0) -1 r(8(0) ((0)) Ci
max (a.a)~тa
(6)
TORA
(R:((T:TPPTTA) :=
Ei=o o² (1- ((00))100))) ((s(().(())) + +C2
max (s.a)~up
|r(s; a)l (7)
6.3 A Mela-Algorithm for Solving PAGAR-Based IL
Given an IRL objective function JIRL, we enforce the constraint r E Rg by adding to JPAGAR(P (r: TP, TA) penalty term (8-Jne JIIRL,,, where À is a Lagrangian parameter. The resulting objective function for optimizing becomes min Jpagar(r; Tp, X (8 (T)).
rE
We initialize À with large value to prioritize satisfying the constraint Rg and update it based on 5 JIRL (the details can be found in Appendix B 4). [ outlines our meta- algo-ihg for PAGAR-basel IL. The algorithm takes an IRL objective JIRL policy and reward learning. In line 3, TA is trained via RL with its own sample set D In line 4, we train xp via the on-and-off policy approach in Section 6 with both D and Tr's 's sample set Dpp Finally, in line 5, JPAGAR is estimated from both A and Dp, while JInL is from D and E.
imput, and alternates between
7 Experiments
The goal of our experiments is to assess whether using PAGAR- can efficiently mitigate reward misalignment under conditions that are not ideal for IRL. We present the main results below and provide details and additional results in Appendix C
7.1 Discrete Navigation Tasks
Benchmarks: We consider a maze navigation environment where the task objective is compatible with [ Our benchmarks include two discrete domain tasks from the Mini-Grid environ- ments Chevalier- Boisvert et al. [2023]: DoorKey-6x6-vé and SimpleCrossing.59.N.. the agent needs to interact with the environmental objects which are randomly positioned in every episode while the agent can only observe small, unblocked area in front of it. The default reward, which always zero unless the agent reaches the target, used to evaluate the performance of learned policies. Due to partial observability and the implicit hierarchical nature of the task, these environments are considered challenging for RL and II,, and have been extensively used for benchmarking curriculum RL and exploration-driver RL.
8

---

Baselines: We compare our approach with two standard baselines: GAIL Ho and Ermon 2016 and Peng al. 2019]. GAIL has been introduced in Section VAIL based on GAIL but additionally optimizes variational discriminator bottleneck (VDB) objective. Our approach uses the IRL. techniques behind those two baseline algorithms, resulting in two versions Algorithm T denoted as PAGAR-GAIL and PAGAR-VAII respectively. More specifically, if the baseline optimizes a JIR objective, we use the same JIRL objective in Algorithm | the reward function from the discriminator D mentioned in Section 3 More details are in Appendix C.T PPO Schulman et al. [2017] used for policy training in GAIL, VAIL, and ours with algorithm with state-of-the-ans (SOTA) IL algorithm. IQ-Learn Garg et al. [202]], which, however, is not compatible with our algorithm because it does not explicitly optimize reward function. The policy and the reward functions all approximated using convolutional networks.
Also, we extract
replay buffer of size 2048. Additionally, we compare
Minus not PAGAROWAA AA-
GRIL
Vel..
Frammax
Drammee
DeorKe (10 demos)
(b) DoorKey-6x6 (I demo)
(c) SimpleCrossingS9NI demos)
(d) SimpleCrossingSNN (I demo)
PACAR-CAL HROAE/VA
NiGh CMi
RACATAW
90
Transfer Envs
SimpleCrossingS9N2
(g) SimpleCrossingS9N3
(h) FourRooms
Figure 2: Comparing Algorithm with baselines in partial observable navigation tasks. The suffix after each 'PAGAR-' indicates which IRL technique is used in Algorithm 1. The axis indicates the average return per episode. The x axis indicates the number of time steps.
IL with Limited Demonstrations. By learning from 10 expert-demoostrrate trajectories with high returns, PAGAR-based IL produces tigh-performance policies with high sample efficiencies as shown in Figure[2 and (c). Furthermore, we compare PAGAR- -AIL with VAIL by reducing the number of demonstrations from 10 to 1. As shown in Figure (b) and (d), PAGAR- PAGAR-VAIL produces high-performance policies with significantly higher sample efficiencies.
II. under Dynamics Mismatch. We demonstrate that PAGAR enables the agent to infer and accomplish the objective of task even environments that are substantially different from the one observed during expert demonstrations. As shown Figure e)) we collect 10 expert demonstrations from the SimpleCrosse Sroppleeerssnn environment. Then we apply Algorithm [ and the baselines, Simple ComplleCossinggSNNN3 and
GAIL, VAIL, and IQ-learn learn policies in SimpleCrossigg99222 FourRooms- The results in Figure (()) show that PAGAR -AGAR-based IL outperforms the baselines in these challenging zero-shot settings.
IL with Different Reward Hypothesis Sets. The foundational theories of GAII. and AIRL indicate that different reward function hypothesis sets can affect the equilibrium of their GAN frameworks. We study whether choosing different reward hypothesis sets can influence the per- formance of Algorithm We compare using a Sigmoid function with Categorical distribution in the output layer of the discriminator net- works in GAIL and PAGAR-GAIL. When using the Sigmoid function, the outputs of D are not normalized, i.e., ΣaEs ((s, # 1. When using Categorical distribution, EaER D(s,a) 1. We test GAIL and PAGAR-GAIL. in Doorrey-6x6-v environment. PAGAR-GAIL outperforms GAIL in both cases by using fewer samples.
Figure 3: PAGAR-GAR GAIL
in different reward spaces
shown in Figure 3

---

14030
9003
500
10030
2300
4003
2300
300
2000
2063
2000
nAL
IAGARIIL
1300
1003
300 125
275
9em
(a) Ant
(b) Hopper
(c) Walker2d
(d) HallCheetah
Figure 4: Comparing Algorithm with f-IRL continuous control tasks. "PAGAR-fIRE indicates
f-IRL is used as the inverse RL algorithm in Algorithm 1. The axis indicates the average return per episode. The x axis indicates the number of time steps the environment.
RECOIL
PAGAR-RECOIL
Hopper-randone
106.87 = 2.69
111.16
malfcheetah-randee
80.84 17.62
92.94 = 0.10
walker2d-random
108.40 + 0.04
108.40 + 0.12
ant-random
113.34 2.78
121 + 5.86
Table I: Offline RL results obtained by combining PAGAR with RECOIL averaged over seeds.
7.2 Continuous Control Tasks
We evaluate JAGAR-based IL on continuous control tasks in both online and offline RL settings, demonstrating its ability to improve IRI-bbased II. performance across different types of tasks.
Benchmarks: We use four continuous control environments from Mujoco. In the online RL setting, both protagonist and antagonist policies are permitted to explore the environment. RL setting, exploration by these policies is restricted. Especially, for offline RL we use the D4RL's 'expert' datasets as the expert demonstrations and the 'random' datasets as the offline suboptimal dataset. Policy performance is evaluated online in both settings by using the default reward function of the environment.
the offline
Baselines: We compare BGGAR-based IL against f-IRL Ni et al. [2021] in the online RL setting and compare with RECOIL, Sikchi et al. [2024] in the offline RI. setting. When comparing with f-IRI, we use f-IRI. as the IRI. algorithm in Algorithm T When comparing with RECOIL, as RECOIL. does not directly learn the reward function but learns the Q and functions, we develop another algorithm for the offline RL setting to combine PAGAR with RECOIL by explicitly learning reward function and using the reward function and function to represent the Q function. The details can found in Appendix B.4
Results: As shown in Figure PAGAR based IL achieves equivalent performance to the baselines with fewer iterations. Furthermore, on the Ant and Walker2d tasks, Algorithm [ matches the performance level of f-IRL using significantly less iterations. Additional results PAGAR with GAIL and VAIL. across other continuous control benchmarks are provided in Appendix further shows that when combined with RECOIL, PAGAR-based II achieves higher performance in most of the tasks than the baseline. These results demonstrate the broader applicability of PAGAR- based IL in both online and offline settings and its effectiveness across different types of environments, further reinforcing the robustness of our approach.
C3 Table ]
8 Conclusion
In this paper, we propose prioritize task alignment over conventional data alignment in IRL-based IL by treating expert demonstrations weak supervision signals to derive a set of candidate reward functions that align with the task rather than only with the data. Our PAGAR-based adversarial mechanism to train policy with this set of reward functions. Experimental results demonstrate that our algorithm can mitigate reward misalignment challenging environments. Our future work will focus employing the PAGAR paradigm to other task alignment problems.
adopts an
10

---

Acknowledgment
This work was supported in part by the U.S. National Science Foundation under grant CCF-2340776.
References
Abbeel and Y. Ng. Apprenticeship learning via inverse reinforcement learning. Proceedings of the Twenty-First International Conference on Machine Learning, ICML '04, page I, New York. NY, USA, 2004, Association for Computing Machinery. ISBN 1581 138385, doi: 10. 145/1011133 1015430.
D. Abel, W. Dabney, A. Harutyunyan, M. K. Ho, M. Littman, D. Precup, and Singh. On the expressivity of markov reward. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances Neural Information Processing Systems, 2021.
D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in Al safety, CoRR, ftbs/1606.06665 2016.
H. Cao, S. Cohen, and L. Szpruch. Identifiability in inverse reinforcement learning. Advances in Neural Information Processing Systems, 34: 12322-13333 2021.
M. Chevalier-Boisvert, Dai, M. Towers, de Lazcano, L.. Willems, S. Lahlou, S. Pal, Castro, and J. Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306. 13831, 2023.
M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine. Emergent com- plexity and zero-shot transfer via unsupervised environment design. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 13049-13061. Curran Associates, Inc., 2020.
C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference machine learning, pages 49-58. PMLR, 2016.
C. Finn, Yu, Fu, P. Abbeel, and Levine. Generalizing skills with semi- sup-suupe reinforcement learning. In International Conference Learning Representations, 2017.
Fu, Luo, and S. Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. IQ-learn: Inverse soft-q learning for imitation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
D. Hadfield-Menell, Milli, P. Abbeel, J. Russell, and A. Dragan. Inverse reward design. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Hejna and Sadigh. Inverse preference learning: Preference-based RL. without reward function. In Thirtyy-eventh Conference on Neural Information Processing Systems, 2023. URL https 77 wwwpaaewwieee.eee/ooomiiiiieggee
J. Ho and S. Ermon. Generative adversarial imitation learning. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Gameett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
J. Hong, K. Bhatia, and A. Dragan. On the sensitivity of reward inference to misspecified human models. In The Eleventh International Conference on Learning Representations, 2023.
Jeon, S. Seo, and K.-F, Kim. A bayesian approach to generative adversarial imitation learning. In Bengio, H. Wallach, Larochelle, K. Grauman, N. Cesa-Bianchi, and Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.

---

D. Lindner, A. Krause, and G. Ramponi. Active exploration for inverse reinforcement learning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
A. M. Metelli, G. Ramponi, A. Concetti, and M. Restelli. Provably efficient learning of transferable rewards. In M. Meila and Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7665-7676. PMLR, 28-24/1 Jul 2021.
A. Y. Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference Machine Learning, ICML '00, pages 663-670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1-55860-707-2.
A. Y. Ng. D. Harada, and S. J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning. ICML '99, pages 278-287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1-55860-61222.
T. Ni, H. Sikchi, Wang. T. Gupta, L. Lee, and B. Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference Robot Learning, pages 529-551 PMLR, 2021.
A. Pan, K. Bhatia, and J. Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022. URL wit.orrmnndyyytGGl117/
https://openreview
X. B. Peng, A. Kanazawa, Toyer, P. Abbeel, and S. Levine. Variational discriminator bottle- neck: Improving imitation learning, inverse RL, and GANs by constraining information flow. In International Conference on Learning Representations, 2019.
D. Ramachandran and E. Amir. Bayesian inverse reinforcement learning. In Proceedings of the 20th International Joint Conference Artifical Intelligence, IJCAI'0, page 2586-2591, San Francisco, CA, USA, 2007. Morgan Kaufmann Publishers Inc.
N. D. Ratliff, J. A. Bagnell, and M. A. Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference Machine learning, pages 729-736, 2006.
Ross and D. Bagnell, Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661-668 2010.
S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635, 2011.
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06341 2017.
H. Sikchi, Q. Zheng, A. Zhang, and S. Niekum. Dual RL: Unification and new methods for reinforcement and imitation learning. In The Twelfth International Conference Learning Representations, 2024. URL /tRt:tttt:://peereeie httpl/gerreeieeeeeeeeoooteee www.:/eee.foorriaee. didtrrunrrrtttttteerrrrr
Skalse and A. Abate. Misspecification in inverse reinforcement learning. arXiv preprint arXiv:2212.03201, 2022.
J. Skalse, M. Farrugia-Ronn Roberts, S. Russell, A. Abate, and A. Gleave. Invariance in policy optimisation and partial identifiability in reward learning. arXiv preprint arXiv:2203.07475. 2022a.
J. M. V. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger. Defining and characterizing reward gaming. In A. H. Oh, Agarwal, D. Belgrave, and K. Cho, editors, Advances Neural Information Processing Systems, 2022b.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. 2018.
12

---

L. Viano, Y.-T. Huang, P. Kamalaruban, A. Weller, and V. Cevher. Robust inverse reinforcement learning under transition dynamics mismatch. In A. Beygelzimer, Y. Dauphin, Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https
7/
W. Zhou and W. Li. awfety-aware apprenticeship learning. In H. Chockler and G. Weissenbacher, ed- itors, Computer Aided Verification, pages 662-680 Cham, 2018. Springer International Publishing.
ISBN
W. Zhou and W. Li. Programmatic reward design by example. Proceedings of the AAAI Conference on Artificial Intelligence. 36(8):9233-9241, Jun. 2022a. doi: 10.1609/aaii v36............200000 20910.
W. Zhou and W. Li. hierarchical Bayesian approach to inverse reinforcement learning with symbolic reward machines. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 27159-211111 17177-7717. PMLR, 17-23 Jul 2022b.
W. Zhou, R. Gao, B. Kim, E. Kang, and W. Li. Runtime-safcty-quidee policy repair. In Runtime Verification: 20th International Conference, RV 2020, Los Angeles, CA, USA, October 6-9, 2020, Proceedings 20, pages 131-150. Springer, 2020.
B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume 3, AAAI'08, pages 1433 1438. AAAI Press, 2008. ISBN 978-1-57735-368-3.
13

---

A Reward Design with PAGAR
This paper does not aim to resolve the ambiguity problem in IRL. but provides a way to circumvent so that reward ambiguity does not lead to reward misalignment in IRL-based IL. PAGAR, the semi- supervised reward design paradigm proposed in this paper, tackles this problem from the perspective of semi-supervised reward design. But the nature of PAGAR is distinct from IRL and IL: assume that a set of reward functions is available for some underlying task, where some of those reward functions align with the task while others are misaligned. PAGAR provides solution for selecting reward functions to train a policy that successfully performs the task, without knowing which reward function aligns with the task. Our research demonstrates that policy training with PAGAR equivalent to learning policy to maximize an affine combination utilities measured under a distribution of the reward functions in the reward function set. With this understanding of PAGAR, integrate it with IL to illustrate its advantages.
A.1 Motivation: Failures in IRL-Based IL
For readers' convenience, we put the theorem from Abel
[202]] here for reference.
Theorem 5. Viano al. (202]) If the demonstration environment has dynamics function Pdemo different from the dynamics function P in the learning environment, the performance gap between the policies TE and Ty. under the ground true reward function Tg satisfies (rrg(TE) - UrE (TT.) S
2-y-max [r((,,))
max Dry (P(- s, Ptemol s,a)).
(1-7)²
6,0
Then, we prove that limited number of demonstrations can also lead to the performance gap as described in Theorem Essentially, we can construct a demonstration dynamics P Pdemo such that all the state transition probability estimated in E match Pdemo with zero error. Then a similar performance gap that caused by dynamics mismatch can be derived.
Proposition 3. Assume that the expert demonstration contains m state-action pairs by only selecting the optimal actions, i.e., select arg max Tg(aas at each s. Then Urg (TE) U (Tyr) S
a
2-d m-xx re(s,a) |
where Prob(d 2 t) S 2 exp(-2 m 2) for any E [p, 1] where = min rg(s,a) - min (as(a, max (a)))))))
(1-7)²
mAX (2(a,)) min rg(e,))
Proof. We translate the limited demonstration case into a dynamics mismatch case where we will leverage the demonstrated state transitions construct a dynamics Pdemol | 8, a) and compare with P(- a).
Assume that for each state-action pair s, in E, i... Ma.a E [m]] the my,a transition instances (s, s1), (s, a, 82), .., (s, a, 5m,.a). ). For each 3 E S, we let Pdemo 8, s, ma.a D I[s, $).
Then, we can view (P a), P. s,))) = |P( s,a) - Paemol 1,, (s,a))l1 as function of m independent random variables, i.e., (s1, (2,,(22))))))))))) a2)...., (Sm.n.mm..a)) sampled from the domain. ( s, a) and (3'mo a) should decrease and increase by m..a respectively. Then, Drv s, a)) P. (-1 ss a)) changes at most by m...
one of the variables, 8,, changed from å to another state 3' * 8,
We can also view the Drv each as E E as functions of the m sampled state-action pairs in E, with a little abuse of notations, denoted as a1), ($2, (2)..... (Sm: am) by flat- tening all the demonstrated trajectories. If state-action pair, (si, az), in one of the trajecto- ries, is changed from (&, â) to another state (Ś',ā). then for the state-action (8--1,0;11) that precedes 8 if 8; is not the initial state, DTv(P( 8, s,a), P. 8,a)) changes most by 18 11-1
1. Also, (P( | 8, 8, P. 8, a)) and Dr 8', 8',ā'), (dd,,Pdemoll Pdemol. | 8', ".) change at most by mas and myra"" Note that if my a O, i.c. (8',d') & E, changing â into 3', equiv- alently leads to taking Drv 8, à, Paemnol 8', a')) into consideration when computing
which can be as large as since it is possible that M8.-1.4--1
14

---

max Drv (P( s, Pdemol- s,a
1 I a)). And Drv (P(- s, Pdemnol- 1 a') <1 Ple'''''',a"
where 8,41 is the state that succeeds s; in the demonstrated trajectroy. Therefore, if changing a Drv s, Pdemol- (- s, a)) can be (s,a)EB
state-action pair in E' into another state-action pair, max
changed as large as by
According to McDiarmid's Inequality, Prob[], max (s,a)E
Drv(P( 8, P Pdemol. s,)))
E[ max (F
Drv(P(- 8, a), Paemol | 8, a))l 2 6) S 2 exp(
2. (1)( = 2 - 2²)
Note that E|max Drv(P( s, a), Pdemol 1s, a))] 0 since the estimation is un-biased. Hence, 8.0
Prob( max (P( a), P. 8, a)) 2 E) 2 exp(- -22x) for (s, E E.
(s,a)E
For state-action pairs that are outside of E, we need to construct Pdemo those state-action pairs such that the action selection in E is always optimal. To start with, we let P.emo a, s) = P( a, s) for any state-action pairs that do not appear in E. For some state-action pairs, the states are not reached in E. We can disregard whether T maintains optimal at those states and focus on the states that do appear in E. By denoting the Q-value of Te under Pdemo Qdame we need to make sure that the Quemooss (s, for a' + arg max g no greater than Querro (s, arg max (E(ass)) Since the state-actions in E may have extremely low rewards stochastic nature of P, consider the worst case Q-value for a* = arg max T 8) i.c., Qdamo (s, a²) 2 T min (' arg max (E(a)))).
a Meanwhile, for a' * arg max (a)s)) we consider the best-case Q-value, i.e., Qdomo (s, S
mmx a). We add dummy, absorbing state that is unreachable from any actton 1-y (s.a)
under the dynamics P and also unreachable from any demonstrated state-action under the constructed
dynamics Pdemo But we let Poemo (s]s, a') > 0 if Qdeme (s, a') > Qdemio a²). ) In other words, we add probability density on transitioning from such (8, a') to g so that Quemo a') drops below Qdemooss ).) We denote this density as p. Then (glm, a')
and Pdemo (8 (5'(s.c) a') = (1- p) PP(s'ss,a') for any other s' E E. As a result, Drv(P(- s, a, (- s, s.a') = Note that adding does not affect the Q-values of the state-action pairs that appear in E. Then we aim to find the p to ensure gdeemoss (s,a') S Qdemo (a,
(s,a"). Considering the worst-case, p (max rn(s, a)
min rE(8, + (1 2) I max (E(S, a) S + min ræ(8, arg max TE(a|s)) gives s.a
5.0
max TR(s,a) min the (staarr max (a(a))
p
max (E(s.a)-uin 7.8" (a(s,))
Combining the analysis on the state-action pairs that appear in E' and not in E, we can conclude that for P. extend the confidence bound Prob( max Drv(P( s, a), Pdemo (- s, s,a)) 2 (s,a)EE
<) 2 exp(e 2 for (s, a) E E from (s. a) E E' to all state-action pairs in S x A. By using a variable d represent max Drv(P(( s, a) PPemmo s, a)), we can use the conclusion drawn s.a
from Theorem 5
Note that adding the dummy does not affect the max (ræ(s, a) since we let 8,0
(n a) min (E(8, a). Our proof is complete. s.a
A.2 Rewa-d Alignment
In this section, we provide proof of the properties of the reward alignment concept that we define in the main text. For readers' convenience, we include our definition of task for reference.
Definition D (Task) Given the policy hypothesis set ΤI, task (IT, Stask; Tacx) is specified by a partial order over IT and non-empty set of acceptable policies IT acc C TI such that VT, E Ta and VTz & Loce. 72 Staek тi always hold.
We have defined in the main text that U, := min U,(TT) is the lowest utility achieved by any rEll.co
acceptable policies under r, and U, == max U, (T) 8.1. VT,, T2 E (m)) S (T(T) S UT(22) +
#ell
(IT1 Stask T) A (IT1 Stask T2) is the highest utility threshold such that any policy achieving utility
15

---

higher than U, also has higher order than those of which utilities are lower than U. We call reward function r a (Ur,U) --allgnde with the task.
Lemma 1. For any task-aligned reward function rt, S.U++ 2 Un
Proof. If U+ U, then for any policy with (T) U,,, it is guaranteed that T E [Tce by definition of - Then, U,+ is a better solution for max (T) s.t. VT T2 E
well
I1, (TI S U+(T)) S UT(T22 + (T Stask т) A (M Stosk T2) than Upt. Hence, U,+ 2 U must hold.
Lemma 2. If m1 Stask 42 + (T(T2) S (T1) holds for any T1, E II, then U, = U, min (T). xell
Proof. Since the policy that has the highest order achieves the lowest utility min U, (x), no other xell
acceptable policy can achieve even lower utility. Hence, by definition U, min (T).
well
Another example is that when optimal policy under r has the lowest order in terms of Stack. and the highest-order policy has the lowest utility under r. U, = U,. In those extreme cases, the reward function can be considered completely misaligned. In the non-extreme cases, if a reward function exhibits misalignment with the task - the threshold U, is lower than the utility of an unacceptable policy U, then must be also lower than the utilities of all the acceptable policies, which forms lower-bound for the size of U.-
Lemma 3. For any reward function r, 3x & nace, U²(T) 2 U. then U, U.
Proof. Since π & Uaace and (T) 2 U, then Va' E Ilaces U,(T') 2 U, must be true, otherwise ER' E [acce n² Stask H, which contradicts the definition of Iaae and U,.
Lemma 4. If fewattt r has U, S U,, r is a task-misaligned reward function.
Proof. If U, < U. then there must exists two policies ni 72 where E IT (m(m)) U,, and U,(T2) 2 U, A オ2 Stask W1. Such must not be acceptable policy. Otherwise, it contradicts the definition of aacc- Hence, must unacceptable, leading to being a task mask-misaligned reward function.
r2, if {T (x) 2 then there must exist a Ti E fx : U,, (T) 2 Un and a T2 E {a (T, 2 U. that satisfy U,, (T2) < Ur, (TT) and T2 Stesk M. while (T)) (22()T))
Proposition 1 For any two reward functions rı, r2,
Un (T (T)
Proof. According to the definition of Un VT E { (T) 2 Ún1 and Vng E { (T) > F/at//t Ur, (T)
(T2) < Un, (m1) and Te Stask Ti must be true. Furthermore, for all pairs of T2 E { (T) > U/ (n) 2 U, and मा E (5 U,, (अ) 2 }. U.2(T) (T2) is true, then (T) (T) U.)) C [U max Ur, (=) is a smaller
xell
non-empty interval than (Ur,, max Ur,(z)) (+)], contradicting the fact that U,, is the highest utility well
threshold under 12. Hence, there must exist a 22 E { (T)(T) 2 Ur.)/t Un 2 U and a mi E { (z) 2 J that satisfy Ur, (T2) & Un, (TI) and (T) S Urg (T2) while T2 Stask mi aforementioned.
Note that from now on, we use the notation r- to denote task-aligned reward functions and r to denote task-misaligned cewardffncttin for short. Furthermore, if reward function satisfies (m)) < (T222) < कं Stask #2, we call this reward function the ground true reward function, and denote it as Ttask for simplicity. Apparently, any Ttask is task-aligned reward function, and has the most trivial misalignment.
16

---

Lemma S. U Trark
max U Urean (T) Urinat (T) has higher order and higher utility than any other policies.
#ll
Proof. By definition, arg max #ell
Lemma 6. If Triooh is optimal under Ttasks for any reward function r, Ur (Treon)) 2 U..
Proof. If U, < U, then there exists a x with its utility (+(T) 2 U. thus Freak Stask n, 17 which contradicts the assumption that Freas is the highest-order policy.
Furthermore, we have the following property.
Theorem Let I be indicator function. For any {min ΣrEn ((U+((T)))) 2 Up+ (TE))), y+
' satisfies EEnen [TU,(T) 2 (+}+))}} < |Haaell for all r E Rg == frl ΣrEn [TU, (TE)) < k). then x² is acceptable policy, i.e., n² E mace. Additionally, if k < [lacel such an acceptable policy n² is guaranteed to exist.
Proof. Since TE an acceptable policy, there must be least one task-aligned reward function r+ that satisfies 1 (,( 2 U,(TE)) s k since k 2 min { U, (π) 2 U,+ (TE) The
greater l is. the more task-aligned reward functions tend to be included. If π² achieves H (T) 2 Un+ (x*)) < Tacel under any task-aligned reward function r+, x must be acceptable policy. Because if n² unacceptable, there must be acceptable policy performing worse than the unacceptable x" under r+, contradicting the definition task- lass-aaigeed reward function. Hence, n must be acceptable policy. Furthermore, for any is E [min Exem I(U (T) U,- (TT(T) aTee)))
the policy Tg itself satisfied Σ I(U.(T)) 2 (T(T)) < [Maacl for all r &RE. which guarantees the existence of feasible x².
A.3 Semi-supervised Reward Design
Designing a reward function can be thought as deciding an ordering of policies. We adopt a concept, called total domination, from unsupervised environment design Dennis et al. [2020), and re-interpret this concept in the context reward design. In this paper, we suppose that the function U,((T) is given measure the performance of a policy and it does not have to be the utility function. While the measurement of policy performance can vary depending on the free variable r, total dominance can be viewed as invariance regardless of such dependency.
Definition 5 (Total Domination). A policy, TI, is totally dominated by some policy T2 w.r.t reward function set R, for every pair of reward functions T1, E R, U,,(m) < Ur,(T2).
If TT totally dominate T2 w.r.t R, n2 can be regarded as being unconditionally better than T1. In other words, the two sets (T) r E R} and (U2) r E R} are disjoint, such that sup(U, r E R). Conversely, if policy æ is not totally dominated by any other policy, it
R) < (U2(2)))) indicates that for any other policy, say 22, (m)( r R} 2 inf (UT( r E R).
Definition 6. A reward function set R aligns with an ordering <R among policies such that mi <R T2 if and only if is totally dominated by T2 w.r.t R.
Especially, designing a reward function r is to establish an ordering <(r) among policies. Total domination can be extended to pondcy-con reward design, where the reward function selected by following a decision rule w(rr) such that Erer (rT)(r) = 1. We let (T) = (r) (T) be an affine combination of (T'T))ss 's with its coefficients specified by W(T). PER
Definition 7. A policy conditioned decision rule w is said to prefer a policy to another policy т2. which is notated πī ~² T2, if and only if Us (T2).
Making a decision rule for selecting reward functions from reward function set to respect the total dominance w.r.t this reward function set an unsupervised learning problem, where no additional external supervision is provided. If considering expert demonstrations as form of supervision
17

---

and using it to constrain the set Rg of reward function via IRL, the reward design becomes semi- supervised.
A.4 Solution to the MinimaxRegret
Without loss of generality, we use R instead of Rg in our subsequent analysis because solving M MinimozRegri R) does not depend on whether there are constraints for R. In order to show such an equivalence, we follow the same routine as in Dennis et al. [2020]. and start by introducing the concept of weakly total domination.
Definition 8 (Weakly Total Domination). A policy л1 is weakly totally dominated w.r.t reward function set R by some policy if and only if for any pair of reward function T1. E R, Ur, (T1) S (T2).
Note that policy и being totally dominated by any other policy is a sufficient but not necessary condition for being weakly totally dominated by some other policy. policy æ1 being weakly R} I inf |r E R}. We
totally dominated by policy 22 implies that sup(U,(TT) (T1) assume that there does not exist a policy æ that weakly totally dominates itself, which could happen if and only if U, (T) is constant. We formalize this assumption as the following.
Assumption 1. For the given reward set R and policy set II, there does not exist a policy æ such that for any two reward functions r1, r2 E R, U,, (T) = (T).
This assumption makes weak total domination non-reflexive relation. It is obvious that weak total domination is transitive and asymmetric. Now we show that successive weak total domination will lead to total domination.
Lemma 7. for any three policies Tı, T2, T3 E II, ii is weakly totally dominated by T2. is weakly totally dominated by T3, then T3 totally dominates my
Proof. According to the definition of weak total domination, max (TTT min (T22 and rER
rER
max Up (T2) S min Un (Tg). If Tı is weakly totally dominated but not totally dominated by na, then rell
max (T1) = min U, (Tx) must be true. However, it implies min (T( = max U, (T2),, which P
r R
rER
FE
violates Assumption
We finish the proof.
Lemma 8. For the set IT not C IT of policies that are not weakly totally dominated by any other
policy in the whole set of policies w.r.t reward function set R, there exists a range U C T. such that for any policy π E Il-wtd. U S [min (T)), max (T)
rER
Proof. For any two policies T1. 72 E II-wtd it cannot be true that max (T(TT)) = min (T(T)) reR nor min (T1) = max (T2),,, because otherwise one of the policies weakly totally dominates reft
rent
reft the other. Without loss of generalization, assume that max (+(m) min UT(T2). In this rER case, max (,(T2) > min (TIT)) must also be true, otherwise कi weakly totally dominates m2- rEit Inductively, min max (T) > max min (T)... Letting ub = min max Ur(T) and well PER 1b = max min (T) any U [lh, ub) shall support the assertion. We finish the proof.
rER
rEA
Hell rER
sell swed rER
wEll wed PER
Lemma 9, For reward function set R, if policy n E Il is weakly totally dominated by some other policy in and there exists a subset H-w S Il of policies that are not weakly totally dominated by any other policy in then max (T)
min max (x(z²)
x' elEllwwid rER
Proof. If ui is weakly totally dominated by a policy Ta E II, then min U,(T2) = max (,(TT). PER If max (TT > min max U²(m'), then min U+ 2 min max (T('',,, making at r'Ell-oud
FER
reR least one of the policies in Iw- being weakly totally dominated by π2. Hence, max Ur(TT) < re
PER
r'Ell-wid PE
min max U²(T'') must be true.
a'elllued reR
18

---

Given policy x and a reward function the regret is represented as Eq
Regret(F, max U,(T') (T)
(8)
Then we represent the RinimarRegree(f. (R) problem in Eq/
MinèmazRegret(At = arg min max Regret(x,)) welt
rER
We denote as r E R the reward function that maximizes (T) among all the r's that achieve the maximization in Eq 9 Formally,
arg arg (+(
s.t. E arg max Regret(z, r'ER
(10)
PER
Then Minimar Regret can be defined as minimizing the worst-case regret as in Eq Next, we want to show that for some decision rule w, the set of optimal policies that maximize Un the solutions to M AnnimozReggett R). Formally,
Minimar ReniieanRegre((PR = arg max (T)
(11)
xell
We design by letting (TT) -( (1 (()))()) RS where RS E A(R) policy conditioned distribution reward functions, 5,- be delta distribution centered at r and ((x)) is a coefficient. We show how to design R. by using the following lemma.
Lemma 10. Given that the reward function set is R. there exists decision rule R : 11 A(R) which guarantees that: 1) for any policy æ that is not weakly totally dominated by any other policy in min (+(m"); 2) for any that is weakly
II, ie., II wid C II, UR(x)) where x'Ell-urd rER
totally dominated by some policy but not totally dominated by any policy, (R(T) = max U, (T)); 3)
FER
if at is totally dominated by some other policy, w(T) is uniform distribution.
Proof. Since the description of R. for the policies in condition 2) and 3) are self-explanatory, we omit the discussion them. For the none weakly totally dominated policies in condition 1), having con- stant UR ( = is possible if and only if for any policy æ E wwed, E [min U²(m'), max (+)). reR As mentioned in the proof of Lemma g can exist within [min (r(T), max UT(T). Hence, reR
rER
max min U,'2)) valid assignment.
x| r/Elll.wse reft
Then by letting (TT) == Regret(x.rg)) c-U,,(1 ( we have the following theorem.
Theorem 6. By letting w(T) == ((T) 8,z + +1 --T)) - Rr with ((T): = Regret(n.r3)
and any
c-U, (7)
R that satisfies Lemma 10
MinimazRegreet(Ri = arg max (T)
(12)
zell
Proof. If a policy π E II is totally dominated by some other policy, since there exists another policy with larger Un, cannot be solution to arg max (T). Hence, there no need for further NEll discussion on totally dominated policies. We discuss none weakly totally dominated policies and the weakly totally dominated but not totally dominated policies (shortened to "weakly totally
19

---

dominated" from now on) respectively. First we expand arg max (T) as in Eg 13 metf
arg max (T) #Ell
[w(T)(r) (T)
arg max
Rell
EEn
Reg r2) (T) + (UR(T) - Urg(T) (2))) UR(T) c- C-Ur;(t) (T)
arg max =ell
Ur;; Un(()) - (UR(T) (x)) (
= arg max =ell
c- UrU (TT)
UR(TT) - -r; (TT) c c- (m)
arg max sEll
(T) Regrel(T,
(13)
1) For the none weakly totally dominated policies, since by design UR = c. [3 is equivalent to arg max Regret(m, which exactly equals Minimar MinimarRegre(()) R). Hence, the equivalence
=Ell,
holds among the none weakly totally dominated policies. Furthermore, if none weakly totally dominated policy л E -ooo achieves optimality in Minimax Regret (R), its (x) is also no less than any weakly totally dominated policy. Because according to Lemma 9 for any weakly totally dominated policy II, its (R(TI) S c, hence (x,)))(;,, (R(TI) c. Since Regret r;) S Regret( ",,)), (T) 2 (. Therefore, (=) assert that if a none weakly totally dominated (T). Additionally,
policy и is solution to M Mnnmaa
it is also a solution to arg max
mEll
to prove that if none weakly totally dominated policy is a solution to arg max Uw (TT), is also x'ell
solution to Minimax Regret( R)) it is only necessary to prove that ग achieve no larger regret than all the weakly totally dominated policies. But we delay the proof to 2).
2) If policy π is weakly totally dominated and is solution to Mnimmar Regret we show that it is also solution to arg max (T)) i.e., its Un (T) is no less than that of any other policy.
#Ell
We start by comparing with non weakly totally dominated policy. for any weakly totally dominated policy Ti E Minimaz (eiimazzeere(A))) Regrel(T2, ) for any T2 E IT that weakly totally dominates Ti- However, it also holds that Regret(Tg, r S Regret(T1, Regret(m r Regret(me,r*, = Regret(Ti, r=g)), implying that I2 is also solution to M MinimoRRegeet(R
must hold true that Regret(m1, S
due to the weak total domination. Therefore,
It also implies that Urz, (T1) = Ur (T2) 2 Urz, (TI) due to the weak total domination. However, by definition Ur. (TI) 2 (TT)- Hence, Ur:, (m1) Ure, (=1)= = U, (12) must hold. Now we discuss two possibilities: a) there exists another pol- icy Ty that weakly totally dominates T2: b) there does not exist any other policy that weakly totally dominates π2. First, condition a) cannot hold. Because inductively it can be derived Ur:, (T1) = Ur: (TT) = Ur:, (I2) = Ur (Tg), while Lemmal) 7 indicates that Ty totally dominates TT, which is contradiction. Hence, there does not exist any policy that weakly totally dominates TT, meaning that condition b) is certain. We note that Ur: (TI) Ur: (TI) Up (T2) and the weak total domination between Ti, T2 imply that T T2 E arg max Ú,(T),, rx, E arg min U, (T2). rER and thus min (T2) & max min U,(T) = c. Again, Ii Minimar Regret (R) makes rER Regret( r S Regret(M1; S Regret(n, r not only hold for æ = T2 but also for any other policy x Nl-wdd then for any policy æ E II-ptd Urz(T1) > U.g (T)
PER
rell-wed reR
min (.(T)). rER
Hence, U2ş(TT) 2 max min (²(T) = c. Since Upg (m)) = min (x27 as aforemen- PER
xell-wid PER
tioned, min (T2) rell-wed refl rell
max min (T) will cause contradiction. Hence, min U²(T2) rER max min U,(T') = c, and max min (T) = c. As a result, UR(T) = (T²(T) x' r'lll- rell max Uw(T'). In other words,
Tell PER UT( c- Regret(T, > max c- Regret(m', ;+)) x'ell .wed wel
if a weakly totally dominated policy и is solution to Minimax Regrel R)), then its (m( is no less than that of any non weakly totally dominated policy. This also complete the proof at the end of
20

---

1), because if a none weakly totally dominated policy Ti is solution to arg max (T) but not so- lution to Minimar Regretl R), then ( must be the solution to M EnimarRRerred( Then, (T2) = c- Regret(T1, r = (TL).
=ell
(T; r > 0 and weakly totally dominated policy лa
which, however, contradicts E arg max (T).
=ell
It is obvious that weakly totally dominated policy E Minimaa Regiaaaaaeeeeeea Rt has (x) no less than any other weakly totally dominated policy. Because for any other weakly totally dominated policy TT. (T1) S c and Regret(M1, vx s Regret(n, r7),, hence U (TT) - Uw(() according to Eq 13
So far we have shown that if a weakly totally dominated policy π is a solution to MenimozRegree( it is also solution to arg max (x()). Next, we need to show that the r'ell
reverse is also true, i.e., if a weakly totally dominated policy π is a solution to arg max (T), =Ell must also be a solution to MinimarRegree [t). In order to prove its truthfulness, we need to show that if π & Minimax MinimarRegred
whether there exists: a) none weakly totally dominated policy Ti, or b) another weakly totally dominated policy æ1, such that T E . Minimax mnimarRegrell R)) and (T(T)) S Uw) If neither of the two policies exists, we can complete our proof. Since it has been proved in 1) that if a none weakly totally dominated policy achieves M MinimozRReree [)), it also achieves arg max (='), ). the policy described in condition a) does not exist. Hence, it is only r'lll necessary to prove that the policy in condition b) also does not exist.
If such weakly totally dominated policy exists, a & MinimarRegretiA R) and Ii E finimazRegrell (t) indicates Regrel(F, r+)) > Regret(zr, Since (T() 2 Un(T),
according to Eq 13 UL(TI) = c ---egre((Ti;F") S Un(T) = -1,())-1,(()))) UR(T)
Regret(z, r.) Thus (n(()) (1))-
(TT) UR ct Regret(m,rt) Regret(mI, > c, which is impossible due to UR S c. Therefore, such b also does not exist. In fact, this can be reasoned from another perspective. If there exists a weakly totally dominated policy कi with Ure, (TI)= = = Un: (T) but Tı & MinimarRegret(C It also indicates max Ur: (x') > max Ur:, (I'). Mcanwhile, Regret(M1, r2) = max Ur (T)) z'ell
c-
then Regret(TTrT) > Regre((mI,*t)
r'Ell
r'Ell
Urz(ma) S Regret(T1, r+))===== = max Ur:, (x')- (0))-Ur, Ur: (T1) == max max (+')) - ((, indicates
x x'ell
rER x'ell
max Ung (TT) max Ure, (x') S Ur (m1) - Urz, (TT). However, we have proved that, for a weakly N'ell
E'Ell
totally dominated policy, गι E Minimax RegreazRegred( (R) indicates Ur: (m)) max U. (TI). Hence,
rEk
Ur,,(x') & (T1) Ur: (TI) S and it contradicts max Uy; (T') > max Uy (x') max x r'ell
x'ell max Ur:, (m'). Therefore, such T does not exist. In summary, we have exhausted all conditions and e'ell
r'Ell
can assert that for any policies, being a solution to M SiiiaarRegrett R) is equivalent to solution to arg max U., (T). We complete our proof.
well
A.5 Collective Validation of Similarity Between Expert and Agent
In Definition 2 and our definition of Regrel in Eq we use the utility function U, to measure the performance of a policy. We now show that we can replace U, with other functions.
Lemma 11. The solution of MerinnanReegre((Rp. Minimax Regret is replace with (x) - ((rr where J can be arbitrary function of
does not change when U, in
Proof. When using (T) -((r instead of (T) to measure the policy performance, solving M Minimox Regret R)) is to solve Eq. 14 which is the same as Eq Misnimor Regrrel)))
arg max min Regrel(T,.)
Rell ER
= arg max min max (U²(T') - -f(r)) - (U,(T) - f(r))
well HR w'ell
arg max min max U.(mT'- - (T(T)
(14)
well rell r'ell
21

---

Lemma 11 implies that we can use the policy-expert margin (T) (,(E) as a measurement of policy performance. This makes the rationale of using PAGAR-based IL for collective validation of similarity between and ı more intuitive.
A.6 Criterion for Successful Policy Learning
To analyze the sufficient conditions for Minimas Regrel to mitigate task-reward misalignment, we start by analyzing the general properties of innmaa Regret
arbitrary input R.
Proposition 4. If the following conditions (1) (2) hold for R. then the optimal protagonist policy
== Minimax MinimorReegeel (egrel(R) satisfies Vr+ E RR U (TP) ) 2 Uy+.
(1) There exists r+ E R. and max {max U, (T) Up1 } < min {U, - -U,- J: rtE
rte
(2) There exists policy x²' such that Vr+ E R. (T') 2 Upt, and Vr E R. Regret(T", ,r) minn {U, - }.
rtER
Proof. Suppose the conditions are met, and a policy лı satisfies the property described in condi- tions 2). Then for any policy क2 E MinimazzRegree Rt), if m2 does not satisfy the mentioned
property, there exists a task-aligned reward function r+ E R such that Up (I2) I - In this case Regret(TT, r+) = max U,+ (अ) U,+ (T2) 2 U,+ Up 2 min Up - Up+e.
rt'R
However, for Tı, it holds for any task-aligned reward function A+ E that Regret(n1, ft) S max Upt (T) U++ S max {max - U. < min mell
{U,-- S Regret(T2, r+),
rt' well
and it also holds for any misaligned reward function r E R that Regret(zi, < min , Regrel((z, contradicting m2
rll
U.tr} S Regret(T22 f+). Hence, Regret(TI,rt) Minimax Renrill It). We complete the proof.
In Proposition 4 condition (1) states that the task-aligned reward functions in R all have low extent of misalignment while condition (2) states that there exists a TT that not only performs well under all rt's (thus being acceptable in the task) but also achieves relatively low regret under all r'.. Note that the more aligned the y rs's the more forgiving the tolerance for high regret on r- Furthermore, Proposition 3 shows that, under a stronger condition on the existence of policy x* performing well under all reward functions in R, M MinimarRege i... satisfying the condition (2) in Definition 3
R) can guarantee to induce an acceptable policy,
Proposition 5 (Strong Acceptance). Assume that condition (1) in Proposition 4 is satisfied. In addition, if there exists a policy x² such that Vr R, Regrel(z*"r) max {max U,+ (T) Upt
rteR HEll
then the optimal protagonist policy Tp == Minimaz satisfies Vrr E R. U (TP) U,+.
Proof. Since mar max U,(T) U,(TP) S mar max (,( - U,(T*) < max fmax U, (T) EER
reft
y-e sEll
U,+), ), we can conclude that for any + E R. U, (TP) 2 U The proof is complete.
Note that the assumptions in Proposition and are not trivially satisfiable for arbitrary R, e.g., if R contains two reward functions with opposite signs, i.e., r, -r E R, no policy can perform well under both r and -r. However, in PAGAR-based IL. using RE in place of arbitrary is equivalent to using E and 5 to constrain the selection of reward functions, which can lead to additional implications.
Theorem 2. (Weak Acceptance) If the following conditions (1) (2) hold for Rg then the optimal protagonist policy xp == M Rinimaz Rg,5) satisfics tr E Rg.5. Up (TP) 2 U,
(1) The condition (1) in Proposition 4
holds
(2) Vrt E Rg L,+ WE < max U,+ (T) - U.+ and Vr E Rg.6. L,- We - 8 < min rteRg,
Mell
(U+t U, }.
22

---

Proof. We consider (+) = (Bra|r(TT) Since We min W1 WI(T, E) = k sup -U,)) n
Irisk
for any K 0, let n² be the policy that achieves the minimality in WF. Then for any r+ E R. the
term L,, WE 8 2 Lep E,t Irl,shhtt
sup
U,(B) - U,(x) - 8 U,+ (B) - U,+ (T) (U, (B)
max Up+ (T)) = max Un+ (+') - Up+ (T). Hence, for all r+ E R, max U, (T') U,+ (T) < r'Ell
r'lll
r'Ell
min U, - U indicates max Upi (T') - U, ,,,,,iie,,, i.e. Up (T*) 2 U2l. Likewise, L, .EE r rtERR r'Ell
that for all r- R, max U, (T²) (T) < min U, U Then, we have recovered the ERg.
s'ell
condition (2) in Proposition As result, we deliver the same guarantees in Proposition 4
Theorem 2 delivers the same guarantee as that of Proposition but differs from Proposition 4 in that Condition (2) implicitly requires that for the policy x² arg min W1 W1(T, E), the performance nlll
difference between E and x²' is small enough under all RE
Theorem 3. (Strong Acceptance) Assume that the condition (1) in Theorem holds for RE If for any RE,8, L, WE 8 min
{max U+-(T) U,+ }. then the optimal protagonist policy
rtERE. =Ell
TP = Regret RE,5) satisfies Vr+ E Rg Up+ (xp) 2 U,+.
Proof. Again, we let x* be the policy that achieves the minimality in WE. E. Then, we have Lyp WE ō 2 Lr +. sup ((E))-U((T))) (U,+ (E) max Up+ (m') 2 max Upt (T') Upt (x for "Illsse
x'ell
x'ell
any Rg.5. We have recovered the condition in Proposition 5
The proof is complete.
A.7 Stationary Solutions
In this section, we show that MinimarRegre is convex for xp. Proposition 6. max Regret(TP,) r) is convex in np. ref
Proof. For any [0, 1] and TT,, R2,, there exists T23 amr,, (1-a) r1, re, TA,1 and
T2, A,2 be the optimal reward and antagonist policy for TP, and Tp2 Then a (max max (T(
rER aIEll
(TTT.) + (1 a)- (max max (,(TA)= (T22)) = aUU,, (TA,I) - U, (Tp,1)) + PE JyEl
(U,,, (TA22) Ur (TP2)) 2 (TA,)(TA,3) (TP,)) (1- (1- (TA,3) - Urs (TP:2) =
(TA3)))) Ur (TT.)) Therefore, max max (TA) (T(T)) is convex in ny. rElt EIEll
A.8 Compare PAGAR-Based IL with IRL-Based II
Assumption 2. max (r) can reach Nash Equilibrium at an optimal reward function y* and its optimal policy ,...
We make this assumption only demonstrate how PAGAR-based II. prevent performance degradation w.r.t IRI.-baedd II., which is preferred when IRIe-based II. does not have a reward misalignment issue under ideal conditions. We draw two assertions from this assumption. The first one considers Maximum Margin IRL-based IL and shows that if using the optimal reward function set R as input to Rinemaz BAGAR-based IL and Maximum Margin IRL-based IL have the same solutions.
Proposition 7. T.. = Rinimiox
Proof. The reward function set Rg and the policy set Ia achieving Nash Equilibrium for arg min JIRL(r) indicates that for any r E Rg, π Hacc, π E arg max (,( - U, (E). Then TER
xET
II Tace will be the solution to arg max min xpell re reRg
max U²(TA) - (F) -U, MAEl
(U.(Fr) U.(F))
23

---

because the policies in Nace achieve zero regret. Then Lemma 11 states that Nace will also be the solution to arg max min Tpell re R
max Ur (T(TTA) (TTp)). We finish the proof. TAEII
The proof can be found in Appendix A.6 The second assertion shows that if IRL-based IL can learn a policy to succeed in the task, Minimar Miniinaz/eegre((PP..
with < 8 can also learn a policy that succeeds in the task under certain condition. The proof can be found in Appendix [..6 This assertion also suggests that the designer should select a 5 smaller than 8* while making 8* - no greater than the expected size of the high-order policy utility interval.
Proposition 8. If r" is a task-aligned reward function and ó 2 8 - - Uy. - the
well
optimal protagonist policy ap = Minimiax Regrett Rg,6) )) is guaranteed to be acceptable for the task.
Proof. If π.- E Minimiaa (eeiimiianegre((RRaeee then π,, can succeed in the task by definition. Now assume that TP # ny.. Since JIRI. achieves Nash Equilibrium r" and TT-, for any other reward function we have max (T( Ur(T.-) S 8" (U,(E) max (+)) S 8 6. We also sEll have max Regrel(r' Tp) z max Regrel(r', .T-) 8* - 8. Furthermore, Regre(r", TP) S PERE,
r'ERE,
max Regret(r', xp)) Hence, Regre(r", TT)) S 6 8* S max Up+(TT) - Upt. In other words, PERR,
well
(TP) E [U,-, max U-- (T)], indicating Tp can succeed in the task. The proof is complete. well
L
A.9
Example 1
S3
S5
S4
3.50
0.00
0.6
$6
Prob(ag
2.90
3 0.00
Q1
0.2
1.90
0.2
22
50
3.30
E: (So, au Sn, S4, Sz. $6),) (S a2 52. S2, Sg. Sg)
2.00
MaxEnt IRL Loss
0.50 1.20
1400
2.00
2.40
2.90
Figure 5: Left: Consider MDP where there two available actions a1 02 initial state so- In other states, actions make no difference: the transition probabilities are either annotated at the transition edges equal by default. States 83 and sq are terminal states. Expert demonstrations are in E. Middle: x-axis indicates the MaxEnt IRL loss bound for RÆ as defined in Section AA.3 The y-axis indicates the probability of the protagonist policy learned via Minimax: Regret Rg.5) choosing 42 at 8g- The red curve shows how different 8's lead to different protagonist policies. The blue dashed curve is for reference, showing the optimal policy under the optimal reward learned via MaxEnt IRL. Right: The curve shows how the MaxEnt IRL. Loss changes with w.
Example 1. Figure 5 Left shows an illustrative example of how PAGAR-based IL mitigates reward misalignment in IRL-based IL. The task requires that a policy must visit 82 and 8g with probabilities no less than 0.5 within steps, i.c. Prob(s2 т) 2 0.5 A Prob(s6 π)) 2 0.5 where Prob(s т is the probability of æ generating a trajectory that contains within the first 5 steps. It can be derived
analytically that a successful policy must choose at so with a probability within [ 188 derivation is as follows.
1251 The
that reach (So, a2 S2, 86). (So, 02 82, 82, 56), Prob(s6 80 02) =
The trajectories
86
after
choosing a2 at 8o include: The total probability equals Then the total probability of reaching 86
(So:
02 $2, 82, S2, 56).
13
31
5
125
equals Prob(ss T) = -1 ( Su))
+
31
Ta2 Su). For Prob(s6 T)) to be no less than 0.5.
125
[aag so) must be no greater than
125
185

---

The reward function hypothesis space is frw rw(s, w. =
row ri(s + w) - rr((s,a} where [0, is parameter, r1, 72 are two features. Specifically, ri (s, a) equals 1 if s 82 and equals O otherwise, and (2 a) equals if 8 = 86 and equals 0 otherwise. Given the demonstrations and the MDP, the maximum negative MaxFnt IRI. loss 8 2.8 corresponds to the optimal parameter w- = 1. This is computed based on Eq.6 in Ziebart et al. [2008). The discount factor is 7 0.99. When computing the normalization term 2 in Eq.4 of Ziebart et al. (2008), we only consider the trajectories within 5 steps. The optimal policy under r. chooses d2 at so with probability 1 and reaches S6 with probability less than 0.25, thus failing to accomplish the task. The optimal protagonist policy TP Minimar RenimazRegrett Rg,5) ()
succeed in the task as indicated by the grey dashed lines in Figure 5 Middle. It chooses 02 at 8y with probability when is close to its maximum 6". However, p $2) decreases d decreases. It turns out that for any 8 < 1.1 the optimal protagonist policy can succeed the task. In Figure 5 Right, we further show how the MaxEnt IRI. loss changes with w.
B Approach to Solving MinimaxRegret
In this section, show how we derive the off-policy RL objective function for Tp. Also, develop a series of theories that lead to two bounds of the Protagonist Antagonist Induced Regret. By using those bounds, we formulate objective functions for solving Imitation Learning problems with PAGAR.
B.1 Off-Policy Objective Function for Protagonist Policy Training
For reader's convenience, we put the Theorem in Schulman al. [015 here.
Theorem 7 (Schulman et al. (2015). D. Lel or max Drv (Ralds Tnew), and let e max En-rnew Amar (8, a)}, then 15 holds.
Up(Tnww) S (Toid) +
Σ
Paota Σ aEA
Tnew (s, + 2ey
a²
(15)
(1 (1-79 7)-
sEB
s|Fola) is the discounted visitation frequency of oold Anata where (s) (s)= y' Prob(((s! P the fudanntagefuncctoon without considering the entropy.
Algorithm in SShulman al. 2015 learns Nnew by maximizing the r.h.s of the inequality Eq 15
which only involves the trajectories and the advantage function of wold- By moving U, from r.h.s of Eq 15 to the left, and replacing Tnew with Tp and Kold with T.
obtain bound for
(TP)) (T(A) as mentioned in Section 6.1 The PPO in Schulman et al. [2017) further simplifies
the r.h.s of the inequality Eq 15 with a clipped importance sampling rate. We derived Ja (TP) by
using the same trick.
B.2 Protagonist Antagonist Induced Regret Bounds
Our theories inspired by the on-policy policy improvement methods in Schulman et al. 20155] The theories in Schulman et al. 2013) are under the setting where entropy regularizer is not considered. In our implementation, we always consider entropy regularized RL of which the objective is to lear policy that maximizes [ r) (+( + (().. Also, since we use GAN-based IRL algorithms, the learned reward function as proved by Fu et al. (0018 is distribution. Moreover, it is also proved in Fu et al. (0188) that a policy π being optimal under r indicates that log ग = r r Aq. We omit the proof and let the reader refer to Fu al. (2018) for details. Although all our theories are about the relationship between the Protagonist Antagonist Induced Regret and the soft advantage function A,, the equivalence between A,, and allows to use the theories formulate our reward optimization objective functions. To start off, we denote the reward function to be optimized as r. Given the intermediate learned reward function r, we study the Protagonist Antagonist Induced Regret between two policies Ti and π2-
Lemma 12. Given a reward function r and a pair of policies Ti and n2.
80
U.(T)) - UT(2
Σ y' A' (o(0)) (a(1))))))
+
Σ
h (T2 (12()1)11))
(16)
25

---

Proof. This proof follows the proof of Lemma in Schulman et al. 2013 where
RL is not entropy-regularizee For huropy-regularized RL, since An (3, a
(+1,01) (r(s,a()) + -
A (s(0) cl10)
7m071
(r(sltt2), al(tt)) ( + VVV (s(t+1)) ( - Vio (6(00)
=
7w011
7'r(s (r(s)))a!)) - (2(00)
7w
1
7 7r((s _al0)
Vr (s(01)
1=0
= TWI
Σ 7'r(s(0, )(())
E TriTT
>'r(((0) ,a(0)) )+ + 7 (T2( (12(11((0))
1=0
1=
= U,(TI) - UU( - E
Σ 1=
7'H (T2(11((1))
TMR2
= U,(TI)- - (x22 - -(222)
Remark 1. Lemma
confirms that E
E As(1) (l(s())))))))) U, (T)) H(T))
We follow Schulman et al. 2015 and denote AA(s)
E (s,
E [A (8, a)) as
a->)((-1s)
the difference between the expected advantages of following क2 after choosing an action respectively by following policy тi and T2 at any state s. Although the setting of Scchulma et al. (2015) differs from ours by having the expected advantage E
[Am (s) equal to due the absence of
a~>r((11s)
entropy regularization, the following definition and lemmas from Schulman et al. 2015 remain valid our setting.
2015). the protagonist policy m and the antagonist policy π2) are a-coupled if they defines a joint distribution over (a, ā) A x A, such that Prob(a # āls) S for all s.
Definition Schulman et al.
Lemma 13. Schulman et al. (2155) Given that the protagonist policy nı and the antagonist policy T2 a-coupled, then for all state
|AA() S 20max Arg (s, a))
(17)
Lemma 14. Schulman et al. 2015 22 are a-coupled, then
Given that the protagonist policy कi and the antagonist policy
E mlm
AA(s(1)]
E a(t)
[AAS
< 4a(1 - -(1 (11)l1((a))))a (s,
(18)
Lemma 15. Given that the protagonist policy iı and the antagonist policy na are a-coupled, then
I
Arz (1))
< 2(1 - (1 - )axx A (8)
(19)
(M) (())
18) (1)
(5.0)
Proof. The proof is similar to that of Lemma 14 in Schulman et al. (0055). Let n. be the number of times that a (t') mi does not equal (t') m2 for t' < t, i.c. the number of times that क and n
26

---

disagree before timestep t. Then for s(1) we have the following.
(s(1) [(o)]]
E
E (a(t)~iT
Ax
P(n
0)
E
E Lalt)riT
Ar (s!1) [[00]] as)),
P(r 0)
E La(1)xx2
A (5(t) (0(0))))
(() nly
2(19 m,30
The expectation decomposes similarly for s(t) ~ x2.
E
(s(0, .a("))
(00 (022 NTT2
P(n+ 0)
E
(er((()),))) + P(n > 0)
E
Ars (s())
(1)) (t)
((1)
a!!) m,=0
(!) ~
na
When computing
(s(0) a(0)]]
E
A (o(1) (,()),(())) the terms with ny
(00 (00
O cancel each other because n4 0 indicates that wi and 22 agreed all timesteps less than That leads to the following.
E a(t)rot
E (aft)ros2
Aro (sit)
E
Ang (s(0) (s(() (s(()))(())
(0))
200
P(n
E (a(l)roT
Ar (s(t) a(1))]
P(n, >> 0)
E
(ms()(0)
(1)) 0:20
(1) (072
(
By definition of a, the probability of Ti and 72 agreeing at timestep t' is no less than 1 - Hence, P(n, > 0) & - (1 - )²) Hence, we have the following bound.
(s(t)) [a,,)
(s (((0_)0)))
a(t) alttiwat
(t) Not (ot
Ars (s(t),
=
P(n > 0)
P(n, > 0)
(1)
all)
n250
a(t)
n,20
(s10)) (..600,0000))
P(n, 0)
+
E
(9) (1)
att) m
(0)
2,
n,30
< 2(1 - -ma')max(e A (s)
(20)
(s.a)
The preceding lemmas lead to the proof for Theorem in the main text.
Theorem 4 Suppose that Ta is the optimal policy in terms of entropy regularized RL under Let a max Drv (m (-Is), m2(+)))) max | (s, (,)))), and AA(s) = E [Ar, (s, a)]
27

---

E (s, a)]. For any policy T1, the following bounds hold.
U. (T() - (U2(
E 8(1)
AA(((()))
5
20rye
(21)
(1 1 7)²
(T() (T2)
E 8(t)
AA(S(1)]
2a7 + 1)c
(22)
(1-7)
Proof. We first leverage Lemma
12 to derive Eq 23
Note that since T2 is optimal under r, Remark |
confirmed that H(TT2) Ero o
[Ara (s(t) a(.))]
a(t)
(11) - (T(
(m)(m))
Us (T(2 - -H((T2)
+ +((22)
E
7 Am, (s(0 .0(1))
+ H((m2)
20
Am, (s(0) (a(0))
E
Ara (s(t) [a(0))
Tw/l
a(t)
[alt)ai2
Arz (s(1) _a(0))
Anz (s (()
E
E
E
a(t)
[a(t)outT
alt)
1=U
E
E Lall)wa
Ara (5
E stu)uoz
E [alt)rusg
An (s(1) 2000)
E
AA(s(1))
git)
Arz (s(1)) (6(6))
[ha[600,0(0]]] (23)
E
E
E 5(0)m0
E [alt)
(a(l)
We switch terms between Eq 23 and UT(TT)
Ur (T22, then use Lemma 15 to derive Eq 24
U,(T:)
UT(TT)
Σ
x
E
AA(S!O]]
( (1))
8
ArT (s(t) .a(1))]]
(s
E
((1)
(o(1)
2(0)
aatt)rssg
20rymax Arr (s,
x 2max (s.a)
Ama (8, (1 - - a)))
(s.4a
(24)
(1 71
1=
28

---

Alternatvell, we can expand U, - (T) into Eq 25 During the process, H(22) is converted
into - -Eis ?f
E
E
(s alo0)]
U (x2)
= (U,(TT) Un(T2) - -H(Tz)) + +H(2T)
2
= E
7 ArA (s(1)a(11)
+ +H(TT2)
rMT1
Ans( (a())] + H(()2)
E
E
aallints s(()AT)
t=0
E
AA(s(t))
+ E
Ars(s((), _ + +7((z2)
((()
t=0
Ang (000))
(s((0) - AA(c(())
E altlaazz
g(()~82
t=
E
AA(s(1))
+ E
(s(0,
E
Ars (s(0) (000)))))))
a(t)
2
(s(1 _aa0)]
(s!
E
2
E
E
+
glthaut
(a())
a(t) (a(t)
As2(s(1) a [00)
AA(s())]
AA(SW)))
E
E (a(t)
E a(1)
E g(0)
(25)
We switch terms between Eq 25 inequality in Eq 26
and Ur (TT) (T22), then base on Lemma 14 and 15 to derive the
AAArs((),
U. U+(T)) - -T((T2)
E a(t)ount
(Ti) - -Ur(T2)
E salt)mat
E
(s((
: .cco)]]
E
E [a(t)-oo
Ars (s(t))
a()) [
ait)
1=0
00
E
E alt)
a(t)
(s _d(1)]]
E
E
E (alt) all2uu
1m0
00
E
AA(s(())]
E a())
(A(((()))
a(t)
Arg (0(0) (o(0) (0000,
(ss((s()) , .a(0)]]
E
E
E Lalt)auzz
((0)
1=1
71
(1 (1 - (4mmax) (8. a)l + 2max Ary (8, an))
(5.a)
1=0
207 (2a + 1)max (s,
8.0
(26)
(1 71
29

---

It is stated in Schulman et al. 2015 that max : xT(-|s) S a. Hence, by letting
== max Drv T2 ( (-()) [ Eq 23 and still hold. Then, we have proved Theorem
B.3
Objective Functions of Reward Optimization
To derive JR.1 and JR22, we let ı тт and T22 TA. Then based on Eq following upper-bounds of (Tr) - (T().).
21 and 22
we derive the
(((T))-U((
x
E BlORK
AA(s(0))]
+
2ar(2a + 1)e
(27)
(1 7)
(,(T)) (TAT
IM
AA(s(1))
207e
(28)
(1 7)2
By our assumption that TA is optimal under r, we have A = enables us to replace An, 's in AA with As for the 2ay(2a+1)e and (1 v12 terms, since the objective
Fu et al. (018)) This equivalence
20rre
(1-7)²
to maximize (T (Ta) (TB). we heuristically estimate the in Eq 27 by using the samples from
TP and the in Eq 28 by using the samples from TA. As a result we have the objective functions
(a(a (d(9) and &
(((1))))))) are the importance
defined as Eq 29 and 30 where (i a) = n (at9/a(0))
(
sampling probability ratio derived from the definition of A::; C1 x
2a and C2 X (I-v]
76 where
[I-?]
is either an estimated maximal KL-divergence between тA and mg since DEL 2 DIv according
to Schulman et al. (2015). or estimated maximal Drv depending on whether the reward function is Gaussian or Categorical. We also note that for finite horizon tasks, we compute the average rewards
instead of the discounted accumulated rewards in Bg and 29
(P) ==
TMTT E 7f (s1(1() d(1)) - 1) ((s(t) [( + C1 max
|n(s, (29) (29)
(s,a)~AA
JR.2(T;TP,TA) =
E 7 (1 (2((()) ((()))))(())) r(s(t), d(t)]] + +C2 max
|n(s, (30)
(a.a)~T
Beside JR,1, Jn. we additionally use two more objective functions based on the derived bounds. W (r: TA, Tp) By denoting the optimal policy under r as n². a max Drv (I* (-1s), AA(-s))
.E8
= max Ar (s, ())) and (s) (s
E (s,a) E Amm as) we have the fol- anna
and
lowing.
U,(TP) - U,(=")
U,(TP) - (,(TA) + U,(AA) - U²(TT'
AA)((((0)]
ya*ge"
U,(TP) - (,(TA) +
E
alti
(1 7)-
1=0
[r(o(O),aco)]]]
U,(TP)
E
E LallasTa
((U)
1=
80
E [oll)NT
Al. (5(0),
-
E
AAx (o(1) a(.)))
+
20"ac"
a00
(1 - 7
t=0
(s(()) a(+)]]
2a y0x"
U,(TP)
E
E (all)rt
althNNT
(1 7-2
1=
E
Σ
()) a(a)))
exp(r(s(()) (
[a(alt)/((0) r(s ,a(t))
2a"
(31)
7omp
TATA
(1 7)2
Let Eg exp(r(s!")) (
be the importance sampling probability ratio. is suggested in Schulman al. 2017 that instead of directly optimizing the objective function Eq 31 optimizing surrogate
(at st
30

---

objective function as in Eq 32 which is an upper-boune of Eq 31 with some small E (0, 1) can be much less expensive and still effective.
JR3(TTTPTTA))
E TrUp
E 7'r(s(t),(1))
E
Ei= 7' min (f3 r(s (( cliplE,, -,,11 rr(st) (a)))
(32)
TOUT
E Alternatively, we let (Ap(s)=
E (s, The according Eq
amop
ams
have the following.
(T(T)) -U(
AApps((1)] + 2x 2a*"(2a" 1)c²
E
8(0) 3(0)xop
(1 (1-
Ar (s (1()) aa)00 + 2a 2a²>(2a" + +1)"
Ag. (s(1), (((()))(())))
E La(D>~p 3(0
E
E a(l)
(1-
t=0
(33)
Then objective function JR, is formulated in Eq 34 where exp(r(e (0
Ap(att)
JAA(F
E Trup
E 7'r(s(0, _
E
Ei=o 7 min (5a . r(((() a () cliplEa, 8,1 + 8) r(s())a(()))
(34)
B.4 Incorporating IRL Algorithms
Online RL Setting, In our implementation, we combine PAGAR with GAIL Ho and Ermon 2016 VAIL Peng et al. 2019]. and f-IRL Ni al. 202T]. respectively. In this section, we use JIRl to indicate the IRL loss to be minimized in place of the notation JIRL in the main text. Accordingly, the target IRI. loss.
f-IRL. We use the FKL of f-IRL in our experiments. Since FKL is explicitly models reward function with a neural network, when PAGAR is combined with FKI,, the meta-algorithm Algorithm
I can be directly implemented without changes except for letting JIna be the FKL loss.
GAIL and VAIL. We take additional steps to incorporate GAN-based algorithms, as they do not explicitly learn reward functions but instead train discriminators. When PAGAR is combined with GAIL, the meta-algorithm Algorithm becomes Algorithm [ When PAGAR is combined with VAIL, it becomes Algorithm 3 Both of the two algorithms are GAN-based IRL, indicating that both algorithms use Eq aa the IRL objective function. In these two cases, we use neural network to approximate D, the discriminator in Eq [ To get the
reward function r, we follow Fu et al. [2018) and denote r(s, a) = log (al()))
D(s.a) - (als))
mentioned in Section [] Hence, the only difference between Algorithm 2 and Algorithm T is in the representation of the reward function. Regarding VAIL, since it additionally Iearns representation for the state-action pairs, bottleneck constraint (D) < i added where the bottleneck JIC is estimated from policy roll-outs. VAIL introduces a Lagrangian parameter & to integrate (D) le in the objective function. As result its objective function becomes (I) + B (Jic
iz). VAIL not only learns the policy and the discriminator but also optimizes In our case, we utilize the samples from both protagonist and antagonist policies to optimize B as in line 10 following Peng et al. [2019).
In our implementation, depending on the difficulty the benchmarks, we choose to maintain as a constant or update À with the IRL loss (r) in most of the continuous control tasks. In HalfCheetah-v2 and the maze navigation tasks, we update by introducing a hyperparameter u. As described in the maintext, treat d as the target IRL loss of (r),()), i.c., JIRL (r) s 6.
31

---

Algorithm 2 GAIL w/ PAGAR
Input: Expert demonstration E, discriminator loss bound ò, initial protagonist policy TP, antagonist
policy тA, discriminator D (representing r(s, a) = log
Ia(als)
=A(als)) Lagrangian parameter
Dis.a]
X,
iteration number 0, maximum iteration number N
Output:
1: while iteration number i N do
2: Sample trajectory sets D ~ J and Dp ~ up with a 4: Optimize AA to maximize (RI,(TA;T).
3: Estimate
5:
Estimate (T with Dp; Jua (TT; (A;) with Dp and DA;
6:
Optimize T to maximize JRL (TT((P;T) JR (T:
7:
Estimate JPA (P, with Dp and DA
8:
Estimate JI (TAiT) (IRL(TAiT)
with DA and E' by following the IRL algorithm minimize J JPAGAN(T; TA) mar(JIRL(7 + 6,0)
9: Optimize
10: end while
return xp
Algorithm 3 VAIL w/ PAGAR
Input: Expert demonstration E, discriminator loss bound 6, initial protagonist policy тp, antagonist policy л A. discriminator D (representing r(s, a) log
Talals)
D(s.a) (als)))) Lagrangian parameter
À for PAGAR, iteration number i = 0, maximum iteration number N, Lagrangian parameter 3 for bottleneck constraint, bounds on the bottleneck penalty i,, learning rate μr.
Output:
1: while iteration number i N do
2: Sample trajectory sets D ~ aA and Dp up
3: Estimate JRz ( with a
4: Optimize aA to maximize JRI((TAF))
5:
Estimate JRI. with Dp; JaN (TP; n A, with Dp and DA;
6:
Optimize тт to maximize (TT; + JR (T))
7:
Estimate JPAGAR(T;; TT, TA) with Dp and DA
8:
Estimate (TAiT) with DA and E' by following the IRL algorithm
9:
Estimate (D) with A, A Dp and E
10:
Optimize D to minimize JPAGAN(T: TP, TA) + A mar(JInl(() - 3-
Update 3 = max 0. 3- - p (Jr - iz)
11:
end while return XP
13:
In all the maze navigation tasks, we initialize with some constant Xo and update À by X exp(μ- (F) after every iteration. In HalfCheetah-v2,, we update À by À == maz Ap. X- exp(a- (F) - 8))) to avoid being too small. Besides, we use PPO Schulman et al. (017]] to train all policies in Algorithm 2 and
2024 The original RECOIL algorithm does not learn reward function but learns Q, V value functions and a policy with neural networks. In order combine PAGAR with RECOIL, we made the following modification RECOIL:
Offline RL Setting. We incorporate PAGAR with RECOIL Sikehi al.
Instead of learning the Q value function, we explicitly learn reward function r SxAXS - IR, which takes the current state 8, action a and the next state s' as input, and outputs a real number as the reward
We use the same loss function as that for optimizing Q in RECOIL to optimize by replacing Q(s, a) with a(s,a, a') VV for every (s, a, s') sampled from an offline dataset D. We denote this loss function for as JTRE (r) as in [3]5 set of some sub-optimal behavioral policy, df F.S indicates mixture between S and E sets,
is an offline trajectory sample
32

---

Algorithm 4 RECOIL w/ PAGAR
Input:
Expert demonstration E, behavioral policy sample set S, reward function value function V, initial protagonist policy иy, antagonist policy TA, maximum iteration number N.
Output: TP
while iteration number N do
2: Train using min (IRL(T) + 0.001 -
3: Train V using min (V)
V
4: Train A using max (TA)
5: Train op using max (Tp)
6: end while
7:
return TP
and [ is the mixing ratio as defined in Sikchi et al. [2024).
JIRI = 3 (F(eea))))slrsss..a (s')) "ernad"-err((ss)) + +V(())) +
0.25 E (s.a, Mi (r(8, a, 8 )'))² (35)
We use the same loss function for optimizing the value function V as in RECOIL to still optimize V, except for replacing the target Q(s,a) with target r(s, a, 8') + (s')) for every (s, a, s') experience sampled from the offline dataset. Consequently, we have the loss function for as defined in 34/6//here is a conservatism parameter defined in Sikchi et al. 2024].
Jv(V)= = ( exp ((r(s, + +V((s' ()+ - oo)) +
(r(s,a,)) + -V()))))))))
(36)
Instead of learning single policy as in RECOIL, we learn a protagonist and antagonist policies тт and TA by using the same SAC-like policy update rule as in RECOIL, except for replacing ((s, a) with r(s, s') + >V (s') for every (s, a, s') experience sampled from the offline dataset. The objective function for learning TA and are defined in Eq 37
(T) == E (s.a,)")aa exp ((r(s, s') vV()') - V(s))) log [[|ss]
(37)
With some heuristic, we construct a PAGAR-loss follows.
B( r(s,a,s') exp(clip(maz((), log (r(als)
JPAGAR(T)
-1, 1)
T
(alss)
((a r(s, a(s,a,s') exp(clip(minn(0, log pa(as)) 1.1) (38)
For simplicity, multiply this PAGAR-loss JPAGAR with fixed Lagrangian parameter
A = le 3 and add the aforementioned loss JIRL, for optimizing r.
We summarize this RECOIL w/ PAGAR algorithm in Algorithm 4
C Experiment Details
This section presents some details of the experiments and additional results.
C.1 Experimental Details
Hardware. All experiments are carried out on a quad-core i7-7700K processor running 3.6 GHz with NVIDIA GeForce GTX 1050 Ti GPU and a 16 GB of memory. Network Architectures. Our algorithm involves a protagonist policy nP, and an antagonist policy n.. our implementation, the two policies have the same structures. Each structure contains two neural networks, an actor network, and critic network. When associated with GAN-based IRI,, we use discriminator 1 represent the reward function mentioned in Appendix B.4
33

---

Protagonist and Antagonist policies. We prepare two versions of actor-critic networks, fully connected network (FCN) version, and CNN version, respectively, for the Mujoco and Mini-Grid benchmarks. The FCN version, the actor and critic networks have 3 layers. Each hidden layer has 100 neurons and a lanh activation function. The output layer output the mean and standard deviation of the actions. In the CNN version, the actor and critic networks share convolutional layers, each having 5, 2, 2 filters, 2 x 2 kernel size, and LU activation function. Then 2 FCNs are used to simulate the actor and critic networks.
The FCNs have one hidden layer, of which the sizes are 64.
Discriminator D for PAGAR-based GAIL in Algorithm 2 We prepare two versions of discriminator networks, an FCN version and CNN version, respectively, for the Mujoco and Mini-Grid benchmarks. The FCN version has linear layers. Each hidden layer has 100 neurons and a lanh activation function. The output layer uses the Sigmoid function to output the confidence. In the CNN version, the actor and critic networks share 3 convolutional layers, each having 5, 2, 2 filters, 2 2 kernel size, and ReLU activation function. The last convolutional layer is concatenated with an FCN with one hidden layer with 64 neurons and tanh activation function. The output layer uses the Sigmoid function as the activation function.
Discriminator D) for PAGAR-based VAIL in Algorithm 3 We prepare two versions of discriminator networks, an FCN version and CNN version, respectively, for the Mujoco and Mini-Grid benchmarks. The FCN version uses linear layers to generate the mean and standard deviation of the embedding of the input. Then a two-layer FCN takes a sampled embedding vector as input and outputs the confidence. The hidden layer this FCN has 100 neurons and tanh activation function. The output layer uses the Sigmoid function to output the confidence. In the CNN version, the actor and critic networks share 3 convolutional layers, each having 5, 2, 2 filters, 2 x 2 kernel size, and ReLU activation function. The last convolutional layer is concatenated with a two-layer FCN. The hidden layer has 64 neurons and uses tanh as the activation function. The output layer uses the Sigmoid function as the activation function.
Hyperparameters The hyperparameters that appear in Algorithm and are summarized in Table 2 where we use N/A to indicate using 8*, in which case we let O. Otherwise, the values of ja and vary depending on the task and IRL algorithm. The parameter X the initial value of X as explained in Appendix B.4
Parameter Policy training batch size Discount factor GAE parameter PPO clipping parameter
Continuous Control Domain
Partially Observable Domain
64
256
0.99
0.99
0.95
0.95
0.2
0.2
Ao
le3
1e3
a
0.2
0.2
ic
0.5
0.5
B
0.0
0.0
μ VAIL(HalfCheetah) 0.5; others: 0.0
VAIL: 1.0; GAIL: 1.0
5 VaILeHalfCheeetah) 1.0; others: N/A
VAIL: 0.8; GAIL: 1.2
Table Hyperparameters used the training processes
Expert Demonstrations. Our expert demonstrations all achieve high rewards in the task. The number of trajectories and the average trajectory total rewards are listed in Table 3
C.2 Additional Results
Continuous Tasks with Non-Binary Outcomes We test PAGAR- based IRL in Mujuco tasks where the task objectives do not have binary outcomes. We append the results in three Mujoco benchmarks: Walker2d-v2, HulfCheeta-v2. Hopper-v2, InvertedPendulumm12 and Swimmer-12 in Figure 6 and 7 Algorithm| I performs similarly to VAIL and GAIL in those two benchmarks. The results show that PAGAR-based II. takes fewer iterations to achieve the same performance as the baselines. In particular, in the HalfCheetah- task, Algorithm| I acchieess the same level of performance compared
34

---

Task
Number of Trajectories
Average Tot. Rewards
Walker2d-v2
10
4133
HalfCheetah-v2
100
1798
Hopper-v2
100
3586
InvertedPendulun-12
10
1000
Swimmer-v2
10
122
DoorKey-6x6-v0
10
0.92
SimpleCrossingS9NN-00
10
0.93
Table 3: The number of demonstrated trajectories and the average trajectory rewards
Average Return Per E9isode
PAGAR-GAIL
MAGAR-VAIL
GAIL
VUL
25000
-1000
0000
2200
-300
-0000
250
300
750
1000
1250
1500
17.00
2000
500
1000
1500
2000
2500
3000
Iterations
Iterations
Figure 6: (Left: Walker2d-v2. Right: HalfCheeta-v2) The y axis indicates the average return per episode.
with GAIL and VAIL by using only half the numbers of iterations. IQ-learn does not perform well in Walker2d-v2 but performs better than ours and other baselines by a large margin.
Average Return Per ESHOCO
Auerage Sesuum Pler Fraannde
$120
PAY
GNL
VALL
Itertionn
Berations
Rerations
(a) Hopper-v2
(b) InvertedPendulum--2
(c) Swimmer-v2
Figure 7: Comparing Algorithm with baselines. The suffix after each 'PAGAR-" indicates which IRL algorithm is utilized in Algorithm 1. The y axis is the average return per step. The axis is the number iterations in GAIL, VAIL, and ours. The policy is executed between each iteration for 2048 timesteps for sample collection. One exception is that IQ-learn updates the policy every timestep, making its actual number of iterations 2048 times larger than indicated in the figures.
C.3 Influence of Reward Hypothesis Space
In addition to the DoorKey-6x6-v0 environment, we also tested PAGAR-GAIL and GAIL in SimpleCroosing59922.. environment. The results are shown in Figure and
35

---

SAlSSrrdd
(ALgtemmad) Get (Catng)
Average Episode
1.4
Average Return Per Episode
IRD4A Dallliigmecd)
0.2
0.6
0.4
J0C000
200030
-000000
A00000 3400000
1000000
200000
200000
RU0000
300000
Frames
Frames
(a) SoanirrddDDorKe--beeee
(b) ShnnGrrd&SimpllCCossnngggW>
Figure 8: Comparing Algorithm D with baselines. The prefix 'protagonist__ol "aaioniitttAAi''' indicates that the IRL algorithm utilized in Algorithm is the same as in GAIL. The 'SSigmoid' and 'CCateg' suffixes indicate whether the output layer of the discriminator is using the Sigmoid function or Categorical distribution. The axis is the number of sampled frames. The axis is the average return per episode.
36

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the task-aligned reward function is hardly predictable in IRL-based setting. As mentioned Section 5 "How to build Re.k", ", it is designer's decision to set value for the parameter k.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
37

---

Answer: [Yes]
Justification: Every lemma, proposition and theorem have the set of assumptions included in them. The proves are included Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We disclosed all the information in Appendix
CT
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
38

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide source code our submission, including the instructions on how to reproduce the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Coaee//ddeeeebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We disclosed all the information in Appendix C.I
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The curves in the plots are smoothed. They reflect how the policies' average returns change as the learning episodes increase.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
39

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We disclosed all the information in Appendix C.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We conform with NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper does not discuss and not intended to cause significant societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
40

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not release data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: This paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
41

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
42


Reviews:
[{"review_id": "ckn7VLkZcn", "reviewer": "Reviewer_yv3o", "summary": "This work investigates the inverse reinforcement learning problem. Previous methods suffer from the weakness: fail to capture the true task objectives from demonstrations. This study proposes deriving a set of candidate reward functions that align with the task rather than merely the data. Using an adversarial learning approach, the policy is trained with this set of reward functions. Then the proposed framework collectively validates the policy’s ability for task accomplishment. The proposed method is theoretically derived with detailed derivation and experimentally implemented on MiniGrid environments with impressive empirical results.", "questions": "In Appendix Figure 5, the proposed approach does not significantly better than the baselines in locomotion tasks (Walker, HalfCheetah, Hopper, Swimmer, etc.). Would the method show more advantages if only 1 demonstration is used instead of 10 or 100?\n\nThe MiniGrid DoorKey 6x6 experiment just requires a demonstration with a short horizon. In larger mazes or more challenging environments with complicated demonstrations, would the proposed approach still perform impressively with only 1 demo?", "limitations": "There is no analysis explaining why the proposed method performs better on MiniGrid (Figure 2) than on locomotion tasks (Appendix Figure 5). Studying the scenarios where the method does not work well and providing explanations would help researchers determine the appropriateness of using this approach.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured, clearly explained, and nicely presented.\n\nThe approach has a solid theoretical foundation.\n\nThe method shows significant advantages over baseline approaches on MiniGrid DoorKey, especially with a single demonstration.", "weaknesses": "The experiments on MiniGrid are not very extensive. Beyond DoorKey, MiniGrid includes more challenging environments such as ObstructedMaze and UnLock. Demonstrating advantages in these environments would be more convincing.\n\nThe proposed approach requires a set of candidate reward functions. While it achieves a higher success rate than baselines, it is unclear if this comes at a higher time cost. Comparing the time efficiency with prior work would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ## MiniGrid includes more challenging environments\n\nAs our approach builds upon existing IRL/IL methods (PAGAR-GAIL, PAGAR-VAIL), its performance is influenced by the integrated IRL/IL algorithms. We found that the more complex MiniGrid environments, such as ObstructedMaze and KeyCorridor, are too challenging for our integrated IRL/IL methods and the baselines to produce meaningful results. Consequently, we did not include these environments. Future work will focus on enhancing our method and the underlying IRL/IL algorithms to address these more challenging tasks.\n \n> ## ... requires a set of candidate reward functions\n\nWe do not explicitly build this set of reward functions. We mention in Section 6.3 that we incorporate the $r\\in R_{E,\\delta}$ constraint into the objective function. Algorithm 1 shows that the policy optimization and reward searching processes are performed alternately, similar to those in GAN-based IRL/IL methods.\n\n> ## Comparing the time efficiency with prior work would be beneficial.\n\nWe appreciate this suggestion and will analyze the time efficiency in the updated version of the paper. The primary potential increase in time complexity arises from the need to optimize two policies, $\\pi_A$ and $\\pi_P$. However, these two policies can explore the environment and be optimized in parallel, mitigating the redundant time complexity. Additionally, the estimation of Equations 6 and 7 introduces some overhead: we evaluate policy $\\pi_A$'s output along $\\pi_P$'s samples and vice versa. This increased time complexity is linear in the size of the replay buffers of the two policies.\n\n> ## would the method show more advantages if only 1 demonstration\n\nWe did attempt to use only one demonstration. However, our approach, like the integrated GAN-based IRL/IL algorithms, suffers when only one demo is provided. Our conjecture is that the $R_{E,\\delta}$ set can be overwhelmed by task-misaligned reward functions that overfit the single trajectory, leading to poor performance. Our approach aims to improve task alignment under general non-ideal situations, but it is not optimized to excel in one-shot IL across baselines.\n\n> ## In larger mazes or more challenging environments with complicated demonstrations, would the proposed approach still perform impressively with only 1 demo?\n\nIn environments like S9N1, IRL/IL baselines struggle with only one demo, while our approach shows better results. However, as tasks become more complex, the partial observability significantly hinders the IRL/IL baselines from producing meaningful results, which can affect the performance of our method, as mentioned earlier. We have tested our algorithm and the baselines in the KeyDoorS3 and ObstructedMaze environments but did not achieve satisfying results.\n\n\n> ## There is no analysis explaining why the proposed method performs better on MiniGrid (Figure 2) than on locomotion tasks\n\nWe discussed this briefly in lines 226-234. The intuition is that MiniGrid tasks, which are primarily about reachability, have a clear binary success or failure criterion. Such tasks present a distinct discrepancy between task-aligned and misaligned reward functions, as well as between acceptable and unacceptable policies. In contrast, locomotion tasks exhibit more quantitative and nuanced differences between policies. Our method aims to learn an **acceptable** policy, which is not necessarily an **optimal** policy under some 'ground truth' reward function -- in Proposition 2, we show that the policy is optimal under a mixture of rewards. \n \nAnother probable explanation is that we did not use high-quality demos in locomotion to simulate non-ideal situations, as shown by the expert's total reward in Table 2. In MiniGrid, even a low-quality demo ends up reaching the goal, and our algorithm captures this property. However, in locomotion, low-quality demos may exhibit different behaviors, such as low speed or perfect balance followed by a crash.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8bjnxRO1BM", "reviewer": "Reviewer_MkJQ", "summary": "This paper addresses the reward misalignment in inverse reinforcement learning and introduces a novel approach to tackle this problem: PAGAR. It introduces a protagonist and an antagonist policy and treats the expert demonstrations as weak supervision to derive a set of reward functions rather than a single reward function. The reward functions and policies are used in a min-max scheme to minimize the regret induced by the two policies. The antagonist policy should maximize the regret whereas the protagonist policy should minimize it. PAGAR is used on adversarial imitation learning settings (more specifically on top of GAIL and VAIL) in combination with PPO on two discrete domain tasks.", "questions": "See weaknesses.", "limitations": "Even though the authors mention a small amount of limitations in the checklist, I think that setting the parameter\n$k$ is not the only limitation of the method (which is the only limitation addressed in the main text). An individual section about the limitations of the approach is missing.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The problem of solving the underlying tasks instead of adapting to the expert data is very relevant.\n* Even though the idea takes inspiration from UED, the proposed approach is novel and significant for the IRL community.\n* The authors provide theoretical results.", "weaknesses": "* Even though the relationship and inspiration to UED is mentioned once, the basis of UED should be explained more.\n* The experimental evaluation is limited. E.g. the experiments only contain environments with discrete action spaces. Since GAIL and VAIL as well as PPO work in continuous action spaces, it should not be a problem to include additional experiments on continuous control tasks.\n* Due to the introduced notation and presented theoretical results, the paper is difficult to follow. Even though the appendix already includes most of the theoretical derivations and proofs, some parts of the main paper could still be included there to give more space for explanation, intuition and experiments. E.g. section 5.1. could be moved to the appendix or integrated in a reduced version in the main text.\n\n* Minor:\n- Missing } in line 270.\n- Line 4 in algorithm 1: It should be \"update $\\pi_{P}$ to maximize...\" instead of $\\pi_{A}$ ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ## the basis of UED should be explained more.\n\n\nOur Eq. 3, which minimizes the worst-case Protagonist Antagonist Induced Regret $Regret$, is inspired by UED in [1]. However, the core of our approach focuses on learning policy with a set of candidate reward functions rather than searching for diverse environments to test the robustness of policies as in [1]. We have developed **novel theories for reward search**. \n\nFurthermore, as mentioned in line 47~52, our contribution is **three-fold**. Besides formulating Eq. 3 for reward search, our **two other contributions are independent of UED**. \n* Our first contribution in establishing the concepets and properties of task-alignment that ultimately lead us to Theorem 1, which justifies the usage of Eq.3. \n* Our third contribution is to develope a practical implementation for solving MinimaxRegret in Section 5.1.  \n\nTherefore, we cited [1] but omitted its details to focus on the distinct aspects of our work. \n\n\n> ## ...additional experiments on continuous control tasks\n\nIn our main text (lines 226-234) and in the general response, we have explained why we did not focus on continuous tasks.\n\nHowever, we did include **continuous control tasks in Appendix C.2**. Furthermore, we have conducted additional experiments on Offline IL with **D4RL continuous control datasets**. The details can be found in our general response\n\n\n>  ## ...E.g. section 5.1. could be moved to the appendix or integrated in a reduced version in the main text.\n\nWe intended for Section 5.1, especially Proposition 2, to highlight the difference between PAGAR-based IL and IRL-based IL: IRL-based IL uses a single reward function derived from an inner optimization problem for policy training, whereas our method employs a mixture of reward functions for policy training. We will consider restructuring our manuscript to provide better clarification rather than presenting all the information as it currently stands.\n\n> ## Missing } in line 270.\n\nThe } is in the middle of the equation.\n \n\n[1] Dennis, M, et al., Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design, NeurIPS 20202.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gJalRdyjOG", "reviewer": "Reviewer_QeNS", "summary": "This paper introduces a novel approach to inverse RL (IRL) by identify weaknesses in current IRL algorithms and proposing the respective solution:  optimizing for *task-related* reward functions. The authors provide a clean theoretical framework for task-related rewards, but lack applicability in practice from my perspective (see below).", "questions": "- How is $\\Pi_{acc}$ actually spanned? It is not really talked about it, which made the following sentence a bit confusing “It is crucial to recognize that rE might not meet the task-aligned reward function criteria specified in Definition 2, even though its optimal policy πE is acceptable. ”\n- Looking at Def 2, the “task relatedness” is only induced by the definition of $\\Pi_{acc}$, is this correct?", "limitations": "My biggest problem is that I dont see why this paper (in practice) proposes a technique for task-aligned reward functions. $k$ and $\\Pi_{acc}$ define task-relatedness in this framework to my understanding. And as the authors say:\n“As the task alignment of each reward function typically remains unknown in IRL settings, this paper proposes treating k as an adjustable parameter – starting with a small k and adjusting based on empirical learning outcome, allowing for iterative refinement for alignment with task requirements.” So what defines task-relatedness is actually set as a hyperparameter in this framework. For me, the method sounds more like a regularization technique for finding less overfitting reward functions. It looks like the auhors also say so:\n“Given the uncertainty in identifying which reward function is aligned, our solution is to train a policy to achieve high utilities under all reward functions in RE,δ to satisfy the conditions in Definition 3.” --> all reward functions does not mean task-related reward functions. Hence, my claim of being a regularization technique rather than an active search for task-related search. For me, this is breaking the storyline of the paper. \n\nFurther points:\n- In section 4.1, consider clarifying the difference between r* and r_E. It took me a bit do understand it.\n- Algorithm 1, line 4: I guess it should be $\\pi_P$ instead of $\\pi_A$.\n- an ablation on $\\delta$ would be interesting \n- add more complex environments. One good fit could be the new locomotion benchmark LocoMuJoCo. It provides motion capture datasets mapped towards different robots. So, there is a dynamics mismatch between the expert dataset and the environment. Also, all baselines (Gail, Vail, IQ-Learn) are already available.\n- Since you have been using IQ-Learn [1] as a baseline, I would also add it together with similar methods using implicit rewards [2, 3] to the related work section with some discussion. Otherwise, it looks out of place in the experiment section. \n- General remark: shorten the theory and make a more comprehensive experimental campaign.\n\n\n[1] Garg et al.; IQ-Learn: Inverse soft-Q Learning for Imitation \\\n[2] Al-Hafez et al.; LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning \\\n[3] Sikchi et al.;  Dual RL: Unification and new methods for reinforcement and imitation learning", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- inverse RL with task-aligned reward functions is an interesting research direction.\n- the paper presents a good theoretical basis for future work.\n- the theoretical work looks plausible to me (only reading the main paper).", "weaknesses": "- The storyline of the paper is not clear. I do not see the emphasis on task-relatedness (see comments below).\n- The paper is cluttered in theory, making it hard to follow. Instead, more focus should be on the story, the motivation, and the experiments.\n- The algorithm looks quite complex; not sure how scaleable it is. (suggestions below)\n- The experimental campaign is concentrated on simple tasks; more complex tasks could prove scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ## The experimental campaign is concentrated on simple tasks... more complex tasks could prove scalability.\n\nThe MiniGrid tasks we employ are challenging benchmarks for RL and IRL although they may look deceptively simple. The agent only makes **partial observation** of the environment -- the 7x7 cells in front of it. **All the objects/walls/doors are randomly placed after every episode.** As a result, it has prohibitively large state space, which has made it a standard benchmark for exploration-driven RL algorithms. For instance, AGAC [1],  the SOTA exploration-driven RL technique, uses this environment as the main benchmark. It takes less than **10 million steps** to attain high performance for a task in VizDoom (a more feature-rich, perception-based environment), but takes **200 million steps** to only achieve a moderate performance on one of the MiniGrid environments. The MiniGrid Github repository also has a long list of papers that use MiniGrid in their empirical evaluations.\n\n\nBesides discrete navigation tasks, we **have also included continuous control tasks, e.g., Mujoco, in Appendix C.2.** Furthermore, we have experimentd on **D4RL dataset** and attached results later in this rebuttal.\n\n  \n> ## How is $\\Pi_{acc}$ spanned ... \n\n$\\Pi_{acc}$ is the set of policies that can accomplish the `task`. How it is spanned depends on what the underlying `task` is. For instance, in some reachability tasks, any policy that can reach a goal state with a higher probability than a threshold can be deemed acceptable. As mentioned in our general response, our definition of `task` is inspired by the NeurIPS 2021 best paper [2], which shows that this definition can characterize a wide range of real-world tasks.\n\n> ## Definition 2 task-relatedness is only induced by the definition of $\\Pi_{acc}$, is this correct?\n\nWhether a reward function is `task-aligned` is determined by $\\Pi_{acc}$: a reward function $r$ is `task-aligned` if **all** the acceptable policies from $\\Pi_{acc}$ achieve higher utilities under $r$ than **all** those not from $\\Pi_{acc}$. Even if the expert policy $\\pi_E$ belongs inside $\\Pi_{acc}$, it does not guarantee that its optimal reward function $r_E$ is `task-aligned`. \n\n> ## ... hyperparameter $k$ ... what defines task-relatedness is actually set as a hyperparameter in this framework.\n\nThe hyperparameter $k$ **does not determine task-relatedness** ('task-alignment'). It is used to determine the **size of the candidate reward functions set $R_{E,k}$**. \n\nOne way to assist understanding the role of $k$ is to draw an analogy to adversarially robust machine learning: the learning model should maintain high accuracy/low error not only at the training data points but also in the 'vicinity' of these points. As the range of adversarial perturbation is generally unknown, the range of this 'vicinity' is often set as a hyperparameter. But this hyperparamter does not determine which point is prone to be attacked. \n \n> ## ...‘finding less overfitting reward functions’ ... regularization technique rather than an active search for task-aligned ...\n\nWe do not actively search for task-aligned reward functions. Our goal is to find an acceptable policy. However, **the concept of task alignment is crucial as it is the backbone of Theorem 1** which justifies two key aspects of our approach: \n\n    1. Collect a set of reward functions, including IRL's sub-optimal rewards, to encompass task-aligned rewards, and \n    2. Train a policy to attain high utility across this reward set. \n    \nWe have not seen literature supporting that **\"finding less overfitting reward functions\"** justifies either of these aspects.\n\nOur formulation in Equation 3 is a **constrained optimization problem**, distinct from merely adding **regularization** to IRL/IL’s loss. The objective is to find a policy that minimizes worst-case regret while the constraint allows us to consider all reward functions with low IRL loss.  \n\n> ## ... breaking the storyline ...\n\nThe main idea of this paper can be summarized as follows:\n* **Goal**: Learn a policy to fulfill an unknown task that fits the description in Definition 1.\n* **Insight**: Only achieving high utility under task-aligned reward functions can ensure task fulfillment.\n* **Problem**: Unknown which reward function is task-aligned\n* **Solution**: \n    1. Treating expert demos as weak supervision, learn a set of reward functions to encompass task-aligned rewards\n    2. Train a policy to achieve high utility across this set of reward functions \n\n> ## the difference between r* and r_E\n\n$r^*$ is the optimal reward function solved via IRL while $r_E$ is the expert reward function. \n\n> ## line 4: I guess it should be $\\pi_P$ instead of $\\pi_A$\n\nYes, we meant to update $\\pi_P$ instead of $\\pi_A$ in line 4.\n\n> ## new locomotion benchmark LocoMuJoCo\n\nAs mentioned earlier, we included continuous control tasks, e.g., Mujoco, in Appendix C.2. However, we appreciate the suggestion and will conduct experiments on LocoMujoco and update the results in the paper.\n\n> ## an ablation on $\\delta$ would be interesting\n\nIn our experiments, we observe that when $\\delta$ is too large, the algorithm does not induce any meaningful results. This has been described in line 164. We plan to include this observation in the updated version of the paper. \n \n\n> ## ...similar methods using implicit rewards \n\nWe will add them to our related work section. In our general response, we have attached our experiemtns on combining our framework with RECOIL in [3]. This marks that compatibility with **implicit reward** and **offline IL** as a new feature of our method.\n\n\n[1] Flet-Berliac, Y., Ferret, J., Pietquin, O., Preux, P.,  Geist, M. Adversarially guided actor-critic. ICLR 2021\n\n[2] Abel, David, et al. On the expressivity of markov reward, NeurIPS 2021\n\n[3] Sikchi et al.; Dual RL: Unification and new methods for reinforcement and imitation learning, ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ABg0OGLpt9", "reviewer": "Reviewer_fY7t", "summary": "This paper aims to alleviate the reward-misalignment issue by introducing task-related objective. The authors theoretically derive sufficient conditions to mitigate the task-reward misalignment issue and design algorithms accordingly. Theoretical analysis of the algorithm is provided to guarantee the reward learning improvement. Two discrete grid-world experiments are provided to empirically validate the proposed algorithms.", "questions": "Please see weakness.", "limitations": "The author discusses the limitation in the paper, which is reasonable.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of introducing task objective to help alleviate the reward ambiguity issue is novel and interesting.\n2. This paper has a complete and solid theoretical structure, which is also the biggest strength.\n3. The empirical results show superior performance over GAIL and VAIL.", "weaknesses": "The main weakness lies in the empirical evaluation:\n1. The environment setups are too simple. The paper only studies two discrete settings where continuous environments are missing, e.g., MuJoCo.\n\n2. The experiment only has two comparison baselines, which is not convincing enough to show the superiority of the proposed algorithm. The paper mentions the IRL based IL and aims to alleviate the reward ambiguity issue, why not directly use some IRL algorithms as baselines to compare, e.g., MaxEnt IRL, ML-IRL, f-IRL, etc.\n\nI know that this paper is a theory paper where the theoretical analysis is the major contribution and should be valued the most. However, the empirical evaluation should at least meet the minimum standard of the current majority, and two discrete gridworlds with fewer than 100 states apparently do not. I highly suggest the authors to add some continuous environments and baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ## ... continuous environments are missing, e.g., MuJoCo.\n\nWe have included continuous control tasks in Appendix C.2.\n\n> ## ... environment setups are too simple ... fewer than 100 states\n\nThe MiniGrid environments are challenging benchmarks for RL and IRL although they may look deceptively simple. The agent only makes **partial observation** of the environment -- the 7x7 cells in front of it. **Furethermore, all the objects/walls/doors are randomly placed in every episode.** As a result, it has **far more than 100 states**. In fact, due to its massive state space, it has become a standard benchmark for exploration-driven RL algorithms. For instance, AGAC [1],  the SOTA exploration-driven RL technique, uses this environment as the main benchmark. It takes less than **10 million steps** to attain high performance for a task in VizDoom (a more feature-rich, perception-based environment), but takes **200 million steps** to only achieve a moderate performance on one of the MiniGrid environments. The MiniGrid Github repository also has a long list of papers that use MiniGrid in their empirical evaluations.\n\n\n\n> ## ... not directly use some IRL algorithms as baselines to compare, e.g., MaxEnt IRL, ML-IRL, f-IRL, etc.\n\nOur approach, named PAGAR-based IL, is an IL algorithm that uses existing IRL-based IL algorithm to specify the set $R_{E,\\delta}$ as mentioned in line 172. We design the experiments mainly to show the compatibility of our algorithm with the IRL parts of the existing IRL-based IL benchmarks, and whether our algorithm can improve the imitation performance by using the same IRL parts as those of the benchmarks. \n\nHowever, we appreciate the reviewrs' suggestions on trying other IRL algorithms. Therefore, we have incorporated our algorithm with the most advanced IRL/IL algorithm, RECOIL [2].  We have attached our results in our general response to all reviewers. Please refer to it for details.\n\n\n[1] Flet-Berliac, Y., Ferret, J., Pietquin, O., Preux, P.,  Geist, M. Adversarially guided actor-critic. ICLR 2021\n\n[2] Sikchi et al.; Dual RL: Unification and new methods for reinforcement and imitation learning, ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers' insights and suggestions. In this general response, we would like to reiterate the motivation of this paper and share our additional experimental results.


> ## PAGAR-Based IL for Task-Alignment

In this paper, we focus on aligning with tasks that fit the format described in Definition 1, which is based on the **NeurIPS 2021 best paper** [1].  A significant characteristic of these tasks is the presence of a binary success or failure criterion between policies. Our method aims to learn an **acceptable** policy that succeeds in such a task, rather than an **optimal** policy under some 'ground truth' reward function. As a result, our approach's effectiveness is more pronounced in tasks where success and failure are distinctly binary, such as in MiniGrid, than in tasks with more quantitative performance metrics, such as continuous control tasks. Therefore, we included our MiniGrid experimental results in the main text and placed the Mujoco experimental results in Appendix C.2. 


> ## Applying PAGAR to offline IL tasks

Our additional experiments combine PAGAR with RECOIL [2], a SOTA **offline IL** algorithm. In these experiments, we used the standard offline datasets from D4RL as baselines. Our results below demonstrate the compatibility of PAGAR with offline IL algorithms and highlight this as a new feature of our method.

|        | RECOIL | (Ours) PAGAR_RECOIL| 
|--------|------- | ------------|  
| hopper-random |$106.87 \pm 2.69$ | $111.16\pm 0.51$ | 
| halfcheetah-random | $80.84\pm17.62$  | $92.94\pm 0.10$ |
| walker2d-random | $108.40\pm 0.04$ | $108.40\pm 0.12$|
| ant-random | $113.34\pm2.78$ | $121\pm 5.86$ |


Here are some details. The original RECOIL algorithm learns a $Q$, $V$ value functions and a policy $\pi$.  In order to combine PAGAR with RECOIL, we made the following modification. 
* Instead of learning the $Q$ value function, we **explicitly** learn the reward  function $r$:
    * The $r$ reward function takes the current state $s$, action $a$ and the next state $s'$ as input, and outputs a real number -- the reward.
    * We use the same loss function as that for optimizing $Q$ in RECOIL to optimize $r$ by replacing $Q(s,a)$ with $r(s,a,s')+\gamma V(s')$ for every $(s,a,s')$ sampled from the offline dataset. 
    * We use the same loss function for optimizaing the $V$ value function as in RECOIL to still optimize $V$, except for replacing the target $Q(s,a)$ with target $r(s,a,s') + \gamma V(s')$ for every $(s,a,s')$ experience sampled from the offline dataset. 
* Instead of learning a single policy as in RECOIL, we learn a protagonist and antagonist policies $\pi_P$ and $\pi_A$. We use the same SAC-like policy update rule as in RECOIL to train each policy, except for replacing $Q(s,a)$ with $r(s,a,s')+\gamma V(s')$ for every $(s,a,s')$ experience sampled from the offline dataset.
* With some heuristic, we construct a PAGAR-loss that is proportional to $R(s,a,s') * max(0, \frac{\pi_P(a|s)}{\pi_A(a|s)})$ for $(s,a,s')$ in the expert demonstrations plus $R(s,a,s') *  min(0, \frac{\pi_P(a|s)}{\pi_A(a|s)})$ for $(s,a,s')$ in the offline random sample set. For simplicity, we multiply this PAGAR-loss with a fixed Lagrangian parameter $\lambda=1e-3$ and add it to the aforementioned loss for optimizing $r$.
* We directly build our implementation upon the code base realsed by RECOIL's author, and tested it on D4RL datasets, with the same configurations as reported in the RECOIL paper [2]. In particular, offline IL uses expert and sub-optimal sample sets to learn policies. We use the D4RL's 'expert' datasets as the expert demonstrations and the 'random' datasets for Mujoco environments as the offline suboptimal dataset. The results in the table above are averaged from 4 seeds.  


While our experimental results also show improved performance in continuous tasks, such as those in Appendix C.2 and the offline IL tasks introduced above, our primary focus remains on task alignment. This paper serves as a debut and theoretical introduction to our task-alignment ideology.

[1] Abel, David, et al. On the expressivity of Markov reward, NeurIPS 2021

[2] Sikchi et al.; Dual RL: Unification and new methods for reinforcement and imitation learning, ICLR 2024

Meta-review: This paper develops a novel approach to inverse RL that prioritizes alignment of the learned reward function with the task objectives rather than the demonstration data, as done in typical IRL methods. The authors provide a solid motivation and theoretical foundation for this approach. There were some initial concerns with clarity and the empirical evaluation, focusing mostly on the limited complexity and scope of the chosen domains. The authors provided additional results and clarifications in their rebuttal that partially addressed these concerns, yet reviewers still felt that a more substantial evaluation with more complex environments are needed for a final version of this paper. The authors are strongly encouraged to incorporate this advice from the reviewers, and their suggestions for clarity improvements, in preparing the next draft of this paper.
Decision: Accept (poster)