Title: Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment
Venue: NeurIPS 2024
Accepted: yes
Keywords: inverse reinforcement learning

imitation learning

reinforcement learning
Abstract: Many imitation learning (IL) algorithms use inverse reinforcement learning (IRL) to infer a reward function that aligns with the demonstration.
However, the inferred reward functions often fail to capture the underlying task objectives.
In this paper, we propose a novel framework for IRL-based IL that prioritizes task alignment over conventional data alignment. Our framework is a semi-supervised approach that leverages expert demonstrations as weak supervision to derive a set of candidate reward functions that align with the task rather than only with the data. It then adopts an adversarial mechanism to train a policy with this set of reward functions to gain a collective validation of the policy's ability to accomplish the task. We provide theoretical insights into this framework's ability to mitigate task-reward misalignment and present a practical implementation. Our experimental results show that our framework outperforms conventional IL baselines in complex and transfer learning scenarios.

Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment

Abstract

Many imitation learning (IL) algorithms use inverse reinforcement learning (IRL) to infer a reward function that aligns with the demonstrations. However, the inferred reward function often fails to capture the underlying task objective. In this paper, we propose a novel framework for IRL-based IL that prioritizes task alignment over conventional data alignment. Our framework is a semi-supervised approach that leverages expert demonstrations as weak supervision signals to derive a set of candidate reward functions that align with the task rather than only with the data. It adopts an adversarial mechanism to train a policy with this set of reward functions to gain a collective validation of the policy's ability to accomplish the task. We provide theoretical insights into this framework's ability to mitigate task-reward misalignment and present a practical implementation. Our experimental results show that our framework outperforms conventional IL baselines in complex and transfer learning scenarios. The complete code are available at https://github.com/zwc662/PAGAR.

1 Introduction

Inverse reinforcement learning (IRL) Ng and Russell [2000], Finn et al. [2017] has become a popular method for imitation learning (IL), allowing policies to be trained by learning reward functions from expert demonstrations Abbeel and Ng [2004], Ho and Ermon [2016]. Despite its widespread use, IRL-based IL faces significant challenges that often stem from overemphasizing data alignment rather than task alignment. For instance, reward ambiguity, where multiple reward functions can be consistent with the expert demonstrations, makes it difficult to identify the correct reward function. This problem persists even when there are infinite data Ng and Russell [2000], Cao et al. [2021], Skalse et al. [2022a,b]. Additionally, limited availability of demonstrations can further exacerbate this problem, as the data may not fully capture the nuances of the task. Misaligned reward functions can lead to policies that optimize the wrong objectives, resulting in poor performance and even reward hacking Hadfield-Menell et al. [2017], Amodei et al. [2016], Pan et al. [2022], a phenomenon where the policy exploits loopholes in the inferred reward function. These challenges highlight the limitation of exclusively pursuing data alignment in solving real-world tasks.

In light of these considerations, this paper advocates for a paradigm shift from a narrow focus on data alignment to a broader emphasis on task alignment. Grounded in a general formalism of task objectives, we propose identifying the task-aligned reward functions that more accurately reflect the underlying task objectives in their policy utility spaces. Expanding on this concept, we explore the intrinsic relationship between the task objective, reward, and expert demonstrations. This relationship leads us to a novel perspective where expert demonstrations can serve as weak supervision signals for identifying a set of candidate task-aligned reward functions. Under these reward functions, the expert

achieves high -- but not necessarily optimal -- performance. The rationale is that achieving high performance under a task-aligned reward function is often adequate for real-world applications.

Building on this premise, we leverage IRL to derive the set of candidate task-aligned reward functions and propose Protagonist Antagonist Guided Adversarial Reward (PAGAR), a semi-supervised framework designed to mitigate task-reward misalignment by training a policy with this candidate reward set. PAGAR adopts an adversarial training mechanism between a protagonist policy and an adversarial reward searcher, iteratively improving the policy learner to attain high performance across the candidate reward set. This method moves beyond relying on deriving a single reward function from data, enabling a collective validation of the policy's similarity to expert demonstrations in terms of effectiveness in accomplishing tasks. Experimental results show that our algorithm outperforms baselines on complex IL tasks with limited demonstrations and in challenging transfer environments. We summarize our contributions below.

Introduction of Task Alignment in IRL-based IL: We present a novel perspective that shifts the focus from data alignment to task alignment, addressing the root causes of reward misalignment in IRL-based IL.

Protagonist Antagonist Guided Adversarial Reward (PAGAR): We propose a new semi-supervised framework that leverages adversarial training to improve the robustness of the learned policy.

Practical Implementation: We present a practical implementation of PAGAR, including the adversarial reward searching mechanism and the iterative policy-improving process. Experimental results demonstrate superior performances in complex and transfer learning environments.

2 Related Works

IRL-based IL circumvents many challenges of traditional IL such as compounding error Ross and Bagnell [2010], Ross et al. [2011], Zhou et al. [2020] by learning a reward function to interpret the expert behaviors Ng et al. [1999], Ng and Russell [2000] and then learning a policy from the reward function via reinforcement learning (RL)Sutton and Barto [2018]. However, the learned reward function may not always align with the underlying task, leading to reward misspecification Pan et al. [2022], Skalse and Abate [2022], reward hacking Skalse et al. [2022b], and reward ambiguity Ng and Russell [2000], Cao et al. [2021]. The efforts on alleviating reward ambiguity include Max-Entropy IRL Ziebart et al. [2008], Max-Margin IRL Abbeel and Ng [2004], Ratliff et al. [2006], and Bayesian IRL Ramachandran and Amir [2007]. GAN-based methods Ho and Ermon [2016], Jeon et al. [2018], Finn et al. [2016], Peng et al. [2019], Fu et al. [2018] use neural networks to learn reward functions from limited demonstrations. However, these efforts that aim to address reward ambiguity fall short of mitigating the general impact of reward misalignment which can be caused by various reasons such as IRL making false assumptions about the relationship between expert policy and expert reward function Skalse et al. [2022a], Hong et al. [2023]. Other attempts to mitigate reward misalignment involve external information other than expert demonstrations Hejna and Sadigh [2023], Zhou and Li [2018, 2022a,b]. Our work adopts the generic setting of IRL-based IL without needing additional information. The idea of considering a reward set instead of focusing on a single reward function is supported by Metelli et al. [2021] and Lindner et al. [2022]. However, these works target reward ambiguity instead of reward misalignment. Our protagonist and antagonist setup is inspired by the concept of unsupervised environment design (UED) Dennis et al. [2020]. In this paper, we develop novel theories in the context of reward learning.

3 Preliminaries

Reinforcement Learning (RL) models the environment as a Markov Decision Process M = ⟨ S , A , P , d 0 ⟩ where S is the state space, A is the action space, P is the transition probability, d 0 is the initial state distribution. A policy π ( a | s ) determines the probability of an RL agent performing an action a at state s . By successively performing actions for T steps from an initial state s (0) ∼ d 0 , a trajectory τ = s (0) a (0) s (1) a (1) . . . s ( T ) is produced. A state-action based reward function is a mapping r : S × A → R . The soft Q-value function of π is Q π ( s, a ) = r ( s, a ) + γ · E s ′ ∼P ( ·| s,a ) [ V π ( s ′ )] where γ ∈ (0 , 1] is a discount factor, V π is the soft state-value function of π defined as V π ( s ) := E [ Q π ( s, a )] + H ( π ( ·| s )) , and H ( π ( ·| s )) is the entropy of π at state s

(a) Task-Reward Alignment

(b) PAGAR-Based IL

Policy Utility

n

U

r+

r+

U

r+

r

Tr*

RL

E

Ⅱ

acc

IRL

π

PAGAR

RE,8

Ⅱ

The soft advantage of performing action a at state s and then following a policy π afterwards is A π ( s, a ) = Q π ( s, a ) - V π ( s ) . The expected return of π under a reward function r is given as U r ( π ) = E τ ∼ π [ ∑ ∞ t =0 γ t · r ( s ( t ) , a ( t ) )] . With a slight abuse of notations, we denote the entropy of a policy as H ( π ) := E τ ∼ π [ ∑ ∞ t =0 γ t · H ( π ( ·| s ( t ) ))] . The standard RL learns an optimal policy by maximizing U r ( π ) . The entropy regularized RL learns a soft-optimal policy by maximizing the objective function J RL ( π ; r ) := U r ( π ) + H ( π ) .

Inverse Reinforcement Learning (IRL) assumes that a set E = { τ 1 , . . . , τ N } of expert demonstrations are sampled from the roll-outs of the expert's policy π E and aims to learn the expert reward function r E . IRL Ng and Russell [2000] assumes that π E is optimal under r E and learns r E by maximizing the margin U r ( E ) -max π U r ( π ) while Maximum Entropy IRL (MaxEnt IRL) Ziebart et al. [2008] maximizes an entropy regularized objective function J IRL ( r ) = U r ( E ) -(max π U r ( π ) + H ( π )) .

Generative Adversarial Imitation Learning (GAIL) Ho and Ermon [2016] draws a connection between IRL and Generative Adversarial Nets (GANs) as shown in Eq.1, where a discriminator D : S × A → [0 , 1] is trained by minimizing Eq.1 so that D can accurately identify any ( s, a ) generated by the agent. Meanwhile, an agent policy π is trained as a generator to maximize Eq.1 so that D cannot discriminate τ ∼ π from τ E . Adversarial inverse reinforcement learning (AIRL) Fu et al. [2018] uses a neural-network reward function r to represent D ( s, a ) := π ( a | s ) exp( r ( s,a ))+ π ( a | s ) , rewrites J IRL as minimizing Eq.1, and proves that the optimal reward satisfies r ∗ ≡ log π E ≡ A π E . By training π with r ∗ until optimality, π will behave just like π E .

4 Task-Reward Alignment

In this section, we formalize the concept of task-reward misalignment in IRL-based IL. We start by defining a notion of task based on the framework from Abel et al. [2021].

Definition 1 ( Task ) . Given the policy hypothesis set Π , a task (Π , ⪯ task , Π acc ) is specified by a partial order ⪯ task over Π and a non-empty set of acceptable policies Π acc ⊆ Π such that ∀ π 1 ∈ Π acc and ∀ π 2 / ∈ Π acc , π 2 ⪯ task π 1 always hold.

Remark: The notions of policy acceptance and order allow the definition of task to accommodate a broad range of real-world tasks 1 including the standard RL tasks (learning the optimal policy from a reward function r ): given a reward function r and a policy hypothesis set Π , the standard RL task can be written as a tuple (Π , ⪯ task , Π acc ) where ⪯ task satisfies ∀ π 1 , π 2 ∈ Π , π 1 ⪯ task π 2 ⇔ U r ( π 1 ) ≤ U r ( π 2 ) , and Π acc = { π | ∀ π ′ ∈ Π .π ′ ⪯ task π } contains all the optimal policies.

Designing reward function(s) that align with the underlying task is essential in RL. Whether a designed reward aligns with the task hinges on how policies are ordered by the task and the utilities of the policies under the reward function. Therefore, we define the task-reward alignment by examining the utility spaces of the reward functions. If the acceptable policy set Π acc of the task is given, we let U r := min π ∈ Π acc U r ( π ) be the minimal utility achieved by any acceptable policy under r .

Definition 2 ( Task-Aligned Reward Functions ) . Areward function is a task-aligned reward function (denoted as r + ) if and only if ∀ π ∈ Π \ Π acc , U r + ( π ) < U r + ( π ) . Conversely, if this condition is not met, it is a task-misaligned reward function (denoted as r -).

The definition suggests that under a task-aligned reward function r + , all acceptable policies for the task yield higher utilities than unacceptable ones. It also suggests that a policy is deemed acceptable as long as its utility is greater than U r + for some task-aligned reward function r + , even if this policy is not optimal. We also examine whether high utility under a reward function r suggests a higher order under ⪯ task . We define U r := max π ∈ Π U r ( π ) s.t. ∀ π 1 , π 2 ∈ Π , U r ( π 1 ) < U r ( π ) ≤ U r ( π 2 ) ⇒

( π 1 ⪯ task π ) ∧ ( π 1 ⪯ task π 2 ) , which is the highest utility threshold such that any policy achieving a higher utility than U r has a higher order than those achieving lower utilities than U r . In Figure 1(a) we illustrate how U r and U r vary between task-aligned and misaligned reward functions.

Proposition 1. Given the policy order ⪯ task of a task, for any two reward functions r 1 , r 2 , if { π | U r 1 ( π ) ≥ U r 1 } ⊆ { π | U r 2 ( π ) ≥ U r 2 } , then there must exist policies π 1 ∈ { π | U r 1 ( π ) ≥ U r 1 } , π 2 ∈ { π | U r 2 ( π ) ≥ U r 2 } such that U r 1 ( π 2 ) ≤ U r 1 ( π 1 ) and π 2 ⪯ task π 1 while U r 2 ( π 2 ) ≥ U r 2 ( π 1 ) .

This proposition implies that a high threshold U r indicates that a high utility corresponds to a high order in terms of ⪯ task . In particular, for any task-aligned reward function r + , { π | U r + ( π ) ≥ U r + } ⊆ Π acc ≡ { π | U r + ( π ) ≥ U r + } (see proof in Appendix A.2). Thus, a small { π | U r + ( π ) ≥ U r + } leads to a large { π | U r + ( π ) ∈ [ U r + , U r + ] } . Hence, a task-aligned reward function r + is more likely to be aligned with the task if it has a wide [ U r + , U r + ] and a narrow [ U r + , max π ∈ Π U r + ( π )] .

4.1 Mitigate Task-Reward Misalignment in IRL-Based IL

In IRL-based IL, a key challenge is that the underlying task is unknown , making it difficult to assert if a learned policy is acceptable. We denote the optimal reward function learned from the demonstration set E as r ∗ , and the optimal policy under r ∗ as π r ∗ . When π r ∗ has a poor performance under r E , it is considered to have a high Regret ( π r ∗ , r E ) which is defined in Eq.2. If Regret ( π r ∗ , r E ) > max π ′ ∈ Π U r E ( π ′ ) -U r E , then π r ∗ is unacceptable and r ∗ is task-misaligned.

Several factors can lead to a high Regret ( π r ∗ , r E ) . For instance, Viano et al. [2021] shows that when expert demonstrations are collected in an environment whose dynamical function differs from that of the learning environment, | Regret ( π r ∗ , r E ) | can be positively related to the discrepancy between those dynamical functions. Additionally, we prove in Appendix A.1 that learning from only a few representative expert trajectories can also result in a large | Regret ( π r ∗ , r E ) | with a high probability.

Our insight for mitigating such potential task-reward misalignment in IRL-based IL is to shift our focus from learning an optimal policy that maximizes the intrinsic r E to learning an acceptable policy π ∗ that achieves a utility higher than U r + under any task-aligned reward function r + . Our approach is to treat the expert demonstrations as weak supervision signals based on the following.

Theorem 1. Let I be an indicator function. For any k ≥ { min r + ∑ π ∈ Π I{ U r + ( π ) ≥ U r + ( π E ) } } , if π ∗ satisfies {∑ π ∈ Π I{ U r ( π ) ≥ U r ( π ∗ ) } } < | Π acc | for all r ∈ R E,k := { r | ∑ π ∈ Π I{ U r ( π ) ≥ U r ( π E ) } ≤ k } , then π ∗ is an acceptable policy, i.e., π ∗ ∈ Π acc . Additionally, if k < | Π acc | , such an acceptable policy π ∗ is guaranteed to exist.

The statement suggests that we can obtain an acceptable policy by training it to attain high performance across a reward function set R E,k that includes all the reward functions where, for each reward function at most k policies outperform the expert policy π E . The minimal value of k is determined by all the task-aligned reward functions in the reward hypothesis set. Appendix A.2 provides the proof.

How to build R E,k ? Building R E,k involves setting the parameter k . If r E is a task-aligned reward function and π E is optimal solely under r E , then the minimal k = 0 , and R E, 0 only contains r E . However, relying on a singleton R E, 0 equates to applying vanilla IRL, which is susceptible to misalignment issues, as noted earlier. It is crucial to recognize that r E might not meet the task-aligned reward function criteria specified in Definition 2, even though its optimal policy π E is acceptable. This situation necessitates a positive k , thereby expanding R E,k beyond a single function and changing the role of expert demonstrations from strong supervision to weak supervision. Note that we suggest letting k ≤ | Π acc | instead of allowing k →∞ because R E, ∞ would then encompass all possible reward functions, and it is impractical to identify a policy capable of achieving high performance across all reward functions. Letting k ≤ | Π acc | guarantees there exists a feasible policy π ∗ , e.g., π E itself. As the task alignment of each reward function typically remains unknown in IRL settings, this paper proposes treating k as an adjustable parameter - starting with a small k and adjusting based on empirical learning outcome, allowing for iterative refinement for alignment with task requirements.

In practice, Π can be uncountable, e.g., a Gaussian policy. Hence, we adapt the concept of k in R E,k to a hyperparameter δ ≤ δ ∗ := max J IRL ( r ) , leading us to redefine R E,k as a δ -optimal reward

r

function set R E,δ := { r | J IRL ( r ) ≥ δ } . This superlevel set includes all the reward functions under which the optimal policies outperform the expert by at most -δ . If δ is appropriately selected such that R E,δ includes task-aligned reward functions, we can mitigate reward misalignment by satisfying the conditions outlined in Definition 3, which are closely related to Definition 2 and Proposition 1.

Definition 3 ( Mitigation of Task-Reward Misalignment). Assuming that the reward function set R E,δ contains task-aligned reward function r + 's, the mitigation of task-reward misalignment in IRL-based IL is to learn a policy π ∗ such that (i) (Weak Acceptance) ∀ r + ∈ R E,δ , U r + ( π ∗ ) ≥ U r + , or (ii) (Strong Acceptance) ∀ r + ∈ R E,δ , U r + ( π ∗ ) ≥ U r + .

While condition (i) states that π ∗ is acceptable for the task, i.e., π ∗ ∈ Π acc , condition (ii) further states that π ∗ have a high order in terms of ⪯ task . Hence, condition (i) is weaker than (ii) because a policy π ∗ satisfying (ii) automatically satisfies (i) according to Definition 2. Given the uncertainty in identifying which reward function is aligned, our solution is to train a policy to achieve high utilities under all reward functions in R E,δ to satisfy the conditions in Definition 3. We explain this approach in the following semi-supervised paradigm, PAGAR.

5 Protagonist Antagonist Guided Adversarial Reward (PAGAR)

PAGAR is an adversarial reward searching paradigm which iteratively searches for a reward function to challenge a policy learner by incurring a high regret as defined in Eq.2. We refer to the policy to be learned as the protagonist policy and re-write it as π P . We then introduce a second policy, dubbed antagonist policy π A , as a proxy of the arg max π ′ ∈ Π U r ( π ′ ) for Eq.2. For each reward function r , we call the regret of π P under r , i.e., Regret ( π P , r ) = max π A ∈ Π U r ( π A ) -U r ( π P ) , the Protagonist Antagonist

Induced Regret . We then formally define PAGAR in Definition 4.

Definition 4 (Protagonist Antagonist Guided Adversarial Reward (PAGAR) ) . Given a candidate reward function set R and a protagonist policy π P , PAGAR searches for a reward function r within R to maximize the Protagonist Antagonist Induced Regret , i.e., max r ∈ R Regret ( π P , r ) .

PAGAR-based IL learns a policy from R E,δ by minimizing the worst-case Protagonist Antagonist Induced Regret via MinimaxRegret ( R E,δ ) as defined in Eq.3 where R can be any input reward function set and is set as R = R E,δ in PAGAR-based IL.

Our subsequent discussion will focus on identifying the sufficient conditions for PAGAR-based IL to mitigate task-reward misalignment as described in Definition 3. In particular, we consider the case where J IRL ( r ) := U r ( E ) -max U r ( π ) . We use L r to denote the Lipschitz constant of r ( τ ) , and

π

W E to denote the smallest Wasserstein 1 -distance W 1 ( π, E ) between τ ∼ π of any π and τ ∼ E , i.e., W E ≜ min π ∈ Π W 1 ( π, E ) . Then, we have Theorem 2.

Theorem 2 (Weak Acceptance) . If the following conditions (1) (2) hold for R E,δ , then the optimal protagonist policy π P := MinimaxRegret ( R E,δ ) satisfies ∀ r + ∈ R E,δ , U r + ( π P ) ≥ ¯ U r + .

This statement shows the conditions for PAGAR-based IL to attain the 'Weak Acceptance' goal described in Definition 3. The condition (1) states that the task-aligned reward functions in R E,δ all have a high level of alignment in matching ⪯ task within their high utility ranges. The condition (2) requires that for the policy π ∗ = arg min π ∈ Π W 1 ( π, E ) , the performance difference between E and

r

π ∗ is small enough under all r ∈ R E,δ . Since for each reward function r ∈ R E,δ , the performance difference between E and the optimal policy under r is bounded by δ , condition (2) implicitly requires that π ∗ not only performs well under any task-aligned reward function r + (thus being acceptable in the task) but also achieve relatively low regret under task-misaligned reward function r -. However, the larger the rage [ U r + , U r + ] is across the task-aligned reward function r + , the less strict the requirement for low regret under r -becomes. The proof can be found in Appendix A.5. The following theorem further suggests that a δ close to its upper-bound δ ∗ := max J IRL ( r ) can help MinimaxRegret ( R E,δ ) gain a better chance of finding an acceptable policy for the underlying task and attain the 'Strong Acceptance' goal described in Definition 3.

Theorem 3 (Strong Acceptance) . Assume that the condition (1) in Theorem 2 holds for R E,δ . If for any r ∈ R E,δ , L r · W E -δ ≤ min r + ∈ R E,δ { max π ∈ Π U r + ( π ) -U r + } , then the optimal protagonist policy

When do these assumptions hold? The condition (1) in Theorem 2 requires all the task-aligned reward functions in R E,δ exhibit a high level of conformity with the policy order ⪯ task . Being task-aligned already sets a strong premise for satisfying this condition. We further posit that this condition is more easily satisfied when the task has a binary outcome, such as in reach-avoid tasks so that the aligned and misaligned reward functions tend to have higher discrepancy than tasks with quantitative outcomes. In the experimental section, we validate this hypothesis by evaluating tasks of this kind. Regarding condition (2) of Theorem 2 and the assumptions of Theorem 3, which basically require the existence of a policy with low regret across R E,δ set, it is reasonable to assume that expert policy meets this criterion.

5.1 Comparing PAGAR-Based IL with IRL-Based IL

We illustrate the difference between IRL-based IL and PAGAR-based IRL in Fig.1(b). While IRLbased IL aims to learn the optimal policy π r ∗ under the IRL-optimal reward r ∗ , PAGAR-based IL learns a policy π ∗ from the reward function set R E,δ . Both PAGAR-based IL and IRL-based IL are zero-sum games between a policy learner and a reward learner. However, while IRL-based IL only aims to reach equilibrium at a single reward function under strong assumptions, e.g., sufficient demonstrations, convex reward and policy spaces, etc., PAGAR-based IL can reach equilibrium with a mixture of reward functions without those assumptions.

Proposition 2. Given arbitrary reward function set R , there exists a constant c and a distribution R over R such that MinimaxRegret ( R ) yields the same pol-

π { }

A detailed derivation can be found in Theorem 6 in Appendix A.4. In a nutshell, R π ( r ) is a baseline distribution over R such that (i) c ≡ E r ∼R π [ U r ( π )] holds for all the π 's that do not always perform worse than any other policy under r ∈ R , (ii) among all the R π 's that satisfy the condition (i), we pick the one with the minimal c ; and (iii) for any other policy π , R π uniformly concentrates on arg max r ∈ R U r ( π ) . Note that in PAGAR-based IL, where R E,δ is used in place of arbitrary R , R π

is a distribution over R E,δ and r ∗ π is constrained to be within R E,δ . Essentially, the mixed reward functions dynamically assign weights to r ∼ R π and r ∗ π depending on π . If π performs worse under

r ∗ π than under many other reward functions ( U r ∗ π ( π ) falls below c ), a higher weight will be allocated to using r ∗ π to train π . Conversely, if π performs better under r ∗ π than under many other reward functions ( c falls below Ur ∗ π ( π ) ), a higher weight will be allocated to reward functions drawn from R π . Furthermore, we prove in Appendix A.7 that the MinimaxRegret objective function defined in Eq.3 is a convex optimization w.r.t the protagonist policy π P .

We also prove in Appendix A.8 that when there is no misalignment issue, i.e., under the ideal conditions for IRL, PAGAR-based IL can either guarantee inducing the same results as IRL-based IL with δ = max r J IRL ( r ) , or guarantee inducing an acceptable π P by making max r J IRL ( r ) -δ no greater than max U r + ( π ) -U r + for r + ∈ R E,δ .

6 A Practical Approach to Implementing PAGAR-based IL

We solve MinimaxRegret ( R E,δ ) , by alternating between policy learning and reward search. Based on Eq.3, we introduce an on-and-off policy learning framework and an adversarial reward search objective. Moreover, we embed the constraint r ∈ R E,δ into the reward search objective using IRL, resulting in a meta-algorithm compatible with various IRL methods.

6.1 Policy Optimization with On-and-Off Policy Samples

Given an intermediate learned reward function r , we use RL to train π P to minimize the regret min π P Regret ( π P , r ) = min π P { max π A U r ( π A ) } -U r ( π P ) as indicated by Eq.3 where π A is trained to serve as the optimal policy under r as noted in Section 5. Since we have to sample trajectories with π A and π P , we propose to combine off-policy and on-policy samples to optimize π P so that we can leverage the samples maximally. Off-Policy: We leverage the Theorem 1 in Schulman et al. [2015] to derive a bound for the utility subtraction: U r ( π P ) -U r ( π A ) ≤ ∑ s ∈ S ρ π A ( s ) ∑ a ∈ A π P ( a | s ) A π A ( s, a

) + C · max s D TV ( π A ( ·| s ) , π P ( ·| s )) 2 where ρ π A ( s ) = ∑ T t =0 γ t Prob ( s ( t ) = s | π A ) is the discounted visitation frequency of π A , A π A is the advantage function without considering the entropy, and C is some constant. Then we follow the derivation in Schulman et al. [2017], which is based on Theorem 1 in Schulman et al. [2015], to derive from the inequality an importance sampling-based objective function J π A ( π P ; r ) := E s ∼ π A [min( ξ ( s, a ) · A π A ( s, a ) , clip ( ξ ( s, a ) , 1 -σ, 1 + σ ) · A π A ( s, a )] where σ is a clipping threshold, ξ ( s, a ) = π P ( a | s ) π A ( a | s ) is an importance sampling rate. The details can be found in Appendix B.1. This objective function allows us to train π P by using the trajectories of π A . On-Policy: We also optimize π P with the standard RL objective function J RL ( π P ; r ) by using the trajectories of π P itself. As a result, the objective function for optimizing π P is max π P ∈ Π J π A ( π P ; r ) + J RL ( π P ; r ) . As for π A , we only use the standard RL objective function, i.e., max π A ∈ Π J RL ( π A ; r ) . Although the computational complexity equals the sum of the complexities of RL update steps for π A and π P , these two RL update steps can be executed in parallel.

6.2 Regret Maxmization with On-and-Off Policy Samples

Given the intermediate learned protagonist and antagonist policy π P and π A , according to MinimaxRegret in Eq.3, we need to optimize r to maximize U r ( π A ) -U r ( π P ) . In practice, we found that the subtraction between the estimated U r ( π A ) and U r ( π P ) can have a high variance. To resolve this issue, we derive two reward improvement bounds to approximate this subtraction.

Theorem 4. Suppose policy π 2 ∈ Π is the optimal solution for J RL ( π ; r ) . Then , the inequalities Eq.4 and 5 hold for any policy π 1 ∈ Π , where α = max s D TV ( π 1 ( ·| s ) , π 2 ( ·| s )) , ϵ = max s,a |A π 2 ( s, a ) | ,

Algorithm 1 An Meta-Algorithm for Imitation Learning with PAGAR

Input : Expert demonstration set E , IRL objective function J IRL , loss bound δ , parameter λ ≥ 0 , initial protagonist policy π P , antagonist policy π A , reward function r , maximum iteration number N . Output : π P

1: for iteration i = 0 , 1 , . . . , N do

2: Sample trajectory sets D A ∼ π A and D P ∼ π P

3: Optimize π A : estimate J RL ( π A ; r ) with D A ; update π A to maximize J RL ( π A ; r )

4: Optimize π P : estimate J RL ( π P ; r ) with D P ; estimate J π A ( π P ; r ) with D A ; update π A to maximize J RL ( π P ; r ) + J π A ( π P ; r )

5: Optimize r : estimate J PAGAR ( r ; π P , π A ) with D P and D A ; estimate J IRL ( r ) with D A and E ; update r to minimize J PAGAR ( r ; π P , π A ) + λ · ( δ -J IRL ( r )) ; then update λ to maximize δ -J IRL ( r )

6: end for

7: return π P

By letting π P be π 1 and π A be π 2 , Theorem 4 enables us to bound U r ( π A ) -U r ( π P ) by using either only the samples of π A or only those of π P . Following Fu et al. [2018], we let r be a proxy of A π 2 in Eq.4 and 5. Then we derive two loss functions J R, 1 ( r ; π P , π A ) and J R, 2 ( r ; π P , π A ) for r as shown in Eq.6 and 7 where C 1 and C 2 are constants proportional to the estimated maximum KL divergence between π A and π P (to bound α Schulman et al. [2015]). The objective function for r is then J PAGAR := J R, 1 + J R, 2 . The complexity equals that of computing the reward along the trajectories sampled from π A and π P .

6.3 A Meta-Algorithm for Solving PAGAR-Based IL

Given an IRL objective function J IRL , we enforce the constraint r ∈ R E,δ by adding to J PAGAR ( r ; π P , π A ) a penalty term λ · ( δ - J IRL ) , where λ is a Lagrangian parameter. The resulting objective function for optimizing r becomes min r ∈ R J PAGAR ( r ; π P , π A ) + λ · ( δ -J IRL ( r )) .

We initialize λ with a large value to prioritize satisfying the constraint r ∈ R E,δ and update it based on δ -J IRL (the details can be found in Appendix B.4). Algorithm 1 outlines our meta-algorithm for PAGAR-based IL. The algorithm takes an IRL objective J IRL as an input, and alternates between policy and reward learning. In line 3 , π A is trained via RL with its own sample set D A . In line 4 , we train π P via the on-and-off policy approach in Section 6.1 with both D A and π P 's sample set D P . Finally, in line 5 , J PAGAR is estimated from both D A and D P , while J IRL is from D A and E .

7 Experiments

The goal of our experiments is to assess whether using PAGAR-based IL can efficiently mitigate reward misalignment under conditions that are not ideal for IRL. We present the main results below and provide details and additional results in Appendix C.

7.1 Discrete Navigation Tasks

Benchmarks: We consider a maze navigation environment where the task objective is compatible with Definition 1. Our benchmarks include two discrete domain tasks from the Mini-Grid environments Chevalier-Boisvert et al. [2023]: DoorKey-6x6-v0 , and SimpleCrossingS9N1-v0 . In both tasks, the agent needs to interact with the environmental objects which are randomly positioned in every episode while the agent can only observe a small, unblocked area in front of it . The default reward, which is always zero unless the agent reaches the target, is used to evaluate the performance of learned policies. Due to partial observability and the implicit hierarchical nature of the task, these environments are considered challenging for RL and IL, and have been extensively used for benchmarking curriculum RL and exploration-driven RL.

Baselines: We compare our approach with two standard baselines: GAIL Ho and Ermon [2016] and VAIL Peng et al. [2019]. GAIL has been introduced in Section 3. VAIL is based on GAIL but additionally optimizes a variational discriminator bottleneck (VDB) objective. Our approach uses the IRL techniques behind those two baseline algorithms, resulting in two versions of Algorithm 1, denoted as PAGAR-GAIL and PAGAR-VAIL, respectively. More specifically, if the baseline optimizes a J IRL objective, we use the same J IRL objective in Algorithm 1. Also, we extract the reward function r from the discriminator D as mentioned in Section 3. More details are in Appendix C.1. PPO Schulman et al. [2017] is used for policy training in GAIL, VAIL, and ours with a replay buffer of size 2048 . Additionally, we compare our algorithm with a state-of-the-art (SOTA) IL algorithm, IQ-Learn Garg et al. [2021], which, however, is not compatible with our algorithm because it does not explicitly optimize a reward function. The policy and the reward functions are all approximated using convolutional networks.

(a) DoorKey-6x6

(10 demos)

(b) DoorKey-6x6

(1 demo)

(c) SimpleCrossingS9N1

(10 demos)

(d) SimpleCrossingS9N1

(1 demo)

(e) Transfer Envs

(f) SimpleCrossingS9N2

(g) SimpleCrossingS9N3

(h) FourRooms

Average Return Per Episode

1.0

0.8

0.6

VAIL

IQ

PAGAR-VAIL

PAGAR-GAIL

Average Return Per Episode

1.0

0.8

0.6

VAIL

PAGAR-VAIL

Average Return Per Episode

0.8

0.6

1.0

IQ

VAIL

PAGAR-VAIL

PAGAR-GAIL

Average Return Per Episode

0.8

0.6

VAIL

PAGAR-VAIL

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

0.0

100000

200000

Frames

300000

400000

00000S

000009

0'0

Z'0

0.4

Frames

0.6

0.8

1.0

1e6

0

100000

200000

Frames

300000

400000

0.0

0.5

Frames

1.0

1.5

2.0

1e6

Average Return Per Episode

1.0

PAGAR-GAIL

Average Return Per Episode

PAGAR-GAIL

Average Return Per Episode

PAGAR-GAIL

PAGAR-VAIL

0.8

PAGAR-VAIL

0.4

PAGAR-VAIL

0.8

0.6

IQ

VAIL

0.6

VAIL

IQ

0.3

VAIL

IQ

0.4

0.4

0.2

0.2

0.2

0.1

0.0

0.0

0.0

0.00

0.50

0.75

Frames

1.00

1.25

1.50

1.75

2.00

1e6

0.00

0.25

0.50

Frames

1.50

1.75

2.00

1e6

0.0

0.5

1.0

Frames

1.5

2.0

2.5

3.0

1e6

IL with Limited Demonstrations. By learning from 10 expert-demonstrated trajectories with high returns, PAGAR-based IL produces high-performance policies with high sample efficiencies as shown in Figure 2(a) and (c). Furthermore, we compare PAGAR-VAIL with VAIL by reducing the number of demonstrations from 10 to 1 . As shown in Figure 2(b) and (d), PAGAR-VAIL produces high-performance policies with significantly higher sample efficiencies.

IL under Dynamics Mismatch. We demonstrate that PAGAR enables the agent to infer and accomplish the objective of a task even in environments that are substantially different from the one observed during expert demonstrations. As shown in Figure 2(e), we collect 10 expert demonstrations from the SimpleCrossingS9N1-v0 environment. Then we apply Algorithm 1 and the baselines, GAIL, VAIL, and IQ-learn to learn policies in SimpleCrossingS9N2-v0, SimpleCrossingS9N3-v0 and FourRooms-v0 . The results in Figure 2(f)-(g) show that PAGAR-based IL outperforms the baselines in these challenging zero-shot settings.

Average Return Per Episode

1.0

PAGAR-GAIL(Sigmoid)

PAGAR-GAIL(Categ)

GAIL(Sigmoid)

0.8

GAIL(Categ)

0.6

0.4

0.2

0.0

O

100000

200000

300000

400000

500000

̸

IL with Different Reward Hypothesis Sets . The foundational theories of GAIL and AIRL indicate that different reward function hypothesis sets can affect the equilibrium of their GAN frameworks. We study whether choosing different reward hypothesis sets can influence the performance of Algorithm 1. We compare using a Sigmoid function with a Categorical distribution in the output layer of the discriminator networks in GAIL and PAGAR-GAIL. When using the Sigmoid function, the outputs of D are not normalized, i.e., ∑ a ∈ A D ( s, a ) = 1 . When using a Categorical distribution, ∑ a ∈ A D ( s, a ) = 1 . We test GAIL and PAGAR-GAIL in DoorKey-6x6-v0 environment. As shown in Figure 3, PAGAR-GAIL outperforms GAIL in both cases by using fewer samples.

(a) Ant

(b) Hopper

(c) Walker2d

(d) HalfCheetah

6000

14000

6000

5000

12000

3000

5000

Return

4000

10000

2500

4000

8000

2000

3000

3000

6000

1500

2000

PAGAR-fIRL

4000

1000

2000

1000

fIRL

2000

500

1000

0.000.25 0.500.75 1.001.25 1.501.75 2.00

0

0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0

0.2

0.4

0.6

0.8

1.0

0

0:0

0.5

1.0

1.5

2.0

2.5

3.0

Steps

1e6

1e6

1e6

1e6

7.2 Continuous Control Tasks

We evaluate PAGAR-based IL on continuous control tasks in both online and offline RL settings, demonstrating its ability to improve IRL-based IL performance across different types of tasks.

Benchmarks: We use four continuous control environments from Mujoco. In the online RL setting, both protagonist and antagonist policies are permitted to explore the environment. In the offline RL setting, exploration by these policies is restricted. Especially, for offline RL we use the D4RL's 'expert' datasets as the expert demonstrations and the 'random' datasets as the offline suboptimal dataset. Policy performance is evaluated online in both settings by using the default reward function of the environment.

Baselines: We compare PAGAR-based IL against f-IRL Ni et al. [2021] in the online RL setting and compare with RECOIL Sikchi et al. [2024] in the offline RL setting. When comparing with f-IRL, we use f-IRL as the IRL algorithm in Algorithm 1. When comparing with RECOIL, as RECOIL does not directly learn the reward function but learns the Q and V functions, we develop another algorithm for the offline RL setting to combine PAGAR with RECOIL by explicitly learning a reward function and using the reward function and V function to represent the Q function. The details can be found in Appendix B.4.

Results: As shown in Figure 4, PAGAR-based IL achieves equivalent performance to the baselines with fewer iterations. Furthermore, on the Ant and Walker2d tasks, Algorithm 1 matches the performance level of f-IRL using significantly less iterations. Additional results of PAGAR with GAIL and VAIL across other continuous control benchmarks are provided in Appendix C.3. Table 1 further shows that when combined with RECOIL, PAGAR-based IL achieves higher performance in most of the tasks than the baseline. These results demonstrate the broader applicability of PAGARbased IL in both online and offline settings and its effectiveness across different types of environments, further reinforcing the robustness of our approach.

8 Conclusion

In this paper, we propose to prioritize task alignment over conventional data alignment in IRL-based IL by treating expert demonstrations as weak supervision signals to derive a set of candidate reward functions that align with the task rather than only with the data. Our PAGAR-based IL adopts an adversarial mechanism to train a policy with this set of reward functions. Experimental results demonstrate that our algorithm can mitigate reward misalignment in challenging environments. Our future work will focus on employing the PAGAR paradigm to other task alignment problems.

Acknowledgment

This work was supported in part by the U.S. National Science Foundation under grant CCF-2340776.

References

Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-First International Conference on Machine Learning , ICML '04, page 1, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581138385. doi: 10.1145/1015330. 1015430.

Abel, W. Dabney, A. Harutyunyan, M. K. Ho, M. Littman, D. Precup, and S. Singh. On the expressivity of markov reward. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in AI safety. CoRR , abs/1606.06565, 2016.

Cao, S. Cohen, and L. Szpruch. Identifiability in inverse reinforcement learning. Advances in Neural Information Processing Systems , 34:12362-12373, 2021.

Chevalier-Boisvert, B. Dai, M. Towers, R. de Lazcano, L. Willems, S. Lahlou, S. Pal, P. S. Castro, and J. Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR , abs/2306.13831, 2023.

Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 13049-13061. Curran Associates, Inc., 2020.

Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning , pages 49-58. PMLR, 2016.

Finn, T. Yu, J. Fu, P. Abbeel, and S. Levine. Generalizing skills with semi-supervised reinforcement learning. In International Conference on Learning Representations , 2017.

Fu, K. Luo, and S. Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations , 2018.

Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. IQ-learn: Inverse soft-q learning for imitation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Hadfield-Menell, S. Milli, P. Abbeel, S. J. Russell, and A. Dragan. Inverse reward design. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Hejna and D. Sadigh. Inverse preference learning: Preference-based RL without a reward function. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https: //openreview.net/forum?id=gAP52Z2dar .

Ho and S. Ermon. Generative adversarial imitation learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

Hong, K. Bhatia, and A. Dragan. On the sensitivity of reward inference to misspecified human models. In The Eleventh International Conference on Learning Representations , 2023.

Jeon, S. Seo, and K.-E. Kim. A bayesian approach to generative adversarial imitation learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Lindner, A. Krause, and G. Ramponi. Active exploration for inverse reinforcement learning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

M. Metelli, G. Ramponi, A. Concetti, and M. Restelli. Provably efficient learning of transferable rewards. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 7665-7676. PMLR, 18-24 Jul 2021.

Y. Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning , ICML '00, pages 663-670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1-55860-707-2.

Y. Ng, D. Harada, and S. J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning , ICML '99, pages 278-287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1-55860-612-2.

Ni, H. Sikchi, Y. Wang, T. Gupta, L. Lee, and B. Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference on Robot Learning , pages 529-551. PMLR, 2021.

Pan, K. Bhatia, and J. Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=JYtwGwIL7ye .

B. Peng, A. Kanazawa, S. Toyer, P. Abbeel, and S. Levine. Variational discriminator bottleneck: Improving imitation learning, inverse RL, and GANs by constraining information flow. In International Conference on Learning Representations , 2019.

Ramachandran and E. Amir. Bayesian inverse reinforcement learning. In Proceedings of the 20th International Joint Conference on Artifical Intelligence , IJCAI'07, page 2586-2591, San Francisco, CA, USA, 2007. Morgan Kaufmann Publishers Inc.

D. Ratliff, J. A. Bagnell, and M. A. Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference on Machine learning , pages 729-736, 2006.

Ross and D. Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 661-668, 2010.

Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635, 2011.

Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017.

Sikchi, Q. Zheng, A. Zhang, and S. Niekum. Dual RL: Unification and new methods for reinforcement and imitation learning. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=xt9Bu66rqv .

Skalse and A. Abate. Misspecification in inverse reinforcement learning. arXiv preprint arXiv:2212.03201 , 2022.

Skalse, M. Farrugia-Roberts, S. Russell, A. Abate, and A. Gleave. Invariance in policy optimisation and partial identifiability in reward learning. arXiv preprint arXiv:2203.07475 , 2022a.

M. V. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger. Defining and characterizing reward gaming. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022b.

S. Sutton and A. G. Barto. Reinforcement learning: An introduction . 2018.

Viano, Y.-T. Huang, P. Kamalaruban, A. Weller, and V. Cevher. Robust inverse reinforcement learning under transition dynamics mismatch. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https: //openreview.net/forum?id=t8HduwpoQQv .

Zhou and W. Li. Safety-aware apprenticeship learning. In H. Chockler and G. Weissenbacher, editors, Computer Aided Verification , pages 662-680, Cham, 2018. Springer International Publishing. ISBN 978-3-319-96145-3.

Zhou and W. Li. Programmatic reward design by example. Proceedings of the AAAI Conference on Artificial Intelligence , 36(8):9233-9241, Jun. 2022a. doi: 10.1609/aaai.v36i8.20910.

Zhou and W. Li. A hierarchical Bayesian approach to inverse reinforcement learning with symbolic reward machines. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 27159-27178. PMLR, 17-23 Jul 2022b.

Zhou, R. Gao, B. Kim, E. Kang, and W. Li. Runtime-safety-guided policy repair. In Runtime Verification: 20th International Conference, RV 2020, Los Angeles, CA, USA, October 6-9, 2020, Proceedings 20 , pages 131-150. Springer, 2020.

D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3 , AAAI'08, pages 1433-1438. AAAI Press, 2008. ISBN 978-1-57735-368-3.

A Reward Design with PAGAR

This paper does not aim to resolve the ambiguity problem in IRL but provides a way to circumvent it so that reward ambiguity does not lead to reward misalignment in IRL-based IL. PAGAR, the semisupervised reward design paradigm proposed in this paper, tackles this problem from the perspective of semi-supervised reward design. But the nature of PAGAR is distinct from IRL and IL: assume that a set of reward functions is available for some underlying task, where some of those reward functions align with the task while others are misaligned, PAGAR provides a solution for selecting reward functions to train a policy that successfully performs the task, without knowing which reward function aligns with the task. Our research demonstrates that policy training with PAGAR is equivalent to learning a policy to maximize an affine combination of utilities measured under a distribution of the reward functions in the reward function set. With this understanding of PAGAR, we integrate it with IL to illustrate its advantages.

A.1 Motivation: Failures in IRL-Based IL

For readers' convenience, we put the theorem from Abel et al. [2021] here for reference.

Theorem 5. Viano et al. [2021] If the demonstration environment has a dynamics function P demo different from the dynamics function P in the learning environment, the performance gap between the policies π E and π r ∗ under the ground true reward function r E satisfies | U r E ( π E ) -U r E ( π r ∗ ) | ≤ 2 · γ · max s,a | r E ( s,a ) | (1 -γ ) 2 · max s,a D TV ( P ( · | s, a ) , P demo ( · | s, a )) .

Then, we prove that a limited number of demonstrations can also lead to the performance gap as described in Theorem 5. Essentially, we can construct a demonstration dynamics P demo such that all the state transition probability estimated in E match P demo with zero error. Then a similar performance gap as that caused by dynamics mismatch can be derived.

Proposition 3. Assume that the expert demonstration contains m state-action pairs by only selecting the optimal actions, i.e., select arg max a π E ( a | s ) at each s . Then | U r E ( π E ) -U r E ( π r ∗ ) | ≤

Proof. We translate the limited demonstration case into a dynamics mismatch case where we will leverage the demonstrated state transitions to construct a dynamics P demo ( · | s, a ) and compare it with P ( · | s, a ) .

Assume that for each state-action pair s, a in E , i.e., m s,a ∈ [ m ] , the m s,a transition instances are ( s, a, s 1 ) , ( s, a, s 2 ) , . . . , ( s, a, s m s,a ) . For each ˆ s ∈ S , we let P demo (ˆ s | s, a ) = 1 m s,a ∑ m s,a i =1 I [ s i = ˆ s ] .

̸

Then, we can view D TV ( P ( · | s, a ) , P demo ( · | s, a )) = 1 2 ||P ( · | s, a ) - P demo ( · | s, a )) || 1 as a function of m independent random variables, i.e., ( s 1 , a 1 ) , ( s 2 , a 2 ) , . . . , ( s m s,a ,a ms,a ) , sampled from the S domain. If one of the variables, s i , is changed from ˆ s to another state ˆ s ′ = ˆ s , P demo (ˆ s | s, a ) and P demo (ˆ s ′ | s, a ) should decrease and increase by 1 m s,a respectively. Then, D TV ( P ( · | s, a ) , P demo ( · | s, a )) changes at most by 1 m s,a .

We can also view the D TV at each ( s, a ) ∈ E as functions of the m sampled state-action pairs in E , with a little abuse of notations, denoted as ( s 1 , a 1 ) , ( s 2 , a 2 ) , . . . , ( s m , a m ) by flattening all the demonstrated trajectories. If a state-action pair, ( s i , a i ) , in one of the trajectories, is changed from (ˆ s, ˆ a ) to another state (ˆ s ′ , ˆ a ′ ) , then for the state-action ( s i -1 , a i -1 ) that precedes s i if s i is not the initial state, D TV ( P ( · | s, a ) , P demo ( · | s, a )) changes at most by 1 m s i -1 ,a i -1 which can be as large as 1 since it is possible that m s i -1 ,a i -1 = 1 . Also, D TV ( P ( · | ˆ s, ˆ a ) , P demo ( · | ˆ s, ˆ a )) and D TV ( P ( · | ˆ s ′ , ˆ a ′ ) , P demo ( · | ˆ s ′ , ˆ a ′ )) change at most by 1 m ˆ s, ˆ a and 1 m ˆ s ′ , ˆ a ′ . Note that if m ˆ s ′ , ˆ a ′ = 0 , i.e., (ˆ s ′ , ˆ a ′ ) / ∈ E , changing ˆ s, ˆ a into ˆ s ′ , ˆ a ′ equivalently leads to taking D TV ( P ( · | ˆ s, ˆ a, P demno ( · | ˆ s ′ , ˆ a ′ )) into consideration when computing

max s,a D TV ( P ( · | s, a ) , P demo ( · | s, a )) . And D TV ( P ( · | ˆ s, ˆ a, P demno ( · | ˆ s ′ , ˆ a ′ )) ≤ 1 -P ( s i +1 | ˆ s ′ , ˆ a ′ )) 2 where s i +1 is the state that succeeds s i in the demonstrated trajectroy. Therefore, if changing a state-action pair in E into another state-action pair, max ( s,a ) ∈ E D TV ( P ( · | s, a ) , P demo ( · | s, a )) can be changed as large as by 1 .

According

E

D

(

s,a

)

E

∈

Note that

McDiarmid's

TV

(

P

E

(

s, a

)

,

·

|

P

D

TV

(

P

(

· |

Inequality, demo

(

Prob s, a

·

|

s, a

)

,

P

demo

(

(

))]

|

· |

(

)

E

∈

ϵ

)

s, a

))] = 0

|

∑

s, a

)

·

2

ϵ

2

=1

i m s, a

|

exp(

))

-

2

ϵ

-

2

(1)

m

2

since the estimation is un-biased. Hence,

̸

a s a . Meanwhile, for a ′ = arg max a π E ( a | s ) , we consider the best-case Q-value, i.e., Q demo π E ( s, a ′ ) ≤ 1 1 -γ · max ( s,a ) r E ( s, a ) . We add a dummy, absorbing state s that is unreachable from any state-action under the dynamics P and also unreachable from any demonstrated state-action under the constructed dynamics P demo . But we let P demo ( s | s, a ′ ) > 0 if Q demo π E ( s, a ′ ) > Q demo π E ( s, a ∗ ) . In other words, we add probability density on transitioning from such ( s, a ′ ) to s so that Q demo π E ( s, a ′ ) drops below Q demo π E ( s, a ∗ ) . We denote this density as p . Then P demo ( s | s, a ′ ) = p and P demo ( s ′ | s, a ′ ) = (1 -p ) · P ( s ′ | s, a ′ ) for any other s ′ ∈ S . As a result, D TV ( P ( · | s, a ′ ) , P demo ( · | s, a ′ ) = p . Note that adding s does not affect the Q-values of the state-action pairs that appear in E . Then we aim to find the p to ensure Q demo π E ( s, a ′ ) ≤ Q demo π E ( s, a ∗ ) . Considering the worst-case, p · (max s,a r E ( s, a ) +

For state-action pairs that are outside of E , we need to construct P demo at those state-action pairs such that the action selection in E is always optimal. To start with, we let P demo ( · | a, s ) ≡ P ( · | a, s ) for any state-action pairs that do not appear in E . For some state-action pairs, the states are not reached in E . We can disregard whether π E maintains optimal at those states and focus on the states that do appear in E . By denoting the Q-value of π E under P demo as Q demo π E , we need to make sure that the Q demo π E ( s, a ′ ) for a ′ = arg max a π E ( a | s ) is no greater than Q demo π E ( s, arg max a π E ( a | s )) . Since the state-actions in E may have extremely low rewards stochastic nature of P , we consider the worst case Q-value for a ∗ = arg max π E ( a | s ) i.e., Q demo π E ( s, a ∗ ) ≥ 1 1 -γ · min r E ( s, arg max π E ( a | s ))

̸

Combining the analysis on the state-action pairs that appear in E and not in E , we can conclude that for ϵ > p , we can extend the confidence bound Prob ( max ( s,a ) ∈ E D TV ( P ( · | s, a ) , P demo ( · | s, a )) ≥

ϵ ) ≤ 2 · exp( -2 ϵ 2 m ) for ( s, a ) ∈ E from ( s, a ) ∈ E to all state-action pairs in S × A . By using a variable d to represent max s,a D TV ( P ( · | s, a ) -P demo ( · | s, a )) , we can use the conclusion drawn from Theorem 5. Note that adding the dummy s does not affect the max s,a | r E ( s, a ) | since we let r E ( s, a ) ≡ min r E ( s, a ) . Our proof is complete.

s,a

A.2 Task-Reward Alignment

In this section, we provide proof of the properties of the task-reward alignment concept that we define in the main text. For readers' convenience, we include our definition of task for reference.

Definition 1 ( Task ) Given the policy hypothesis set Π , a task (Π , ⪯ task , Π acc ) is specified by a partial order over Π and a non-empty set of acceptable policies Π acc ⊆ Π such that ∀ π 1 ∈ Π acc and ∀ π 2 / ∈ Π acc , π 2 ⪯ task π 1 always hold.

We have defined in the main text that U r := min π ∈ Π acc U r ( π ) is the lowest utility achieved by any acceptable policies under r , and U r := max π ∈ Π U r ( π ) s.t. ∀ π 1 , π 2 ∈ Π , U r ( π 1 ) ≤ U r ( π ) ≤ U r ( π 2 ) ⇒ ( π 1 ⪯ task π ) ∧ ( π 1 ⪯ task π 2 ) is the highest utility threshold such that any policy achieving a utility

)

.

D

2

TV

·

(

P

exp(

(

·

-

,

P

demo

(

) = 2

·

·

higher than U r also has a higher order than those of which utilities are lower than U r . We call a reward function r a ( U r , U r ) -aligned with the task.

Lemma 1. For any task-aligned reward function r + , U r + ≥ U r + .

Proof. If U r + < U r + , then for any policy π with U r ( π ) ∈ [ U r + , U r + ) , it is guaranteed that π ∈ Π \ Π acc by definition of U r + . Then, U r + is a better solution for max π ∈ Π U r ( π ) s.t. ∀ π 1 , π 2 ∈ Π , U r ( π 1 ) ≤ U r ( π ) ≤ U r ( π 2 ) ⇒ ( π 1 ⪯ task π ) ∧ ( π 1 ⪯ task π 2 ) than U r + . Hence, U r + ≥ U r + must hold.

Lemma 2. If π 1 ⪯ task π 2 ⇔ U r ( π 2 ) ≤ U r ( π 1 ) holds for any π 1 , π 2 ∈ Π , then U r = U r = min π ∈ Π U r ( π ) .

Proof. Since the policy that has the highest order achieves the lowest utility min π ∈ Π U r ( π ) , no other acceptable policy can achieve even lower utility. Hence, by definition U r = min π ∈ Π U r ( π ) .

Another example is that when optimal policy under r has the lowest order in terms of ⪯ task , and the highest-order policy has the lowest utility under r , U r = U r . In those extreme cases, the reward function can be considered completely misaligned. In the non-extreme cases, if a reward function r exhibits misalignment with the task - the threshold U r is lower than the utility of an unacceptable policy U r then must be also lower than the utilities of all the acceptable policies, which forms a lower-bound for the size of U r .

Lemma 3. For any reward function r , if ∃ π / ∈ Π acc , U r ( π ) ≥ U r , then U r ≤ U r .

Proof. Since π / ∈ Π acc and U r ( π ) ≥ U r , then ∀ π ′ ∈ Π acc , U r ( π ′ ) ≥ U r must be true, otherwise ∃ π ′ ∈ Π acc , π ′ ⪯ task π , which contradicts the definition of Π acc and U r .

Lemma 4. If a reward function r has U r ≤ U r , r is a task-misaligned reward function.

Proof. If U r ≤ U r , then there must exists two policies π 1 , π 2 where π 1 ∈ Π \ Π acc , U r ( π 1 ) ∈ [ U r , U r ] and U r ( π 2 ) ≥ U r ∧ π 2 ⪯ task π 1 . Such π 2 must not be an acceptable policy. Otherwise, it contradicts the definition of Π acc . Hence, it must be unacceptable, leading to r being a task-misaligned reward function.

Proposition 1 For any two reward functions r 1 , r 2 , if { π : U r 1 ( π ) ≥ U r 1 } ⊆ { π : U r 2 ( π ) ≥ U r 2 } , then there must exist a π 1 ∈ { π : U r 1 ( π ) ≥ U r 1 } and a π 2 ∈ { π : U r 2 ( π ) ≥ U r 2 } that satisfy U r 1 ( π 2 ) ≤ U r 1 ( π 1 ) and π 2 ⪯ task π 1 while U r 2 ( π 1 ) ≤ U r 2 ( π 2 ) .

Proof. According to the definition of U r 1 , ∀ π 1 ∈ { π : U r 2 ( π ) ≥ U r 1 } and ∀ π 2 ∈ { π : U r 2 ( π ) ≥ U r 2 } / { π : U r 1 ( π ) ≥ U r 1 } , U r 1 ( π 2 ) ≤ U r 1 ( π 1 ) and π 2 ⪯ task π 1 must be true. Furthermore, if for all pairs of π 2 ∈ { π : U r 2 ( π ) ≥ U r 2 } / { π : U r 1 ( π ) ≥ U r 1 } and π 1 ∈ { π : U r 1 ( π ) ≥ U r 1 } , U r 2 ( π 1 ) > U r 2 ( π 2 ) is true, then { U r 2 ( π ) | U r 1 ( π ) ≥ U r 1 } ⊂ [ U r 2 , max π ∈ Π U r 2 ( π )] is a smaller non-empty interval than [ U r 2 , max π ∈ Π U r 2 ( π )] , contradicting the fact that U r 2 is the highest utility threshold under r 2 . Hence, there must exist a π 2 ∈ { π : U r 2 ( π ) ≥ U r 2 } / { π : U r 1 ( π ) ≥ U r 1 } and a π 1 ∈ { π : U r 1 ( π ) ≥ U r 1 } that satisfy U r 1 ( π 2 ) ≤ U r 1 ( π 1 ) and U r 2 ( π 1 ) ≤ U r 2 ( π 2 ) while π 2 ⪯ task π 1 as aforementioned.

Note that from now on, we use the notation r + to denote task-aligned reward functions and r -to denote task-misaligned reward functions for short. Furthermore, if a reward function r satisfies U r ( π 1 ) ≤ U r ( π 2 ) ⇔ π 1 ⪯ task π 2 , we call this reward function the ground true reward function , and denote it as r task for simplicity. Apparently, any r task is a task-aligned reward function, and it has the most trivial misalignment.

Proof. By definition, arg max U r task ( π ) has higher order and higher utility than any other policies.

π ∈ Π

Lemma 6. If π r task is optimal under r task , for any reward function r , U r ( π r task ) ≥ U r .

Proof. If U r (Π acc ) < U r , then there exists a π with its utility U r ( π ) ≥ U r , thus π r task ⪯ task π , which contradicts the assumption that π r task is the highest-order policy.

Furthermore, we have the following property.

Theorem 1 Let I be an indicator function. For any k ≥ { min r + ∑ π ∈ Π I{ U r + ( π ) ≥ U r + ( π E ) } } , if π ∗ satisfies {∑ π ∈ Π I{ U r ( π ) ≥ U r ( π ∗ ) } } < | Π acc | for all r ∈ R E,k := { r | ∑ π ∈ Π I{ U r ( π ) ≥ U r ( π E ) } ≤ k } , then π ∗ is an acceptable policy, i.e., π ∗ ∈ Π acc . Additionally, if k < | Π acc | , such an acceptable policy π ∗ is guaranteed to exist.

Proof. Since π E is an acceptable policy, there must be at least one task-aligned reward function r + that satisfies |{ π : U r ( π ) ≥ U r ( π E ) }| ≤ k since k ≥ min r + |{ π : U r + ( π ) ≥ U r + ( π E ) }| . The greater k is, the more task-aligned reward functions tend to be included. If π ∗ achieves |{ π : U r + ( π ) ≥ U r + ( π ∗ ) } < | Π acc | under any task-aligned reward function r + , π ∗ must be acceptable policy. Because if π ∗ is unacceptable, there must be an acceptable policy performing worse than the unacceptable π ∗ under r + , contradicting the definition of task-aligned reward function . Hence, π ∗ must be acceptable policy. Furthermore, for any k ∈ [min r + ∑ π ∈ Π I{ U r + ( π ) ≥ U r + ( π E ) } , | Π acc | ) ,

the policy π E itself satisfied ∑ π ∈ Π I{ U r ( π ) ≥ U r ( π E ) } < | Π acc | for all r ∈ R E,k , which guarantees the existence of a feasible π ∗ .

A.3 Semi-supervised Reward Design

Designing a reward function can be thought as deciding an ordering of policies. We adopt a concept, called total domination , from unsupervised environment design Dennis et al. [2020], and re-interpret this concept in the context of reward design. In this paper, we suppose that the function U r ( π ) is given to measure the performance of a policy and it does not have to be the utility function. While the measurement of policy performance can vary depending on the free variable r , total dominance can be viewed as an invariance regardless of such dependency.

Definition 5 (Total Domination) . A policy, π 1 , is totally dominated by some policy π 2 w.r.t a reward function set R , if for every pair of reward functions r 1 , r 2 ∈ R , U r 1 ( π 1 ) < U r 2 ( π 2 ) .

If π 1 totally dominate π 2 w.r.t R , π 2 can be regarded as being unconditionally better than π 1 . In other words, the two sets { U r ( π 1 ) | r ∈ R } and { U r ( π 2 ) | r ∈ R } are disjoint, such that sup { U r ( π 1 ) | r ∈ R } < inf { U r ( π 2 ) | r ∈ R } . Conversely, if a policy π is not totally dominated by any other policy, it indicates that for any other policy, say π 2 , sup { U r ( π 1 ) | r ∈ R } ≥ inf { U r ( π 2 ) | r ∈ R } .

Definition 6. Areward function set R aligns with an ordering ≺ R among policies such that π 1 ≺ R π 2 if and only if π 1 is totally dominated by π 2 w.r.t. R .

Especially, designing a reward function r is to establish an ordering ≺ { r } among policies. Total domination can be extended to policy-conditioned reward design, where the reward function r is selected by following a decision rule ω ( π ) such that ∑ r ∈ R ω ( π )( r ) = 1 . We let U ω ( π ) = ∑ r ∈ R ω ( π )( r ) · U r ( π ) be an affine combination of U r ( π ) 's with its coefficients specified by ω ( π ) .

Definition 7. A policy conditioned decision rule ω is said to prefer a policy π 1 to another policy π 2 , which is notated as π 1 ≺ ω π 2 , if and only if U ω ( π 1 ) < U ω ( π 2 ) .

Making a decision rule for selecting reward functions from a reward function set to respect the total dominance w.r.t this reward function set is an unsupervised learning problem, where no additional external supervision is provided. If considering expert demonstrations as a form of supervision

and using it to constrain the set R E of reward function via IRL, the reward design becomes semisupervised.

A.4 Solution to the MinimaxRegret

Without loss of generality, we use R instead of R E,δ in our subsequent analysis because solving MinimaxRegret ( R ) does not depend on whether there are constraints for R . In order to show such an equivalence, we follow the same routine as in Dennis et al. [2020], and start by introducing the concept of weakly total domination .

Definition 8 (Weakly Total Domination) . A policy π 1 is weakly totally dominated w.r.t a reward function set R by some policy π 2 if and only if for any pair of reward function r 1 , r 2 ∈ R , U r 1 ( π 1 ) ≤ U r 2 ( π 2 ) .

Note that a policy π being totally dominated by any other policy is a sufficient but not necessary condition for π being weakly totally dominated by some other policy. A policy π 1 being weakly totally dominated by a policy π 2 implies that sup { U r ( π 1 ) | r ∈ R } ≤ inf { U r ( π 2 ) | r ∈ R } . We assume that there does not exist a policy π that weakly totally dominates itself, which could happen if and only if U r ( π ) is a constant. We formalize this assumption as the following.

Assumption 1. For the given reward set R and policy set Π , there does not exist a policy π such that for any two reward functions r 1 , r 2 ∈ R , U r 1 ( π ) = U r 2 ( π ) .

This assumption makes weak total domination a non-reflexive relation. It is obvious that weak total domination is transitive and asymmetric. Now we show that successive weak total domination will lead to total domination.

Lemma 7. for any three policies π 1 , π 2 , π 3 ∈ Π , if π 1 is weakly totally dominated by π 2 , π 2 is weakly totally dominated by π 3 , then π 3 totally dominates π 1 .

Proof. According to the definition of weak total domination, max r ∈ R U r ( π 1 ) ≤ min r ∈ R U r ( π 2 ) and max r ∈ R U r ( π 2 ) ≤ min r ∈ R U r ( π 3 ) . If π 1 is weakly totally dominated but not totally dominated by π 3 , then max r ∈ R U r ( π 1 ) = min r ∈ R U r ( π 3 ) must be true. However, it implies min r ∈ R U r ( π 2 ) = max r ∈ R U r ( π 2 ) , which violates Assumption 1. We finish the proof.

Lemma 8. For the set Π ¬ wtd ⊆ Π of policies that are not weakly totally dominated by any other policy in the whole set of policies w.r.t a reward function set R , there exists a range U ⊆ R such that for any policy π ∈ Π ¬ wtd , U ⊆ [min r ∈ R U r ( π ) , max r ∈ R U r ( π )] .

Proof. For any two policies π 1 , π 2 ∈ Π ¬ wtd , it cannot be true that max r ∈ R U r ( π 1 ) = min r ∈ R U r ( π 2 ) nor min r ∈ R U r ( π 1 ) = max r ∈ R U r ( π 2 ) , because otherwise one of the policies weakly totally dominates the other. Without loss of generalization, we assume that max r ∈ R U r ( π 1 ) > min r ∈ R U r ( π 2 ) . In this case, max r ∈ R U r ( π 2 ) > min r ∈ R U r ( π 1 ) must also be true, otherwise π 1 weakly totally dominates π 2 . Inductively, min π ∈ Π ¬ wtd max r ∈ R U r ( π ) > max π ∈ Π ¬ wtd min r ∈ R U r ( π ) . Letting ub = min π ∈ Π ¬ wtd max r ∈ R U r ( π ) and lb = max π ∈ Π ¬ wtd min r ∈ R U r ( π ) , any U ⊆ [ lb, ub ] shall support the assertion. We finish the proof.

Lemma 9. For a reward function set R , if a policy π ∈ Π is weakly totally dominated by some other policy in Π and there exists a subset Π ¬ wtd ⊆ Π of policies that are not weakly totally dominated by any other policy in π , then max r ∈ R U r ( π ) < min π ′ ∈ Π ¬ wtd max r ∈ R U r ( π ′ )

Proof. If π 1 is weakly totally dominated by a policy π 2 ∈ Π , then min r ∈ R U r ( π 2 ) = max r ∈ R U r ( π ) . If max r ∈ R U r ( π ) ≥ min π ′ ∈ Π ¬ wtd max r ∈ R U r ( π ′ ) , then min r ∈ R U r ( π 2 ) ≥ min π ′ ∈ Π ¬ wtd max r ∈ R U r ( π ′ ) , making at least one of the policies in Π ¬ wtd being weakly totally dominated by π 2 . Hence, max r ∈ R U r ( π ) < min π ′ ∈ Π ¬ wtd max r ∈ R U r ( π ′ ) must be true.

Given a policy π and a reward function r , the regret is represented as Eq.8

Then we represent the MinimaxRegret ( R ) problem in Eq.9.

We denote as r ∗ π ∈ R the reward function that maximizes U r ( π ) among all the r 's that achieve the maximization in Eq.9. Formally,

Then MinimaxRegret can be defined as minimizing the worst-case regret as in Eq.9. Next, we want to show that for some decision rule ω , the set of optimal policies that maximize U ω are the solutions to MinimaxRegret ( R ) . Formally,

We design ω by letting ω ( π ) := ω ( π ) · δ r ∗ π + (1 -ω ( π )) · R π where R π ∈ ∆( R ) is a policy conditioned distribution over reward functions, δ r ∗ π be a delta distribution centered at r ∗ π , and ω ( π ) is a coefficient. We show how to design R by using the following lemma.

Lemma 10. Given that the reward function set is R , there exists a decision rule R : Π → ∆( R ) which guarantees that: 1) for any policy π that is not weakly totally dominated by any other policy in Π , i.e., π ∈ Π ¬ wtd ⊆ Π , U R ( π ) ≡ c where c = max π ′ ∈ Π ¬ wtd min r ∈ R U r ( π ′ ) ; 2) for any π that is weakly totally dominated by some policy but not totally dominated by any policy, U R ( π ) = max r ∈ R U r ( π ) ; 3) if π is totally dominated by some other policy, ω ( π ) is a uniform distribution.

Proof. Since the description of R for the policies in condition 2) and 3) are self-explanatory, we omit the discussion on them. For the none weakly totally dominated policies in condition 1), having a constant U R ( π ) ≡ c is possible if and only if for any policy π ∈ Π ¬ wed , c ∈ [min U r ( π ′ ) , max U r ( π ′ )] .

r ∈ R r ∈ R As mentioned in the proof of Lemma 8, c can exist within [min r ∈ R U r ( π ) , max r ∈ R U r ( π )] . Hence,

Then by letting ω ( π ) := Regret ( π,r ∗ π ) c -U r ∗ π ( π ) , we have the following theorem.

Theorem 6. By letting ω ( π ) := ω ( π ) · δ r ∗ π +(1 -ω ( π )) · R π with ω ( π ) := Regret ( π,r ∗ π ) c -U r ∗ π ( π ) and any R that satisfies Lemma 10,

Proof. If a policy π ∈ Π is totally dominated by some other policy, since there exists another policy with larger U ω , π cannot be a solution to arg max π ∈ Π U ω ( π ) . Hence, there is no need for further discussion on totally dominated policies. We discuss the none weakly totally dominated policies and the weakly totally dominated but not totally dominated policies (shortened to "weakly totally

dominated" from now on) respectively. First we expand arg max π ∈ Π U ω ( π ) as in Eq.13.

1) For the none weakly totally dominated policies, since by design U R ≡ c , Eq.13 is equivalent to arg max π ∈ Π 1 -Regret ( π, r ∗ π ) which exactly equals MinimaxRegret ( R ) . Hence, the equivalence holds among the none weakly totally dominated policies. Furthermore, if a none weakly totally dominated policy π ∈ Π ¬ wtd achieves optimality in MinimaxRegret ( R ) , its U ω ( π ) is also no less than any weakly totally dominated policy. Because according to Lemma 9, for any weakly totally dominated policy π 1 , its U R ( π 1 ) ≤ c , hence U R ( π ) -U r ∗ π ( π ) c -U r ∗ π ( π ) · U R ( π 1 ) ≤ c . Since Regret ( π, r ∗ π ) ≤ Regret ( π 1 , r ∗ π 1 ) , U ω ( π ) ≥ U ω ( π 1 ) . Therefore, we can assert that if a none weakly totally dominated policy π is a solution to MinimaxRegret ( R ) , it is also a solution to arg max π ∈ Π U ω ( π ) . Additionally, to prove that if a none weakly totally dominated policy π is a solution to arg max π ′ ∈ Π U ω ( π ′ ) , it is also a solution to MinimaxRegret ( R ) , it is only necessary to prove that π achieve no larger regret than all the weakly totally dominated policies. But we delay the proof to 2).

2) If a policy π is weakly totally dominated and is a solution to MinimaxRegret ( R ) , we show that it is also a solution to arg max π ∈ Π U ω ( π ) , i.e., its U ω ( π ) is no less than that of any other policy.

We start by comparing with non weakly totally dominated policy. for any weakly totally dominated policy π 1 ∈ MinimaxRegret ( R ) , it must hold true that Regret ( π 1 , r ∗ π 1 ) ≤ Regret ( π 2 , r ∗ π 2 ) for any π 2 ∈ Π that weakly totally dominates π 1 . However, it also holds that Regret ( π 2 , r ∗ π 2 ) ≤ Regret ( π 1 , r ∗ π 2 ) due to the weak total domination. Therefore, Regret ( π 1 , r ∗ π 1 ) = Regret ( π 2 , r ∗ π 2 ) = Regret ( π 1 , r ∗ π 2 ) , implying that π 2 is also a solution to MinimaxRegret ( R ) . It also implies that U r ∗ π 2 ( π 1 ) = U r ∗ π 2 ( π 2 ) ≥ U r ∗ π 1 ( π 1 ) due to the weak total domination. However, by definition U r ∗ π 1 ( π 1 ) ≥ U r ∗ π 2 ( π 1 ) . Hence, U r ∗ π 1 ( π 1 ) = U r ∗ π 2 ( π 1 ) = U r ∗ π 2 ( π 2 ) must hold. Now we discuss two possibilities: a) there exists another policy π 3 that weakly totally dominates π 2 ; b) there does not exist any other policy that weakly totally dominates π 2 . First, condition a) cannot hold. Because inductively it can be derived U r ∗ π 1 ( π 1 ) = U r ∗ π 2 ( π 1 ) = U r ∗ π 2 ( π 2 ) = U r ∗ π 3 ( π 3 ) , while Lemma 7 indicates that π 3 totally dominates π 1 , which is a contradiction. Hence, there does not exist any policy that weakly totally dominates π 2 , meaning that condition b) is certain. We note that U r ∗ π 1 ( π 1 ) = U r ∗ π 2 ( π 1 ) = U r ∗ π 2 ( π 2 ) and the weak total domination between π 1 , π 2 imply that r ∗ π 1 , r ∗ π 2 ∈ arg max r ∈ R U r ( π 1 ) , r ∗ π 2 ∈ arg min r ∈ R U r ( π 2 ) , and thus min r ∈ R U r ( π 2 ) ≤ max π ∈ Π ¬ wtd min r ∈ R U r ( π ) = c . Again, π 1 ∈ MinimaxRegret ( R ) makes Regret ( π 1 , r ∗ π ) ≤ Regret ( π 1 , r ∗ π 1 ) ≤ Regret ( π, r ∗ π ) not only hold for π = π 2 but also for any other policy π ∈ Π ¬ wtd , then for any policy π ∈ Π ¬ wtd , U r ∗ π ( π 1 ) ≥ U r ∗ π ( π ) ≥ min r ∈ R U r ( π ) . Hence, U r ∗ π ( π 1 ) ≥ max π ∈ Π ¬ wtd min r ∈ R U r ( π ) = c . Since U r ∗ π ( π 1 ) = min r ∈ R U r ( π 2 ) as aforementioned, min r ∈ R U r ( π 2 ) > max π ∈ Π ¬ wtd min r ∈ R U r ( π ) will cause a contradiction. Hence, min r ∈ R U r ( π 2 ) = max π ∈ Π ¬ wtd min r ∈ R U r ( π ) = c . As a result, U R ( π ) = U r ∗ π ( π ) = max π ′ ∈ Π ¬ wtd min r ∈ R U r ( π ′ ) = c , and U ω ( π ) = c -Regret ( π, r ∗ π ) ≥ max π ′ ∈ Π ¬ wtd c -Regret ( π ′ , r ∗ π ′ ) = max π ′ ∈ Π ¬ wtd U ω ( π ′ ) . In other words, if a weakly totally dominated policy π is a solution to MinimaxRegret ( R ) , then its U ω ( π ) is no less than that of any non weakly totally dominated policy. This also complete the proof at the end of

1), because if a none weakly totally dominated policy π 1 is a solution to arg max π ∈ Π U ω ( π ) but not a solution to MinimaxRegret ( R ) , then Regret ( π 1 , r ∗ π 1 ) > 0 and a weakly totally dominated policy π 2 must be the solution to MinimaxRegret ( R ) . Then, U ω ( π 2 ) = c > c -Regret ( π 1 , r ∗ π 1 ) = U ω ( π 1 ) , which, however, contradicts π 1 ∈ arg max π ∈ Π U ω ( π ) .

It is obvious that a weakly totally dominated policy π ∈ MinimaxRegret ( R ) has a U ω ( π ) no less than any other weakly totally dominated policy. Because for any other weakly totally dominated policy π 1 , U R ( π 1 ) ≤ c and Regret ( π 1 , r ∗ π 1 ) ≤ Regret ( π, r ∗ π ) , hence U ω ( π 1 ) ≤ U ω ( π ) according to Eq.13.

So far we have shown that if a weakly totally dominated policy π is a solution to MinimaxRegret ( R ) , it is also a solution to arg max π ′ ∈ Π U ω ( π ′ ) . Next, we need to show that the reverse is also true, i.e., if a weakly totally dominated policy π is a solution to arg max π ∈ Π U ω ( π ) , it must also be a solution to MinimaxRegret ( R ) . In order to prove its truthfulness, we need to show that if π / ∈ MinimaxRegret ( R ) , whether there exists: a) a none weakly totally dominated policy π 1 , or b) another weakly totally dominated policy π 1 , such that π 1 ∈ MinimaxRegret ( R ) and U ω ( π 1 ) ≤ U ω ( π ) . If neither of the two policies exists, we can complete our proof. Since it has been proved in 1) that if a none weakly totally dominated policy achieves MinimaxRegret ( R ) , it also achieves arg max π ′ ∈ Π U ω ( π ′ ) , the policy described in condition a) does not exist. Hence, it is only necessary to prove that the policy in condition b) also does not exist.

If such weakly totally dominated policy π 1 exists, π / ∈ MinimaxRegret ( R ) and π 1 ∈ MinimaxRegret ( R ) indicates Regret ( π, r ∗ π ) > Regret ( π 1 , r ∗ π 1 ) . Since U ω ( π 1 ) ≥ U ω ( π ) , according to Eq.13, U ω ( π 1 ) = c -Regret ( π 1 , r ∗ π 1 ) ≤ U ω ( π ) = U R ( π ) -U r ∗ π ( π ) c -U r ∗ π ( π ) · U R ( π ) -Regret ( π, r ∗ π ) . Thus U R ( π ) -U r ∗ π ( π ) c -U r ∗ π ( π ) ( π ) · U R ≥ c + Regret ( π, r ∗ π ) -Regret ( π 1 , r ∗ π 1 ) > c , which is impossible due to U R ≤ c . Therefore, such π 1 also does not exist. In fact, this can be reasoned from another perspective. If there exists a weakly totally dominated policy π 1 with U r ∗ π 1 ( π 1 ) = c = U r ∗ π ( π ) but π 1 / ∈ MinimaxRegret ( R ) , then Regret ( π, r ∗ π ) > Regret ( π 1 , r ∗ π 1 ) . It also indicates max π ′ ∈ Π U r ∗ π ( π ′ ) > max π ′ ∈ Π U r ∗ π 1 ( π ′ ) . Meanwhile, Regret ( π 1 , r ∗ π ) := max π ′ ∈ Π U r ∗ π ( π ′ ) -U r ∗ π ( π 1 ) ≤ Regret ( π 1 , r ∗ π 1 ) := max π ′ ∈ Π U r ∗ π 1 ( π ′ ) -U r ∗ π 1 ( π 1 ) := max r ∈ R max π ′ ∈ Π U r ( π ′ ) -U r ( π 1 ) indicates max π ′ ∈ Π U r ∗ π ( π ′ ) -max π ′ ∈ Π U r ∗ π 1 ( π ′ ) ≤ U r ∗ π ( π 1 ) -U r ∗ π 1 ( π 1 ) . However, we have proved that, for a weakly totally dominated policy, π 1 ∈ MinimaxRegret ( R ) indicates U r ∗ π 1 ( π 1 ) = max r ∈ R U r ( π 1 ) . Hence, max π ′ ∈ Π U r ∗ π ( π ′ ) -max π ′ ∈ Π U r ∗ π 1 ( π ′ ) ≤ U r ∗ π ( π 1 ) -U r ∗ π 1 ( π 1 ) ≤ 0 and it contradicts max π ′ ∈ Π U r ∗ π ( π ′ ) > max π ′ ∈ Π U r ∗ π 1 ( π ′ ) . Therefore, such π 1 does not exist. In summary, we have exhausted all conditions and can assert that for any policies, being a solution to MinimaxRegret ( R ) is equivalent to a solution to arg max π ∈ Π U ω ( π ) . We complete our proof.

A.5 Collective Validation of Similarity Between Expert and Agent

In Definition 2 and our definition of Regret in Eq.2, we use the utility function U r to measure the performance of a policy. We now show that we can replace U r with other functions.

Lemma 11. The solution of MinimaxRegret ( R E,δ ∗ ) does not change when U r in MinimaxRegret is replace with U r ( π ) -f ( r ) where f can be arbitrary function of r .

Proof. When using U r ( π ) -f ( r ) instead of U r ( π ) to measure the policy performance, solving MinimaxRegret ( R ) is to solve Eq. 14, which is the same as Eq.9.

Lemma 11 implies that we can use the policy-expert margin U r ( π ) -U r ( E ) as a measurement of policy performance. This makes the rationale of using PAGAR-based IL for collective validation of similarity between E and π more intuitive.

A.6 Criterion for Successful Policy Learning

To analyze the sufficient conditions for MinimaxRegret to mitigate task-reward misalignment, we start by analyzing the general properties of MinimaxRegret on arbitrary input R .

Proposition 4. If the following conditions (1) (2) hold for R , then the optimal protagonist policy π P := MinimaxRegret ( R ) satisfies ∀ r + ∈ R,U r + ( π P ) ≥ ¯ U r + .

There exists r + ∈ R , and max r + ∈ R { max π ∈ Π U r + ( π ) -¯ U r + } < min r + ∈ R { ¯ U r + -¯ U r + } ;

There exists a policy π ∗ such that ∀ r + ∈ R , U r + ( π ∗ ) ≥ ¯ U r + , and ∀ r -∈ R , Regret ( π ∗ , r -) < min r + ∈ R { ¯ U r + -¯ U r + } .

Proof. Suppose the conditions are met, and a policy π 1 satisfies the property described in conditions 2). Then for any policy π 2 ∈ MinimaxRegret ( R ) , if π 2 does not satisfy the mentioned property, there exists a task-aligned reward function r + ∈ R such that U r + ( π 2 ) ≤ U r + . In this case Regret ( π 2 , r + ) = max π ∈ Π U r + ( π ) -U r + ( π 2 ) ≥ U r + -U r + ≥ min + ′ U r + ′ -U r + ′ .

r ∈ R However, for π 1 , it holds for any task-aligned reward function ˆ r + ∈ R that Regret ( π 1 , ˆ r + ) ≤ max π ∈ Π U ˆ r + ( π ) -U ˆ r + ≤ max r + ′ ∈ R { max π ∈ Π U r + ′ ( π ) -U r + ′ } < min r + ′′ ∈ R { U r + ′ -U r + ′ } ≤ Regret ( π 2 , r + ) , and it also holds for any misaligned reward function r -∈ R that Regret ( π 1 , r -) < min r + ′ ∈ R { U r + ′ -

In Proposition 4, condition (1) states that the task-aligned reward functions in R all have a low extent of misalignment while condition (2) states that there exists a π ∗ that not only performs well under all r + 's (thus being acceptable in the task) but also achieves relatively low regret under all r -'s. Note that the more aligned the r + 's, the more forgiving the tolerance for high regret on r -. Furthermore, Proposition 5 shows that, under a stronger condition on the existence of a policy π ∗ performing well under all reward functions in R , MinimaxRegret ( R ) can guarantee to induce an acceptable policy, i.e., satisfying the condition (2) in Definition 3.

Proposition 5 (Strong Acceptance) . Assume that condition (1) in Proposition 4 is satisfied. In addition, if there exists a policy π ∗ such that ∀ r ∈ R , Regret ( π ∗ , r ) < max r + ∈ R { max π ∈ Π U r + ( π ) -¯ U r + } , then the optimal protagonist policy π P := MinimaxRegret ( R ) satisfies ∀ r + ∈ R , U r + ( π P ) ≥ ¯ U r + .

Note that the assumptions in Proposition 4 and 5 are not trivially satisfiable for arbitrary R , e.g., if R contains two reward functions with opposite signs, i.e., r, -r ∈ R , no policy can perform well under both r and -r . However, in PAGAR-based IL, using R E,δ in place of arbitrary R is equivalent to using E and δ to constrain the selection of reward functions, which can lead to additional implications.

Theorem 2. (Weak Acceptance) If the following conditions (1) (2) hold for R E,δ , then the optimal protagonist policy π P := MinimaxRegret ( R E,δ ) satisfies ∀ r + ∈ R E,δ , U r + ( π P ) ≥ U r + .

The condition (1) in Proposition 4 holds

∀ r + ∈ R E,δ , L r + · W E -δ ≤ max π ∈ Π U r + ( π ) -U r + and ∀ r -∈ R E,δ , L r -· W E -δ < min r + ∈ R E,δ { U r + -U r + } .

Theorem 2 delivers the same guarantee as that of Proposition 4 but differs from Proposition 4 in that Condition (2) implicitly requires that for the policy π ∗ = arg min π ∈ Π W 1 ( π, E ) , the performance difference between E and π ∗ is small enough under all r ∈ R E,δ .

Theorem 3. (Strong Acceptance) Assume that the condition (1) in Theorem 4 holds for R E,δ . If for any r ∈ R E,δ , L r · W E -δ ≤ min r + ∈ R E,δ { max π ∈ Π U r + ( π ) -U r + } , then the optimal protagonist policy

∗

Proof. Again, we let π be the policy that achieves the minimality in W E . Then, we have L r · W E -δ ≥ L r · 1 L r sup | r | L ≤ L r U r ( E ) -U r ( π ∗ ) -( U r + ( E ) -max π ′ ∈ Π U r + ( π ′ )) ≥ max π ′ ∈ Π U r + ( π ′ ) -U r + ( π ∗ ) for any r ∈ R E,δ . We have recovered the condition in Proposition 5. The proof is complete.

A.7 Stationary Solutions

In this section, we show that MinimaxRegret is convex for π P .

Proof. For any α ∈ [0 , 1] and π P, 1 , π P, 2 , there exists a π P, 3 = απ P, 1 +(1 -α ) π P, 2 . Let r 1 , π A, 1 and r 2 , π A, 2 be the optimal reward and antagonist policy for π P, 1 and π P, 2 Then α · (max r ∈ R max π A ∈ Π U r ( π A ) -U r ( π P, 1 )) + (1 -α ) · (max r ∈ R max π A ∈ Π U r ( π A ) -U r ( π P, 2 )) = α ( U r 1 ( π A, 1 ) -U r 1 ( π P, 1 )) + (1 -α )( U r 2 ( π A, 2 ) -U r 2 ( π P, 2 )) ≥ α ( U r 3 ( π A, 3 ) -U r 3 ( π P, 1 )) + (1 -α )( U r 2 ( π A, 3 ) -U r 3 ( π P, 2 )) = U r 3 ( π A, 3 ) -U r 3 ( π P, 3 ) . Therefore, max r ∈ R max π A ∈ Π U r ( π A ) -U r ( π P ) is convex in π P .

A.8 Compare PAGAR-Based IL with IRL-Based IL

Assumption 2. max r J IRL ( r ) can reach Nash Equilibrium at an optimal reward function r ∗ and its optimal policy π r ∗ .

We make this assumption only to demonstrate how PAGAR-based IL can prevent performance degradation w.r.t IRL-based IL, which is preferred when IRL-based IL does not have a reward misalignment issue under ideal conditions. We draw two assertions from this assumption. The first one considers Maximum Margin IRL-based IL and shows that if using the optimal reward function set R E,δ ∗ as input to MinimaxRegret , PAGAR-based IL and Maximum Margin IRL-based IL have the same solutions.

Proposition 7. π r ∗ = MinimiaxRegret ( R E,δ ∗ ) .

Proof. The reward function set R E,δ ∗ and the policy set Π acc achieving Nash Equilibrium for arg min r ∈ R J IRL ( r ) indicates that for any r ∈ R E,δ ∗ , π ∈ Π acc , π ∈ arg max π ∈ Π U r ( π ) -U r ( E ) . Then

because the policies in Π acc achieve zero regret. Then Lemma 11 states that Π acc will also be the solution to arg max π P ∈ Π min r ∈ R E,δ ∗ { max π A ∈ Π U r ( π A ) } -U r ( π P ) . We finish the proof.

The proof can be found in Appendix A.6. The second assertion shows that if IRL-based IL can learn a policy to succeed in the task, MinimaxRegret ( R E,δ ) with δ < δ ∗ can also learn a policy that succeeds in the task under certain condition. The proof can be found in Appendix A.6. This assertion also suggests that the designer should select a δ smaller than δ ∗ while making δ ∗ -δ no greater than the expected size of the high-order policy utility interval.

Proposition 8. If r ∗ is a task-aligned reward function and δ ≥ δ ∗ -(max π ∈ Π U r ∗ ( π ) -U r ∗ ) , the optimal protagonist policy π P = MinimiaxRegret ( R E,δ ) is guaranteed to be acceptable for the task.

̸

Proof. If π r ∗ ∈ MinimiaxRegret ( R E,δ ) , then π r ∗ can succeed in the task by definition. Now assume that π P = π r ∗ . Since J IRL achieves Nash Equilibrium at r ∗ and π r ∗ , for any other reward function r we have max π ∈ Π U r ( π ) -U r ( π r ∗ ) ≤ δ ∗ -( U r ( E ) -max π ∈ Π U r ( π )) ≤ δ ∗ -δ . We also have max r ′ ∈ R E,δ Regret ( r ′ , π P ) ≤ max r ′ ∈ R E,δ Regret ( r ′ , π r ∗ ) ≤ δ ∗ -δ . Furthermore, Regre ( r ∗ , π P ) ≤ max r ′ ∈ R E,δ Regret ( r ′ , π P ) . Hence, Regre ( r ∗ , π P ) ≤ δ -δ ∗ ≤ max π ∈ Π U r + ( π ) -U r + . In other words, U r ∗ ( π P ) ∈ [ U r ∗ , max π ∈ Π U r ∗ ( π )] , indicating π P can succeed in the task. The proof is complete.

A.9 Example 1

S3

S5

1.00

1.00

S1

0.80

0.80

0.6

S6)

0

ob(a2)

0.60

3

0.60

a1

0.2

0.2

0.40

0.40

P

S2

a2

So

0.20

0.20

Protagonist Policy

MaxEnt IRL Loss

{(So, a1, S1, S4, S5,S6),

0.00

MaxEnt IRL-Based IL Policy

0.00

E:

(So,a2,S2,S2,S2,S6))

0.00

0.40

0.80

1.20

8

1.60

2.00

2.40

2.80

0.80

1.20

MaxEnt IRL Loss

1.60

2.00

2.40

2.80

Example 1 . Figure 5 Left shows an illustrative example of how PAGAR-based IL mitigates reward misalignment in IRL-based IL. The task requires that a policy must visit s 2 and s 6 with probabilities no less than 0 . 5 within 5 steps , i.e. Prob ( s 2 | π ) ≥ 0 . 5 ∧ Prob ( s 6 | π ) ≥ 0 . 5 where Prob ( s | π ) is the probability of π generating a trajectory that contains s within the first 5 steps. It can be derived analytically that a successful policy must choose a 2 at s 0 with a probability within [ 1 2 , 125 188 ] . The derivation is as follows.

The trajectories that reach s 6 after choosing a 2 at s 0 include: ( s 0 , a 2 , s 2 , s 6 ) , ( s 0 , a 2 , s 2 , s 2 , s 6 ) , ( s 0 , a 2 , s 2 , s 2 , s 2 , s 6 ) . The total probability equals Prob ( s 6 | π ; s 0 , a 2 ) = 1 5 + 1 5 2 + 1 5 3 = 31 125 . Then the total probability of reaching s 6 equals Prob ( s 6 | π ) = (1 -π ( a 2 | s 0 )) + 31 125 · π ( a 2 | s 0 ) . For Prob ( s 6 | π ) to be no less than 0 . 5 , π ( a 2 | s 0 ) must be no greater than 125 188 .

The reward function hypothesis space is { r ω | r ω ( s, a ) = ω · r 1 ( s, a ) + (1 -ω ) · r 2 ( s, a ) } where ω ∈ [0 , 1] is a parameter, r 1 , r 2 are two features. Specifically, r 1 ( s, a ) equals 1 if s = s 2 and equals 0 otherwise, and r 2 ( s, a ) equals 1 if s = s 6 and equals 0 otherwise. Given the demonstrations and the MDP, the maximum negative MaxEnt IRL loss δ ∗ ≈ 2 . 8 corresponds to the optimal parameter ω ∗ = 1 . This is computed based on Eq.6 in Ziebart et al. [2008]. The discount factor is γ = 0 . 99 . When computing the normalization term Z in Eq.4 of Ziebart et al. [2008], we only consider the trajectories within 5 steps. The optimal policy under r ω ∗ chooses a 2 at s 0 with probability 1 and reaches s 6 with probability less than 0 . 25 , thus failing to accomplish the task. The optimal protagonist policy π P = MinimaxRegret ( R E,δ ) can succeed in the task as indicated by the grey dashed lines in Figure 5 Middle. It chooses a 2 at s 0 with probability 1 when δ is close to its maximum δ ∗ . However, π P ( a 2 | s 2 ) decreases as δ decreases. It turns out that for any δ < 1 . 1 the optimal protagonist policy can succeed in the task. In Figure 5 Right, we further show how the MaxEnt IRL loss changes with ω .

B Approach to Solving MinimaxRegret

In this section, we show how we derive the off-policy RL objective function for π P . Also, we develop a series of theories that lead to two bounds of the Protagonist Antagonist Induced Regret. By using those bounds, we formulate objective functions for solving Imitation Learning problems with PAGAR.

B.1 Off-Policy Objective Function for Protagonist Policy Training

For reader's convenience, we put the Theorem 1 in Schulman et al. [2015] here.

Theorem 7 (Schulman et al. [2015]) . Let α = max s D TV ( π old , π new ) , and let ϵ = max s E a ∼ π new [ A π old ( s, a )] , then Eq.15 holds.

where ρ π old ( s ) = ∑ T t =0 γ t Prob ( s ( t ) = s | π old ) is the discounted visitation frequency of π old , A π old is the advantage function without considering the entropy.

Algorithm 1 in Schulman et al. [2015] learns π new by maximizing the r.h.s of the inequality Eq.15, which only involves the trajectories and the advantage function of π old . By moving U r ( π old ) from r.h.s of Eq.15 to the left, and replacing π new with π P and π old with π A , we obtain a bound for U r ( π P ) -U r ( π A ) as mentioned in Section 6.1. The PPO in Schulman et al. [2017] further simplifies the r.h.s of the inequality Eq.15 with a clipped importance sampling rate. We derived J π A ( π P ) by using the same trick.

B.2 Protagonist Antagonist Induced Regret Bounds

Our theories are inspired by the on-policy policy improvement methods in Schulman et al. [2015]. The theories in Schulman et al. [2015] are under the setting where entropy regularizer is not considered. In our implementation, we always consider entropy regularized RL of which the objective is to learn a policy that maximizes J RL ( π ; r ) = U r ( π ) + H ( π ) . Also, since we use GAN-based IRL algorithms, the learned reward function r as proved by Fu et al. [2018] is a distribution. Moreover, it is also proved in Fu et al. [2018] that a policy π being optimal under r indicates that log π ≡ r ≡ A π . We omit the proof and let the reader refer to Fu et al. [2018] for details. Although all our theories are about the relationship between the Protagonist Antagonist Induced Regret and the soft advantage function A π , the equivalence between A π and r allows us to use the theories to formulate our reward optimization objective functions. To start off, we denote the reward function to be optimized as r . Given the intermediate learned reward function r , we study the Protagonist Antagonist Induced Regret between two policies π 1 and π 2 .

Lemma 12. Given a reward function r and a pair of policies π 1 and π 2 ,

Proof. This proof follows the proof of Lemma 1 in Schulman et al. [2015] where RL is not entropy-regularized. For entropy-regularized RL, since A π ( s, a ( t ) ) = E s ′ ∼T ( ·| s,a ( t ) ) [ r ( s, a ( t ) ) + γ V π ( s ′ ) -V π ( s ) ] ,

Remark 1 . Lemma 12 confirms that E τ ∼ π [∑ ∞ t =0 γ t A π ( s ( t ) , a ( t ) ) ] = U r ( π ) -U r ( π )+ H ( π ) = H ( π ) .

Wefollow Schulman et al. [2015] and denote ∆ A ( s ) = E a ∼ π 1 ( ·| s ) [ A π 2 ( s, a )] -E a ∼ π 2 ( ·| s ) [ A π 2 ( s, a )] as the difference between the expected advantages of following π 2 after choosing an action respectively by following policy π 1 and π 2 at any state s . Although the setting of Schulman et al. [2015] differs from ours by having the expected advantage E a ∼ π 2 ( ·| s ) [ A π 2 ( s, a )] equal to 0 due to the absence of

entropy regularization, the following definition and lemmas from Schulman et al. [2015] remain valid in our setting.

̸

Definition 9. Schulman et al. [2015], the protagonist policy π 1 and the antagonist policy π 2 ) are α -coupled if they defines a joint distribution over ( a, ˜ a ) ∈ A × A , such that Prob ( a = ˜ a | s ) ≤ α for all s .

Lemma 13. Schulman et al. [2015] Given that the protagonist policy π 1 and the antagonist policy π 2 are α -coupled, then for all state s ,

Lemma 14. Schulman et al. [2015] Given that the protagonist policy π 1 and the antagonist policy π 2 are α -coupled, then

Lemma 15. Given that the protagonist policy π 1 and the antagonist policy π 2 are α -coupled, then

Proof. The proof is similar to that of Lemma 14 in Schulman et al. [2015]. Let n t be the number of times that a ( t ′ ) ∼ π 1 does not equal a ( t ′ ) ∼ π 2 for t ′ < t , i.e., the number of times that π 1 and π 2

disagree before timestep t . Then for s ( t ) ∼ π 1 , we have the following.

The expectation decomposes similarly for s ( t ) ∼ π 2 .

When computing E s ( t ) ∼ π 1 [ E a ( t ) ∼ π 2 [ A π 2 ( s ( t ) , a ( t ) ) ] ] -E s ( t ) ∼ π 2 a ( t ) ∼ π 2 [ A π 2 ( s ( t ) , a ( t ) ) ] , the terms with n t =

0 cancel each other because n t = 0 indicates that π 1 and π 2 agreed on all timesteps less than t . That leads to the following.

By definition of α , the probability of π 1 and π 2 agreeing at timestep t ′ is no less than 1 -α . Hence, P ( n t > 0) ≤ 1 -(1 -α t ) t . Hence, we have the following bound.

The preceding lemmas lead to the proof for Theorem 4 in the main text.

Theorem 4. Suppose that π 2 is the optimal policy in terms of entropy regularized RL under r . Let α = max s D TV ( π 1 ( ·| s ) , π 2 ( ·| s )) , ϵ = max s,a |A π 2 ( s, a ( t ) ) | , and ∆ A ( s ) = E a ∼ π 1 [ A π 2 ( s, a )] -

E a ∼ π 2 [ A π 2 ( s, a )] . For any policy π 1 , the following bounds hold.

Proof. We first leverage Lemma 12 to derive Eq.23. Note that since π 2 is optimal under r , Remark 1 confirmed that H ( π 2 ) = -∑ ∞ t =0 γ t E s ( t ) ∼ π 2 [ E a ( t ) ∼ π 2 [ A π 2 ( s ( t ) , a ( t ) ) ] ] .

We switch terms between Eq.23 and U r ( π 1 ) -U r ( π 2 ) , then use Lemma 15 to derive Eq.24.

We switch terms between Eq.25 and U r ( π 1 ) -U r ( π 2 ) , then base on Lemma 14 and 15 to derive the inequality in Eq.26.

It is stated in Schulman et al. [2015] that max s D TV ( π 2 ( ·| s ) , π 1 ( ·| s )) ≤ α . Hence, by letting α := max s D TV ( π 2 ( ·| s ) , π 1 ( ·| s )) , Eq.23 and 26 still hold. Then, we have proved Theorem 4.

B.3 Objective Functions of Reward Optimization

To derive J R, 1 and J R, 2 , we let π 1 = π P and π 2 = π A . Then based on Eq.21 and 22 we derive the following upper-bounds of U r ( π P ) -U r ( π A ) .

By our assumption that π A is optimal under r , we have A π A ≡ r Fu et al. [2018]. This equivalence enables us to replace A π A 's in ∆ A with r . As for the 2 αγ (2 α +1) ϵ (1 -γ ) 2 and 2 αγϵ (1 -γ ) 2 terms, since the objective is to maximize U r ( π A ) -U r ( π B ) , we heuristically estimate the ϵ in Eq.27 by using the samples from π P and the ϵ in Eq.28 by using the samples from π A . As a result we have the objective functions defined as Eq.29 and 30 where ξ 1 ( s, a ) = π P ( a ( t ) | s ( t ) ) π A ( a ( t ) | s ( t ) ) and ξ 2 = π A ( a ( t ) | s ( t ) ) π P ( a ( t ) | s ( t ) ) are the importance sampling probability ratio derived from the definition of ∆ A ; C 1 ∝ -γ ˆ α (1 -γ ) and C 2 ∝ γ ˆ α (1 -γ ) where ˆ α is either an estimated maximal KL-divergence between π A and π B since D KL ≥ D 2 TV according to Schulman et al. [2015], or an estimated maximal D 2 TV depending on whether the reward function is Gaussian or Categorical. We also note that for finite horizon tasks, we compute the average rewards instead of the discounted accumulated rewards in Eq.30 and 29.

Beside J R, 1 , J R, 2 , we additionally use two more objective functions based on the derived bounds. W J R,r ( r ; π A , π P ) . By denoting the optimal policy under r as π ∗ , α ∗ = max s ∈ S D TV ( π ∗ ( ·| s ) , π A ( ·| s ) , ϵ ∗ = max ( s,a ( t ) ) |A π ∗ ( s, a ( t ) ) | , and ∆ A ∗ A ( s ) = E a ∼ π A [ A π ∗ ( s, a )] -E a ∼ π ∗ [ A π ∗ ( s, a )] , we have the following.

Let ξ 3 = exp( r ( s ( t ) ,a ( t ) )) π A ( a ( t ) | s ( t ) ) be the importance sampling probability ratio. It is suggested in Schulman et al. [2017] that instead of directly optimizing the objective function Eq.31, optimizing a surrogate

objective function as in Eq.32, which is an upper-bound of Eq.31, with some small δ ∈ (0 , 1) can be much less expensive and still effective.

Alternatively, we let ∆ A ∗ P ( s ) = E a ∼ π P [ A π ∗ ( s, a )] -E a ∼ π ∗ [ A π ∗ ( s, a )] . The according to Eq.27, we have the following.

Then a new objective function J R, 4 is formulated in Eq.34 where ξ 4 = exp( r ( s ( t ) ,a ( t ) )) π P ( a ( t ) | s ( t ) ) .

B.4 Incorporating IRL Algorithms

Online RL Setting. In our implementation, we combine PAGAR with GAIL Ho and Ermon [2016], VAIL Peng et al. [2019], and f-IRL Ni et al. [2021], respectively. In this section, we use J IRL to indicate the IRL loss to be minimized in place of the notation J IRL in the main text. Accordingly, δ is the target IRL loss.

f-IRL . We use the FKL of f-IRL in our experiments. Since FKL is explicitly models a reward function with a neural network, when PAGAR is combined with FKL, the meta-algorithm Algorithm 1 can be directly implemented without changes except for letting J IRL be the FKL loss.

GAIL and VAIL . We take additional steps to incorporate GAN-based algorithms, as they do not explicitly learn reward functions but instead train discriminators. When PAGAR is combined with GAIL, the meta-algorithm Algorithm 1 becomes Algorithm 2. When PAGAR is combined with VAIL, it becomes Algorithm 3. Both of the two algorithms are GAN-based IRL, indicating that both algorithms use Eq.1 as the IRL objective function. In these two cases, we use a neural network to approximate D , the discriminator in Eq.1. To get the reward function r , we follow Fu et al. [2018] and denote r ( s, a ) = log ( π A ( a | s ) D ( s,a ) -π A ( a | s ) ) as mentioned in Section 1. Hence, the only difference between Algorithm 2 and Algorithm 1 is in the representation of the reward function. Regarding V AIL, since it additionally learns a representation for the state-action pairs, a bottleneck constraint J IC ( D ) ≤ i c is added where the bottleneck J IC is estimated from policy roll-outs. V AIL introduces a Lagrangian parameter β to integrate J IC ( D ) -i c in the objective function. As a result its objective function becomes J IRL ( r ) + β · ( J IC ( D ) -i c ) . VAIL not only learns the policy and the discriminator but also optimizes β . In our case, we utilize the samples from both protagonist and antagonist policies to optimize β as in line 10 following Peng et al. [2019].

In our implementation, depending on the difficulty of the benchmarks, we choose to maintain λ as a constant or update λ with the IRL loss J IRL ( r ) in most of the continuous control tasks. In HalfCheetah-v2 and all the maze navigation tasks, we update λ by introducing a hyperparameter µ . As described in the maintext, we treat δ as the target IRL loss of J IRL ( r ) , i.e., J IRL ( r ) ≤ δ .

Algorithm 2 GAIL w/ PAGAR

Input : Expert demonstration E , discriminator loss bound δ , initial protagonist policy π P , antagonist policy π A , discriminator D (representing r ( s, a ) = log ( π A ( a | s ) D ( s,a ) -π A ( a | s ) ) ), Lagrangian parameter

λ , iteration number i = 0 , maximum iteration number N Output : π P

1: while iteration number i < N do

2: Sample trajectory sets D A ∼ π A and D P ∼ π P

3: Estimate J RL ( π A ; r ) with D A

4: Optimize π A to maximize J RL ( π A ; r ) .

5: Estimate J RL ( π P ; r ) with D P ; J π A ( π P ; π A , r ) with D P and D A ;

6: Optimize π P to maximize J RL ( π P ; r ) + J π A ( π P ; π A , r ) .

7: Estimate J PAGAR ( r ; π P , π A ) with D P and D A

8: Estimate J IRL ( π A ; r ) with D A and E by following the IRL algorithm

9: Optimize D to minimize J PAGAR ( r ; π P , π A ) + λ · max ( J IRL ( r ) + δ, 0)

10: end while

11: return π P

Algorithm 3 VAIL w/ PAGAR

Input : Expert demonstration E , discriminator loss bound δ , initial protagonist policy π P , antagonist policy π A , discriminator D (representing r ( s, a ) = log ( π A ( a | s ) D ( s,a ) -π A ( a | s ) ) ), Lagrangian parameter

λ for PAGAR, iteration number i = 0 , maximum iteration number N , Lagrangian parameter β for bottleneck constraint, bounds on the bottleneck penalty i c , learning rate µ .

Output : π P

1: while iteration number i < N do

2: Sample trajectory sets D A ∼ π A and D P ∼ π P

3: Estimate J RL ( π A ; r ) with D A

4: Optimize π A to maximize J RL ( π A ; r ) .

5: Estimate J RL ( π P ; r ) with D P ; J π A ( π P ; π A , r ) with D P and D A ;

6: Optimize π P to maximize J RL ( π P ; r ) + J π A ( π P ; π A , r ) .

7: Estimate J PAGAR ( r ; π P , π A ) with D P and D A

8: Estimate J IRL ( π A ; r ) with D A and E by following the IRL algorithm

9: Estimate J IC ( D ) with D A , D P and E

10: Optimize D to minimize J PAGAR ( r ; π P , π A ) + λ · max ( J IRL ( r ) -δ, 0) + β · J IC ( D )

11: Update β := max ( 0 , β -µ · ( J IC ( D ) 3 -i c ) )

12: end while

13: return π P

In all the maze navigation tasks, we initialize λ with some constant λ 0 and update λ by λ := λ · exp( µ · ( J IRL ( r ) -δ )) after every iteration. In HalfCheetah-v2 , we update λ by λ := max ( λ 0 , λ · exp( µ · ( J IRL ( r ) -δ ))) to avoid λ being too small. Besides, we use PPO Schulman et al. [2017] to train all policies in Algorithm 2 and 3.

Offline RL Setting. Weincorporate PAGAR with RECOIL Sikchi et al. [2024]. The original RECOIL algorithm does not learn a reward function but learns a Q , V value functions and a policy π with neural networks. In order to combine PAGAR with RECOIL, we made the following modification to RECOIL:

Instead of learning the Q value function, we explicitly learn a reward function r : S × A × S → R , which takes the current state s , action a and the next state s ′ as input, and outputs a real number as the reward

Weuse the same loss function as that for optimizing Q in RECOIL to optimize r by replacing Q ( s, a ) with r ( s, a, s ′ ) + γV ( s ′ ) for every ( s, a, s ′ ) sampled from an offline dataset D . We denote this loss function for r as J IRL ( r ) as in Eq.35 where S is an offline trajectory sample set of some sub-optimal behavioral policy, d E,S mix indicates a mixture between S and E sets,

Algorithm 4 RECOIL w/ PAGAR

Input : Expert demonstration E , behavioral policy sample set S , reward function r , value function V , initial protagonist policy π P , antagonist policy π A , maximum iteration number N .

Output : π P

1: while iteration number i < N do

2: Train r using min r J IRL ( r ) + 0 . 001 · J PAGAR ( r )

3: Train V using min V J V ( V )

4: Train π A using max π A J RL ( π A )

5: Train π P using max π A J RL ( π P )

6: end while

7: return π P

and β is the mixing ratio as defined in Sikchi et al. [2024].

We use the same loss function for optimizing the value function V as in RECOIL to still optimize V , except for replacing the target Q ( s, a ) with target r ( s, a, s ′ ) + γV ( s ′ ) for every ( s, a, s ′ ) experience sampled from the offline dataset. Consequently, we have the loss function for V as defined in Eq.36 where σ is a conservatism parameter defined in Sikchi et al. [2024].

Instead of learning a single policy as in RECOIL, we learn a protagonist and antagonist policies π P and π A by using the same SAC-like policy update rule as in RECOIL, except for replacing Q ( s, a ) with r ( s, a, s ′ ) + γV ( s ′ ) for every ( s, a, s ′ ) experience sampled from the offline dataset. The objective function for learning π A and π P are defined in Eq.37.

With some heuristic, we construct a PAGAR-loss as follows.

For simplicity, we multiply this PAGAR-loss J PAGAR with a fixed Lagrangian parameter λ = 1 e -3 and add it to the aforementioned loss J IRL for optimizing r .

We summarize this RECOIL w/ PAGAR algorithm in Algorithm 4.

C Experiment Details

This section presents some details of the experiments and additional results.

C.1 Experimental Details

Hardware. All experiments are carried out on a quad-core i7-7700K processor running at 3.6 GHz with a NVIDIA GeForce GTX 1050 Ti GPU and a 16 GB of memory. Network Architectures . Our algorithm involves a protagonist policy π P , and an antagonist policy π A . In our implementation, the two policies have the same structures. Each structure contains two neural networks, an actor network, and a critic network. When associated with GAN-based IRL, we use a discriminator D to represent the reward function as mentioned in Appendix B.4.

Protagonist and Antagonist policies . We prepare two versions of actor-critic networks, a fully connected network (FCN) version, and a CNN version, respectively, for the Mujoco and Mini-Grid benchmarks. The FCN version, the actor and critic networks have 3 layers. Each hidden layer has 100 neurons and a tanh activation function. The output layer output the mean and standard deviation of the actions. In the CNN version, the actor and critic networks share 3 convolutional layers, each having 5 , 2 , 2 filters, 2 × 2 kernel size, and ReLU activation function. Then 2 FCNs are used to simulate the actor and critic networks. The FCNs have one hidden layer, of which the sizes are 64 .

Discriminator D for PAGAR-based GAIL in Algorithm 2 . We prepare two versions of discriminator networks, an FCN version and a CNN version, respectively, for the Mujoco and Mini-Grid benchmarks. The FCN version has 3 linear layers. Each hidden layer has 100 neurons and a tanh activation function. The output layer uses the Sigmoid function to output the confidence. In the CNN version, the actor and critic networks share 3 convolutional layers, each having 5 , 2 , 2 filters, 2 × 2 kernel size, and ReLU activation function. The last convolutional layer is concatenated with an FCN with one hidden layer with 64 neurons and tanh activation function. The output layer uses the Sigmoid function as the activation function.

Discriminator D for PAGAR-based VAIL in Algorithm 3 . We prepare two versions of discriminator networks, an FCN version and a CNN version, respectively, for the Mujoco and Mini-Grid benchmarks. The FCN version uses 3 linear layers to generate the mean and standard deviation of the embedding of the input. Then a two-layer FCN takes a sampled embedding vector as input and outputs the confidence. The hidden layer in this FCN has 100 neurons and a tanh activation function. The output layer uses the Sigmoid function to output the confidence. In the CNN version, the actor and critic networks share 3 convolutional layers, each having 5 , 2 , 2 filters, 2 × 2 kernel size, and ReLU activation function. The last convolutional layer is concatenated with a two-layer FCN. The hidden layer has 64 neurons and uses tanh as the activation function. The output layer uses the Sigmoid function as the activation function.

Hyperparameters The hyperparameters that appear in Algorithm 3 and 3 are summarized in Table 2 where we use N/A to indicate using δ ∗ , in which case we let µ = 0 . Otherwise, the values of µ and δ vary depending on the task and IRL algorithm. The parameter λ 0 is the initial value of λ as explained in Appendix B.4.

Expert Demonstrations. Our expert demonstrations all achieve high rewards in the task. The number of trajectories and the average trajectory total rewards are listed in Table 3.

C.2 Additional Results

Continuous Tasks with Non-Binary Outcomes We test PAGAR-based IRL in 5 Mujuco tasks where the task objectives do not have binary outcomes. We append the results in three Mujoco benchmarks: Walker2d-v2 , HalfCheeta-v2 , Hopper-v2 , InvertedPendulum-v2 and Swimmer-v2 in Figure 6 and 7. Algorithm 1 performs similarly to VAIL and GAIL in those two benchmarks. The results show that PAGAR-based IL takes fewer iterations to achieve the same performance as the baselines. In particular, in the HalfCheetah-v2 task, Algorithm 1 achieves the same level of performance compared

1000

4000

Episode

PAGAR-GAIL

3500

PAGAR-VAIL

0

3000

Per

2500

VAIL

-1000

Return

2000

-2000

1500

Average

1000

-3000

500

-4000

0

0

250

500

750

010001250

1500

1750

2000

0

500

1000

1500

2000

2500

3000

Iterations

Iterations

with GAIL and VAIL by using only half the numbers of iterations. IQ-learn does not perform well in Walker2d-v2 but performs better than ours and other baselines by a large margin.

(b) InvertedPendulum-v2

(c) Swimmer-v2

Episode

3500

Episode

1200

Average Return Per Episode

125

3000

1000

100

Per

2500

Per

800

75

Return

2000

AverageReturn

600

50

1500

PAGAR-GAIL

400

25

AverageR

1000

PAGAR-VAIL

200

0

500

VAIL

25

IQ

0

0

50

0

200

400

Iterations

600

800

1000

0

100

200

Iterations

300

400

500

600

0

200

400

Iterations

600

800

1000

C.3 Influence of Reward Hypothesis Space

In addition to the DoorKey-6x6-v0 environment, we also tested PAGAR-GAIL and GAIL in SimpleCrossingS9N2-v0 environment. The results are shown in Figure 6 and 8.

(a) MiniGrid-DoorKey-6x6-v0

(b) MiniGrid-SimpleCrossingS9N2-v0

GAIL(Sigmoid)

Episode

1.0

GAIL(Categ)

PAGAR-GAIL(Sigmoid)

PAGAR-GAIL(Categ)

0.8

PerE

0.6

Return

0.4

AverageR

0.2

0.0

0

100000

200000

300000

400000

500000

Frames

GAIL(Sigmoid)

Return Per Episode

GAIL(Categ)

0.8

PAGAR-GAIL(Sigmoid)

PAGAR-GAIL(Categ)

0.6

0.4

AverageR

0.2

0.0

0

100000

200000

300000

400000

500000

600000

Frames

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the task-aligned reward function is hardly predictable in IRL-based IL setting. As mentioned in Section 5, " How to build R E,k ", it is designer's decision to set value for the parameter k .

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Every lemma, proposition and theorem have the set of assumptions included in them. The proves are included in Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclosed all the information in Appendix C.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the source code in our submission, including the instructions on how to reproduce the results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We disclosed all the information in Appendix C.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The curves in the plots are smoothed. They reflect how the policies' average returns change as the learning episodes increase.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We disclosed all the information in Appendix C.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform with NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper does not discuss and is not intended to cause significant societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not release data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper aims to alleviate the reward-misalignment issue by introducing task-related objective. The authors theoretically derive sufficient conditions to mitigate the task-reward misalignment issue and design algorithms accordingly. Theoretical analysis of the algorithm is provided to guarantee the reward learning improvement. Two discrete grid-world experiments are provided to empirically validate the proposed algorithms.

STRENGTHS:
1. The idea of introducing task objective to help alleviate the reward ambiguity issue is novel and interesting.
2. This paper has a complete and solid theoretical structure, which is also the biggest strength.
3. The empirical results show superior performance over GAIL and VAIL.

WEAKNESSES:
The main weakness lies in the empirical evaluation:
1. The environment setups are too simple. The paper only studies two discrete settings where continuous environments are missing, e.g., MuJoCo.

2. The experiment only has two comparison baselines, which is not convincing enough to show the superiority of the proposed algorithm. The paper mentions the IRL based IL and aims to alleviate the reward ambiguity issue, why not directly use some IRL algorithms as baselines to compare, e.g., MaxEnt IRL, ML-IRL, f-IRL, etc.

I know that this paper is a theory paper where the theoretical analysis is the major contribution and should be valued the most. However, the empirical evaluation should at least meet the minimum standard of the current majority, and two discrete gridworlds with fewer than 100 states apparently do not. I highly suggest the authors to add some continuous environments and baselines.

QUESTIONS:
Please see weakness.

LIMITATIONS:
The author discusses the limitation in the paper, which is reasonable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to inverse RL (IRL) by identify weaknesses in current IRL algorithms and proposing the respective solution:  optimizing for *task-related* reward functions. The authors provide a clean theoretical framework for task-related rewards, but lack applicability in practice from my perspective (see below).\n\nSTRENGTHS:\n- inverse RL with task-aligned reward functions is an interesting research direction.\n- the paper presents a good theoretical basis for future work.\n- the theoretical work looks plausible to me (only reading the main paper).\n\nWEAKNESSES:\n- The storyline of the paper is not clear. I do not see the emphasis on task-relatedness (see comments below).\n- The paper is cluttered in theory, making it hard to follow. Instead, more focus should be on the story, the motivation, and the experiments.\n- The algorithm looks quite complex; not sure how scaleable it is. (suggestions below)\n- The experimental campaign is concentrated on simple tasks; more complex tasks could prove scalability.\n\nQUESTIONS:\n- How is $\\Pi_{acc}$ actually spanned? It is not really talked about it, which made the following sentence a bit confusing “It is crucial to recognize that rE might not meet the task-aligned reward function criteria specified in Definition 2, even though its optimal policy πE is acceptable. ”\n- Looking at Def 2, the “task relatedness” is only induced by the definition of $\\Pi_{acc}$, is this correct?\n\nLIMITATIONS:\nMy biggest problem is that I dont see why this paper (in practice) proposes a technique for task-aligned reward functions. $k$ and $\\Pi_{acc}$ define task-relatedness in this framework to my understanding. And as the authors say:\n“As the task alignment of each reward function typically remains unknown in IRL settings, this paper proposes treating k as an adjustable parameter – starting with a small k and adjusting based on empirical learning outcome, allowing for iterative refinement for alignment with task requirements.” So what defines task-relatedness is actually set as a hyperparameter in this framework. For me, the method sounds more like a regularization technique for finding less overfitting reward functions. It looks like the auhors also say so:\n“Given the uncertainty in identifying which reward function is aligned, our solution is to train a policy to achieve high utilities under all reward functions in RE,δ to satisfy the conditions in Definition 3.” --> all reward functions does not mean task-related reward functions. Hence, my claim of being a regularization technique rather than an active search for task-related search. For me, this is breaking the storyline of the paper. \n\nFurther points:\n- In section 4.1, consider clarifying the difference between r* and r_E. It took me a bit do understand it.\n- Algorithm 1, line 4: I guess it should be $\\pi_P$ instead of $\\pi_A$.\n- an ablation on $\\delta$ would be interesting \n- add more complex environments. One good fit could be the new locomotion benchmark LocoMuJoCo. It provides motion capture datasets mapped towards different robots. So, there is a dynamics mismatch between the expert dataset and the environment. Also, all baselines (Gail, Vail, IQ-Learn) are already available.\n- Since you have been using IQ-Learn [1] as a baseline, I would also add it together with similar methods using implicit rewards [2, 3] to the related work section with some discussion. Otherwise, it looks out of place in the experiment section. \n- General remark: shorten the theory and make a more comprehensive experimental campaign.\n\n\n[1] Garg et al.; IQ-Learn: Inverse soft-Q Learning for Imitation \\\n[2] Al-Hafez et al.; LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning \\\n[3] Sikchi et al.;  Dual RL: Unification and new methods for reinforcement and imitation learning", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the reward misalignment in inverse reinforcement learning and introduces a novel approach to tackle this problem: PAGAR. It introduces a protagonist and an antagonist policy and treats the expert demonstrations as weak supervision to derive a set of reward functions rather than a single reward function. The reward functions and policies are used in a min-max scheme to minimize the regret induced by the two policies. The antagonist policy should maximize the regret whereas the protagonist policy should minimize it. PAGAR is used on adversarial imitation learning settings (more specifically on top of GAIL and VAIL) in combination with PPO on two discrete domain tasks.\n\nSTRENGTHS:\n* The problem of solving the underlying tasks instead of adapting to the expert data is very relevant.\n* Even though the idea takes inspiration from UED, the proposed approach is novel and significant for the IRL community.\n* The authors provide theoretical results.\n\nWEAKNESSES:\n* Even though the relationship and inspiration to UED is mentioned once, the basis of UED should be explained more.\n* The experimental evaluation is limited. E.g. the experiments only contain environments with discrete action spaces. Since GAIL and VAIL as well as PPO work in continuous action spaces, it should not be a problem to include additional experiments on continuous control tasks.\n* Due to the introduced notation and presented theoretical results, the paper is difficult to follow. Even though the appendix already includes most of the theoretical derivations and proofs, some parts of the main paper could still be included there to give more space for explanation, intuition and experiments. E.g. section 5.1. could be moved to the appendix or integrated in a reduced version in the main text.\n\n* Minor:\n- Missing } in line 270.\n- Line 4 in algorithm 1: It should be \"update $\\pi_{P}$ to maximize...\" instead of $\\pi_{A}$ ?\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nEven though the authors mention a small amount of limitations in the checklist, I think that setting the parameter\n$k$ is not the only limitation of the method (which is the only limitation addressed in the main text). An individual section about the limitations of the approach is missing.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates the inverse reinforcement learning problem. Previous methods suffer from the weakness: fail to capture the true task objectives from demonstrations. This study proposes deriving a set of candidate reward functions that align with the task rather than merely the data. Using an adversarial learning approach, the policy is trained with this set of reward functions. Then the proposed framework collectively validates the policy’s ability for task accomplishment. The proposed method is theoretically derived with detailed derivation and experimentally implemented on MiniGrid environments with impressive empirical results.\n\nSTRENGTHS:\nThe paper is well-structured, clearly explained, and nicely presented.\n\nThe approach has a solid theoretical foundation.\n\nThe method shows significant advantages over baseline approaches on MiniGrid DoorKey, especially with a single demonstration.\n\nWEAKNESSES:\nThe experiments on MiniGrid are not very extensive. Beyond DoorKey, MiniGrid includes more challenging environments such as ObstructedMaze and UnLock. Demonstrating advantages in these environments would be more convincing.\n\nThe proposed approach requires a set of candidate reward functions. While it achieves a higher success rate than baselines, it is unclear if this comes at a higher time cost. Comparing the time efficiency with prior work would be beneficial.\n\nQUESTIONS:\nIn Appendix Figure 5, the proposed approach does not significantly better than the baselines in locomotion tasks (Walker, HalfCheetah, Hopper, Swimmer, etc.). Would the method show more advantages if only 1 demonstration is used instead of 10 or 100?\n\nThe MiniGrid DoorKey 6x6 experiment just requires a demonstration with a short horizon. In larger mazes or more challenging environments with complicated demonstrations, would the proposed approach still perform impressively with only 1 demo?\n\nLIMITATIONS:\nThere is no analysis explaining why the proposed method performs better on MiniGrid (Figure 2) than on locomotion tasks (Appendix Figure 5). Studying the scenarios where the method does not work well and providing explanations would help researchers determine the appropriateness of using this approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-22"}