Understanding Constraint Inference in Safety-Critical Inverse Reinforcement Learning
Authors: Bo Yue
Shufan Wang
Ashish Gaurav
Jian Li
Pascal Poupart
Guiliang Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Constraint Inference
Training Efficiency
Cross-environment Transferability
Primary area: reinforcement learning
TL;DR: We offer theoretical and empirical insights into constraint inference via comparison of ICRL and IRC for their training efficiency and corss-environment transferability.

Abstract: In practical applications, the underlying constraint knowledge is often unknown and difficult to specify. To address this issue, recent advances in Inverse Constrained Reinforcement Learning (ICRL) have focused on inferring these constraints from expert demonstrations. However, the ICRL approach typically characterizes constraint learning as a tri-level optimization problem, which is inherently complex due to its interdependent variables and multiple layers of optimization.
Considering these challenges, a critical question arises: *Can we implicitly embed constraint signals into reward functions and effectively solve this problem using a classic reward inference algorithm?* The resulting method, known as Inverse Reward Correction (IRC), merits investigation. In this work, we conduct a theoretical analysis comparing the sample complexities of both solvers. Our findings confirm that the IRC solver achieves lower sample complexity than its ICRL counterpart.
Nevertheless, this reduction in complexity comes at the expense of generalizability. Specifically, in the target environment, the reward correction terms may fail to guarantee the safety of the resulting policy, whereas this issue can be effectively mitigated by transferring the constraints via the ICRL solver. 
Advancing our inquiry, we investigate conditions under which the ICRL solver ensures $\epsilon$-optimality when transferring to new environments. Empirical results across various environments validate our theoretical findings, underscoring the nuanced trade-offs between complexity reduction and generalizability in safety-critical applications.

Published as a conference paper at ICLR 2025
UNDERSTANDING CONSTRAINT INFERENCE IN SAFETY- CRITICAL INVERSE REINFOR LEARNING
Bo Yue', Shufan Wang", Ashish Gaurav Jiin Li,, Pascal Poupart Guiliang Liu 1 Schooo of Data Science, The Chinese University of Hong Kong, Shenzhen, Stony Brook University, University of Waterloo, *Vector Institute boyuel pruuelling ink. cuhk. edu. cn, Litguiliang@euueeueeooo edu. cn, cuhk (shufan wang, jian 3) @stoorbrook. edu, (a5gaurav, ppoupart "osgaraa,ppouparte @wwawerloo.ccoo
ABSTRACT
In practical applications, the underlying constraint knowledge is often unknown and difficult specify. To address this issue, recent advances in Inverse Constrained Reinforcement Learning (ICRL) have focused on inferring these constraints from expert demonstrations. However, the ICRL approach typically characterizes con- straint learning as a tri-level optimization problem, which is inherently complex due its interdependent variables and multiple layers of optimization. Considering these challenges, a critical question arises: Can we implicitly embed constraint sig- nals into reward functions and effectively solve this problem using a classic reward inference algorithm? The resulting method, known as Inverse Reward Correction (IRC), merits investigation. In this work, we conduct a theoretical analysis com- paring the sample complexities of both solvers. Our findings confirm that the IRC solver achieves lower sample complexity than its ICRL counterpart. Nevertheless, this reduction in complexity comes at the expense of generalizability. Specifically, in the target environment, the reward correction terms may fail to guarantee the safety of the resulting policy, whereas this issue can be effectively mitigated by transferring the cost functions via the ICRL solver. Advancing our inquiry, we investigate conditions under which the ICRL solver ensures E-optimality when transferring to new environments. Empirical results across various enviromments validate our theoretical findings, underscoring the nuanced trade-off's between complexity reduction and generalizability in safety-critical applications.
INTRODUCTION
Aligning the decision process with the underlying constraints in the environment is erucial prerequi- site for solving decision-making problems in safety-critical applications. To realize this vision, safe Reinforcement Learning (RL) algorithms typically optimize a control policy based on known or manually-speciferl constraint (Liu et al., 2021; Gu et al., 2022). However, the ground truth constraints are often unknown in many real-world applications. Moreover, given the inherent complexity of environmental dynamics, safety constraints must accurately model the interdependencies among numerous variables, which are difficult to capture solely with prior knowledge.
To resolve the above challenges, Inverse Constrained Reinforcement Learning (ICRL) designs data-driven constraint inference method to learn the constraints from expert demonstrations (Scobee & Sastry, 2020). Specifically, an ICRL algorithm (Malik et al., 2021) typically addresses tri-level optimization problem involving the update of 1) the feasibility functions to represent constraints. 2) the Lagrange parameters balance reward maximization and constraint satisfaction, and 3) the policy function to guide the agent's 's behaviors. Under this setting, the variables subject to optimization are interdependent, and the sub-optimality of one variable can influence the performance of the others. To mitigate the complexity of this problem, Hugessen et al. (2024) proposed to simplify the ICRL solver by incorporating the impact of both constraint and Lagrange parameters into a reward correction term. This modification reduces the ICRL solver to bi-level solver known Inverse Reward Correction (IRC) (Li et al., 2023). Hugessen et al. (2024) empirically demonstrated that
"Corresponding author: Guiliang Liu

---

Published as a conference paper at ICLR 2025
such simplification does not compromise the performance of constraint learning. These observations raise significant questions about the necessity of explicitly modeling constraints. It is still uncertain whether the canonical reward learning framework is sufficient to capture agent's preferences within a Constrained Markov Decision Process (CMDP).
To address this issue, in this work, conduct rigorous study to understand the impact of modeling constraints. Specifically, we theoretically and empirically compare the performance of IRC and ICRL from the following perspectives:
Training Efficiency. Unlike previous studies, which primarily compared IRC and ICRL through empirical evaluations (Hugessen al., 2024), are the first to provide theoretical quantification of training efficiency for both IRC and ICRL solvers in capturing the safety preferences in expert agents' decisions. However, achieving this objective is inherently challenging due the unidentifiability of the optimal solution, as multiple solutions can explain expert demonstrations (Ng et al., 2000). To address this, rather than modeling pointwise solutions, we adopt the approach first proposed by Metelli et al. (2021), which introduces a theoretical framework that characterizes a complete set of feasible solutions. This approach enables us to determine the intrinsic sample complexity of both IRC and ICRL without being obfuscated by further restrictions to ensure the uniqueness of optimal solutions. Our results indicate that ICRL requires more training samples, consistent with the empirical findings in Hugessen al. (2024). Moreover, our theoretical analysis provides a deeper understanding of the increased complexity. Specifically, we show that ICRL solvers capture constraint-vio viiiaaiin movements by leveraging known reward signals. During the update of a constrained RL policy, these movements correspond to decision-making patterns that drive an increase in the Lagrange multipliers.
Cross-enviromment Transferability. While previous constraint learning methods primarily focus on imitating experts' behavior (Scobee & Sastry, 2020; Liu et al., 2023; Hugessen et al., 2024; Liu et al., 2024), key motivation for inferring experts' safety preferences is to generalize this knowledge for guiding policy learning in similar contexts (Feng et al., 2023; Zhang et al., 2024). In this study, we compare IRC and ICRL in terms of guarantceing safety and optimality of policies based on inferred terms across different environments. We begin with an illustrative example to demonstrate situations where IRC could potentially induce unsafe behaviors. We identify and summarize the conditions under which IRC leads to such unsafe behaviors and explain why the performance of ICRL is more robust by explicitly modeling constraints. Regarding the optimality of generalized policies, we theoretically investigate how mismatches in environmental dynamics and reward signals influence the acquisition of optimal policies based on learned cost functions by the ICRL solver. In the absence of constraint satisfaction, these results cannot be extended to IRC.
Contributions. We compare the training efficiency and cross-environmen transferability of IRC and ICRL solvers address the critical question: Can we implicitly embed constraint signals into reward functions and effectively solve this problem using a classic reward inference algorithm?
Following the theoretical framework from Metelli et al. (2021), we introduce the IRC solver (Definition 3.3) to overcome the limitation of the IRL solver, which lacks a mechanism to leverage existing reward signals and is not compatible with different rewards. Furthermore, we analyze the sample complexity of IRC (Section 4.1) and compare it with existing results for ICRL (Yue et al., 2024), showing that IRC achieves lower sample complexity than ICRL under the same optimality criterion (Theorem 4.2).
We conduct formal study of transferabillty in safety to compare IRC and ICRL. We show that transferred cost functions by ICRL are guaranteed to preserve safety in overlapping critical regions (Lemma 5.2), whereas transferred reward correction terms by IRC can be offset by the difference in reward functions and transition dynamics between source and target environments (Theorem 5.3).
We analyze the optimality of transferred cost functions in target environments, by extending the transferability definition from Schlaginhaufen & Kamgarpour (2024, Definition 3.1) in regular MDP settings to CMDP settings. Specifically, we define the suboptimality gap under CMDP settings (Definition 5.5). Building on this, we derive a sufficient condition that limits the similarity between source and target environments guarantee s-optimality for ICRL (Theorem 5.7).
Finally, we empirically validate our results on training efficiency and cross-enviromment transferability in various environments (Section 6).
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Inverse Constrained Reinforcement Learning (ICRL). Unlike IRI. which focuses primarily on recovering reward functions, ICRL seeks to align with expert agents' preferences by inferring the constraints they adhere to. The majority of existing ICRL algorithms update cost functions by maximizing the likelihood of generating expert demonstrations under the maximum (casual) entropy framework (Scobee & Sastry, 2020). Subsequent works have extended this approach from discrete to continuous state-action spaces (Malik et al., 2021; Qiao et al., 2023; Xu & Liu, 2024b; Quan et al., 2024; Zhao et al., 2025). To enhance training efficiency, Liu Zhu (2022); Gaurav et al. (2023) combined ICRL with bi-level optimization techniques. Towards theoretical groundings of ICRL. Yue et al. (2024) recently proposed efficient constraint inference through exploration strategies, achieving tractable sample complexity. However, these works primarily evaluate their performance by applying inferred constraints within the same environment used for learning. While Xu & Liu (2024a) proposed an approach to learning robustly safe policy across set of pre-defined and limitedly varied transition dynamics, the challenge of transferring constraints to new environments remains largely unexplored.
Transferability in Inverse Reinforcement Learning (IRL.. A significant application of IRL. al- gorithms is to guide policy learning in similar environments. However, obtaining guarantees of transferability in unregularized settings is more challenging than in entropx-regularized contexts. To facilitate knowledge transfer in unregularied settings, Metelli et al. (2021) assumed that the feasible reward functions recovered remain valid in target environments, and Amin et al. (2017) reduced the dimension of the reward class state-only rewards. In contrast, entropy-regularized settings have been explored more extensively. Cao et al. (2021) and Skalse et al. (2023) demonstrated that under entropy regularization, the expert's s reward can be identified up to potential shaping transformations (Ng et al., 1999). In addition, Rolland et al. (2022) showed that to guarantee transferability across any transition laws, the expert's reward must be identified up to a constant, Building upon these insights, Cao et al. (2021) and Rolland et al. (2022) learned the reward function from multiple experts who shared rewards but had sufficiently different transition laws distinguished by a specific rank condition. Continuing this line of research, Schlaginhaufen & Kamgarpour (2024) extended this approach to an offline setting and derived a sufficient condition for transferability considering local changes in the transition law when learning from a single expert. However, these methods focus on transferring reward functions in regular MDPs, without addressing the generalization of constraints in CMDPs.
PRELIMINARIES AND PROBLEM FORMULATION
Notation. Let X' and ) be two sets. The notation y represents the set of functions f X' + y. The set of probability measures over X denoted A = {v E [0, 1|* E v(x) 1] and we denote A as the set of functions: y - A We define min (x) returns the minimum positive value of f over X. For a linear operator T, its image is denoted by im Let represent the ground- truth cost function obeyed by the expert, and y¹ denote the expert occupancy measure. Let 1()) denote the indicator function. The expansion operator E R + SxA satisfies (Ef)(s,a) = f(s).
The complete notation is provided in Appendix Table 1.
Constrained Markoy Decision Process (CMDP). The environment is modeled as a stationary CMDP
M. := (S, A, Pr, r, c), where S and A are the finite state and action spaces, with cardinalities S = [S] and A = [A]; Pr (s,a) a) E A defines the transition dynamic; r E [0, Kmax]sxx and E [0, Cmax] SxxA denote the reward and cost functions. The set {e, mo,7} represents additional environmental configurations, where e defines the threshold of the constraint, Ho E A denotes the initial state distribution, and 7 E [0, 1) is the discount factor. The agent's 's policy on the infinity planning horizon, and our theoretical results are based on a discrete state-action space.
Ag. We focus
Value and advantage functions. We denote action value functions in a CMDP MM for costs and rewards as QiA, and Q'AA.- The superscript e or r specifies the actual costs or rewards evaluated. The reward action-value function Q (s, ) = Ex.ry E 7'r(st, )=) 8, 40 = =a] and the a)= QM. (s, a) VVV.. (s). where the reward state-value function
reward advantage function AM, (s, a)
VM, (s) E. (MM, (8,a)), a)). The same notation scheme applies to cost value functions in Me by
3

---

Published as a conference paper at ICLR 2025
replacing the superscript with c, i.., QM, and V.. This scheme also applies to CMDP without knowing the cost, i.e., M= by replacing the subscript Me with M, i.e., QM VM" and A
Constrained Reinforcement Learning (CRL). Within a CMDP environment, CRL learns a policy that maximizes the cumulative rewards subject known constraint:
20
CRL(r, c) = max Ep
Σ y a:)
s.t. Et
Σ
r'e(s) Se,
(PI)
t=0
where indicates a hard constraint and 0 represents soft constraint.
Inverse Constraint Inference (ICI). In many practical applications, constraints are not readily available, so we need to infer the constraints followed by expert agents based on their behaviors. An ICI problem is a pair q = xM, where T E A is the expert's policy.
A common solver for this ICI problem under the RL setting is known Inverse Constrained Reinforcement Learning (ICRL), which is formally defined as follows:
Definition 3.1. (ICRL solver for ICI (Malik et al., 2021 An ICRL solver is denoted as SICRL- A cost function e is a feasible solution for an ICI problem @ if and only if n" an optimal policy for CMDP M We denote by Cpp the set of feasible cost functions derived by SICRL (7).
Previous ICRL solvers (Scobee & Sastry, 2020; Malik al., 2021) explicitly model constraints and infer the cost function by alternatively optimizing the policy and the cost function. In the phase of policy optimization, they commonly solve CRL problem (PI) by studying its Lagrangian dual:
D DICRL(r, c)] min max J (TT, Ac) + Xe, A30
(DI)
where J(x - Ac) = Eo Ei_ (r(s), a.) - Ac(st, a
Theorem 3.2. (CRL has zero duality gap (Puternain et al., 2019)). Suppose that and are bounded and the Slater's 's condition holds for (PI). then strong duality holds for (PI), i.e., PI" = DI"
Accordingly, the optimal policy in CRL objective (PI) can be equivalently solved by utilizing an unconstrained objective (DI). Based on the dual representation CRL problem, ICRL solvers are essentially solving a tri-level optimization problem (Kim et al., 2023):
max max min J F. A
Ae)- J(I, Ac).
Given the complexity of tri-level optimization, Hugessen et al. (2024) recently explored whether we can 1) apply IRL algorithm
recover F = Ac r by optimizing À and collectively in == de if the range of 7 is a convex cone, and 2) learn an imitation policy by directly maximizing the cumulative rewards EI 7 (st, at]] without considering the constrained optimization objective. In this work, we formally define this method as Inverse Reward Correction (IRC) as follows:
Definition 3.3. (IRC solver for ICI (Li al., 2023)). An IRC solver is denoted STRC- A reward correction term Ar is a feasible solution for an ICI problem & if and only if oh is an optimal policy for (M') S where corrected rewards r= a) r(s, a) Ar(s, a), (ss a). We denote by Rgs the set of feasible reward correction terms derived by SIRC
For the sake of clarity, simplify (Mir) y M in the remainder of the paper. Under this setting, the correction term can play the role of negative collective cost function such that 4r = -č. If the negative optimal ē can be represented within the bounded range of the correction term, i.e., - = X*e E range(Ar)) where X" and e* are optimal solutions (1), the tri-level optimization can be transferred bi-level one defined in the following:
max min J E + Ar) J(x Ar).
Hugessen et al. (2024) demonstrated that this simplification results in a more performant solver. In the following sections, we will provide a more formal comparison of these solvers, focusing on their sample complexity and transferability, i.c., performance in transferring to new environments.
4 TRAINING EFFICIENCY: A FORMAL STUDY OF SAMPLE COMPLEXITY
In this section, we compare the training efficiency of ICRL and IRC solvers by deriving their sample complexity and analyzing their performance gaps.
4

---

Published as a conference paper at ICLR 2025
4.1 SAMPLE COMPLEXITY OF IRC SOLVER
To compute the sample complexity of the IRC solver, we adopt the theoretical framework from Metelli et al. (2021) and define the feasible of reward correction terms follows:
Lemma 4.1. (Feasible reward correction implicit). Let & be ICI problem and SiRe be the IRC solver. Ar is feasible reward correction term, i.e., Ar E Rop if and only if a) E A:
Ar,x (i) if " (als) > 0, then Omuurt Ar) (8, (i) 1f (als) O, then (u((+An) Ar) (s, S (Mu(++Ar) (s).
Vr+Ar,
"uir+Ar) (8),
vr+
Intuitively, in CMDP, reward function r alone does not align with the expert policy π due to potential constraint violations, i.e. (als) = 0 but QMM (8,a) > VM= (s). The correction term Ar adjusts the reward r so that + Ar collectively ensures the optimality of the expert policy. This approach differs from the IRL solver (Metelli ct al., 2021; ; Lindner et al., 2022) in two key aspects: 1) IRL lacks mechanism leverage the known reward function for constraint inference, and 2) IRL is not compatible with different reward signals in new environments.
To provide fair comparison of sample complexity between the two solvers, we study uniform sampling strategy, detailed in Appendix Algorithm 1. This strategy queries the generative model to sample the state-action space, enabling the estimation of the transition dynamics and the expert policy as 3 (M, R F.), where M = U Py. Let 5 E (0, be the significance level. This strategy guarantees that with probability greater than 1 8, the Hausdorff distance dæ between the ground truth and estimated feasible set is upper bounded by an amount related to the number of samples.
In the of the IRC solver, this Hausdorff distance between Raz and R.z is upper bounded by
de (Rp, R₂j) S max I (s,(s,) with I (s, I4 2y 20
(3)
(s.0)ESxA
1-y
N (
where Not (s, a) the positive cumulative count of visitations to (s,a) a) (formally defined in Appendix B.2) and (s+1(s,0 a) log (12SA + /8). Towards reducing this upper bound below a targeted accuracy, we derive the sample complexity of the IRC solver.
Theorem 4.2. (Sample Complexity IRC Solver). If the IRC solver stops at iteration K with updated accaracy KK, then with probability at least 1 - 6 it converges, with the number of samples upper where Ô notation suppresses logarithmic terms.
bounded by n S δ R2 SA
4.2 SAMPLE COMPLEXITY OF AN ICRL SOLVER
Following a similar theoretical framework, Yue et al. (2024) derived the sample complexity for ICRL solver by defining the feasible cost set. We briefly recap and discuss the results below. Appendix B.6 provides a detailed review.
Lemma 4.3, (Feasible cost set implicit (Yue et al., 2024, Lemma 4.3)). Under (Yue et al., 2024, Assumption 4.1), let & be ICI problem and SICRe be the ICRL solver, then e is feasible cost, i.e., e E Cup if and only if V(8 a) S x A:
(i) F(ls) > 0, Qu
(ue(() =s)=
(ii) (
and A A (u (s) a) O. (Myy
V
(s) > 0:
Mus
(ii)) if (als) and A (s
Qu vax (s) 3)50 0.
Remark 4.4. Unlike the feasible reward correction terms in Lemma 4.1, the feasible cost function is closely related the reward advantage function under an expert policy, i.e., A If A..=" A a) O, the action a yields higher rewards than the expert. Such actions must violate the underlying constraints (case (ii)); otherwise, the expert policy could be further improved, contradicting its optimality.
Similarly, to compute the sample complexity of the ICRL solver, we utilize the uniform sampling strategy in Appendix Algorithm 1. Guaranteed by the strategy, the Hausdorff distance du between
5

---

Published as a conference paper at ICLR 2025
the ground truth and estimated feasible cost set, i.e. Cas and Cip. is upper bounded by
cx+1 (3, 2. k+1 (s,
d1r(Cop, Co)) S
max (s.a)E5xA
(8, with 1
where a m3 (Rmax + /)/ min
| (1 (1-7)).
Theorem 4.5. (Sample Complexity of ICRL solver (Yue al., 2024, Theorem C.9)). the ICRL solver terminates at iteration K with the updated accuracy ЕK, then with probability at least 1 - 6, it 2
converges with a number of samples upper bounded by n S (1-7)08k
Discussion. By comparing Theorem 4.2 with Theorem 4.5, we observe that the sample complexity of ICRL solver exceeds that of IRC solver by factor of 1/(1 -))2 This increase stems from
estimating the reward advantage function A which indictates whether to impose additional costs on any state-action pairs. Specifically, in the tri-level optimization in (I), establishing the feasible cost functions relies on the convergence of the Lagrange multiplier À to its optimum while increases whenever A A (s, a) to penalize unsafe policies.
CROSS-ENVIRONMENT TRANSFERABILITY. GENERALIZING THE SAFETY AND OPTIMALITY OF CONSTRAINTS
Beyond cloning the behaviors of expert agents, a critical application of the inverse optimization methods (including IRI., ICRL, and IRC) is infer generalizable oracle signals, such as rewards or constraints, that can effectively guide agent behaviors in similar environments. We use teal and purple colors to distinguish the source CMDP M. MU c = (S, A, PT, and the target CMDP M = MM' U (c') (S, A, Pf, (c)) where they can differ in the reward function, the transition model and underlying cost functions (i.e., r' # Pt * Py and cB * E)) Under this setting, our study adheres to the standard ICI problem fashion, assuming access to a single expert within a specific environment. We investigate the transferability of learned constraints across source and target CMDPs, focusing on two key aspects: safety and optimality.
5.1 GENERALIZING SAFETY GUARANTEES ACROSS DIVERSE ENVIRONMENTS
Transferring a recovered reward correction term Ar or a cost function introduces new challenges that remain largely unexplored in the ICI literature since there is no guarantee that Ar or c with the new reward function r' and new transition model P will make the new expert policy (T') constraint satisfying in the target CMDP M' That is to say, transferred Ar or may lead to unsafe policies (let alone optimal policies) in the target environment.
Challenges in Ensuring Safety with IRC. Although the IRC solver is more sample-efficient in training in the source environment, struggles to guarantee safety in the target vironment, as illustrated in Figure I. Consider a hard constraint scenario; soft constraints will be discussed later. In the source environment, trajectory (with a reward (s(T)) 2) has larger reward than 72 (rs(72) 1), but the expert agent prefers since Ti passes through an unsafe state 8 To align with the expert's demonstra- tion, the IRC solver learns feasi- ble reward correction term Ar(s")= -1 - 3 (where 3 > 0), ensuring that 71 achieves lower rewards than
Source Env
Target Env
==(:))2,,3(t))))
Highe Bewars
infer
Doan
T
Tzll
IRC: AP = -1---11- -1-B
ICRL: (c(s") Mc
transfer
Tx(T,) 2t -r- Ø
Figure 1: An example showing that transferring Ar and constraint 1 Mc(s = 0 learned in the source environment (left) to the target environment (right) induces different opti- mal policies (71 denotes the optimal path and denotes the second- best path).

---

Published as a conference paper at ICLR 2025
T2: (s(s1) + = - rs(2)) + Ar(12) (note that 4r(r) = 0 since T2 does not pass through s²). However, when this correction term is transferred to a target environment where reward functions and transition dynamics slightly differ (!T(T)) = 2 + 3, (T())) = 1 / it renders the unsafe path 71 as deceptively safe since hi now achieves higher rewards than 72: (r(()) + Ar(n) 1 2 1 = (r((2) + Ar(12)- This misguides the agent prefer π
In essence, the reward correction terms reflect the extent of penalty applied to constraint- -ivilaiin actions in the source environment. These penalties can be easily offset by increasing the gains on penalized actions or decreasing the gains on other actions in the target environment, Comparably, the ICRL solvers do not have such difficulty since they learn hard constraint that invalidates any visit to s" (e.g., the feasibility 1 (s") 0). The cost function directly decides the boundaries of safe region, a modification that cannot be compensated for in the same manner as correction terms.
A Formal Study of Transferability in Safety. Transferability with single expert knowledge requires similarity restrictions between source and target environments (Metelli et al., 2021, Assumption 4.1) (Schlaginhaufen & Kamgarpour, 2024, Theorem 3.10). In the context of safety, we expect the learned constraint knowledge to remain at least partially active the target environment, i.e., the two constrained regions overlap so that the learned constraint information can be effectively reused. This property is characterized as follows:
Assumption 5.1. (Similarity). Let (M, and (M'()') represent the source and target ICI problems. The intersection of their cunstraintvviolati region is not empty: g {{ss a)) A
(*')==) Mu.. (8, a) > 0} n {(s,a) a) A M (s.a) > 0} + Ø.
Under this assumption, we begin by addressing the hard constraint scenario, in which the ICRL solver outputs set of feasible cost functions, all of which be safely transferred to the target CMDP, i.e., ensuring safety within G. This property is illustrated in the following lemma:
Lemma 5.2. Suppose a hard constraint scenario. For any (s', d') E G, the feasible cost function è inferred by the ICRL solver can prevent the visitation to (s', a') in the larget CMDP.
However, the reward correction term learned by the IRC solver fails to guarantee safety in the target environment. In the following, we identify conditions under which Ar is not transferable.
Theorem 5.3. Suppose a hard constraint scenario. At state 8, let a denote the expert action, a denote the action that satisfies (s. a) E G and a denote the other actions. Vr' E [0, Rumxx SAA and PP E A if is S, V a a E A, 3 A that satisfies the following conditions, then a reward correction term Ar constructed by such Q-functions leads unsafety in the target CMDP,
O artAn Mu(rtar) (e (0(a))) ((n,0))) 2 Q TA, M...MTTT MM(r+Ar) (e( - ((g,0))) 2 0,
Art Mu(r+ar) (e (0..0)) ()sa0)) -(s,a))<<(()) (Y p/ -r 11++---' (1" Ar, (U(uuttnn))
(C(s,a)) ()),a)))
Qu Mu(r+Ar) (e(s (((a))) <(8))- // - r +Y (urrrt)) (e(s,a)) ((s,a)))
where Y = (ISXA- - -ISxA-7PPT Y'= (ISXA (P) and ((,,)) denotes a vector with value of 1 at index (s, a) and O elsewhere
The first two inequalities, obtained from Lemma 1, ensure the optimality of the source expert policy F. The last two inequalities state that unsafe actions will be chosen if increments in the Q function of constraint-violatin actions are larger than other actions after transfer. We provide a numerical validation of the above theorem in Appendix B.7, regarding the example in Figure 1.
Extension to Soft Constraint. The above results pertain to hard constraint scenarios. In cases of the soft constraint, i.e., threshold c > 0 in (PI), any (s, a) can be visited by feasible policies because the cost of visiting (s, a) can always be mitigated via the discount factor y < 1. In this sense, cost functions, like reward correction terms, reflect the degree of penalization for constraint- violating actions that can be compensated by new transition dynamics and expert policies. Although the recovered cost function no longer guarantees safety in G. it still outperforms inferred reward correction terms because it resists variations between source and target reward functions. We provide detailed analysis for soft constraints in Appendix B.8.2.
7

---

Published as a conference paper at ICLR 2025
5.2 GENERALIZING OPTIMALITY GUARANTEES ACROSS DIVERSE ENVIRONMENTS
The above analysis primarily emphasizes ensuring safety by satisfying constraints in the target environment. However, it does not guarantee that the resulting policy optimal with respect to constrained reward maximization. In this context, a trivial solution involves blocking all state-action pairs deviating from expert trajectories. However, such an approach compromises the optimality of the resulting policy. Building upon safety guarantees, this section establishes the theoretical foundations for policy optimality in the target environment under the estimated hard constraint. As safety takes precedence over optimality, the IRC solver is excluded from this discussion.
Transferability in Optimality for ICRL Solver. Guaranteeing the optimality of transferred cost functions to target environments requires sufficient knowledge of the environment landscape. To achieve this goal, we recognize the difficulty of obtaining optimality guarantees under unregularized settings (Schlaginhaufen & Kamgarpour 2024, Remark 3.1 and emphasize that exploration is crucial for obtaining theoretical guarantees of transferability.
Assumption 5.4. (Exploration). Let F= E Ba E A: c(s, a) = 0} be the state feasible region, (1 m)m AASxA be the set of occupancy measures that satisfies the Bellman flow constraints. We assume that for any s E F and l E Q. the state occupancy measure uus) := >n (ss, a) is lower bounded by positive constant, i.e., pls) Mmin 0.
and Q = {u T (E - PP)
To ensure the policy can exhibit exploratory behavior, we adjust the CRL objective (PI) by assuming the optimal policy maximizes the regularized objective in the following:
oc
?' (r(st. ux) - X c(se, at) + ((T((1)))))
π arg max Ep PT
where the Shannon entropy regularizer ((T(-1))) := -A [log [a|s)] and the weighting term >> O. Additionally, we model the influence of policy by deriving the occupancy measure ju. As a result, the dual representation of the CRL problem (DI) can be recast as a convex optimization problem (Altman, 2021; Puterman, 2014):
CRLp, (r, c) := arg max J J(r, X", c, p)) with J(r, c. = (r - x=,, - E [x]h(T"))
(5)
M
where u(s,a) = (1 - 7)
2 = s,a, aa] and TH (als) = uus, a(s.a)/u(ss)
Under this formulation, to quantify the performance of given occupancy measure μ under an inferred cost function c, we define the sub-optimality gap as follows:
f
(c,
max J(r, Xr,,"',,,,, p', N'EQ
(6)
Building upon this, we extend the transferability definition from Schlaginhaufen & Kamgarpour (2024, Definition 3.1) from regular MDP settings to CMDP settings.
Definition 5.5. (e-transferability for cost). For some fixed & > O, we say the inferred cost function è is e-transferabble to some new reward function r' E [0, XKmaa]Sx4 and transition law Pf E A if ('0"" (c, (RL( S.
Intuitively, e-transferability quantifies how well the inferred cost function è captures the preference of expert behaviors in different environment. When the gap is large, it indicates that certain feasible state-action pairs are overly penalized, potentially leading to suboptimal policy decisions. Conversely, when the gap is small, the estimated cost function è accurately explains the expert's 's behaviors,
key element in bounding this suboptimality gap in target environments is constrain the similarity between source and target transition laws. To represent the feasible cost set Cacai: we map a transition law Pr to a subspace Ur, == im E >Pr) via potential shaping transformation (Ng et al., 1999). A detailed discussion of this transformation for cost equivalence is provided in Appendix B.28. We utilize principal angles as a generalization of angles between higher-dimension planes to provide a more refined measure of similarity and dissimilarity between them.
Definition 5.6. (Principal angles (Galántai, 2013)) Let V, W C R. be two subspaces of dimension
m < n. The principal angles 0- < (V, W) S S ,.. (V, W) =: m I x/2 between V and W are defined recursively via
max (v, w) s.t. ||0||| ||ul||2 = 1, (u, 0z) (yy) = 0, j = 1, ...i 4-1, 1, cos(0; (V, (V,W)) veV.wEW
8

---

Published as a conference paper at ICLR 2025
where by, wy are the maximizers corresponding to the angle 8z. For two transition laws Pri, Pr2. we define (e(PT,, == 8i (((urr, (Upr2) and m ( PT2) == 8 (upx (
Theorem 5.7. Let P'T be the target transition law, the source and turget ground-truth constraints are the same (d) dº and d1 Il[e Elupy, 1l2. Suppose that Assumption 5.4 holds. If
of' 2'' (e, )R εt, then is e-transferable to the target environment with PpT
E = 2 max {df sin (Omaxx(P',, f2, 22,221/00 /oR /m-
where JR. and are regularity constants, given in Appendix B.25.
The theorem asserts that if the two transition laws are close and the recovered cost has small suboptimality gap in the target environment, then e-optimality of the recovered cost is guaranteed. More specifically, we observe that the transferability in optimality of cost is affected by two key factors. The first factor is the discrepancy between the source and target CMDP, which encompasses: 1) the difference rewards and Lagrange multipliers, captured by E and 2) the discrepancy in transition dynamics, indicated by 0max- The second factor is the estimation error associated with the recovered cost é, denoted by d1.
EMPIRICAL EVALUATION
Gridnariea
GR
We empirically evaluate the ICRL solver against the IRC solver in four different constrained Grid- world environments. For each Gridworld, source and target environments differ in reward func- tions and transition dynamics. Code is available at https: / //gggttubb com Booon//Bouuelll /wt////onsaranntt..//e
Figure 2: Four different Gridworld environments.
IRL.
Experiment Setting. We focus on evaluating the training efficiency and transferability of the ICRL and IRC solvers. The results are assessed using two key metrics:1) discounted cumulative rewards, which quantify the total rewards achieved by the learned policy. 2) discounted cumulative costs, which calculate the total costs incurred by the learned policy. We compare the uniform sampling strategy (Appendix Algorithm 1) of the ICRL and IRC solvers.
In the environment, the agent's goal is navigate from a starting location (blue) to target location (red) while avoiding constraints (black). The expert policy is trained under ground truth constraints. Four source environments exhibit a stochasticity of p 0.05, i.c., the agent takes uniformly randomized action with that probability, while four target environments feature higher stochasticity of p All rewards are assigned at the target location, with identical values of in four source environments and 2, 7, 7, 15 in four respective target environments.
Figure demonstrates four learned cost functions by the ICRL solver at each state (top) and four learned reward correction terms by the IRC solver at each state (bottom), in the source environments of Figure 4 demonstrates the accumulated
four Gridworld settings (from left right, Gridworll-1,2.333 rewards and costs of resulting policies learned under inferred reward correction terms by the IRC solver and cost functions by the ICRL solver at each iteration. In the source environments, we observe that the reward and cost curve of the IRC solver converges more quickly than the ICRL solver, indicating higher training efficiency. However, in the target environments, we find that inferred reward correction terms lead unsafe policies (costs exceeding 0), whereas recovered cost functions ensure both safety (costs converging to 0) and optimality (rewards converging to the expert).
For continuous environments, we use the maximum entropy framework of ICRL (Malik al., 2021) and simplified IRL framework for constraint inference (Hugessen et al., 2024) where two frameworks recover constraint knowledge that best explains the expert demonstrations from an offline dataset. We report additional experimental results and implementation details in Appendix C, D and E.
7 CONCLUSION
Summary. In this paper, we study the novel challenge of transferring learned constraint information from source to target environments. Constraint information can be either implicit reward correction

---

Published as a conference paper at ICLR 2025
Iteration255
15
Iteration:50
Iteration-3 35
12.5
6
Iteration550
2.5
10.0
10
1.5
3
5.0
2
1.0
2.5
0
0.0
0.2
keration-35
Iteration-00
0.0
0.0
-0.2
3
-022
-0.2
0.4
-0.4
00.4
Figure 3: Learned constraint information by the ICRL solver (top) and the IRC solver (bottom).
Legan
1.20
couice
Eran
222
Sraen Lage
Boratin
Tordoon
Insttn
Laye
levis
Pantoe
Figure 4: Training curves of discounted cumulative rewards (top), costs (bottom) for the ICRL (red) and IRC (blue) solvers across four Gridworld environments. The expert's 's rewards and costs are represented in grey. Solid and dashed lines correspond to source and target environments, respectively.
terms by IRC solvers or explicit cost functions by ICRL solvers. compare both solvers regarding training efficiency and transferabilityy Although the IRC solver is guaranteed lower sample complexity, the additional sample complexity required by the ICRL solver proves crucial for ensuring both safety and optimality in target environments. Under hard constraints, the recovered cost functions strictly prevent the agent in target environments from entering overlapping constrained regions, whereas the reward correction terms can be easily offset by variations in transition laws and reward functions, leading to insufficient constraint representation. We also derive conditions that limit the similarity between source and target environments to ensure the optimality for the ICRL solver. Empirical studies across various environments validate our findings.
Limitations and Future Work. Our research initiates intriguing avenues for future studies. First, extending our approach to incorporate demonstrations from multiple experts. including sub-optimal ones, would be valuable contribution the field. Second, we believe that extensions more complex and scalable tasks, particularly those in real-world applications, such as safety issues in robotics and Large Language Models, could offer valuable insights into the practical challenges and opportunities involved in transferring constraint information.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported in part by Shenzhen Fundamental Research Program (General Pro- gram) under grant 0111202208001442220000
Muungdong-Shennzher Joint Research Fund under grant 2023A1515100111 Guangdong Basic and Applied Basic Research Foundation under grant 2024A1515012103, and Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023B (212010001) Poupart and Gaurav were also supported by the Canada CIFAR AI Chair program, discovery grant from the Natural Sciences and Engineering Research Council of Canada and grant from ITTP & MSIT of Korea (No. Hub Project).
Al Research
REFERENCES
Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
Kareem Amin, Nan Jiang, and Satinder Singh. Repeated inverse reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
Haoyang Cao, Samuel Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learning. In Advances Neural Information Processing Systems (NeurIPS), volume 34, pp. 12362-12373, 2021.
Zeyu Feng, Bowen Zhang, Jianxin Bi, and Harold Soh. cansty-con policy transfer with successor features. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7219-7225, 2023.
Aurél Galántai. Projectors and projection methods, volume 6. Springer Science & Business Media, 2013.
Ashish Gaurav, Kasra Rezaee, Guiliang Liu, and Pascal Poupart. Learning soft constraints from constrained expert demonstrations. In International Conference on Learning Representations (ICLR), 2023.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois C. Knoll. A review of safe reinforcement learning: Methods, theory and applications. arxiv preprint arxiv: 2205.10330, 10330, 2022.
Adriana Hugessen, Harley Wiltzer, and Glen Berseth. Simplifying constraint inference with inverse reinforcement learning. In the First Reinforcement Learning Safety Workshop at Reinforcement Learning Conference (RLC), 2024.
Konwoo Kim, Gokul Swamy, Zuxin Liu, Ding Zhao, Sanjiban Choudhury, and Zhiwei Steven Wu. Learning shared safety constraints from multi-task demonstrations. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang Mind the gap: Offline policy optimization for imperfect rewards. In International Conference Learning Representations (ICLR), 2023.
David Lindner, Andreas Krause, and Giorgia Ramponi. Active exploration for inverse reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 5843-588 5553-585,,00 2022.
Guiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezace, and Pascal Poupart. Benchmarking constraint inference in inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2023.
Guiliang Liu. Sheng Xu, Shicheng Liu, Ashish Gaurav, Sriram Ganapathi Subramanian, and Pascal Poupart. A comprehensive survey inverse constrained reinforcement learning: Definitions, progress and challenges. arXiv preprint 2rXiv:2409.07569, 2024.
Shicheng Liu and Minghui Zhu. Distributed inverse constrained reinforcement learning for multi- agent systems. In Advances in Neural Information Processing Systems (NeurIPS), 2022.

---

Published as a conference paper at ICLR 2025
Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints in model-free reinforce- ment learning: A survey. In International Joint Conference on Artificial Intelligence (IJCAI), pP- 4508 1508-4515, 2021.
Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International Conference on Machine Learning (ICML), pp. 7390-7399, 2021.
Alberto Maria Metelli, Giorgia Ramponi, Alessandro Concetti, and Marcello Restelli. Provably efficient learning of transferable rewards. In International Conference on Machine Learning (ICML), pp. 7665-7676, 2021.
A. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning (ICML), 1999.
Andrew Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning (ICML), pp. 663-670, 2000.
Santiago Paternain, Luiz F. O. Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. In Advances in Neural Information Processing Systems (NeurIPS), pp. 7553-7563, 2019.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Guanren Qiao, Guiliang Liu, Pascal Poupart, and ZhiQiang Xu. Multi-modal inverse constrained reinforcement learning from a mixture of demonstrations. Advances Neural Information Processing Systems (NeurIPS), 2023.
Guorui Quan, Zhiqiang Xu, and Guiliang Liu. Learning constraints from offline demonstrations via superior distribution correction estimation. In International Conference on Machine Learning (ICML), 2024.
Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science Business Media, 2009,
Paul Rolland, Luca Viano, Norman Schürhoff, Boris Nikolov, and Volkan Cevher. Identifiability and generalizability from multiple experts in inverse reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 550-564, 2022.
Andreas Schlaginhaufen and Maryam Kamgarpour. Towards the transferability of rewards recovered via regularized inverse reinforcement learning. arxiv preprint arxiv: 2406.0/793, 2024.
Dexter R. R. Scobee and S. Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2020.
Joar Max Viktor Skalse, Matthew Barrugia-Roberts, Stuart Russell, Alessandro Abate, and Adam Gleave. Invariance in policy optimisation and partial identifiability in reward learning. In Interna- tional Conference on Machine Learning (ICML), pp. 32033-32058, 2023.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Sheng Xu and Guiliang Liu. Robust inverse constrained reinforcement learning under model misspecification. In International Conference on Machine Learning (ICML), 2024a.
Sheng Xu and Guiliang Liu. Uncertainty-awan constraint inference in inverse constrained reinforce- ment learning. In International Conference on Learning Representations (ICLR), 2024b.
Bo Yue, Jian Li, and Guiliang Liu. Provably efficient exploration in inverse constrained reinforcement learning. arXiv preprint arXiv:2409. /5963, 2024.
Quanqi Zhang, Chengwei Wu, Haoyu Tian, Yabin Gao, Weiran Yao, and Ligang Wu. Safety reinforcement learning control via transfer learning. Automatica, 166:111714 2024. ISSN 0098
12

---

Published as a conference paper at ICLR 2025
Runyi Zhao, Sheng Xu, Bo Yue, and Guiliang Liu. Toward exploratory inverse constraint inference with generative diffusion verifiers, In International Conference on Learning Representations (ICLR), 2025.
13

---

Published as a conference paper at ICLR 2025
Table of Contents
A General Notations and Nomenclature
14
Definitions, Theorems, and Proofs
14
B.1
Additional Definitions
14
B.2
Estimating the Transition Model and Expert Policy
.i..............................
15
B.3
Uniform Sampling Strategy for Each Solver
16
B.4
Theoretical Results of TRL Solver
16
B.5
Theoretical Results of IRC Solver
17
B.6
Theoretical Results of ICRL Solver
19
B.7
Numerical Analysis of the Example in Figure
20
B.8
Theoretical Results of Transferability in Safety
21
B.8.1 The Hard Constraint Scenario
21
B.8.2 The Soft Constraint Scenario
23
B.9 Theoretical Results of Transferability in Optimality
25
C
Comparison in Soft Constraint Scenarios
26
Comparison in Continuous Environments
27
Experimental Details
28
A GENERAL NOTATIONS AND NOMENCLATURE
In Table 1, we report the general notations and nomenclature used in paper.
B DEFINITIONS, THEOREMS, AND PROOFS
B. ADDITIONAL DEFINITIONS
Definition B.1. (Hausdorff distance). (Rockafellar & Wets, 2009). Let (M, d) be metric space. The
Hausdorff distance d between two non-empty subsets C M and B C M with distance function d is defined as:
dn (A, B) max sup inf d(a, b), sup inf d(b.a) BEB aEA
(aeA beB
where:
infse d(a, denotes the shortest distance from point a E A all points set
suPaea infeeB d(a, b) is the maximum of these shortest distances for all points in A.
Similarly, supbe infaed d(b, a) represents the maximum shortest distance for all points in to A.
Definition B.2. (Operators). Let RS and g e S We abuse Py and π as the operators in-
duced by the transition model and by the policy म. (P..(()))) = =Eves Pr(''ss, a)fss) and (s) = EneA (a)))) a). Moreover, the expansion operator (Ef)(s, = f(s). Given म e A denote with (B" (s, a) g(s,a) a) (=(as)>>0) and (B" (), g(s, (=(als) 0).
14

---

Published as a conference paper at ICLR 2025
Table 1: General Notations and Nomenclature
Symbol
Name
Signature
State space
A
Action space
T/r
Source/Targel reward function
SKA AS
PT
Source Target transition model
(TT)
Cource/Target expert policy
e
Expert cost function
IS
Cost function
TR S
x.A
Ar
Reward correction term
SKA
Q
Reward action-value function for of π in M Reward state-value function for of in M Reward advantage function for of in M Cost action-value function for c π in Me Cost state-value function for of π in M Reward action-value function for r of π in Me Reward state-value function for r of π in Me Reward advantage function for r of π in Me
SxA
V
IS
SXA
IKA
Vea
I
IKA
M
V
IRS
M
SRA
M.
ICI problem (M, B)
SiRC
IRC solver
Ry SICRL
Set of feasible reward correction terms
{IR S
ICRL solver
Cp
Set of feasible cost functions
Target accuracy
R-
Significancy
(0, 1)
Principal angle
[0, T/2
|| 1 ol
Infinity norm
Expansion operator
R$S R.SAA
im (T)
Image of a linear operator T
Definition B.3. (Image of linear operator). Let T'; V W be a linear operator from vector space V to vector space W. The image of T, denoted im) is defined as:
im( = (T(b) u V
This is the set of all vectors in W that can be written T(r) for subspace of W. Definition B.4. (Inifinity norm). For vector a, we define the vector infinity norm as ||a|bx [ax. For a matrix M, we define the matrix infinity norm as || = max; E, Magl
V. The image of T is
B.2 ESTIMATING THE TRANSITION MODEL AND EXPERT POLICY
We define how estimate the transition model and the expert policy with generative model in Algorithm When the generative model is queried about a state-action pair (s, a), it responds with transition triple (s,a, s') and an expert action a . (s)
In each iteration of uniform sampling, we query the generative model with each state-action pair multiple times. At iteration k. denote by ns,s, a, s') the number of times observe the transition tuple (s, a, s'). We further denote ( a) = Eves ( a, s'). At iteration k, we denote by n n a) the number of times we observe action as an expert decision at state s. We further denote n (s) EaEa n (8,a).
We define four cumulative counts from iteration 1 to as Na(s,a,s') = =E ny(s, a,s) and (s D ()) (s, Σ nf a) and N E}= nf((s). Finally, the
15

---

Published as a conference paper at ICLR 2025
estimated transition model and the estimated expert policy defined as:
Pra(s'|s,a)= N,(s,,,s') (als)) No. (s, a) N (s)
(8)
(s,
where x+ = max(1,
B.3 UNIFORM SAMPLING STRATEGY FOR EACH SOLVER
To acquire desired information, the agent utilizes the uniform sampling strategy to query a generative model. Specifically, the agent can always query generative model about state-action pair (s, a) to receive a next state s' ~ P(-ss, a) and an expert action a x (-1s).
Algorithm Uniform Sampling Strategy With a Generative Model
Input: significance 8, target accuracy E, maximum number of samples per iteration Ryaa Initialize k +- 0, En I-
while Ek > E do
Collect Amuxx samples from each (s,a)= E 8 A
For IRC solver, update accuracy Ex+1 I-7 max
IA" (+1((,,0) (sa)ESxA
For ICRL solver, update accuracy Ek+1 1-7
max
I (8,00)
(s.0)ESxA
Update PT and 7 according to Eq. (8)
l + +1
end while
B.4 THEORETICAL RESULTS IRL SOLVER
In this part, we reformulate the IRL solver for ICI based on Metelli et al. (2021).
Inverse Reinforcement Learning (IRL). Typically, IRL solvers recover the reward function from expert demonstrations, where the environment is always considered to be safe. To employ IRL for constraint inference in CMDP, we modify the original formalization of the IRL problem (Ng et al., 2000) as follows:
Definition B.5. (IRL solver ICI). An IRL solver is denoted as SIRL- A corrected reward function r is a feasible solution for an ICI problem B if and only if T an optimal policy for M Ur. We denote by Roy the set of feasible corrected reward functions.
Lemma B.6. (Feasible reward set implicit) (Metelli et al.. 2021, Lemma 3.1). Let D be an ICI problem and SmI be the IRL solver, r a feasible corrected reward function, i.e, r E Rop if and only (s, a) E S x A:
T 0. if (als) o,
=
mT Mure (s).
S Vr st
Mure (s).
Lemma B.7. (Metelli al., 2021, Lemma B.1), Let LetR be an ICI problem and SIRL be the IRL solver.
A Q-function satisfies the condition of Lemma B.6 if and only there exist Ç RSAA and V" g such that:
T = EVE.. Muu
(9)
Furthermore, |/ S [l 1o and the the expansion operator E satisfies (Ef) (()).
Lemma B.8. (Feasible reward set explicit (Metelli et al., 2021, Lemma 3.2)). Let & be an ICI problem and SYRL be the IRL solver. A reward function r E Rog if and only if there exist ( e 20 and V E R$ such that
S
r --
(10)
16

---

Published as a conference paper at ICLR 2025
Lemma B.9. (Error Propagation (Metelli al., 2021, Theorem 3.1)). Let & xM,x)) and
T (M, be two ICI problems. Then, for any r" Rop such that = (T and ||r I Rmax. there exists F E Rg such that dlement-wise it holds that:
I Ml SISB" B s+ (Pr Pr) v.l
(11)
Furthermore, Ol S Buus and |VT S R.NN.
T-y
Theorem B.10. (Sample Complexity of the IRL Solver (Metelli et al., 2021, Theorem 5.1)). If the IRL solver stops iteration K. with accuracy EK, then with probability least 1 5 it converges, with number of samples upper bounded by:
S Ô 2 R2 R SA
(12)
71
Proof. The above result can be obtained by setting M' M and 7 = in (Metelli et al., 2021, Theorem 5.1).
Remark B.11. In IRL. solvers, although r in the CMDP is independent of the cost function c, r indeed captures the information of underlying constraint signals for enabling the imitating agent to reproduce R.. However, the IRL solver lacks a mechanism to leverage the known reward function r for constraint inference. Moreover, since r relies the current reward function, directly transferring rf to another CMDP with different reward function does not make sense. Therefore, it is necessary adjust the setting of IRL solver accommodate potential changes the reward function. et al. (2023) propose modifying imperfect reward functions to align them with expert behaviors, but this approach lacks tractable sample complexity. To address these limitations, we propose a modified version of the IRL solver from a feasible set perspective, namely the Inverse Reward Correction (IRC) solver in the main paper.
B.5 THEORETICAL RESULTS OF IRC SOLVER
In this part, we provide additional results regarding the IRC solver for ICI.
Inverse Reward Correction (IRC). Based the known IRC solvers learn reward correction term Ar to capture constraint signals. The goal of IRC solvers is to enable the imitating agent to match expert demonstrations by following the corrected rewards: r"(ss, a) r(s, Ar(s, a), V(s x A.
Lemma 4.1. (Feasible reward correction set implicit). Let be an ICI problem and Sinc be the IRC solver. Ar is a feasible reward correction term, i.e., Ar E Rop if and only if a) E x A:
(i) if (als) 0, Qx (8,
vrtAr=" MU((+Ar)) (s).
a (ii) (fls) O. extrr (s. Muirtar) (s).
Proof. From Lemma B.6 and the decomposition that r", a) r(s.)) Ar(ss V(s, a), we have,
(uurtan) (s,a) vr+ara" " (ii if (als) = 0, artrr
(i if if 0, Oxtrr"" "
Muirta))( (8).
(Mr(rAr) (8, & vetdr. (M((+Ar))))) (s).
Corollary B.12. (Feasible reward correction implicit). Let T be ICI problem and Siee be the IRC solver. Ar is feasible reward correction term, i.e., Ar E Ry if and only if for all (s, a) E S x A. it holds that:
(i) (als) 0. R R
(1 (s.a) - -rr
Muur+A)) (s. + v
Mu (s),
(ii) (als) O. Q (8,0) -Aj (s, + A
Mu(+
17

---

Published as a conference paper at ICLR 2025
Proof. Note that, for any given policy и, Q-function and V-function are both linear towards re- ward function, i.c. @uurri Q +r()+ + Mr(n and V Mu()) Thus, inheriting from Lemma 4.1, we obtain, VP2,5
V
Mu(r Mr(n+
(i if (als) 0,
Ar.ae Q Mu(r+An) Mu(r+An)) (8,a)
Qra
Vn V-draa MMU(++A)) *Mu(++dr) (s),
(ii) if 8 (a|s) 0. Q Qu55 Mu(r+Ar) (0,0) S (u(+++r)) Mu((+Ar) (5).
V -Arr,
By simple transposition, we derive,
(i if (als) 00,
An. Quuirtar) (8, = Q Mu + Va (u(r+s))(() Mu((+ar)) (8),
VAn
(ii) if (a|s)
(5, S - Q Mu(r+Ar) (8.0) + Vha Mu(++An) (8) v
Mu(r+ar) (8).
By the definition of advantage function A (s,a) = Quiun (s,a) a) - Mun (s), we obtain,
i x B (als) > 0, Q Mn (8,a) -A A (8,))))))))))))))))))))))))))) + V
(8.0)
Mu(r+Ar) (8),
(ii) if (als) = 0.
Ana Qu
-MM(++ar)) (8.0) + VArge MM (5).
Lemma B.13. Let & be an ICI problem and Sinc be the IRC solver. Q-function satisfies the condition of Lemma 4.1 if and only there exist C ISAA and V RRS such that:
Q
An Mu(r+AM) -B" A
Mu(r+An) + EV".
(13)
Proof. The proof can be easily derived from from (Metelli et al., 2021, Lemma B.1).
Lemma B.14. (Feasible reward correction set explicit). Let B be an ICI problem and Sixe be the IRC solver. Ar is a feasible reward correction term, ie., Ar E Rop if and only if there exist Ç E SxSAA and V ERS such that:
20
= - c-r+ (E PP) V Mu(r+An) + +B--7r)VV* V
(14)
Proof. From Bellman equation (Sutton & Barto, 2018), we have,
Ar,zE
Ar -Pp Muir+Ar))
r - (1sxA-7PTT7) 2
Q'uu(taay
Thus, we obtain.
Ar (ISXA- - 7PTmF) (u(r+Ar)
= -ASXAA - 7PPx (-B"* <
A Mu(r+ar) + EVT)
5 --B²
A
MU(r+AA) EV- + BPPT=B A
MU(r+Ar) VP
B= < (IsxA-7PP7 B) A
AMU((tan) + -PT) V
B"
- " ( (IsxA-7PPPp B) (QMu(r+sn)
EV R
MU(r+Ar) + +E V
BB" (TxXA 7P Q
(mu(t+a)) + (ISXA 7PTx F.V
MU(++A)) + (F (F- (8--PT) V
=-B" --r+ (E - 7PT) V
Mu(r+Ar) + + VT.
(15)
where the last equality utilizes _ Is-
n
18

---

Published as a conference paper at ICLR 2025
Lemma B.15. (Error Propagation). Let & = (M, )) and ②3 (M, be two ICI problems. Then, for any Ar Raz such that Ar =BB < --r+ (E- vn there exists Ar E R such that element-wise it holds that:
MM(r+Ar) (E-yPT)V V
Ar Ar B'B
E B S
(pr7((pr-P))) (V + +Vm)
(16)
Furthermore, 11 I 2imaax 1-7
V
Mu(r+Ar)
1oo Rmxx and ||o I Blax I-9
I-7
Proof. From Lemma B.14, we can express Ar and Ar as,
Ar = -B" ( T- (E- -PP) VT e
Mr(r+Ar) + (E-TP) V
(17)
2h Ç r Ar -B
(E- pp-yPr)vuR Mu(r+Ar) + (E--
(18)
Since look for the existence of Ar, provide specific choice of ? and Vi: /= B" Ç and
V vr= Vr + Mu(r+Ar) Mu(
v
Ar Ar -B & 1 <-> (
(VAT (U + +
(19)
By taking the absolute value and applying the triangular inequality, we obtain:
Ar
S B
B < + (Pr (Pr-Pr) (Viuir+Ar) + +
(20)
Note that the Loo norms of value function Lemma B.13, 11
||
| Mu((+rr)lle S
E
Buss and | S Rmss 1-y
Then, by
1-7
2Rmax 1-7
Theorem 4.2. (Sample Complexity of the IRC Solver). If an IRC solver stops at iteration K with accuracy εx, then with probability least 5 it converges, with number of samples upper bounded by:
4-, R2 ReAA SA (1 7)
n S
(21)
Proof. Compare Lemma B.15 for the IRC solver with Lemma B.9 for the IRL solver. Using the proof techniques from (Metelli et al., 2021, Theorem 5.1), we derive the following sample complexity for the IRC solver.
Remark B. Compared to the IRL solver, the IRC solver utilizes the known reward signals for constraint inference. This method treats r as "imperfect" rewards and learns a correction term Ar(s, a) to incorporate constraint signals. Note that the IRC solver does not explicitly model the constraints during inference since the solver considers an unconstrained RL problem instead of CRL during policy updates.
B.6 THEORETICAL RESULTS ICRL SOLVER
In this part, we provide additional results regarding the ICRL solver for constraint inference.
Inverse Constrained Reinforcement Learning (ICRL). ICRL solvers infer the constraint respected
by the expert agents from their demonstrations. An ICRL solver admits the following assumptions.
Assumption B.17. Either of the following two statements holds:
(i) The constraint in Eq. (PI) is a hard constraint such that = 0;
(ū) The constraint in Eq. (PI) is a soft constraint such that e > 0, and the expert policy is deterministic.
Lemma 4.3. (Feasible cost implicit (Yue al., 2024, Lemma 4.3)). Under Assumption B.17, let Q be an ICI problem and SICRI. be the ICRL solver, then c a feasible cost, i.e., e E Cyy if and only if (s,a) E 8 A:
19

---

Published as a conference paper at ICLR 2025
(i) if i... (s,a) follows the expert policy:
_ (Muc(s, Mue (s) = 0.
(22)
(ii if (als) = 0 and A a) i.i.e, (s,a) violates the constraint:
E (s Vu (s) > O.
(23)
(iii (als) = and A (s.a) & i., (s, a) is in the non-critical region:
Q
Mus S.
(24)
Lemma B.18. Let F be an ICI problem and SICRL be the ICRL solver. A Q-function satisfies the condition of Lemma 4.3 if and only there exists Ç E Sx and V R$ such that:
R Q = A 5+ + EVE.
(25)
where the expansion operator E satisfies (Ef)(s) f(s
Lemma B.19. (Feasible cost set explicit (uue al., 2024, Lemma 4.4)). Let @ be an ICI problem and SICRL be the ICRL solver. e is feasible cost, i.e., E Cp if and only if there exist ( E S and V E RsS such that:
20
A MuG5 (E- (E-?Pr)Ve.
(26)
Similarly, we show how the estimation error on the environmental dynamic and on the expert policy propagates the cost function.
Lemma B.20. (Error Propagation (Yue et al., 2024, Lemma 4.5)). Let T = x, and के =
#M be two ICI problems. For any e Cp satisfying c AMuss + (E - pPr)V and
||c]boo S Crux. there exists é Cip such that element-wise it holds that:
- 2p - PT] + A - A
Mur 5.
(27)
Furthermore, (+)lo) & Cmax/(1 - 7) and 1151 S /wax\/ min (s,a) AMruett
Lemma B.21. (Yue et al., 2024, Lemma 4.6) For a given policy n, let A Mur denote the reward
advantage function based on the original MDP M. For an estimated policy R. let A a Muc
denote the
reward advantage function based the estimated MDP M. Then, have
A - ATT
<
27
-
PTF-PT)V'
7(1+7) Pn + 1-7 (T-
(28)
MUe
Theorem 4.5. (Sample Complexity of ICRL solver Yue et al., 2024, Theorem C.9)). If ICRL solver terminates at iteration with the updated accuracy ex, then with probability at least 1 5, converges with number of samples upper bounded by
7022SA 15
n 5
(29)
where V3aCmax (Rmax(3 + / min+ AMueel
+ (1-
Remark B.22. Compared with the IRC solver, the ICRL solver considers a CMDP environment instead of an MDP one, so it explicitly models constraints during policy updates.
B.7 NUMERICAL ANALYSIS OF THE EXAMPLE IN FIGURE
As illustrated in Figure the basic environment contains 2 x grids, where the agent navigates from the starting location (0. 0) (the left bottom corner) to the target location (0, (the left top comer) with four possible actions, i.e., Up. Down, Left and Right. the source environment, (0, 4) (colored blue) is assigned with reward of 10, 8f (0, 2) (colored orange) contains hard constraint that
20

---

Published as a conference paper at ICLR 2025
must not be accessed, while all other locations are assigned with 0 rewards and 0 costs. The discount factor 7 = 0.7 and there is no stochasticity in the source environment. For better readability, in this part, we simplify Qu +Ar,5
(u((r/+*ar) (s, a) as Qr/r'+Ar and vefr'+Ara
MM((/r'-Ar) (r//// (s.0) a) as Ve/r' (a).
If we do not add the reward correction term Ar, under the current reward function r((0, 4)) 10, the agent chooses to go Up at (0, 1), which violates the hard constraint and deviates from the expert policy. To guide the agent to go right at (0, 1) instead of going up, a feasible reward correction term Ar((0,2) 2)) = -3 and are zero at all the other locations, which, together with reward function r((0, 4 = 10, leads to
Q *rt ((0, Up) = Ar((0,2) + + Ar, ((022) = -3 + 0.7 X 7 = 1.9, Qrt e ((0, 1), Right) = Ar((1, + Ar+ ((1, = + x 3.43 = 2.401.
This ensures the optimality of the expert policy at state (0, since
Q Qr+am E ((0, 10,1),up Up)
Qr+an ((0, 1), Right),
meaning the safe and optimal action Right is chosen. Figure illustrates the source environment, the optimal value function regarding r + Ar, and the optimal policy each state.
Suppose we have a target environment, with r'((0,4) 4)) = 20, and stochasticity of 0.3 to state 41, (1,3), (1, and (0, 1) of the environment and there remains no stochasticity to all the other states. Figure 6 (middle) shows the new reward value function regarding r' + Ar. Figure 6 (right) shows the policy generated by r' with Ar((0, 2)) -3. We observe that Ar((0, 2)) = -3 is not enough to penalize the going up action at (0, 1), since
ar'AAr=& F ((0, 1), Up) = Ar((0, 2)) + 7 V + ((0, 2)) = -3 3 0.7 x 14 6.8,
Q +r,= ((0, 1), Right) Ar((1, 1)) + 7 + (1, = -0+ 0.7 x 5.78 4.046, Q''+Ar,=" ((0, 1), Up) > Q Ar Ar ((0, Right),
meaning the constraint-violating action Up chosen.
Next, we validate Theorem 5.3. At state (0, 1). a = Right, ac = Up, of = Down, og Left. Through numerical calculation, we derive the following results, which match the results of Theorem 5.3.
Qu+ar(a Ar = 0.501 2 0;
@r+ar(aF) 2r 1.225 0: - ( 0.720 > 0:
@r+)r(aB) - t0 (a = 0.501 < <V - Y')Qr+Ar - -a = 3.701;
@rtar(aP) 0r (a -0.224 (X -') (aC af 4.312;
ar+ar(a7) - art (a 0.219 < (Y') t' 0 4.060.
B.8 THEORETICAL RESULTS OF TRANSFERABILITY IN SAFETY
B.8.1 THE HARD CONSTRAINT SCENARIO
Lemma 5.2. Suppose hard constraint scenario. For any (s', a') E G. the feasible cost function è inferred by the ICRL solver (3.1) can prevent the visitation to (s', d') in the target CMDP.
Proof. For any (s', aa) e 9 and efficiently small cost estimation error E, there must be č(s', a') > 0 by the ICRL solver, which bans action a' state s'. In a hard constraint scenario, when ê is transferred to a target CMDP, this ö(s', a²) will also ban this action at state s', since any policy with rraalls') > 0 leads violation of the hard constraint.
21

---

Published as a conference paper at ICLR 2025
Env
Reward Value Function
Expert Policy
10.0
7.0
U
7.0
4.9
U
U
2.4
3.43
R
U
1.68
2.4
U
U
Figure 5: The basic environment (left), the source reward value function under and Py (middle), and the expert policy of the example in Figure (right).
Env
Reward Value Function
Expert Policy
18.08
20.0
12.96
U
L
14.0
9.07
U
U
6.8
5.78
u
U
4.76
3.78
u
Figure 6: The basic environment (left), the target reward value function under r' and Pf (middle), and the generated policy (right).
Theorem 5.3. Suppose hard constraint scenario. At state let E denote the expert action a denote the action that satisfies (s, a E and a denote the other actions. Yr' E [0, R S and VP E A 3s S, v a a E A, 3 A that satisfies the following condition, then the reward correction term Ar constructed by such Q-functions violates safety the target CMDP,
Orrr Mu(r+Ar)
((s,a))) 2 0, (Mlrttr)(e(aaat) (0(xaa)) ((na))) 2 0, Mu(r+Ar) (e(s,a 20
Art Mu(ttrr (e (0(,,05) ((s,00)) < (Y') -r ++ (++(Y--')) -Y-YY""""""" -"") (MM)uttaa)))) (e(s,a (),a
Ar.
F (Mu(tArr) (e( ((saa)) <( / --r+ (Y V') Argm
Mu(rtar) (e(s,a (ssaa)))
where Y = (ISxA - D Y' Y'= (1sxA - (7) and ((s.a) denotes a vector with value of 1 at the index (s, and 0 elsewhere.
Proof. For better readability, in this part, we denote Q" = Q
M'u("'+rr) and Q" Ar,
Muir+ar) Since, at each state E the agent chooses its action based on the maximum Q-function, we study the Q-function in the source and target CMDP.
Q' (s (JsxA B (8,a) (IsxA (IsxA-
(,,
Ar(s,
22

---

Published as a conference paper at ICLR 2025
Q' (s. (1 (Is) Ar(s.
We next show how the Q-function in the target CMDP is shifted from the source CMDP.
('(s,a) = (ISXA 1r+
(30)
Q' (8,a) (ISXA - -rP{(m) (s,
= (ISXA - -)p{(t) (ISXA -PPT* (8,
(x')~1(6 1(' +r)+
-(")) [v'==)=VVV=++++]]]]
(31)
Based on Eq. (31), we have
-+- +
- (1) 1-0--')01 (8,
+ (s
(Y Y')QI (s,a Q (s,
(32)
The first two inequalities result from Lemma 4.1, aiming to ensure the optimality of expert policy.
The third and fourth inequalities are derived by substituting Eqs. (32) into Q' (s,a*) Q (s.a²) and Q' (x, a) < Q" (s,a²) construct target Q-functions that drive the agent choose constraint- violating actions a
Given Y (ISXA FP and Y -(ssAA PP% (PT(()) (T)) we have
Y - Y' = (PP((()) PT
= ( PT + -
- P
-
- (T)) -=)((())====)
(33)
transidinn difference
expert policy difference
A +
Based on (33), we can further split the increment from Q" (s, in (31) into:
(s) ('(sa))QQ(ssa)) = (-
AQ'
(Q'
(s,
(34)
reward transfer shift
Iransition transfer shift
expert policy transfer shift
We observe how transfer from the source CMDP the target CMDP affects the Q-value function, driven by variations in the reward functions, transition dynamics, and the expert policy.
B.8.2 THE SOFT CONSTRAINT SCENARIO
In this part, we extend our theory from the hard constraint scenario to the soft constraint scenario. Unlike the hard constraint scenario, where we construct constraint
state-action pairs (i.e.,
{(s, a)(c(s,a) a) > 0}} that sensible agent should never visit, the soft constraint scenario is more challenging. Given a threshold
0, matter how large cost c(s, a) is. the agent can always visit (s, a) after sufficiently large number of steps as the effective cost of visiting (s, a) can be reduced below the threshold via the discount factor 7 raised the power of [logo(s.a) c] + 1.
For IRC solvers, in the soft constraint scenario, safety is also not guaranteed by the transferred reward correction term in the target environment. Lemma 5.3 still applies, as long as +
For ICRL solvers, the key distinction between the hard and soft constraint scenarios is that the inferred cost function no longer guarantees safety, since Q (s,a) va(('') 8 M'ue (s) > 0 (case (ii) in Lemma 4.3) is necessarily satisfied for (8, a) E G. However, demonstrate that the
23

---

Published as a conference paper at ICLR 2025
ICRL solver still offers better transferability than the IRC solver in safety, as it remains unaffected by reward transfer shifts.
Since at each state S, the agent examines whether action is safe or not based the cost Q-function, we first study the cost Q-function in the source and target CMDP, which are defined as:
('(8, MUe [[1sxA - - (8,a)
(35)
Q' as) Co((=) Quirle Q'uu
(P)))) (8,
(36)
Similar to the transferability of reward correction terms, further discuss the influence of different reward functions, transition models, and expert policies on the transferability of cost functions. We next show how the cost Q-function in the larget CMDP is shifted from the source CMDP.
From (35) and (36), we obtain
Q (s.a) -X(')""00(()
= =I Y' (o ()))(((aa)
= -XX)"((X-X))))) Q
(37)
Given (33), the increment in cost Q-functions can be split into:
(Q")
(Q'
(s,a)
(38)
transition transfer shift
expert policy transfer shift
By comparing (34) with (38), and cost functions become evident. The transfer of cost functions by the ICRL solver is influenced by: (1) transition shifts and (2) expert policy shifts. In contrast, the transfer of reward correction terms by the IRC solver is affected by: (1) reward shifts, (2) transition shifts, and (3) expert policy shifts.
clearly the differences in transferring reward correction terms
Next, identify the condition under which cost function is not transferable. At state for a E G. we further distinguish two cases. Let a Co denote the action that satisfies (s, a: E
{(s, a)) A) (s, > 0} n ((s, a) MMlle)) Mr a) > 0} and a² denote the action that satisfies (s, 22) {(s, A (s. 0/ (or)"
Based on Eq. (37), we have
Q'
Q (s = +r==vV==100(s=a===000
Q (s. (V))= +
(s = (x) =
+
(1)) (',, (a')
(39)
sufficient condition that the ICRL solver fails would be choosing a1 in the target CMDP. Specifi- cally, we should show the condition under which with new reward function and new transition Pf. 3s E S, V _ A and V a E A. ci E A: Q (s, (a')E) > Q (s, a1) (a does not violate the constraint but achieves higher rewards). We formally state this condition the following lemma.
Lemma B.23, Vr' E [0, Rmax]SXA and VP E A Is E S, a E and V a E A. C2 E A that satisfies the following condition, then a cost function e. constructed by such cost Q-functions is mon-transferable in the target CMDP,
a (,,,, (., ()ssaa)<< (s,
Q3, a (s, (.. Q's,a) - > 1(X') - LIIIX-Y''QQ(ee, (") - -().())))))))))))))))))
(40)
where ((s,a) denotes vector with value of at the index (8,a) and O elsewhere.
24

---

Published as a conference paper at ICLR 2025
Proof. The first three inequalities result from Lemma 4.3, aiming to ensure the optimality of expert policy in the source environment. For the fourth inequality, based on Eqs. (39), the condition
Q (a)) Q (s,
equivalent
By simple transpositions, we obtain
(d.)) - Q7 a >
-
(c,(()))))
(41)
Corollary B.24. If two additional assumptions are satisfied: the transition model does not change, Le., Pa = PT; 2) the optimal policy of source CMDP is the optimal policy of target CMDP, iee. HM, S Ll' then feasible cost function inferred by the ICRL solver (3.1) can prevent any visitation (8',a')
(.)')))) G + Ø the targei CMDP.
Proof. Suppose (s', a') E + Ø, at state s', since a' can bring more rewards both in the source CMDP and in the target CMDP, a' should be abandoned in both CMDPs. This means, in soft constraint scenario, the expert policy state must reach the threshold (Yue al., 2024, Lemma 4.2). Since the constraint condition in case (ii) of Lemma 4.3 should be satisfied both in the source and
target CMDP, for v0 E (0, 1), policy x' xa(ss) {
=aad
violates the soft constraint
- 0, a _ = (
in target CMDP. As result, action a' is abandoned in the target CMDP after transferring.
B.9 THEORETICAL RESULTS TRANSFERABILITY OPTIMALITY
Proposition B.25. Let H., = 1/(1 7), = max ||*| and D ra || 1121- Suppose that a < 2D.then for the Shannon entropy regularization, and OR can be obtained from (Schlaginhaufen & Kangarpour, 2024, Proposition D.8):
Pmin (1 - 2D
a exp
22RH,
11 Offmminn H and OR
(42)
A|
Lemma B.26. Suppose Assumptions 5.4 hold, and let r, E R. Then, we have
OR |1rru 1r ((r',RL(r)) Da. RL())) <
1 21
11
1r
2
for some problem-dependent constant oR >
Proof. This lemma directly comes from (Schlaginhaufen & Kamgarpour, 2024, Lemma 3.5). Lemma B.27. Consider I, y E R" and two subspaces V, W c R2 of dimension m < n. Then, [[ [y]wll ||llw llwll || 2ll2 + |[[x] - [v]vllg
where [[w - Hy = sin (6max (1, W)).
Proof. This lemma directly comes from (Schlaginhaufen Kamgarpour, 2024, Proposition D.10).
Definition B.28. (Cost equivalence). We extend the results of reward equivalence (feasible reward set) in IRI. settings (Schlaginhaufen & Kamgarpour, 2024) to cost equivalence (feasible cost set) ICRL settings. Given linear subspace V C S the quotient space SSAA /// is the set of all equivalence classes Ir - Xccly := (r- X SxA, (²(d - e) E V). On quotient space SxA /V, V, we define the quotient norm ||x]|||| = min || + vil2 1|y1 and we say that and ê are close in xgSxA/1 given X², (1r--xc)--(r-xte
[1(r-\** (r- (r-x'c)) [[[x (c'=))c v)]
25

---

Published as a conference paper at ICLR 2025
is small. The expert's cost is said to be recovered by è up to Fly if X [r- x- lv. In this paper, we consider the equivalence relations induced by the subspace of potential shaping transformations (Ng et al., 1999), i... V= =Up, im E PT). Revisiting Eq. (26), this subspace is equivalent to the way we construct feasible cost set.
Theorem 5.7. Let Pt be the target transition law, the source and target ground-trutt constraints are the same (c) = c and dj = -Ilce - Eupy ll2. Suppose that Assumption 5.4 holds. If
e g2000 (RL(c))) S E then è E-transferable to the target environment with
max { d sin (Omax (T''Pr)) /2,221 3/1, where oR and n are regularity constants, given Appendix B.25.
(43)
Proof. For better readability, denote (f) = r' - (X) 8 and r It follows - (ury
her, (f)' in subspace Upz in Lemma B.27, we then have that
from Lemma B.26 that
[(F))
((F)ur, S / By choosing the closest (Ft) and
[(')) [(r)) [u) 11. S sin (0max (P,, PT)) ||(())== - Jung 11a + ||(())) - -[FF) Juer 11 PT))
-
S di sin (0 PT)) + v 1/21//0
(44)
Hence, applying Lemma B.26 again yields
E 6'00' (c. (RRRLL,((())))))
= e '0'0 (c, CRLL;(
1 | lung [((F)) ((F))eepg
12
2
21
(dl sin (Omax (PT', ,P)) /221/OR)
2m
2 max (ossin sin (Omax(PP's (Pr', /2, 201/OR
(45)
11
C COMPARISON IN SOFT CONSTRAINT SCENARIOS
We evaluate the performance of the ICRL solver in soft constraint scenarios in two aspects. In aspect one, only reward functions are different between source and target. In aspect two, only transition dynamics different between source and target.
Aspect One: In soft constraint scenarios, the ICRL solver still outperforms the IRC solver in the sense that it resists the variation between source and target reward functions. Table 2 illustrates whether the inferred reward correction terms by the IRC solver violate the constraint in the target environment (safe / or not X). We can see that with the increase in reward functions, the inferred reward correction terms tend to become unsafe while the inferred cost functions by ICRL solvers are safe still. Threshold 0.015 and ground truth costs are 1.
Table 2: Safety (safe / or not X) of inferred reward correction terms by the IRC solver under different rewards in the target environments of Gridworld-1. represents the terminal location (6,6) in Gridworld-1.
different
r())) ((T) =1 r(T)) 1.2 r(T) 1.4
r(T) 1.6 r(T)
1.8
r(T) 2
IRC solver
X
X
ICRL solver
/
26

---

Published as a conference paper at ICLR 2025
Aspect Two: In soft constraint scenarios, solver solver can violate constraints due compensated penalization by new transition dynamics. We provide a case study to help the readers better understand this aspect. Reconsider the example in Figure I in soft constraint scenario. We only set c((0.2)) 1 and é = 0.8 x (0.7)-² 2 > O. Suppose the source environment, there is no randomness for any chosen actions. The true cost function the expert follows is c ((0,
In this case, the shortest path, i.e. going straight upward from (0,0) to (0.5), is forbidden by the expert because the path induces cumulative cost of (0.7) 2 e. One choice of the feasible cost function is c((0, 2)) = 0.85, since 0.85 x (0.7) 2 > is sufficient to ban the shortest path. Now we transfer c( 2)) 0.85 from the source the target environment. Suppose the target environment only differs from the source environment in the transition model. In the target environment, if the location (0.1) alters its transition model to be ( (), 1), Up) 0.9 and (T1 (), 1), Up) 0.1. The inferred optimal policy based on (0.0)))))))))))) = 0.85 visits (0, 2) because the shortest path has a cumulative cost of 0.85 0.9 x (0.7)
0.765 x (0.7)-2 < E but should forbidden because the policy of going straight upwards in the left column has a cumulative ground truth cost of 0.9 x (0.7)
D COMPARISON CONTINUOUS ENVIRONMENTS
For continuous environments, we apply offline setting to compare the transferability of constraint knowledge inferred by IRC or ICRL solvers. The offline setting is different from the online setting in discrete environments. The expert policy for online estimation is replaced by expert demonstrations in given dataset. For ICRL, The goal is to recover the minimum constraint set that best explains the expert data. Existing ICRL works commonly follow the Maximum Entropy framework (Malik et al., 2021). IRC solvers in this setting follow the same framework but solve bi-level optimization problem (Hugessen al., 2024).
We build the codebases from Hugessen et al. (2024) and Liu et al. (2023) to compare the transferability performance of the IRC and ICRL solvers. We adapt the code to enable both solvers to infer constraint knowledge-such as reward correction terms cost cunctions-withee the source environment while evaluating the feasibility of this knowledge in the target environment. Using the blocked half-cheetah environment deviation in Figure 7 with three random seeds. The definition of metrics and detailed source and target environment specifications are explained in Section E.
testbed, report the results with mean + standard
We find that IRC solver has better training efficiency in the source environment, i.c., achieving zeo violation rate with considerate feasible rewards more quickly than the ICRL solver. However, after transferring constraint knowledge into the target environment, inferred correction terms by the IRC solver fail ensure safety (avoid constraint violation) while the cost function inferred by the ICRL solver has better generalizability.
Rewands
Vieiitin Rate
Feasbbe Rewards
KXL
CVL
Sturro anvineemment
16.
10c 1200
1227 207330055
Rewardss
Vialetion
Tresibll Desaards
3000
CAL
Legget Srnttrrmee
2005
Itaration
360
3.5
Itaratoos
305
Figure 7: Training curves of rewards (left), violation rate (middle), and feasible rewards (right) for the ICRL (red) and IRC (blue) solvers. The top row shows the results for the source environment, and the bottom row shows the results for the target environment.
27

---

Published as a conference paper at ICLR 2025
E EXPERIMENTAL DETAILS
We ran experiments on a desktop computer with Intel(R) Core(TM) i5- 14400F and NVIDIA GeForce RTX 4060 Ti.
Details about Gridworld. In this paper, we construct a map with dimensions of 7 x units and define four distinct settings, shown in Figure 2. Locations are represented by two coordinates, with the first corresponding to the vertical axis and the second to the horizontal axis. The agent's objective is to navigate from starting point to target location while avoiding specified constraints. The agent begins in the bottom-left corner at position (0, 0) and has eight possible actions: four cardinal directions (up, down, left, right) and four diagonal directions (upper-left, lower-left, upper- right, lower-right). The target location and reward are positioned in the upper-right cell (6, 6) in the first, second, and fourth Gridworld environments, while in the third environment, the target is located in the upper-left cell (6, 0). If the environment has stochasticity of the agent has probability of p to move randomly any feasible direction, with each direction having a probability of p/ hum_of___a__ _f__ccions. The reward is only provided at the target cell, with all other cells yielding zero reward. A cost of is incurred if the agent enters a constrained location. Each policy rollout continues for maximum of 50 time steps. In Figure 4, present the mean and the 68% confidence interval (I (I-sigma error bar), calculated using three random seeds. Table presents utilized hyperparameters in Gridworld experiments.
Table 3: List of the utilized hyperparameters in the Gridworld environment.
Parameters
Gridworld
Max Episode Length
50
Discount Factor
0.7
Stopping Threshold
0.001
Stochasticity
0.05
Nu Max Clamp
1
Penalty Initial Value
0.1
Penalty Learning Rate
0.1
Source Terminal Rewards
1,1,1,1
Target Terminal Rewards
2,7,7,15
Ground Truth Costs
Details about Half-Cheetah The Blocked Half-Cheetah task is built on Mujoco, where the agent controls a two-legged robot. The reward is determined by the distance the robot travels between consecutive time steps, penalized by the magnitude of the input action. Each episode ends after a maximum of 5000 time steps. To impose constraint, we block the region where the X-coordinate should satisfy x x_pos < -3, restricting the robot's movement to the region where X-coordinate is between -3 and +00 The source environment follows the setup described above. In the target environment, rewards are scaled by a factor of 1.1, and Gaussian noise with a mean of 0 and a standard deviation of 0.1 is added to each observation. We utilize three metrics for evaluations: I) rewards are defined as the total returns for an episode, regardless of constraints; 2) feasible rewards the aggregated returns for episode up to the first constraint violation; 3) the violation rate is calculated as the percentage of episodes in which one or more constraint violations occur. Table presents utilized hyperparameters in Gridworld experiments. Other hyperparameters follow previous codebases (Hugessen et al., 2024; Liu et al., 2023).
Table 4: List of the utilized hyperparameters in the Half-Cheetah environment.
Parameters
Half-Cheetah
Training Epoch
320
Testing Epoch
320
Max Episode Length
5000
IRC Solver
IRL-base
28


Reviews:
[{"review_id": "my2ku5YUta", "reviewer": "Reviewer_MoLg", "summary": "This work introduces the IRC solver to overcome the limitation of the IRL solver, which generally lacks a mechanism to leverage existing reward signals and may not be compatible with different rewards. The authors also give theoretical analysis and achieve a lower sample complexity. Besides, they also study and extend the transferability. The results of the experiments demonstrate its efficiency.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2. The related work is well organized.\n3. The authors give a strong theoretical. analysis of the advantages and shortcomings between IRC and ICRL solvers, i.e., convergence, and sample complexity. Besides, the authors clearly presented their theory contributions.", "weaknesses": "1. What are the wall-clock running times of your method and the other baselines in the experiments?\n2. In Theorem 3.2, Can we have such that directly maximizing the cumulative rewards and considering the constrained optimization objective?\n3. In B.8.2 subsection, Can we provide experimental results to evaluate the soft constraint scenario?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GfzlznQyiC", "reviewer": "Reviewer_foPH", "summary": "The paper addresses the task of inferring safety constraints from expert demonstrations where the expert maximizes a known reward subject to unknown constraints. Although inverse constrained reinforcement learning (ICRL) has developed specialized methods for this task, recent work has shown that casting the problem into the framework of inverse reinforcement learning (IRL) via \"inverse reward correction\" (IRC) could offer advantages. This paper provides a deeper analysis of these claims, showing that:\n1) Inverse reward correction can achieve better sample efficiency than ICRL for constraint inference;\n2) The implicit constraints inferred through IRC, however, may transfer poorly across environments with different dynamics or reward structures compared to constraints explicitly modeled by ICRL; and\n3) ICRL’s explicit constraint modeling offers conditions under which its inferred constraints allow for approximately optimal and safe policies when transferred to new environments, even under varied dynamics.", "questions": "- what role does the superscript $r$ play in  $Q^{r,\\pi}_{\\mathcal{M}\\cup r'}$  ?\n- in Definition 3.3 of an IRC solver, you introduce $\\mathcal{C}_{\\text{IRL}}$, \"the set of feasible reward correction terms derived by [the solver]\", which seems to imply that the IRC solver at least *may* recover a whole set of feasible reward correction terms. Then, having this set, one may try to produce a policy that is as robust as possible with respect to that set. Using your own example from Fig. 3, you write that the solver learns a error correction term $-1-\\beta,\\beta>0$, If understood as a set of feasible correction terms, a robust policy would avoid this state. Instead, you seem to force a choice of one particular reward correction. What do you think of using the full set of feasible error correction terms? Would this eliminate the advantage you point out? Alternatively, Bayesian IRL methods would give a posterior over the different feasible error correction terms, again allowing for producing robust policies.\n- could you please point out which results and proofs mostly just follow prior work on related concepts, and which parts (e.g. giving line ranges) are indeed novel and specific to this context?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- the task of inferring unknown safety constraints is important\n- the recent challenge that the field of ICLR with its growing literature may be redundant deserves further scrutiny, and the results in this paper add important nuance to this challenge, showing that such redundancy is not clear cut\n- the paper shows potentially useful results relating inverse reward correction and ICRL that deserve a place in the literature", "weaknesses": "- unfortunately, the presentation is bad, making deciphering the paper a painful challenge even to a motivated reader closely familiar with the work this paper builds on:\n    - for me, this point alone currently warrants rejection, but I may be swayed if the authors manage to upload a substantial revision\n    - here is a non-exhaustive list of issues:\n    \t- for $Q^{r,\\pi}_{\\mathcal{M}\\cup r'}$ (and similarly V), it's never made clear how the value depends on $r$ in the superscript, which remains confusing throughout, especially as other values including the cost c get swapped into that place and this value is quite central in the paper and used frequently throughout the text\n    \t- the paper initially introduces a soft-constraint setting (allowing for costs up to $\\epsilon\\geq 0$, with the hard-constraint case $\\epsilon=0$ as a special case) but then, in places, assumes the hard-constraint setting without alerting the reader to the fact adding further confusion\n    \t- several pieces of notation are not defined in the main text (e.g. $N^+_{k+1}$) on page 5\n    \t- l.219 mentions advantage, followed by a definition of what appears to be a state-value function instead with advantage never defined\n    \t\n\t- regarding minor issues:\n\t\t- I'd advise using similar notation for different concepts: e.g. depending on sub/superscript, $\\mathcal{C}$ is sometimes a set of feasible cost functions, sometimes a (not directly related) scalar value in the sample complexity bound, or $\\Delta$ is once a set of measures, another time a reward correction term. I'd advise using at least a different font in the two cases. \n\t\t- several times, the main text refers to numbered Tables or Figures, which are in the appendix but this is not pointed out - by default, a reader will search in the main text\n\t\t- there is also a fair amount of typos\n\n- the theoretical results very closely follow similar prior results on the sample complexity of IRL and the transferrability of the recovered rewards (Metelli 2021, Schlaginhaufen 2024). Per se, I don't see that as a huge problem - clearly porting these results to the context of constraint inference still has value. However, if the main contribution of the paper is translating the results into another context, presentation would seem to be one of the main possible contributions, so doing a poor job at that takes away most of the value", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wt7DeUKVEd", "reviewer": "Reviewer_i1Qn", "summary": "The paper explores inverse reward correction (IRC) that, instead of solving a three-level optimization in inverse constrained RL (ICRL), tries to combine the constraints as part of the reward function so as to solve only a reward optimization problem. The paper proves IRC has lower sample complexity but worse generalizability than ICRL solvers. The work investigates conditions in the transition laws, cost/reward functions, and Lagrangian multipliers that guarantee epsilon-optimality of IRC when transferring to a different environment.", "questions": "- what are the implications of this in few-shot learning after transferring an IRC/ICRL policy?\n- What would these bounds on the sample complexity/generalizability of IRC and ICRL mean in real world tasks where S and A are continuous spaces? \n- Furthermore, do these bounds consider any measure (like complexity) of the dynamics/transition function? Why or why not?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- well-written and clear\n- problem and questions are well-motivated\n- interesting theoretical guarantees and analysis of IRC", "weaknesses": "- could benefit from comparison on more complex environments than grid-world\n- it would be nice to have a comparison of the pseudocode of the IRC and ICRL algorithms used in experiment\n- consider adding hyperparameters of the algorithms in the experiments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oS0JNEVkEo", "reviewer": "Reviewer_87Lh", "summary": "This paper focuses on two approaches for safety-critical inverse RL: Inverse Constrained Reinforcement Learning (ICRL) and Inverse Reward Correction (IRC). It derives the upper bounds of these two methods and concludes the advantage of IRC in terms of sample efficiency. The paper also discusses potential constraint violations when learning a reward correction term or cost in source environments and transferring them to new environments. Theoretical analysis demonstrates that ICRL is more robust to these issues compared to IRC and examines the optimality of ICRL in target environments. Finally, empirical studies on gridworlds validate the theoretical findings for both methods.", "questions": "- The paper compares sample efficiency using theoretical upper bounds. I haven’t fully examined the details, but I am curious about the tightness of these estimations and whether they reliably represent real sample efficiency in practice.\n- Although the experiments are conducted in gridworlds to validate the theoretical findings, I wonder if the results hold true for more complex, scalable tasks. Discussing potential challenges in such contexts could offer guidance and inspiration for the practical application of ICRL and IRC.\n- I am a little confused about $\\min_{\\triangle r}$ in Eq. (2), which seems to minimize the expert's returns. Should it be $\\max$ instead?\n- The term 'iteration' appears in several theorems. It would be helpful to briefly explain what ICRL and IRC do in each iteration, such as taking a fixed number of gradient steps based on Eq. (1) and (2) ?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a comprehensive comparison of ICRL and IRC approaches from multiple perspectives, including sample complexity and transferability. This reveals a tradeoff between the methods, offering interesting insights into their applications.\n- The theoretical formulations and derivations are detailed and well-defined.\n- The empirical evaluations are convincing and consistent with theoretical results.", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)