Simplifying Constraint Inference with Inverse Reinforcement Learning
Authors: Adriana Hugessen
Harley Wiltzer
Glen Berseth
Venue: NeurIPS 2024 poster (2024)
Keywords: reinforcement learning
inverse reinforcement learning
safe reinforcement learning
constrained reinforcement learning
Primary area: reinforcement_learning
TL;DR: Solving a constrained MDP in the forward pass of inverse RL is an unnecessary complication for inferring constraints - constraints can be inferred as well or better through simple IRL methods.

Abstract: Learning safe policies has presented a longstanding challenge for the reinforcement learning (RL) community. Various formulations of safe RL have been proposed; However, fundamentally, tabula rasa RL must learn safety constraints through experience, which is problematic for real-world applications. Imitation learning is often preferred in real-world settings because the experts' safety preferences are embedded in the data the agent imitates. However, imitation learning is limited in its extensibility to new tasks, which can only be learned by providing the agent with expert trajectories. For safety-critical applications with sub-optimal or inexact expert data, it would be preferable to learn only the safety aspects of the policy through imitation, while still allowing for task learning with  RL. The field of inverse constrained RL, which seeks to infer constraints from expert data, is a promising step in this direction. However, prior work in this area has relied on complex tri-level optimizations in order to infer safe behavior (constraints). This challenging optimization landscape leads to sub-optimal performance on several benchmark tasks. In this work, we present a simplified version of constraint inference that performs as well or better than prior work across a collection of continuous-control benchmarks. Moreover, besides improving performance, this simplified framework is easier to implement, tune, and more readily lends itself to various extensions, such as offline constraint inference.

Simplifying Constraint Inference with Inverse Reinforcement Learning
Adriana Hugessen Mila, Université de Montréal
Harley Wiltzer
Glen Berseth
Mila, McGill University
Mila, Université de Montréal
(adriam) addiaana.e Srrcchbull-hugessee wiltzerh, glen. berseth) @minammmmmaa....mmmmmmmmmmm quebec
Abstract
Learning safe policies has presented a longstanding challenge for the reinforcement learning (RI.) community. Various formulations of safe RI. have been proposed; However, fundamentally, tabula rasa RI. must learn safety constraints through experience, which is problematic for real-world applications. Imitation learning is often preferred in real-worl worldw settings because the experts' safety preferences are embedded in the data the agent imitates. However, imitation learning limited its extensibility to new tasks, which can only be learned by providing the agent with expert trajectories. For safety-critical applications with sub-optimal or inexact expert data, it would be preferable to learn only the safety aspects of the policy through imitation, while still allowing for task learning with RL. The field inverse constrained RL, which seeks to infer constraints from expert data, is promising step in this direction. However, prior work in this area has relied on complex tri-level optimizations in order to infer safe behavior (constraints). This challenging optimization landscape leads to sub-optimal performance on several benchmark tasks. In this work, present simplified version of constraint inference that performs as well or better than prior work across a collection of continuous-control benchmarks. Moreover, besides improving performance, this simplified framework is easier to implement, tune, and more readily lends itself to various extensions. such as offline constraint inference. Our code is made available at httpps///gtubb.boom//guu////gppp
Introduction
Reinforcement learning (RL) has made significant advances in recent years, yet real-world ap- plications RI. remain limited due to various challenges, including particularly safety concerns (Dulac-Amold et al. 2202T. One common setting where RI. holds promise for real-world deploy- ments is replacing existing, possibly sub-optimal, buman-manage the field of power network control, decisions are often made by combination of automatic control policies combined with careful human monitoring and intervention [Marot et al. 2022]. However, concerns regarding the deployment of autonomous systems on safety-critical tasks have hindered the adoption of RL for replacing legacy control systems. This is particularly true for deep RL, which functions black box controller.
control policies. For example, in
In the setting of replacing legacy controllers, however, the challenge is simplified due to access to data produced by the current control system. While existing control policies may be sub-optimal, they are nonetheless "safe" in the sense that they obey some explicitly or implicitly defined constraints corresponding to human understanding of safety, Ideally, we would like be able learn an RL policy that can outperform the current system, in terms of optimizing the reward, while continuing to respect these constraints. When these constraints are unknown (i.e. defined implicitly through human
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

actions), the implicit safety procedures that are being followed may not be explicitly known or have an obvious encoding. Thus, it would be beneficial infer the constraints directly from trajectories produced by the existing control policy.
Typically, in settings with large-scale offiine data, imitation learning (Schaal 1996 offline RL [Kumar et al. 2020 Kostrikov et al. [022] might be used. However, these methods only extract a single policy that satisfies constraints and optimizes a reward function. Moreover, imitation learning generally cannot outperform the expert, which is prohibiting in cases where we wish to improve over current control policies. Offline RL can learn policies that outperform the demonstration (Kumar iet al., 2020). however, offline RL cannot ensure that safety constraints remain satisfied without access to the constraint function or constraint violation annotations in the dataset.
Ideally, we would like to extract safety constraints from the data based on the expert behavior, which can then be used downstream to constrain task-specific learning. Learning constraints from expert trajectories is the purview of the field of constraint inference. Various methods have been proposed in this domain, with early work focusing on simple settings such as tabular MDPs (Scobee and Sastry] 2020]. More generally, inverse constrained reinforcement learning can infer constraints in continuous settings using parameterized constraint functions and adapting IRL methods such as maximum entropy IRL. (Ziebart et al., 2008 to the constrained setting by solving a constrained MDP in the inner optimization. However, this tri-level optimization creates a challenging landscape for constraint inference.
In this work, we demonstrate that the constrained MDP inner loop is an unnecessary complication and that regular IRI. techniques can recover as good and sometimes better solutions to constraint inference problems than these more complicated methods. This result is significant because it allows us to simplify the training dynamics and complexity of constraint inference methods and implies that advances in sub-domains of IRL can be directly applied to the constraint inference case. For example, recent progress in offiine IRL (uue et al. 2023 Kim et al. 2023] can be readily adapted to the constraint inference case infer constraints entirely offline, significantly increasing the scope of applicability in real-world settings.
In particular, we make the following contributions. First, we show that inverse constrained RL and inverse RI. are equivalent under certain classes of constraint functions. Next, we experimentally validate this claim. Finally, we propose some practical modifications to adapt IRL to constraint inference tasks and conduct ablations over these algorithmic choices to understand how to improve the stability and performance of constraint inference.
2 Related Work
General imitation learning The problem of imitation learning (IL) is primarily concerned with learning policy that produces similar behavior to class of reference policies. Specifically, given transition data from "'experts", the goal is to produce a new policy that, in a sense, generates a similar transition distribution. The simplest approach to IL is behavior cloning (BC) [Pomerleau] [1888. which estimates a policy via maximum likelihood to match the conditional distribution on actions conditioned on state observations. BC tends to perform well in the regime of massive data but suffers from compounding errors over long trajectories as state observations veer outside the coverage of the dataset (Ross al. 201T]. Methods such as DAgger (Ross al., T01T show that, given the ability to interact with the environment during II. and to query expert trajectories, strong policies can be learned with relatively few queries the expert. More recently. behavior cloning losses have been integrated into standard reinforcement learning methods to enhance offline RL performance - that is, RL from a fixed (offline) dataset [Fujimoto and Gul 2021]. Alternative approaches to IL have focused on learning a policy that induces a similar visitatiin distribution to the dataset [Pirotta et al. 2023). Such an objective explicitly accounts for the long-term behavior of a policy as opposed to the myopic predictions of BC methods, resulting in more coherent learned behaviors over longer horizons with less data. IL methods as discussed above can generally learn performant imitation policies, but provide no insight about the reward function that the expert is implicitly optimizing: this is the focus of inverse reinforcement learning, which will be useful for the purpose of learning constraint functions.
2

---

Inverse Reinforcement Learning Inverse reinforcement learning (IRL) learns policy that can mimic expert trajectories and also aims infer a reward function that explains the expert behavior [gg et al. 2006]. The IRI. problem is considerably more difficult than the II. problem; in particular, there is no unique reward function that maximally explains the expert behavior. The work of Abbeel
and Ng] [004] alleviates this issue with maximum margin constraint in the optimization. The work of Ziebart et al. [2008) resolves this problem by inferring the most likely reward function under a particular probabilistic model; this approach is widely known as maximum entropy IRL (MaxEnt IRL). One approach for achieving strong guarantees in IRL is to approximate the expert occupancy distribution in the operator norm that is, to have low discrepancy between expected returns under any reward function. Achieving this generally involves a bilevel optimization procedure where one alternates between optimizing a policy for given reward function, and then adversarially training the reward function [Gleave and Toyer 2022]. The work of Garg et al. [2021] reduces this problem in the maximum entropy framework by estimating soft action-value function (Q), inferring the reward function from a novel "inverse Bellman operator" applied to Q and showing that the policy x- s) x exp(Q(s,)) -)) is MaxEnt-optimal. Effectively, their method reduces the bilevel optimization discussed above into a simpler RL optimization. Alternatively, (Swamy et al. [2021) assumes that the reward function lies in a reproducing kernel Hilbert space, which allows them to obtain an unbiased estimate of the adversarial reward function from data for any policy - this is yet another method for eliminating one of the layers of the bilevel optimization discussed above. However, these aforementioned IRL methods have not considered the issue of modeling environmental constraints which can be transferred when learning policies for novel tasks.
Inverse Constrained Reinforcement Learning Early work in constraint inference focused primar- ily on particular settings such as convex constraints [Menner al. [019 Miryoosefi et al., 2019), or tabular RL (Scobee and Sastry] [220 McPherson et ]l.] [202]] Chou et al. [2220] More recent methods have integrated the power of deep learning within the constraint inference framework through adap- tions of MaxEnt IRL to the constrained setting through inverse constrained reinforcement learning (ICRL) Malik et al. 202T Liu al. 2023a Kim et al. 2023). These methods essentially replace the forward RL inner loop of IRL with a constrained version of the problem by casting it as constrained MDP (CMDP) and solving by Lagrangian or other methods. This adds additional complexity to the IRL problem which is already difficult due to the bi-level optimization and identifiability issues GGlaave and Toyer 2022]. Though these prior works have sometimes considered simple IRL. baselines MMalik et 2021 Liu et al. [003a]]. none have specifically outlined the connections between IRL and ICRL, nor have they attempted to optimize IRL for constraint inference. A related line of work from the control theory community also considers learning constraints from expert data as control barrier functions, [Robey et al., 2020 [asttaneaa et al., 2023 Lindemann et al., 2024), however these methods rely on access to a known or learned model of the dynamics, which we do not assume.
3 Background
Imitation Learning. Imitation learning the process of training new policy to reproduce expert policy behavior. Given a set of expert trajectories D. = (7e ( y which follow a policy Tg, a new policy can be learned to match these trajectories using supervised learning, i.e. maximizing the expectation: Frg Er log (a |st, 0) While this simple method can work well, it often suffers
from distribution mismatch issues leading to compounding errors as the learned policy deviates from the expert's 's behavior [Ross et al. (011T. Inverse reinforcement learning avoids this issue by
extracting a reward function from observed optimal behavior [Ng ct al. 2000].
As discussed in Section 2 there are several formulations of IRL. Here we describe maximum entropy IRL (Ziebart al. 2008 Ziebart 2010). Given a set of expert trajectories De = {TE (=() o, which follow a policy Tg. a policy æ can be trained to produce similar trajectories by solving the min-max problem
Maximum Entropy IRL
===0-
Er(se, Enton.a)))
T ai)) 1)
min max E x
F
Σ r( at) + H(x)
Eng
where r a learned reward function and (TTT) ) is a causal entropy term. Hence, IRL is searching for reward function that is high for the expert me and low for other policies.
3

---

Constrained RL One way to formulate constrained MDPs (CMDPs) is through trajectory- constraints. this formulation, the agent must solve the following objective
T Σ r(s,, at)
s.L. Ex Σ dss, an)
max ER
S 6,
(2)
where c(8t, (x) is a cost function with budget.
The most common method for solving this problem is Lagrangian variations of RI. methods, such as SAC-Lag [Ha et al., 2020]. where Equation 2 is converted to a min-max problem
T
T
min max ET X
y Ar(se, an)
B
Ae(se, oz) Σ
- -
(3)
Inverse constrained RL Inverse constrained reinforcement learning (ICRL) is class methods used to infer constraints from expert trajectories. As in imitation learning and IRI., it is assumed that we have access to set of expert trajectories D. = {Te N Ji=0- which follow policy Tg. however, is now assumed that the expert policy is optimal under a CMDP instead of an MDP. It is generally assumed that the agent has access the unconstrained (nominal) MDP with which interact, and that the (constraint-agnostio reward is observed by the agent when interacting with the environment.
Kim et al. 2023 present general formulation of ICRL as game-solving. though we note that their formulation was preceded by that of Malik et al. [2021). and, though motivated differently, generally matches it, up some implementation details.
As shown prior work (Swamy et al. 2023), the inverse RL problem can be cast as a two-player zero-sum game
OptRRl [II min sup J(T f) JTT, =e1l feF
where convex and compact and J(T, f) = Ex E f(st, f(st.a.)).
Similarly, the constrained RL problem in Equation 2 with single constraint can also be cast as two-player zero-sum game
min max max- c() ()
xell X20
Finally, Kim et al. [2033] show that for inverse constrained RI., these two games can be combined into a three-player game of the following form (where a given reward function class F,)
OptiCRL (TC (I,,F,,,) Fo) sup max min JTT Xc) J(T, Ac).
cEF, A20 xell
Practically speaking. solving this game involves running IRL where the inner loop optimization solves a constrained MDP using Lagrangian version of RL, such as SAC-Lag [Ha et al. 2020
4 Method
Starting from Equation (6). we now demonstrate that this tri-level optimization is equivalent to simpler bi-level optimization under certain classes of constraint functions and that this bi-level optimization equivalent to the IRL game-solving formulation in Equation (4)
4.1 Inverse constrained RL as IRL
Note that there are two outer maximizations in Equation (6) the first constraints in the class Fo and the second over À > 0. Our key insight is that, under a broad class of constraint functions Fa, this tri-level optimization can be cast as a bi-level optimization, reducing ICRL to IRL.
Theorem 4.1 Let T II and reward function e F, be given. Consider the objective
Opts-IcRL Fr, Fc) max min J( - c) - J[T, 1-,,--0 -cc). cc. ceF. #lll

---

Then, if Fe is a convex cone (that is, c E Fc 7 Ac E For any A 2 0), it holds that OptICRL (II, Fc) Opts-Icn ICRL (II, Fo).
Moreover, suppose IT is compuct and Fr = Fc= = is a vector space with elements I : (8, a) m (Ō(s, wy), where & S x A + [0, 1]4 is a fixed (not necessarily known) feature map and wy E I& idemiifiex the elements F.
F. Then any solution to OptIRL (II,F - T is a solution to OptiCRL FI), that is, the ICRL problem is simply an instance of IRL.
Proof. See Appendix B
The intuition for this is the following. The only difference between solving constrained MDP with Lagrange multiplier À and solving an unconstrained MDP with a cost penalty term weighted by Á that, the former case À is optimized to ensure constraint satisfaction. Indeed, given the optimal X a priori, constrained MDPs can be solved exactly by optimizing the later problem. This distinction is generally important when the cost function is fixed and unknown, however in the case of inverse learning where we learn the cost function from class closed to scalar multiplication, it is possible to learn a cost that directly ensures constraint satisfaction, without scaling by the Lagrange multiplier. Note that optimizing over a convex cone Fe can pose challenges when exact optimization required-for instance, convex cones violate a compactness condition required to prove regret bounds for inverse constrained RI. given by Kim et al. [003]. However, in large scale applications that approximately optimize a constraint model represented by deep neural network, one is often already in the setting where these regret bounds do not hold. Hence, when employing parameterized constraint inference through deep learning, it is sufficient to select an appropriate class as the output activation of our neural network and use a bi-level optimization to learn the scaled constraint.
4.2 Techniques for stabilizing constraint learning
Given that IRL and ICRL are mathematically equivalent under the class of constraint functions described above, the question of how best to perform constraint inference through inverse learning becomes largely a practical one. Is the optimization landscape smoother if we explicitly optimize for À as a Lagrange multiplier or implicitly as part of the constraint function? Do previous imple- mentations of Lagrangian methods add additional algorithmic components that could explain the improved performance of ICRL over IRL in prior work? Moreover, are there other regularizations or modifications that we could consider? this regard. we suggest the following practical modifications to IRL for the constraint learning case, which we will test in Section 5
Bounding rewards One advantage of prior methods' use of a binary classifier for constraint representations is that it restricts learned costs within range of positive values [Malik et al.) 2021). In practice, is advantageous for interpretability and transferability to restrict the constraint function to be strictly positive, so that the learned constraint can only discourage the agent from visiting certain states not visited by the expert. Bounding the rewards above can also be beneficial for preventing divergences during training. However, in the case of IRL for constraint inference, it is necessary to optimize over a convex cone and hence a binary activation function cannot be used.
We propose and evaluate two possible solutions. As a baseline solution, we consider simply clipping the output values of a linear activation function to within some fixed positive range, while also re-scaling rewards using the rolling mean and variance. Though clipping technically violates the convex cone assumption, in practice the reward scaling should restrict the range of the constraint function necessary to ensure constraint satisfaction.
This baseline solution is potentially problematic, however. Besides violating the convex cone assumption, hard clipping the output of neural network can be challenging due to the potential for vanishing gradients. Alternatively, we propose softly enforcing the constraint function to the positive range by using a Leaky ReLU activation and adding an L2 regularization term to the imitation gap loss, similar to Malik et al. 2021]. so that the loss becomes
L( c) [ c) J( c) + Eg,oNTg ((ss.a))
(8)
where E. Es [c(s, a)? is an L2 regularization on the learned cost of the expert trajectories. Unlike in Malik et al. [2021]. we only regularize the output of the constraint function on the expert data,
'The notation F refers to the set (f - (rr
5

---

allowing the constraint function to grow unconstrained. except on states visited by the expert, where the cost should be zero by the assumption that the expert trajectories are safe, i.c. non-violating.
IRL with separate critics One component of prior methods for constraint inference that is implicitly added when utilizing a Lagrangian method for policy optimization, is the use of separate critics for the reward and constraint functions. Separate critics may potentially be beneficial for learning, since separating the critics disentangles the value function learning of the reward (which is not changing during training) with the constraint function (which changes the constraint is learnt). By Q-decomposition [Russell and Zimdars 2003], additíve rewards can also be evaluated by separate additive critics and hence it is straightforward modification to implement separate critics in any actor-critic IRL implementation without the additional Lagrangian optimization, which we outline in Algorithm
Algorithm 1 IRL for ICRL- Separate Critics
1: Initialize cost network parameters 6, policy parameters & and cost and reward critic parameters and a. Let M be the number of cost function updates and be the number of policy updates. 2: for m 0, ... M do
3: for n = 0..... N do
4: Collect experience (s,, dt, 8t) from nominal environment and add to buffer 3
5: Sample batch {(s,, di, , r.))
B.
6:
Compute learned cost for batch C; Ro(si, a.) Vi E B
7: Update cost critic parameters 0 from c; and reward critic parameters a from Ti.
8: Update policy parameters using critic ((s,a) = Qa(s, a) Qo(s,a)
9: end for
Update cost function using loss E. [R Exg [R
11: end for
Last-layer policy resetting Recent work has demonstrated that the primacy bias caused by plasticity loss can be a significant challenge for training deep RL agents [Nikishin et al. [022]. This challenge exacerbated the case of IRL and particularly ICRL since the learned cost function changes
constantly during training while the true rewards are provided from the enviromment, potentially biasing early policy training towards learning the unconstrained policy. To combat this plasticity loss
issue, we adopt the recommendations from Lyle et al. (2023) for MLPs, which advises periodically resetting the final layer of the network during training.
Experiments
In this section, we conduct a series of experiments across several environments in order to answer the following questions: (1) How does IRL perform on constraint inference tasks compared to Lagrangian methods? and (2) How do the proposed modifications or regularizations over vanilla IRL improve performance on constraint inference tasks?
Environments For our experiments, we consider the virtual environments for benchmarking inverse constraint learning, introduced by Liu et al.) 2023a] since these were specially designed to test the performance of constraint inference tasks and also provide a recent baseline for Lagrangian- based constraint inference methods, including expert data. The environments include five MuJoCo environments, Ant, Half Cheetah, Walker 2D, Swimmer and Inverted Pendulum, modified to include constraints, which are primarily binary restrictions the x-position of the agent. For more details, see LeeTii Liu et al. [202a]]
Evaluation metrics Evaluating constrained RL is challenging due the dual objectives of optimiz- ing rewards and satisfying constraints, which are typically in conflict with one another. Hence, in our evaluation, we report two metrics, following Liu et al. [2023a], (. feasible rewards and violation rate. Feasible rewards are calculated as the total returns for an episode up to the point of the first constraint violation. The violation rate is the percentage of episodes with one or more constraint violations. Using these metrics allows us compare the basclines along the same axis prior work.

---

We compare all methods according to average performance in the last 50 testing episodes and report statisties (IQM, Median, Mean and Optimality Gap) with bootstrapped 95% confidence intervals computed across five seeds according to the method recommended in Agarwal et al. [202T]. We provide the full learning curves in Appendix C.2
Prior methods for comparison We compare our results against two baselines from Liu et al. 2023a]. The first is their implementation of MaxEnt ICRL [Malik et al. [02T] (denoted MECL), which is the canonical method for Lagrangian- based constraint inference that provides the primary baseline for our method. The second is their implementation of GAIL for constraint inference (denoted GACL), introduced as a baseline for MECL Malik et al. [2021). This is a basic version of IRL for constraint inference, which uses the GAIL algorithm Ho and Ermon 2016] with an additive log term for the constraint. In Liu et al.) [2023a], they test additional two algorithms which, for conciseness, we do not include in this work, as one has been shown to underperform other methods in previous work and the other introduces Bayesian inference into the constraint learning, which is not considered here. We obtain results for MECL and GACL using the implementation and hyperparameters provided by the authors of Liu et al. 2023a]. We note that for our implementation of constraint inference, we use SAC as the forward RL algorithm, similar to Kim et al. [2023] whereas
Malik et al. [2021] and Liu et al. [2023a] use PPO with an entropy regularization term.
Experimental Setup As mentioned, we use SAC for policy optimization. As the IRL algorithm, we utilize a version of maximum entropy IRL as implemented in Zeng et al. [022]. The learnt constraint function is parameterized as two-layer MLP with linear ouiput activation. We adopt the hyperparameters for SAC as used in Achiam 2018), except that we use automatic a-tunnin [Haarnoja et al. 2018). All the hyperparameters, for IRL were set to those used in /Zeng et al. [2022). except the learning rate on the constraint function which we tune for each environment. We find that when training with separate critics, warm stam-starting the policy learning by training without the constraint critic initially is beneficial, so we warm-start configurations with separate critics with 500k environment steps. We train all variations in all environments for 5M environment steps across five seeds. Full hyperparameter configurations are included in Appendix A.1
In the following sections, the modifications proposed in Section 4.2 are labeled as IRL-Base for IRL with clipping and no modifications, L2 for using L2 regularization in place of clipping, SC for adding separate critics and PR for policy reset. IRI-Pluus includes all proposed modifications (1.2, SC, PR).
5.1 IRL versus ICRL
First, we compare our IRL implementation to the baseline methods. In Figure Twe present summary of our findings across environments, showing the final expert-normalized performance, aggregated across all five environments, with 95% bootstrapped confidence intervals computed across five seeds.
Notably, basic IRL (IRL-Base) already performs quite favorably versus MECL. Statistics on the feasible rewards are generally improved for IRL-Base versus MECL, though this improvement does not always have statistical significance. On the other hand, violation rate is improved over MECL with statistical significance across all but one statistic. Notably, the base IRL. method performs similarly to the prior baseline IRL method GACL, with no statistical difference in feasible rewards across all four statistics, though it generally outperforms in terms of the violation rate.
With the added modifications, IRL-Plus outperforms MECL with statistical significance across all four metrics in feasible rewards and all but one metric in violation rate. IRL-Plus also outperforms the baseline IRL method GACL with statistical significance across almost all metrics for both feasible rewards and violation rate, with the median being the only exception.
5.1.1 Sub-optimal Expert Trajectories
Previous work has found that constraint inference with Lagrangian inner loop is more robust to suboptimal expert trajectories as compared to previous IRL methods which cannot learn effectively [Liu et al. 2023a]. We conduct experiments on the suboptimal dataset for Half Cheetah provided in
Mallle TTooug we have attempted to re-implement Malik et al. 2021 using SAC, our performance is low compared
Li et al. (023a]. We include full results of our SAC implementation in Appendix C.T
"uu code is available at: https//gghhbbbco///gggkimippooop
7

---

IQM
Median
Mean
Optimality Gap
IRL-Plus (Ours)
(Durs)
MECL
GACL
0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00
(a) Feasible Rewards
IQM
Median
Mean
Optimality Gap
IRL-Pius (Ours)
IRL-Base (Ours)
MECL
GACL
0,00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75
(b) Violation Rate
Figure I: Normalized final performance aggregated across the five MuJoCo environments, where feasible rewards are normalized by the expert returns. Both IRL-Base and IRL-Plus perform better than the baseline ICRL method MECL on average, with IRI-PPus out-performing by statistically significant margin across most metrics, for both feasible rewards and violation rate.
Liu et al. 2023a to test whether this remains true with our implementation. The sub-optimal datasets constructed contain varying percentages of unsafe expert trajectories, i.e. expert trajectories with one or more constraint violations. For these experiments, we test the combination of all proposed modifications (IRL-Plus), i.c. L2 regularization on the expert rewards, separate critics and policy resets versus the two baselines MECL and GACL.
Feasible Rewards
Feasible Rewards
Feasible Rewards
IRL-Plus (Ours)
MECL
GACL
2500 5000 7500
0 2500 5000 7500
2500 5000 7500
Violation Rate
Violation Rate
Violation Rate
IRL-Plus (Ours)
MECL
GACL
0.00 0.25 0.50 0.75
1.00
0.00 0.25 0.50 0.75 1.00
0.00
0.25 0.50 0.75 1.00
(a) 20%
(b) 50%
(c) 80%
Figure 2: Interquartile mean (IQM) of final performance on Half- Chef-Chhettah with sub-optimal trajecto- ries in ratios of 20%, 50% and 80%. In all three scenarios, IRL-PPuu out-performs MECL in terms of feasible rewards, though only with statistical significance in the 20% scenario. Neither MECL nor IRI-PPlus clearly outperforms in terms of violation rate, with both methods achieving near zero.
As shown in Figure 2 in all cases, method performs much better than the prior IRL baseline method GACL, which does not achieve any feasible rewards. Our method also generally outperforms the Lagrangian-basee method MECL though with notably high variance, that the improved performance is only statistically significant in the 20% scenario. Overall, however, it is clear that the ICRL method MECL does outperform IRL-Plus statistically significant basis.
5.2 Impact of Modifications
Here, we more closely examine the impact of the proposed modifications (Section constraint inference with IRL. Overall, while the impact of the modifications varies somewhat across environ- ments, can be seen in Figure A aggregated across all environments on normalized basis, the combination all modifications, Tee 1.2 regularization, separate crities and policy resets (IRI.-Plus) results in increase in feasible rewards IRL-Base which is statistically significant in all statistics

---

IQM
Median
Mean
pptimaily Gap
PRL+
(IRL-Plus)
IRL+12+SC
IRL+L2+PR
IRL+12
(IRL-Base)
0.00 0.25 0.50
0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00
(a) Feasible Rewards
IQM
Median
Mean
Optimallty Gap
IRL+
(IRL-Plus)
IRL*L2+SC
IRL+L2+PR
IRL (IRL-Base)
0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00
(b) Violation Rate
Figure 3: Normalized final performance for all ablations aggregated across the five MuJoCo envi- ronments, where feasible rewards normalized by the expert returns. IRL with L2-regularization, separate critics and policy resets (IRL-Plus) generally performs the best terms of feasible rewards with only a slight increase in violation rate versus IRL-Base.
except the median, with only slight increase in the violation rate which is not statistically significant any of the statistics.
1.2 regularization versus clipping with reward normalization Overall, the most substantial increase in performance comes from the 1.2 regularization versus clipping with reward normalization. This modification improves feasible rewards by a statistically significant amount measured by three of the four metrics, excepting the median. At the same time, it results in small increase in violation rate, however this increase is not statistically significant for any of the four statistics. The addition of L2 regularization has the largest impact on Walker2D as can be seen in Figure 4
Policy reset and separate critics Adding both separate crities and policy reset (SC+PR) slightly improves performance over IRL+L2, across most statistics, in both feasible rewards and violation rate, though the impact is marginal and only statistically significant in some metrics. Interestingly, neither the separate critics modification nor the policy reset modification alone results in an improvement over IRL+L2, suggesting it the interaction of these two modifications that are beneficial. For example, separate critics without policy resets (IRL+L2+SC) slightly hurts performance in terms of feasible rewards with similar violation rate, and policy resets without separate crities (IRL+L2+PR) slightly increases violation rate without much improvement in feasible rewards, though we note that these effects are generally policy resets we reset the last layer of only the constraint critic and not the reward critic. This allows the reward critic learn the unchanging reward function without resets, while the constraint critic can adapt more quickly through resets to the changing constraint function, which may explain the performance boost observed when combining these two methods.
statistically significant. Notably, when combining separate critics with
While the impact of separate critics and policy resets is marginal overall, in certain environments such Inverted Pendulum and Half Cheetah, does confer meaningful improvement in feasible rewards, as can be seen in Figure though with notably high variance across seeds.
Discussion and Conclusions
Overall, we have demonstrated that Lagrangian- methods for constraint inference are theoret- ically equivalent to IRL for a broad constraint class. Moreover, these Lagrangian methods do not appear to confer performance benefits over simpler IRI. methods, when IRL. implemented using MaxEnt IRL with constraint functions parameterized by neural networks with appropriate activation

---

GACL
MECL
IRL-Base
IRL-12
IRL-L2+PR
IRL-L2
SCL
Fessible Rewands
Fezsible Rowards
Fessible Rowords
Foasibic Rowards
Feasible Rewords
3000 6000
9000
5000
10000 1 15000
10
20
30
1600
2400
400
600
Violation
Vialation
Violaiio
0.00 0.25 0.50 0.75
1.00
0.00 0.25 0.50
0.75 1.00
0.00 0.25 0.50
0.75
1.00
0.00 0.25 0.50 0.75
1.00
0.00 0.25 0.50 0.75 1.00
(a) Half Cheetah
(b) Ant
(c) Inv. Pendulum
(d) Walker
(e) Swimmer
Figure 4: Interquartile mean (IQM) of final performance for all proposed modifications and baselines in each individual MuJoCo environment. All methods display relatively high variance including MECL. Even so, in three of the five environments, at least one method achieves CI that do not overlap with the baseline ICRL method MECL in feasible returns while being within the CI of the baseline in violation rate. In the remaining two environments, confidence intervals for both feasible rewards and violation rate overlap between the baseline ICRL method MECL.
functions. Further, with the addition of some cumple-to-implement practical modifications to the IRL algorithm we can further improve performance of IRL over previous Lagrangian-base -based ICRL methods for constraint inference tasks.
Advantages of IRL over ICRL Besides performance improvements, using IRL for constraint inference confers several benefits to the field. reduces the complexity of the optimization problem versus ICRL by removing a level in the nested optimization, making a simpler to implement solution that can more easily be applied to new domains. For example, we highlight that did not have to perform any hyperparameter tuning, except on the constraint function learning rate, to achieve decent results in all environments. In contrast, previous ICRL methods use many environment-speen hyperparameters [Liu et al. [2023a]. Additionally, the use of IRL methods to solve constraint inference problems could facilitate applying the much broader field of research in IRL and its sub-fields to various extensions of constraint inference that have not yet been fully explored. These could include, in particular, offline constraint inference using offline IRL techniques Yuue al. 2023].
Importantly, we emphasize that the constraint function learned through our modified IRL procedure is equivalent the constraint function learned through Lagrangian- based ICRL methods, up to scalar multiplicative constant, and hence can be treated as a constraint function in downstream transfer tasks. More specifically, though the constraint function is learned an additive term in the reward of an unconstrained MDP, the learned function can be transferred as a constraint in a constrained MDP. Solving this constrained MDP will require scaling the learned constraint by new Lagrangian term when optimizing for the new transfer task. Hence, there is no loss in the transferability of the constraint when performing constraint inference with IRL versus ICRL.
Limitations and Future Work While we show the feasibility of IRL for constraint inference an alternative to ICRL, we note that there is substantial variance in the solutions produced by IRL on a per environment basis, as can be observed in Figure 4 Reducing this variance should be priority of future work. Moreover, ongoing work is still needed to improve the performance of constraint inference across more challenging environments such as Inverted Pendulum.
Additionally, while our results indicate that our IRL-Plus method outperforms previous ICRL method MECL, we note that there are several algorithmic differences between our implementation and that Liu et al. 2023a], such as the use of SAC vs PPO as the RL policy optimization algorithm. While we believe have effectively demonstrated certain implementations of IRL can be competitive with ICRL we cannot conclude that IRL is strictly better than ICRL in all cases.
As mentioned, an exciting extension for future work would be to explore various subdomains of IRL. in the context of constraint inference, particularly the offline IRL. case, and to apply these techniques to real datasets.
10

---

Acknowledgments and Disclosure of Funding
We want to acknowledge funding support from NSERC, FRQNT and CIFAR, as well as compute support from Digital Research Alliance of Canada, Mila IDT, and NVidia.
References
Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning, Proceedings of the twenty-first international conference on Machine learning, page 1, 2004.
J. Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge the statistical precipice. Advances in neural information processing systems, 34:29304-293333 2021.
A. Barreto, W. Dabney, R. Munos, J. Hunt, T. Schaul, H. P. Hasselt, and D. Silver. Successor features for transfer reinforcement learning. Advances in neural information processing systems, 30, 2017.
F. Castaneda, H. Nishimura, R. T. McAllister, K. Sreenath, and A. Gaidon. In-distribution barrier functions: Self-supervised policy filters that avoid out-of-distribution states. In Learning for Dynamics and Control Conference, pages 286-299. PMLR, 2023.
G. Chou, D. Berenson, and N. Ozay. Learning constraints from demonstrations. In Algorithmic Foundations of Robotics XIII: Proceedings of the 13th Workshop on the Algorithmic Foundations of Robotics 13, pages 228 245. Springer, 2020.
R. Dadashi, A. A. Taiga, N. Le Roux, D. Schuurmans, and M. G. Bellemare. The value function polytope in reinforcement learning. In International Conference on Machine Learning, pages 1486-1495. PMLR, 2019.
G. Dulac-Amold, N. Levine, D. J. Mankowitz, Li, C. Paduraru, S. Gowal, and Hester. Challenges of real-world reinforcement learning; definitions, benchmarks and analysis. Machine Learning, 110(9):241-2244 2468, 2021.
S. Fujimoto and S. S. Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145 2021.
D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-leam: Inverse soft-q learning for imitation, Advances in Neural Information Processing Systems, 14:4028-4034 2021.
A. Gleave and Toyer. A primer on maximum causal entropy inverse reinforcement learning. arXiv preprint ar arXiv:2203. 2022.
S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan. Learning to walk in the real world with minimal human effort. arXiv preprint arXiv:2002.08550. 2020.
Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, Tan, V. Kumar, H. Zhu, A. Gupta,
P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 181005905 2018.
J. Ho and Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
K. Kim, G. Swamy, Z. Liu, D. Zhao, S. Choudhury, and S. Z. Wu. Learning shared safety constraints from multi-task demonstrations. Advances Neural Information Processing Systems, 36, 2023.
I. Kostrikov, Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. The Tenth International Conference on Learning Representatiins, ICLR 2022, Virtual Event, April 25- 29, 2022. OpenReview.net, 2022. URL
A. Kumar, A. Zhou, Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning, Advances in Neural Information Processing Systems, 33:1179-1191, 2020.

---

L. Lindemann, A. Robey, L. Jiang, S. Das, S. Tu, and N. Matni. Learning robust output control barrier functions from safe expert demonstrations. IEEE Open Journal of Control Systems, 2024.
G. Liu, Y. Luo, A. Gaurav, K. Rezaee, and P. Poupart. Benchmarking constraint inference in inverse reinforcement learning. In The Eleventh International Conference Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a. URL https://openrevicen net wttpr//iwiw.eetepfffd/i/i/.. http:ppdffddviI___veee
Z. Liu, Z. Guo, H. Lin, Y. Yao, Zhu, Z. Cen, H. Hu, W. Yu, T. Zhang. J. Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.093300 2023b.
C. Lyle, Z. Zheng, E. Nikishin, B. A. Pires, R. Pascanu, and Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, pages 23190-23211. PMLR, 2023.
S. Malik, U. Anwar, A. Aghasi, and A. Ahmed. Inverse constrained reinforcement learning. In International conference on machine learning, pages 7390-7399. PMLR, 2021.
A. Marot, A. Kelly, M. Naglic, Barbesant, J. Cremer, A. Stefanov, and Viebahn. Perspectives on future power system control centers for energy transition. Journal of Modern Power Systems and Clean Energy, 10(2):32-3344 2022. doi: ...5833/MMCCE.221..0000
D. L. McPherson, K. C. Stocking, and S. S. Sastry. Maximum likelihood constraint inference from stochastic demonstrations. In 202/ IEEE Conference on Control Technology and Applications (CCTA), pages 1208-1213. IEEE, 2021.
M. Menner, P. Worsnop, and M. N. Zeilinger. Constrained inverse optimal control with application to human manipulation task. IEEE Transactions Control Systems Technology, 29(2):826-334, 2019.
S. Miryoosefi, K. Brantley, H. Daume III, M. Dudik, and R. E. Schapire. Reinforcement learning with convex constraints. Advances in Neural Information Processing Systems, 32, 2019.
A. Y. Ng. Russell, et al. Algorithms for inverse reinforcement learning. In leml, volume 1, page 2, 2000.
A. Y. Ng. Coates, M. Diel, V. Ganapathi, Schulte, B. Tse, E. Berger, and Liang. Au- tonomous inverted helicopter flight via reinforcement learning. In Experimental robotics IX: The 9th international symposium on experimental robotics, pages 363-372. Springer, 2006.
E. Nikishin, M. Schwarzer, P. D'Oro, P.-L. Bacon, and A. Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pages 16828 16847. PMLR, 2022.
M. Pirotta, A. Tirinzoni, A. Touati, A. Lazaric, and Y. Ollivier. Fast imitation via behavior foundation models. In The Twelfth International Conference on Learning Representations, 2023.
D. A. Pomerleau. Alvin:: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
A. Robey, H. Hu, L. Lindemann, H. Zhang, D. V. Dimarogonas, S. Tu, and N. Matni. Learning control barrier functions from expert demonstrations, In 2020 59th IEEE Conference on Decision and Control (CDC), pages 3717-3724. IEEE, 2020.
Ross, G. Gordon, and D. Bagnell. A reduction imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings. 2011.
S. J. Russell and A. Zimdars. Q-decomposition for reinforcement learning agents. In Proceedings of the 20th international conference on machine learning (ICML-03), pages 656-663, 2003.
Schaal. Learning from demonstration. Advances neural information processing systems, 1996.
12

---

D. R. R. Scobee and S. S. Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.ner, 2020.
id=RJliakStvH
G. Swamy, S. Choudhury. J. A. Bagnell, and Wu. Of moments and matching: A theorttcc framework for closing the imitation gap. In International Conference Machine Learning, pages 10022-10032. PMLR, 2021.
G. Swamy, D. Wu, S. Choudhury, D. Bagnell, and S. Wu. Inverse reinforcement learning without reinforcement learning. In International Conference on Machine Learning, pages 33299-33318. PMLR, 2023.
J. Weng, H. Chen, D. Yan, K. You, A. Dubureq, M. Zhang, Y. Su, H. Su, and J. Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1 6,,, 2022. http: org/ hwtgg/app////////////ee/eea/ssee 2ttp///2/2111111
html
S. Yue, G. Wang. W. Shao, Z. Zhang, S. Lin, J. Ren, and J. Zhang. Clare: Conservative model-based reward learning for offline inverse reinforcement learning. International Conference on Learning Representations (ICLR), 2023.
S. Zeng, C. Li, A. Garcia, and M. Hong. Maximum-likelihode inverse reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 35:10122-1011 2022.
B. D. Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforcement learning. Agai, volume 8, pages 1433-1438. Chicago, II., USA, 2008
13

---

A Experiment Details
A.1 Hyperparameters
All of our code is based the Tianshou [Weng al. 2022 and FSRL Liu ct al. 2023b] imple- mentations of SAC and SAC-Lagrangian, respectively. Here we include all the hyperparameter configurations for our experiments. Any hyperparameters not listed here the default hyperparam- eters in their respective libraries (Tianshou version 1.0.0 and FSRL version 0.1.0).
SAC Parameters
0.005
0.99
a LR
0.0003
or optimizer
Adam
Critic hidden layers
[256, 256]
Critic optimizer
Adam
Critic learning rate
0.001
Actor hidden layers
[256, 256]
Actor learning rale
0.001
Actor optimizer
Adam
n-step returns
Training Parameters
Max epochs
1000
Batch size
128
Environment steps per epoch
5000
Gradient steps per environment step
Test episodes per epoch
Episodes between gradient updates Buffer size
1000000
Constraint Learning Parameters
IRL-Base
IRL-L2
Output clip range
[0,20]
None
Output activation
Linear
Leaky ReLU
Hidden layers
[64, 64]
[64, 64]
Optimizer
Adam
Adam
Weight decay
0,001
0.001
Batch size
5000
5000
Gradient steps per epoch
1
L2 regularization coefficient
0
0.1
The constraint function learning rate tuned over two values (0.001, 0.0001) per environment with
the following final parameters:
Half Cheetah
Ant
Inverted Pendulum
Walker
Swimmer
0.0001
0.0001
0.001
0.0001
0.001
A.2
Computational Resources
Each run (consisting of five seeds) was trained on a node with a single GPU (varying GPU re- sources were used), CPUs and 6GB of RAM per CPU. A single run across three seeds under this configuration took approximately 1.5 days to complete training.
B Proof of Theorem
We prove both claims individually.
14

---

B.1 Step 1: OptiCRL Opts-
We begin by proving the first claim, that OPTICLL Fr, Fc) = Opts JCRL Fy, Fo) when Fe is convex cone. Firstly, it is simple to show that OptiCRL (II, Fo) 2 Opts
-PRts-ICLiP ICRL Fo,
OptICRL Fe,
max max min J (TT, Xe) - J(T, Ac) o A20 =ell
2 max min J cefe fell
-
Opts ICRL Fr,
remains to show that OpTICL (T, & Opts-IcRL (T,, We have
Opts-ICRL Fr, = max min J ( J(T, c)
cef TEll
max min J( Xc) J( Xc) V120
c well
2 max max min [TEE r- Ae) - J[ -x Ac) cEF: X20 sell
OptiCRL Fo,
where the first inequality holds since Ac E Fe whenever F. by the hypothesis that F is a convex cone. It follows that OptICRL FII, Opts-IcRL ICRL Fr.
B.2 Step 2: IRL Solves ICRL
now prove the second claim. For any policy æ, we define the successor features [gg et 2000
Ziebart et al. 2008 Barreto et al. 2017 16 S 12 according to
3 (8) =
Σ 120
(S,, AA)
So =8
(9)
where A, x(. S.). Then, for any initial state distribution nio and E F, we have J(n,r) = Exger (1) (so); wr}], where r(x, a) = (Ø(2, a), wr) this holds by the linearity of expectation.
Now, define V () value function under policy π for the reward function di. for any i E [d]. Thus, since II is assumed be compact and Φ is bounded, Dadashi al. (2019) shows that the set
II) C Rd. Note that
15 (s) is equivalent to the
{wb
E II]
convex and compact for each i [d]. By continuity, V. {B th? (so) E IT) is convex
compact. Therefore, holds that as a product convex and compact sets, V = x (=1 V; is convex and compact.
d
Next, suppose (Tr* e* - r) realizes OPTIRL (II, F - r). Since is compact and convex and F convex, we apply Sion's 's minimax theorem and observe that, for thg = Eso t (so),
15

---

J(x',c² -r) [T c² -r) = R (I, r)
min sup J
J J[ c
cEJ
min sup (v) D We w+)
NEW W.ERd
min sup (N' v. Wy we)
NEW w,e3"
sup min v1 w, -we)
Sion's minimax theorem
w.eRd NEV
sup min J (TE, c) - -J( oE sell
Opts-IcRL (1F
OptICAL FFF)
Step
:.TE J(T",, c*) OptiCAL F,F,
where the penultimate step holds because F, as vector space, a convex cone. Thus, we conclude that solving the IRL problem over the reward class F and policy class Il solves the inverse constrained RL problem of interest.
C
Additional Results
C.1 ICRL SAC Implementation Results
We attempted to reproduce the results of Liu et al. 2023a using SAC for policy learning. We tested six settings for the Lagrangian and reward learning rates shown in Figure All other hyperparameters were kept the same as Appendix I except we ran each configuration for 3M timesteps. Notably, had considerable difficulty reproducing the results of al. [023al using the SAC algorithm. While Kim et al. [2023] provided a SAC implementation of a similar algorithm, we note that their implementation added various additional components the algorithm to improve convergence. including in particular, interleaving reward learning with behavior cloning. We did not use these techniques here.
C.2 Learning Curves
Here we provide the full learning curves for all ablations and all environments. An additional configuration, IRL-Basc+BN is included which tested but included in the main results due to poor performance. This configuration added batch normalization to the output of the reward function potentially help combat issues with vanishing gradients.
16

---

reward.rr-0.0000 lambda.ir-O..c
reward.in-D.ook lambda./r-0.00
reward 0001,,
reward..r=@,0001. lambba..rr0..p =0.01
reward.Ir=O.000 0.001, lambba.ir=0001
wwward.InwO..o 0000000., lambda 100000001
Feasible Roward
Feasible Reward
Fessible Reward
3000
15200
5500
10300
2000
5300
0.5
3.5
2.5
1.0
2.0
Vislation Rate
Violation Rate
Vinlation Rate
1.0
2.0
0.0
0.8
0.6
0.4
0.4
0.2
5.2
0.0
0.5
1.0
2.0
2.5
3.3
0.5
1.0
2.5
2.5
3.0
3.0
1.5
2.0
2.5
(a) Half Cheetah
(b) Ant
(c) Inverted Pendulum
Feasible Reward
Feasible Reward
1202
250
003
200
600
250
300
166
36B
Violation Rale
Vislation Rate
3.8
0.3
322
2.3
0.
3.3
3.2
(d) Walker2D
(c) Swimmer
Figure 5: Training curves for feasible rewards (top) and violation rate (below) for ICRL with SAC with varying learning rates for the reward function and Lagrange multiplier across the five MuJoCo environments. None of the hyperparameter settings perform consistently well across environments.
17

---

RRL-245CC+PR
IRL-L2-PR
IRL-Base
MECL
IRLL2+SC
IRL-12
IRL-Base
GACL
Reasible Rewards
Feasible Rowards
Feasible Rewards
30030
20003
93
15003
42
20002
4030
22
2010
Violation
2.8
0.6
0.6
3.6
0.5
3.4
0.2
0.0
2.0
(a) Half Cheetah
(b) Ant
(c) Pendulum
Feus
700
2002
1502
100
2002
200
300
166
146
Molation Rate
Violation Rate
0.5
2.8
62
52
0.
2.0
Walker2D
(c) Swimmer
Figure 6: Training curves for feasible rewards (top) and violation rate (below) for all variations across the five MuJoCo environments.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that simple IRL methods can perform as well or better than ICRL methods across environments. This is shown in our experiments comparing MECL to IRL (see Figure D
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include limitations section (see Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes].
We provide a complete and correct proof of our theoretical result Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provided full hyperparameter details for our experiments in Appendix A. Besides this, we reference exact implementations for each component of our algorithm (i.e.
IRL from Zeng et al.
(2022] and we utilize standard and well-maintained libraries (Weng
ct al. 2022 Liu et al. 2023b the basis for our code. The datasets used available from Liu et al. [203a Finally, we provide public Github for the code.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We do not include code directly in the supplemental materials, however link to the publicly available Github repository is provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
http:/.cccpubbbc/uuuueeee Codeeubmii ssi crrionPPoiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provided the most relevant configuration details in Section 5 and full hyperparameter details for experiments in Appendix A.T
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We include error bars in all figures (see Figures I to computed across five seeds (and aggregated across five environments in the case of Figures and 3) using bootstrapped 95% confidence intervals as recommended Agarwal al. 202T
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

http:///cppppccccccccciiieeeeeee The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details of the computational resources used in Appendix A.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Section we discuss possible real-world applications for constraint inference such as in power grid control, that could confer societal benefits. We do not see any negative societal impacts that would come from our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not release any data or models with high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The only assets we use in our experiments are the data provided by Liu et al. [2023a] which cite extensively, including particularly in Section to note that
using these datasets in our experiments. We also cite the relevant code packages used to implement experiments in Appendix A.I
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not use crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do use human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "MeZxJfhkfz", "reviewer": "Reviewer_iin6", "summary": "The authors propose a method for learning the constraints from demonstrations. To achieve this goal they take note of the similarities between inverse reinforcement learning (IRL) and inverse constraint reinforcement learning (ICRL). The authors aim to reduce the tri-level optimization of constrained inverse reinforcement learning to a bi-level optimization using a special class of constraints.", "questions": "* The clarity of the paper can be significantly improved by adding diagrams or pseudocode describing their specific modifications to the previous IRL methods.\n* Question mark instead of a citation on line 98 (Related Works -> Inverse Reinforcement learning).\n* Eq. (1) describes the IRL objective. In its current form it aims to find a reward function that minimizes the difference between the return of the expert and the current policy while maximizing this difference. The objective should be the opposite where the policy aims to reduce the difference between the return on the expert and the current policy and the reward should aim to maximize the difference between the return of the expert and the current policy.\n* Given that this paper focuses on learning policies that violate fewer constraints and given that the claims are supported mainly by experimental results, the experiments section should focus on a more diverse set of environments that require safety constraints or at least describe in more details the constraints present in the Mujoco environments used for the experiments.\n* If the tri-level optimization can be reduced to a bi-level IRL problem, can it then be further reduced to a single level optimization using methods such as Inverse Q-Learning [1] or IQ-Learn [2]? How does the constraint violation rate of the proposed method compare to such  algorithms?\n\n[1] Kalweit, G., Huegle, M., Werling, M., & Boedecker, J. (2020). Deep Inverse Q-learning with Constraints. Advances in Neural Information Processing Systems, 33, 14291-14302.\n[2] Garg, D., Chakraborty, S., Cundy, C., Song, J., & Ermon, S. (2021). IQ-Learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34, 4028-4039.", "limitations": "* High variance in the experimental results (acknowledged by the authors).\n* The number of constrained environments and the discussion on how these constraints are present in each environment is lacking.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The main claim of the paper with regard to reducing the ICRL to IRL under certain classes of constraint functions is original and can be quite impactful.", "weaknesses": "* The authors' claim that the tri-level optimization can be reduced to a bi-level has a mathematical motivation but it not supported by a mathematical derivation. The authors aim to support the claim via experimental results but the results have high variance and are thus not fully convincing.\n* The paper's main claims are tested using methods developed by prior work (MECL, GACL). The specifics of the modifications done by the authors are somewhat unclear however. The authors mention using SAC instead of PPO for the IRL implementation of the prior method which they compare with. Later it is mentioned that their results are directly taken from the prior works due to lower performance of their SAC-based implementation. \n* The paper combines various modifications to prior work (batch and reward normalization, separate critics, policy reset) in order to examine their benefits and support the main claims. However due to large variance in results in limited locomotion environments, it is difficult to accept the results with high degree of certainty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful consideration of our work. We would hope that the improvements we have made and highlighted in the overall response will alleviate some of your concerns, but we would like to address each of your concerns here specifically:\n\n- “The authors' claim that the tri-level optimization can be reduced to a bi-level has a mathematical motivation but it not supported by a mathematical derivation.” \n  - We hope the provided proof in the general response alleviates your concerns here.\n- “The specifics of the modifications done by the authors are somewhat unclear”, “The clarity of the paper can be significantly improved by adding diagrams or pseudocode describing their specific modifications to the previous IRL methods.”\n  - We have added pseuocode for the separate critics IRL method for inferring constraints in the attached PDF. The additional modifications (policy reset, batch norm and reward normalization) are implemented as standard.\n- “Later it is mentioned that their results are directly taken from the prior works due to lower performance of their SAC-based implementation.” \n  - Indeed, we used SAC in our implementation because the ICRL method we compare against is motivated by maximum-entropy RL. In their implementation, they use an entropy bonus with a PPO objective. Though we tried various learning rates for the ICRL method in SAC (see Appendix 1), we needed to tune all hyperparameters from scratch due to the change in algorithm. In the code of Liu et al 2023, it appears that hyperparameters were tuned quite specifically to each environment, and this full hyperparameter tuning was beyond the scope of our work. Hence, we compare directly to the reported results from Liu et al 2023, without reimplementing in SAC. \n- “However due to large variance in results in limited locomotion environments, it is difficult to accept the results with high degree of certainty.”\n  - We hope that our general response has alleviated the concerns regarding the variance in the results. Regarding the diversity in environments, we use these environments because of their precedent in prior work on ICRL (Liu et al 2023, Malik et al 2021)\n- “Eq. (1) The objective should be the opposite”\n  - We believe the equation is correct, however, another reviewer pointed out that it would be more intuitive to report the negation of these equations. We will do this in the final version.\n- \"If the tri-level optimization can be reduced to a bi-level IRL problem, can it then be further reduced to a single level optimization using methods such as Inverse Q-Learning [1] or IQ-Learn [2]? How does the constraint violation rate of the proposed method compare to such algorithms? \"\n  - Thank you for raising this question. It should potentially be possible to do this for IL purposes, however, in ICRL the goal is generally to recover the constraint function. It is not entirely straightforward how to disambiguate rewards and constraints in a method like IQ-learn which learns the Q function directly. This may be an interesting direction for future work.\n\n\nG. Liu, Y. Luo, A. Gaurav, K. Rezaee, and P. Poupart. Benchmarking constraint inference in inverse reinforcement learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. \n\nS. Malik, U. Anwar, A. Aghasi, and A. Ahmed. Inverse constrained reinforcement learning. In International conference on machine learning, pages 7390–7399. PMLR, 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VHbtjc1yZp", "reviewer": "Reviewer_fdQQ", "summary": "The paper explains how inverse constrained RL (ICRL) - the task of recovering (safety) constraints from expert behaviour respecting those constraints - is equivalent to (straight) inverse RL (IRL) - the task of recovering a reward function from expert behaviour approximately optimizing that reward. This allows the ICRL problem to be solved using the wide array of techniques available in IRL. The authors then go on to empirically show superior performance of using a basic IRL algorithm on the IRL formulation relative to specialized ICRL algorithms.", "questions": "Suggestions for improvement:\n- There is often missing punctuation after equations (e.g. full stop at the end of line 150 or commas after equations 1 and 6). In general, equations are part of the sentence structure and should be punctuated as such (and don't necessarily need to be preceded by a colon).\n- Weakly held opinion: equations 5 and 6 would feel more natural to me in negated form, i.e. finding a policy that maximizes the return subject to reward that minimizes the return of the policy relative to the expert. (yes, I'm taking into consideration also that this then leads to equations 7,8). Like this I find them a bit confusing at first without explanation.\n- The colour-coding of Figure 1 seems a bit unfortunate and doesn't help me much in reading the figure - could e.g. positive values be shades of green and negative ones shades of red with white at zero (not fussy about particular colors, but something that creates more contrast)? Please add x axis labels to Figures 2,3,4. Not all of the horizontal lines in Fig 3 are visible (maybe add a slight offset to prevent perfect overlap?). I'd also prefer seeing confidence interval on the mean across more than 3 seeds, rather than standard deviation. \n\n\nMinor typos:\n- l.3 \"However\" -> \"however\"\n- l. 98 has a missing reference\n- l. 126 has double space", "limitations": "I think the paper has a clear scope corresponding to the scope of previous work and doesn't introduce significant limitations beyond that.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper is rising an important point: ICRL has been arising as a subarea of ML in the past few years with its own methods. Showing that it's equivalent to an existing, more developed area of research (IRL), is certainly useful since it (1) unlocks the use of IRL methods in ICRL (2) puts into question the need for ICRL to exist as a separate subarea and certainly puts more burden on ICRL researchers to show that their methods indeed solve the problem better than vanilla IRL and why. The results from this paper suggest that currently that's not the case.\n- The paper is clear and easy to understand.\n- The paper is a prime illustration of the maxim that a machine learning conference paper should ideally make a single crisp point.\n- Experiments support the claims made in the paper.", "weaknesses": "- In some places, the paper is a bit too wordy and the same thing could be said as clearly with a shorter sentence (an LLM will surely be happy to provide suggestions, of which at least some may be good).\n- the statistical methodology is somewhat underwhelming in Section 5, but not a deal-breaker for me. Fig 1 could give confidence intervals on the mean. Similarly, it'd be useful to have a confidence interval on the mean in the other two figures and ideally across more seeds than 3 (since supposedly the hypothesis we're testing in is whether the mean is better than the baselines). Especially results in 5.2 are somewhat messy. Maybe estimating something like Shapley values may be helpful here? Also, normalizing the results and then aggregating across the environments could help paint a clearer picture (you could work with a normalized effect size relative to the IRL baseline as the main quantity). I definitely find mostly the final performance interesting, so I wouldn't be against moving the training curves into the appendix if you want to free up space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful consideration of our work. We are very glad that you found our work valuable and greatly appreciate your suggestions for improvement.\n\nRegarding your main concern, “the statistical methodology is somewhat underwhelming in Section 5”, we have incorporated many of your suggestions in the overall response, namely (1) we report only final performance in Fig  2 in the rebuttal PDF, (2) We compute confidence intervals rather than standard deviation (3) we include 5 seeds rather than 3 and (4) We have included Fig 1 in the rebuttal PDF which aggregates across all the environments using an expert normalized score. \nRegarding your minor suggestions:\n\n- “Equations 5 and 6 would feel more natural to me in negated form”: Another reviewer was also confused by this - we will include equations in negated form in the final paper\n- “The colour-coding of Figure 1 seems a bit unfortunate and doesn't help me much in reading the figure “ - We will replace Figure 1 with Figure 1 from the rebuttal PDF in the final report.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a1CE0Csx0O", "reviewer": "Reviewer_xRd9", "summary": "The paper proposes a novel inverse constraint learning approach that leverages inverse reinforcement learning (IRL). Prior work by Kim et al. and Malik et al. proposed a game-theoretic approach to the constraint learning problem, where the resulting optimization problem is a tri-level optimization problem involving the policy, the constraint function, and the Lagrange multiplier. The algorithm is an iterated optimization that alternates between constrained RL and constraint function learning. In contrast, the present work leverages the equivalence between the original problem and a simpler IRL problem, which was overlooked in prior work. This reduction removes the Lagrange multiplier from the decision variables and enables the use of an existing IRL algorithm for the purpose of simultaneous constraint learning and constrained policy optimization. Simulation results in the Mujoco environment suggest the feasibility of the IRL approach.", "questions": "* In equation (4), do we need the outer optimization in $\\lambda$ to be over non-negative real values, not the entire real line (because of the inequality constraint)?\n\n* In the IRL formulation, the class of constraint functions $\\mathcal{F}_c$ is more restrictive than the original formulation of Kim et al. (which is convex and compact). Can the authors comment on possible negative implications of this restriction?\n\n* I did not fully follow the reward scaling part in Section 4.2. I wonder if the authors can elaborate on what they meant by “the constraint function is learned independently of the reward scale and hence may be more robust to different reward scales.”", "limitations": "* The limitations are properly discussed in Section 6, but the inconclusive nature of the analysis can be possibly alleviated in the revision, as discussed above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The present work provides a key insight that the inverse constraint learning problem of Kim et al. is equivalent to inverse reinforcement learning, which was overlooked in the prior work. Although the equivalence requires an additional assumption that the set of constraint functions must be closed under multiplication by positive scalars, we can apply game-theoretic IRL algorithms to solve the constraint learning problem as long as we conform to the constraint function class.", "weaknesses": "Although the present work provides an interesting insight into constraint learning problems in RL, the paper in its current form possesses multiple issues.\n\n### Major Issues\nThe most concerning issue is the inconclusive and inconsistent nature of the analysis of experimental results, as detailed below.\n\n* The feasible rewards and the violation rate are two competing evaluation metrics employed in the paper, which are equally important considering the natural trade-off between performance and safety. Nevertheless, Figure 1 only focuses primarily on  the feasible rewards to compare different methods; most notably, it reports information on the violation rate only when it is worse than the proposed vanilla IRL method. Such a partial reporting can easily induce bias in the interpretation of the results, since it effectively ignores circumstances where the violation rate is improved for baselines. The authors should fully report both the feasible rewards and the violation rate, and investigate whether any method pareto-dominates the others in the two evaluation criteria. No definitive conclusion can be drawn without this analysis.\n\n* No detailed analysis of Figure 2 is presented in Section 5.1. In fact, the authors did not reference Figure 2 in the main body of Section 5, and the caption does not provide any analysis either. By looking at Figure 2, it is not clear whether any conclusion can be drawn about the relative performance between the proposed algorithms and the two baselines (i.e. MECL and GACL).\n\n* Speaking of the performance of MECL and GACL, It is not clear what “a best estimate” means in the footnote on page 6. In the first place, the authors should either reproduce the prior work with PPO or report the missing results as they are, with preference given to the former; they should never estimate missing values. If reproduction is impossible, I would contact the authors and ask for raw data.\n\n* Figure 3 is incomplete. Specifically, the left plot on Feasible Reward is missing GACL 80% and GACL 50%. Similarly, the right plot on Violation Rate is missing MECL 50%.\n\n* The paper examines whether various techniques for stabilizing constraint learning actually help the IRL approach, and provides some analysis in Figure 4 and Section 5.2. Unfortunately, its credibility is highly questionable given the small sample size (with only 3 seeds) and the fact that the error bars are largely overlapping in Figure 4. Rather than reporting standard deviation, the authors should report more appropriate uncertainty information, such as the standard error of the mean or a confidence interval. If time permits, please also conduct the experiment with more seeds, which may help separate the error bars and aid for more comprehensible results. (Note that the standard error reduces as the sample size is increased, as opposed to the standard deviation.)\n\n* The conclusion is partly self-conflicting. Specifically, in Section 5.2 the authors mention batch normalization (BN) and reward normalization (RN). They state that “we find these modifications are generally more harmful than beneficial. Including batch normalization, reward normalization or both, … tended to hurt performance over basic IRL in a majority of environments.” On the contrary, in Section 6 the authors write “certain combinations of additional simple regularization such as batch normalization and reward normalization can produce significantly better results in several environments.”\n\n\n### Minor Issues\n#### Problem Formulation\n* In equation (6), the Lagrange dual variable $\\lambda$ should always be the outer optimization variable. I believe that it should be max min, not min max (unless the minimax theorem holds.)\n\n* Equation (8) and its subsequent analysis are central to the development of this paper, and hence may be worth stated as a proposition or a theorem.\n\n#### Related Work\n* The authors seem to consider general Imitation Learning (IL) and IRL as two separate problems. A common view is that there are several categories in IL, of which behavioral cloning (BC) and IRL are among the most popular [1][2].\n\n* There is one reference that is missing on Page 3, line 98.\n\n* The authors state in Section 1 that “we would like to extract safety constraints from the data based on the expert behavior, which can then be used downstream to constrain task-specific learning.” Besides the RL approaches, there is a thread of prior work in learning-based control that considers this problem through the use of barrier functions and fully decoupling downstream task-specific learning from constraint learning (e.g. [3][4][5]), which is missing from the literature review.\n\n[1] Zare, Maryam, Parham M. Kebria, Abbas Khosravi, and Saeid Nahavandi. \"A survey of imitation learning: Algorithms, recent developments, and challenges.\" arXiv preprint arXiv:2309.02473 (2023).\n\n[2] Osa, Takayuki, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. \"An algorithmic perspective on imitation learning.\" Foundations and Trends® in Robotics 7, no. 1-2 (2018): 1-179.\n\n[3] Robey, Alexander, Haimin Hu, Lars Lindemann, Hanwen Zhang, Dimos V. Dimarogonas, Stephen Tu, and Nikolai Matni. \"Learning control barrier functions from expert demonstrations.\" In 2020 59th IEEE Conference on Decision and Control (CDC), pp. 3717-3724. IEEE, 2020.\n\n[4] Lindemann, Lars, Alexander Robey, Lejun Jiang, Satyajeet Das, Stephen Tu, and Nikolai Matni. \"Learning robust output control barrier functions from safe expert demonstrations.\" IEEE Open Journal of Control Systems (2024).\n\n[5] Castaneda, Fernando, Haruki Nishimura, Rowan Thomas McAllister, Koushil Sreenath, and Adrien Gaidon. \"In-distribution barrier functions: Self-supervised policy filters that avoid out-of-distribution states.\" In Learning for Dynamics and Control Conference, pp. 286-299. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful consideration of our paper and the helpful comments and suggestions. We would hope that the improvements we have made and highlighted in the overall response will alleviate some of your concerns, but we would like to address each of your concerns here specifically:\n\n- “The authors should either reproduce the prior work with PPO or report the missing results as they are” \n  - The performance of MECL and GACL reported in Fig 2 and Fig 4 are taken directly from the tables reported in Appendix D of Liu et al. 2023. However, the results of MECL and GACL for Fig 3 were not reported in a table and hence we estimated them based off the plots in Liu et al 2023. We agree that this is insufficient and instead have rerun the author’s code to generate all baseline performance estimates. We provide some of these results in Fig 1 and 2 of the rebuttal PDF which shows IQM on final performance of our proposed modifications versus MECL.\n- “Figure 1 only focuses primarily on the feasible rewards to compare different methods; most notably, it reports information on the violation rate only when it is worse” \n  - You are correct that Fig 1 primarily focuses on feasible rewards and only partially reports violation rate, however, we have included both violation rate and feasible rewards in full, both Fig 2 and Fig 3. We have replaced Fig 2 with the IQM plots in Fig 1 in the attached PDF, which will help make the reporting of the violation rate more clear. Additionally, we would like to point out that feasible rewards is a metric that attempts to capture both safety and performance in one metric, as feasible rewards are the rewards obtained in a trajectory only up to the first constraint violation. This is the metric that is proposed for benchmarking ICRL methods in Liu et al 2023.  \n- “No detailed analysis of Figure 2 is presented in Section 5.1”. \n  -Fig 2 is a reduced version of Fig 4 which shows only the best overall and best per-environment modifications, versus the baselines. We did this to make comparison easier, however, we recognize that this may seem redundant and be confusing. Following the suggestion of another reviewer we have moved Fig 4 to the Appendix and replaced it with Fig 2 in the PDF (IQM plots of final performance). \n- “Figure 3 is incomplete”\n  -Thank you for pointing this out. The issue is that some of the curves are overlapping, which makes them difficult to see. We will clarify this in the revision. \n- \"Unfortunately, its credibility is highly questionable given the small sample size (with only 3 seeds) and the fact that the error bars are largely overlapping\"\n  - We understand your criticism of the statistical analysis, particularly the small number of seeds and reporting of standard deviation. We hope that we have addressed these concerns in our overall response and by including additional random seeds.\n- “The conclusion is partly self-conflicting” [re the impact of batch norm and reward normalization]. \n  -Thank you for pointing out the confusing wording here. What we meant to say is that overall (across all environments) BN and RN are not generally beneficial. However, in specific environments (eg. Cheetah and Walker) we do see benefits. So they are helpful, but only on an environment-specific basis. We will make this more clear in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2pyeXDPKRF", "reviewer": "Reviewer_ZV2P", "summary": "This paper proposes a way to reduce the tri-level stucture of ICRL to bi-level, and uses solid experiment results to validate that this bi-level reformulation achieves better expirical results. The authors also intuitively explain that this is due to the fact that the tri-level optimization has a more complicated optimization landscape. In general, this paper clearly delivers its idea, and is easy to follow. However, the major weakness is that this simplification of tri-level to bi-level is trivial, and can be further exploited.", "questions": "Please see weakness.", "limitations": "The limitation is discussed in the paper and is reasonable.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper provides solid empirical results to evaluate its idea. This paper answers a fundamental question for ICRL, i.e., there is no fundamental difference between IRL and ICRL. In other words, we can use IRL algorithms to solve ICRL problems.", "weaknesses": "The main weakness lies in the core contribution, i.e., the simplification of tri-level to bi-level formulation. This simplication comes from the observation that we can always learn a cost function $c'$ that captures both the dual variable $\\lambda$ and the original cost function $c$, i.e., $c'=\\lambda c$. Therefore, we can learn a single cost function $c'$ to replace $\\lambda c$.\n\nThis idea is correct, and I agree that learning a single cost function $c$ is enough. However, this idea is trivial and can be futher exploited. In fact, I have seen similar trick in an ICRL literature [1]. In reference [1], the authors also remove the dual variable $\\lambda$ and only learn a single cost function $c$. Indeed, the reference [1] does not highlight this modification as their novelty, so that it is totally fine for this paper to highlight this trick as a core contribution.\n\nHowever, this \"c'=\\lambda c\" idea is more like an engineering trick, and more contribution is needed to support this trick. For example, as mentioned in the paper, compared to the tri-level formulation, the bi-level reduction has a simpler optimization landscape and thus is expected to have better optimization result. It will be great if the authors can theoretically support this claim, i.e., theoretically prove that this bi-level formulation is easier to achieve better result than the tri-level formulation.\n\n[1] Liu \\& Zhu, \"Meta inverse constrained reinforcement learning: convergence guarantee and generalization analysis\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful consideration of our paper. We understand that your primary concern with the paper is that our principal claim is trivial. While we understand that the derivation is relatively straightforward, we do not think the reduction is necessarily trivial, as evidenced by the number of peer-reviewed papers which propose complex methods for performing ICRL (Malik et al 2021, Liu et al 2023, Kim et al 2023).. Hence, our work would benefit the community by providing a reference and experimental justification for bypassing more complicated ICRL methods in favor of IRL methods for constraint inference. \n\nRegarding your suggestion, “It will be great if the authors can theoretically support this claim, i.e., theoretically prove that this bi-level formulation is easier to achieve better result than the tri-level formulation” - we believe that this is outside the scope of this paper to prove this generally. However, we note that there is already evidence for this in the literature. In particular, GANs are notoriously difficult to train due to the dynamics induced by the adversarial game (Goodfellow et al 2014). Generally, iterates of gradient descent do not converge to saddle points (Freund and Schapire 1997). Some solutions exist for bi-level optimizations / two-player games (Rakhlin and Sridharan 2012, Moskovitz et al. 2023), but we are unaware of solutions to the tri-level optimization problem.\n\nGenerative Adversarial Networks, Goodfellow et al. 2014, Training GANs with Optimism, Daskalakis et al. 2017\n\nA Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting, Freund and Schapire 1997\nOnline Learning with Predictable Sequences, Rakhlin and Sridharan 2012\n\nReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs, Moskovitz et al. 2023\n\nG. Liu, Y. Luo, A. Gaurav, K. Rezaee, and P. Poupart. Benchmarking constraint inference in inverse reinforcement learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. \n\nS. Malik, U. Anwar, A. Aghasi, and A. Ahmed. Inverse constrained reinforcement learning. In\nInternational conference on machine learning, pages 7390–7399. PMLR, 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful comments and suggestions. We agree with the reviewers that the main claim of our paper is that treating ICRL as a separate problem from IRL may not confer a particular benefit and, in fact, it should be beneficial to not segment the problem class because it means we can make use of the wide literature in IRL to solve ICRL problems. We are glad that most reviewers agree this is an interesting idea. We also provide empirical analysis to validate this claim and provide details on how to best use IRL for ICLR. We understand the primary concern of several reviewers is the statistical robustness of these results, including the high variance, the low number of seeds and the reporting of the statistical significance. We have taken the following steps to alleviate these concerns:

- We have rerun all experiments across 5 seeds to improve the robustness of our results. This is consistent with the number of seeds used by Liu et al 2023.
- We believe that the way we are currently reporting results may overstate the variability of the results. In Figs 2,3,4 we are showing error bars as standard deviation across both seeds and the time smoothing window. Hence, the error bars contain both aleatoric uncertainty (variation across episodes in a single agent) and epistemic uncertainty (variation across seeds). Following several reviewers suggestions we propose replacing Figs 1 and 4 with Figs 1 and 2 in the attached PDF, respectively. This makes the following adjustments:
  - In both figures, we use the following procedure. We first compute the mean of the last 50 testing episodes. We then compute bootstrapped confidence intervals of the IQM (interquartile mean) across the 5 seeds using the methodology proposed in Agarwal et al 2021
  - In Fig 1 in attached, we have additionally normalized by expert performance and reported an overall performance curve for all environments, as recommended in Agarwal et al 2021. This further increases the sample size to reduce variance. Since the baseline MECL is trained with PPO, and so has a different training time, we take only the IQM of final performance and report this as a sold dashed line (without CI) as a comparison. We believe this very clearly illustrates that IRL methods pareto-dominate the baseline ICRL method in these environments. 
  - In Fig 2 in attached, we now report only final performance on a per-environment basis, which we hope facilitates readability when comparing across modifications and environments.
- We have also rerun the author’s provided code for the baseline method MECL from Liu et al 2023 and included the same IQM with confidence intervals for this method, computed over 5 seeds in Fig 2 attached. This allows us to also compare the variance of our method to that of the baseline MECL. We note that the variance in the results of MECL are also very high as can be seen in Fig 2 in the attached PDF.  In general, high variance can be an issue for adversarial IRL, however, we do not think this issue is unique to our method. We also note that upon re-running the author’s code, we find that the reported performance of ICRL is not reproducible with the code provided, using the final performance (the author's may have reported best performance). This further improves the relative performance of IRL over the ICRL baseline.

Finally, we understand that several reviewers would like a more rigorous proof of the equivalence between ICRL and IRL. We offer the following, which is added to the Appendix of the paper:

**Theorem**. Let $\Pi$ denote a class of policies and let $\pi_E\in \Pi$ and $r:\mathcal{X}\times\mathcal{A}\to\mathbb{R}$ be a fixed (expert) policy and reward function, respectively. For any class of constraint functions $\mathcal{F}\subset \mathbb{R}^{\mathcal{X}\times\mathcal{A}}$, we define the following objectives:

$\mathsf{OPTicrl}(\mathcal{F}) = \max_{c\in\mathcal{F}}\max_{\lambda \geq  0}\min_{\pi\in\Pi}\left[J(\pi_E, r - \lambda c) - J(\pi, r - \lambda c)\right]$

$\mathsf{OPTsimple}(\mathcal{F}) = \max_{c\in\mathcal{F}}\min_{\pi\in\Pi}\left[J(\pi_E, r - c) - J(\pi, r - c)\right]$

where $J(\pi, f)$ denotes the expected return (averaged over initial states) earned by the policy $\pi$ for the reward function $f$. Then, if $\mathcal{F}$ is a convex cone, it holds that $\mathsf{OPTicrl}(\mathcal{F}) = \mathsf{OPTsimple}(\mathcal{F})$.

**Proof**. Suppose $\mathcal{F}$ is a convex cone. We will first show that $\mathsf{OPTsimple}(\mathcal{F})\geq \mathsf{OPTicrl}(\mathcal{F})$.
Let $\mu^\pi$ denote the (discounted) occupancy measure of policy $\pi$. It is well known that $J(\pi, f) = \mu^\pi f$. Then, we have

$\mathsf{OPTsimple}(\mathcal{F}) = \max_{c\in\mathcal{F}}\min_{\pi\in\Pi}(\mu^{\pi_E} - \mu^\pi)(r - c)$

$\geq \max_{c\in\mathcal{F}}\min_{\pi\in\Pi}(\mu^{\pi_E} - \mu^\pi)(r - \lambda c)  \forall \lambda > 0$

$\geq \max_{c\in\mathcal{F}}\max_{\lambda\geq 0}\min_{\pi\in\Pi}(\mu^{\pi_E} - \mu^\pi)(r - \lambda c)$

$= \mathsf{OPTicrl}(\mathcal{F})$


where the first inequality holds since $c\in\mathcal{F}\implies \lambda c\in\mathcal{F}$ by the hypothesis that $\mathcal{F}$ is a convex cone.
It remains to show that $\mathsf{OPTsimple}(\mathcal{F})\leq\mathsf{OPTicrl}(\mathcal{F})$. This is simply shown by

$\mathsf{OPTsimple}(\mathcal{F}) = \max_{c\in\mathcal{F}}\min_{\pi\in\Pi}(\mu^{\pi_E} - \mu^\pi)(r - 1\cdot c)$

$\leq \max_{c\in\mathcal{F}}\max_{\lambda\geq 0}\min_{\pi\in\Pi}(\mu^{\pi_E} - \mu^\pi)(r - \lambda c)$

$= \mathsf{OPTicrl}(\mathcal{F})$


We will address each reviewer’s particular concerns in our individual responses. Thank you again for your thoughtful consideration of our paper.

Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.

 Liu,, Guiliang et al: Benchmarking Constraint Inference in Inverse Reinforcement Learning. ICLR 2023

Meta-review: The paper's main contribution is an observation that in order to infer RL constraints from data, instead of solving a tri-level optimization problem one can solve a simpler bi-level one, under reasonable assumptions on the constraint functions. 

Originally the reviewers had several concerns, mainly focused on the paper's lack of theoretical justification and the statistical significance of the experimental results. However, the authors successfully addressed most of them in the rebuttals, including by adding a formal result about the algorithm's correctness. While there are still presentation issues to be addressed and, overall, some reviewers find the paper's contribution straightforward, the metareviewer thinks this contribution is valuable nonetheless, trusts that the authors will improve the manuscript per the reviewers' suggestions, and recommends this work for acceptance.
Decision: Accept (poster)