Title: Simplifying Constraint Inference with Inverse Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: reinforcement learning

inverse reinforcement learning

safe reinforcement learning

constrained reinforcement learning
Abstract: Learning safe policies has presented a longstanding challenge for the reinforcement learning (RL) community. Various formulations of safe RL have been proposed; However, fundamentally, tabula rasa RL must learn safety constraints through experience, which is problematic for real-world applications. Imitation learning is often preferred in real-world settings because the experts' safety preferences are embedded in the data the agent imitates. However, imitation learning is limited in its extensibility to new tasks, which can only be learned by providing the agent with expert trajectories. For safety-critical applications with sub-optimal or inexact expert data, it would be preferable to learn only the safety aspects of the policy through imitation, while still allowing for task learning with  RL. The field of inverse constrained RL, which seeks to infer constraints from expert data, is a promising step in this direction. However, prior work in this area has relied on complex tri-level optimizations in order to infer safe behavior (constraints). This challenging optimization landscape leads to sub-optimal performance on several benchmark tasks. In this work, we present a simplified version of constraint inference that performs as well or better than prior work across a collection of continuous-control benchmarks. Moreover, besides improving performance, this simplified framework is easier to implement, tune, and more readily lends itself to various extensions, such as offline constraint inference.

Simplifying Constraint Inference with Inverse Reinforcement Learning

Abstract

Learning safe policies has presented a longstanding challenge for the reinforcement learning (RL) community. Various formulations of safe RL have been proposed; However, fundamentally, tabula rasa RL must learn safety constraints through experience, which is problematic for real-world applications. Imitation learning is often preferred in real-world settings because the experts' safety preferences are embedded in the data the agent imitates. However, imitation learning is limited in its extensibility to new tasks, which can only be learned by providing the agent with expert trajectories. For safety-critical applications with sub-optimal or inexact expert data, it would be preferable to learn only the safety aspects of the policy through imitation, while still allowing for task learning with RL. The field of inverse constrained RL, which seeks to infer constraints from expert data, is a promising step in this direction. However, prior work in this area has relied on complex tri-level optimizations in order to infer safe behavior (constraints). This challenging optimization landscape leads to sub-optimal performance on several benchmark tasks. In this work, we present a simplified version of constraint inference that performs as well or better than prior work across a collection of continuous-control benchmarks. Moreover, besides improving performance, this simplified framework is easier to implement, tune, and more readily lends itself to various extensions, such as offline constraint inference. Our code is made available at https://github.com/ahugs/simple-icrl.

1 Introduction

Reinforcement learning (RL) has made significant advances in recent years, yet real-world applications of RL remain limited due to various challenges, including particularly safety concerns [Dulac-Arnold et al., 2021]. One common setting where RL holds promise for real-world deployments is replacing existing, possibly sub-optimal, human-managed control policies. For example, in the field of power network control, decisions are often made by a combination of automatic control policies combined with careful human monitoring and intervention [Marot et al., 2022]. However, concerns regarding the deployment of autonomous systems on safety-critical tasks have hindered the adoption of RL for replacing legacy control systems. This is particularly true for deep RL, which functions as a black box controller.

In the setting of replacing legacy controllers, however, the challenge is simplified due to access to data produced by the current control system. While existing control policies may be sub-optimal, they are nonetheless 'safe' in the sense that they obey some explicitly or implicitly defined constraints corresponding to a human understanding of safety. Ideally, we would like to be able to learn an RL policy that can outperform the current system, in terms of optimizing the reward, while continuing to respect these constraints. When these constraints are unknown (i.e. defined implicitly through human

actions), the implicit safety procedures that are being followed may not be explicitly known or have an obvious encoding. Thus, it would be beneficial to infer the constraints directly from trajectories produced by the existing control policy.

Typically, in settings with large-scale offline data, imitation learning [Schaal, 1996] or offline RL [Kumar et al., 2020, Kostrikov et al., 2022] might be used. However, these methods only extract a single policy that satisfies constraints and optimizes a reward function. Moreover, imitation learning generally cannot outperform the expert, which is prohibiting in cases where we wish to improve over current control policies. Offline RL can learn policies that outperform the demonstration [Kumar et al., 2020], however, offline RL cannot ensure that safety constraints remain satisfied without access to the constraint function or constraint violation annotations in the dataset.

Ideally, we would like to extract safety constraints from the data based on the expert behavior, which can then be used downstream to constrain task-specific learning. Learning constraints from expert trajectories is the purview of the field of constraint inference. Various methods have been proposed in this domain, with early work focusing on simple settings such as tabular MDPs [Scobee and Sastry, 2020]. More generally, inverse constrained reinforcement learning can infer constraints in continuous settings using parameterized constraint functions and adapting IRL methods such as maximum entropy IRL [Ziebart et al., 2008] to the constrained setting by solving a constrained MDP in the inner optimization. However, this tri-level optimization creates a challenging landscape for constraint inference.

In this work, we demonstrate that the constrained MDP inner loop is an unnecessary complication and that regular IRL techniques can recover as good and sometimes better solutions to constraint inference problems than these more complicated methods. This result is significant because it allows us to simplify the training dynamics and complexity of constraint inference methods and implies that advances in sub-domains of IRL can be directly applied to the constraint inference case. For example, recent progress in offline IRL [Yue et al., 2023, Kim et al., 2023] can be readily adapted to the constraint inference case to infer constraints entirely offline, significantly increasing the scope of applicability in real-world settings.

In particular, we make the following contributions. First, we show that inverse constrained RL and inverse RL are equivalent under certain classes of constraint functions. Next, we experimentally validate this claim. Finally, we propose some practical modifications to adapt IRL to constraint inference tasks and conduct ablations over these algorithmic choices to understand how to improve the stability and performance of constraint inference.

2 Related Work

General imitation learning The problem of imitation learning (IL) is primarily concerned with learning a policy that produces similar behavior to a class of reference policies. Specifically, given transition data from 'experts', the goal is to produce a new policy that, in a sense, generates a similar transition distribution. The simplest approach to IL is behavior cloning (BC) [Pomerleau, 1988], which estimates a policy via maximum likelihood to match the conditional distribution on actions conditioned on state observations. BC tends to perform well in the regime of massive data but suffers from compounding errors over long trajectories as state observations veer outside the coverage of the dataset [Ross et al., 2011]. Methods such as DAgger [Ross et al., 2011] show that, given the ability to interact with the environment during IL and to query expert trajectories, strong policies can be learned with relatively few queries to the expert. More recently, behavior cloning losses have been integrated into standard reinforcement learning methods to enhance offline RL performance - that is, RL from a fixed (offline) dataset [Fujimoto and Gu, 2021]. Alternative approaches to IL have focused on learning a policy that induces a similar state-visitation distribution to the dataset [Pirotta et al., 2023]. Such an objective explicitly accounts for the long-term behavior of a policy as opposed to the myopic predictions of BC methods, resulting in more coherent learned behaviors over longer horizons with less data. IL methods as discussed above can generally learn performant imitation policies, but provide no insight about the reward function that the expert is implicitly optimizing; this is the focus of inverse reinforcement learning, which will be useful for the purpose of learning constraint functions.

Inverse Reinforcement Learning Inverse reinforcement learning (IRL) learns a policy that can mimic expert trajectories and also aims to infer a reward function that explains the expert behavior [Ng et al., 2006]. The IRL problem is considerably more difficult than the IL problem; in particular, there is no unique reward function that maximally explains the expert behavior. The work of Abbeel and Ng [2004] alleviates this issue with a maximum margin constraint in the optimization. The work of Ziebart et al. [2008] resolves this problem by inferring the most likely reward function under a particular probabilistic model; this approach is widely known as maximum entropy IRL (MaxEnt IRL). One approach for achieving strong guarantees in IRL is to approximate the expert occupancy distribution in the operator norm -that is, to have a low discrepancy between expected returns under any reward function. Achieving this generally involves a bilevel optimization procedure where one alternates between optimizing a policy for a given reward function, and then adversarially training the reward function [Gleave and Toyer, 2022]. The work of Garg et al. [2021] reduces this problem in the maximum entropy framework by estimating a soft action-value function ( Q ), inferring the reward function from a novel 'inverse Bellman operator' applied to Q and showing that the policy π ( · | s ) ∝ exp( Q ( s, · )) is MaxEnt-optimal. Effectively, their method reduces the bilevel optimization discussed above into a simpler RL optimization. Alternatively, Swamy et al. [2021] assumes that the reward function lies in a reproducing kernel Hilbert space, which allows them to obtain an unbiased estimate of the adversarial reward function from data for any policy this is yet another method for eliminating one of the layers of the bilevel optimization discussed above. However, these aforementioned IRL methods have not considered the issue of modeling environmental constraints which can be transferred when learning policies for novel tasks.

Inverse Constrained Reinforcement Learning Early work in constraint inference focused primarily on particular settings such as convex constraints [Menner et al., 2019, Miryoosefi et al., 2019], or tabular RL [Scobee and Sastry, 2020, McPherson et al., 2021, Chou et al., 2020]. More recent methods have integrated the power of deep learning within the constraint inference framework through adaptions of MaxEnt IRL to the constrained setting through inverse constrained reinforcement learning (ICRL) [Malik et al., 2021, Liu et al., 2023a, Kim et al., 2023]. These methods essentially replace the forward RL inner loop of IRL with a constrained version of the problem by casting it as a constrained MDP (CMDP) and solving by Lagrangian or other methods. This adds additional complexity to the IRL problem which is already difficult due to the bi-level optimization and identifiability issues [Gleave and Toyer, 2022]. Though these prior works have sometimes considered simple IRL baselines [Malik et al., 2021, Liu et al., 2023a], none have specifically outlined the connections between IRL and ICRL, nor have they attempted to optimize IRL for constraint inference. A related line of work from the control theory community also considers learning constraints from expert data as control barrier functions, [Robey et al., 2020, Castaneda et al., 2023, Lindemann et al., 2024], however these methods rely on access to a known or learned model of the dynamics, which we do not assume.

3 Background

Imitation Learning. Imitation learning is the process of training a new policy to reproduce expert policy behavior. Given a set of expert trajectories D e = { τ ( i ) e } N i =0 , which follow a policy π E , a new policy π can be learned to match these trajectories using supervised learning, i.e. maximizing the expectation: E π E [ ∑ T t =0 log π ( a t | s t , θ ) ] . While this simple method can work well, it often suffers from distribution mismatch issues leading to compounding errors as the learned policy deviates from the expert's behavior [Ross et al., 2011]. Inverse reinforcement learning avoids this issue by extracting a reward function from observed optimal behavior [Ng et al., 2000].

Maximum Entropy IRL As discussed in Section 2 there are several formulations of IRL. Here we describe maximum entropy IRL [Ziebart et al., 2008, Ziebart, 2010]. Given a set of expert trajectories D e = { τ ( i ) e } N i =0 , which follow a policy π E , a policy π can be trained to produce similar trajectories by solving the min-max problem

where r is a learned reward function and H ( π ) is a causal entropy term. Hence, IRL is searching for a reward function r that is high for the expert π E and low for other policies.

Constrained RL One way to formulate constrained MDPs (CMDPs) is through trajectory-level constraints. In this formulation, the agent must solve the following objective

where c ( s t , a t ) is a cost function with δ budget.

The most common method for solving this problem is Lagrangian variations of RL methods, such as SAC-Lag [Ha et al., 2020], where Equation (2) is converted to a min-max problem

Inverse constrained RL Inverse constrained reinforcement learning (ICRL) is a class of methods used to infer constraints from expert trajectories. As in imitation learning and IRL, it is assumed that we have access to a set of expert trajectories D e = { τ ( i ) e } N i =0 , which follow a policy π E , however, it is now assumed that the expert policy is optimal under a CMDP instead of an MDP. It is generally assumed that the agent has access to the unconstrained (nominal) MDP with which to interact, and that the (constraint-agnostic) reward is observed by the agent when interacting with the environment.

Kim et al. [2023] present a general formulation of ICRL as game-solving, though we note that their formulation was preceded by that of Malik et al. [2021], and, though motivated differently, generally matches it, up to some implementation details.

As shown in prior work [Swamy et al., 2023], the inverse RL problem can be cast as a two-player zero-sum game

where F is convex and compact and J ( π, f ) = E π [ ∑ T t =0 f ( s t , a t ) ] .

Similarly, the constrained RL problem in Equation (2) with a single constraint can also be cast as a two-player zero-sum game

Finally, Kim et al. [2023] show that for inverse constrained RL, these two games can be combined into a three-player game of the following form (where r is a given reward function in a class F r )

Practically speaking, solving this game involves running IRL where the inner loop optimization solves a constrained MDP using a Lagrangian version of RL, such as SAC-Lag [Ha et al., 2020].

4 Method

Starting from Equation (6), we now demonstrate that this tri-level optimization is equivalent to a simpler bi-level optimization under certain classes of constraint functions and that this bi-level optimization is equivalent to the IRL game-solving formulation in Equation (4).

4.1 Inverse constrained RL as IRL

Note that there are two outer maximizations in Equation (6), the first over constraints in the class F c and the second over λ > 0 . Our key insight is that, under a broad class of constraint functions F c , this tri-level optimization can be cast as a bi-level optimization, reducing ICRL to IRL.

Theorem 4.1 Let π E ∈ Π and a reward function r ∈ F r be given. Consider the objective

Then, if F c is a convex cone (that is, c ∈ F c = ⇒ λc ∈ F c for any λ ≥ 0 ), it holds that Opt ICRL (Π , F r , F c ) = Opt S -ICRL (Π , F r , F c ) .

Moreover, suppose Π is compact and F r = F c = F is a vector space with elements f : ( s, a ) ↦→ ⟨ ϕ ( s, a ) , w f ⟩ , where ϕ : S × A → [0 , 1] d is a fixed (not necessarily known) feature map and w f ∈ R d identifies the elements of F . Then any solution to Opt IRL (Π , F r ) 1 is a solution to Opt ICRL (Π , F , F ) -that is, the ICRL problem is simply an instance of IRL.

The intuition for this is the following. The only difference between solving a constrained MDP with Lagrange multiplier λ and solving an unconstrained MDP with a cost penalty term weighted by λ is that, in the former case λ is optimized to ensure constraint satisfaction. Indeed, given the optimal λ ∗ a priori, constrained MDPs can be solved exactly by optimizing the later problem. This distinction is generally important when the cost function is fixed and unknown, however in the case of inverse learning where we learn the cost function from a class closed to scalar multiplication, it is possible to learn a cost that directly ensures constraint satisfaction, without scaling by the Lagrange multiplier. Note that optimizing over a convex cone F c can pose challenges when exact optimization is required-for instance, convex cones violate a compactness condition required to prove regret bounds for inverse constrained RL given by Kim et al. [2023]. However, in large scale applications that approximately optimize a constraint model represented by a deep neural network, one is often already in the setting where these regret bounds do not hold. Hence, when employing parameterized constraint inference through deep learning, it is sufficient to select an appropriate class as the output activation of our neural network and use a bi-level optimization to learn the scaled constraint.

4.2 Techniques for stabilizing constraint learning

Given that IRL and ICRL are mathematically equivalent under the class of constraint functions described above, the question of how best to perform constraint inference through inverse learning becomes largely a practical one. Is the optimization landscape smoother if we explicitly optimize for λ as a Lagrange multiplier or implicitly as part of the constraint function? Do previous implementations of Lagrangian methods add additional algorithmic components that could explain the improved performance of ICRL over IRL in prior work? Moreover, are there other regularizations or modifications that we could consider? In this regard, we suggest the following practical modifications to IRL for the constraint learning case, which we will test in Section 5.

Bounding rewards One advantage of prior methods' use of a binary classifier for constraint representations is that it restricts learned costs within a range of positive values [Malik et al., 2021]. In practice, it is advantageous for interpretability and transferability to restrict the constraint function to be strictly positive, so that the learned constraint can only discourage the agent from visiting certain states not visited by the expert. Bounding the rewards above can also be beneficial for preventing divergences during training. However, in the case of IRL for constraint inference, it is necessary to optimize over a convex cone and hence a binary activation function cannot be used.

We propose and evaluate two possible solutions. As a baseline solution, we consider simply clipping the output values of a linear activation function to within some fixed positive range, while also re-scaling rewards using the rolling mean and variance. Though clipping technically violates the convex cone assumption, in practice the reward scaling should restrict the range of the constraint function necessary to ensure constraint satisfaction.

This baseline solution is potentially problematic, however. Besides violating the convex cone assumption, hard clipping the output of a neural network can be challenging due to the potential for vanishing gradients. Alternatively, we propose softly enforcing the constraint function to the positive range by using a Leaky ReLU activation and adding an L2 regularization term to the imitation gap loss, similar to Malik et al. [2021], so that the loss becomes

where E s,a ∼ τ E [ c ( s, a ) 2 ] is an L2 regularization on the learned cost of the expert trajectories. Unlike in Malik et al. [2021], we only regularize the output of the constraint function on the expert data,

allowing the constraint function to grow unconstrained, except on states visited by the expert, where the cost should be zero by the assumption that the expert trajectories are safe, i.e. non-violating.

IRLwith separate critics One component of prior methods for constraint inference that is implicitly added when utilizing a Lagrangian method for policy optimization, is the use of separate critics for the reward and constraint functions. Separate critics may potentially be beneficial for learning, since separating the critics disentangles the value function learning of the reward (which is not changing during training) with the constraint function (which changes as the constraint is learnt). By Q-decomposition [Russell and Zimdars, 2003], additive rewards can also be evaluated by separate additive critics and hence it is a straightforward modification to implement separate critics in any actor-critic IRL implementation without the additional Lagrangian optimization, which we outline in Algorithm 1.

Algorithm 1 IRL for ICRL - Separate Critics

Last-layer policy resetting Recent work has demonstrated that the primacy bias caused by plasticity loss can be a significant challenge for training deep RL agents [Nikishin et al., 2022]. This challenge is exacerbated in the case of IRL and particularly ICRL since the learned cost function changes constantly during training while the true rewards are provided from the environment, potentially biasing early policy training towards learning the unconstrained policy. To combat this plasticity loss issue, we adopt the recommendations from Lyle et al. [2023] for MLPs, which advises periodically resetting the final layer of the network during training.

5 Experiments

In this section, we conduct a series of experiments across several environments in order to answer the following questions: (1) How does IRL perform on constraint inference tasks compared to Lagrangian methods? and (2) How do the proposed modifications or regularizations over vanilla IRL improve performance on constraint inference tasks?

Environments For our experiments, we consider the virtual environments for benchmarking inverse constraint learning, introduced by Liu et al. [2023a] since these were specially designed to test the performance of constraint inference tasks and also provide a recent baseline for Lagrangianbased constraint inference methods, including expert data. The environments include five MuJoCo environments, Ant , Half Cheetah , Walker 2D , Swimmer and Inverted Pendulum , modified to include constraints, which are primarily binary restrictions on the x-position of the agent. For more details, see Liu et al. [2023a].

Evaluation metrics Evaluating constrained RL is challenging due to the dual objectives of optimizing rewards and satisfying constraints, which are typically in conflict with one another. Hence, in our evaluation, we report two metrics, following Liu et al. [2023a], feasible rewards and violation rate . Feasible rewards are calculated as the total returns for an episode up to the point of the first constraint violation. The violation rate is the percentage of episodes with one or more constraint violations. Using these metrics allows us to compare the baselines along the same axis as prior work.

We compare all methods according to average performance in the last 50 testing episodes and report statistics (IQM, Median, Mean and Optimality Gap) with bootstrapped 95% confidence intervals computed across five seeds according to the method recommended in Agarwal et al. [2021]. We provide the full learning curves in Appendix C.2.

Prior methods for comparison We compare our results against two baselines from Liu et al. [2023a]. The first is their implementation of MaxEnt ICRL [Malik et al., 2021] (denoted MECL ), which is the canonical method for Lagrangian-based constraint inference that provides the primary baseline for our method. The second is their implementation of GAIL for constraint inference (denoted GACL ), introduced as a baseline for MECL in Malik et al. [2021]. This is a basic version of IRL for constraint inference, which uses the GAIL algorithm [Ho and Ermon, 2016] with an additive log term for the constraint. In Liu et al. [2023a], they test an additional two algorithms which, for conciseness, we do not include in this work, as one has been shown to underperform other methods in previous work and the other introduces Bayesian inference into the constraint learning, which is not considered here. We obtain results for MECL and GACL using the implementation and hyperparameters provided by the authors of Liu et al. [2023a]. We note that for our implementation of constraint inference, we use SAC as the forward RL algorithm, similar to Kim et al. [2023], whereas Malik et al. [2021] and Liu et al. [2023a] use PPO with an entropy regularization term. 2 .

Experimental Setup As mentioned, we use SAC for policy optimization. As the IRL algorithm, we utilize a version of maximum entropy IRL as implemented in Zeng et al. [2022]. The learnt constraint function is parameterized as a two-layer MLP with linear output activation. We adopt the hyperparameters for SAC as used in Achiam [2018], except that we use automatic α -tuning [Haarnoja et al., 2018]. All the hyperparameters for IRL were set to those used in Zeng et al. [2022], except the learning rate on the constraint function which we tune for each environment. We find that when training with separate critics, warm-starting the policy learning by training without the constraint critic initially is beneficial, so we warm-start configurations with separate critics with 500k environment steps. We train all variations in all environments for 5M environment steps across five seeds. Full hyperparameter configurations are included in Appendix A.1. 3

In the following sections, the modifications proposed in Section 4.2 are labeled as IRL-Base for IRL with clipping and no modifications, L2 for using L2 regularization in place of clipping, SC for adding separate critics and PR for policy reset. IRL-Plus includes all proposed modifications ( L2 , SC , PR ).

5.1 IRL versus ICRL

First, we compare our IRL implementation to the baseline methods. In Figure 1 we present a summary of our findings across environments, showing the final expert-normalized performance, aggregated across all five environments, with 95% bootstrapped confidence intervals computed across five seeds.

Notably, basic IRL ( IRL-Base ) already performs quite favorably versus MECL . Statistics on the feasible rewards are generally improved for IRL-Base versus MECL , though this improvement does not always have statistical significance. On the other hand, violation rate is improved over MECL with statistical significance across all but one statistic. Notably, the base IRL method performs similarly to the prior baseline IRL method GACL , with no statistical difference in feasible rewards across all four statistics, though it generally outperforms in terms of the violation rate.

With the added modifications, IRL-Plus outperforms MECL with statistical significance across all four metrics in feasible rewards and all but one metric in violation rate. IRL-Plus also outperforms the baseline IRL method GACL with statistical significance across almost all metrics for both feasible rewards and violation rate, with the median being the only exception.

5.1.1 Sub-optimal Expert Trajectories

Previous work has found that constraint inference with a Lagrangian inner loop is more robust to suboptimal expert trajectories as compared to previous IRL methods which cannot learn effectively [Liu et al., 2023a]. We conduct experiments on the suboptimal dataset for Half Cheetah provided in

0.00 0.25 0.50 0.75 1.00

GACL

MECL

IRL-Base (Ours)

IRL-Plus (Ours)

IQM

0.00 0.25 0.50 0.75 1.00

Median

0.00 0.25 0.50 0.75 1.00

Mean

0.00 0.25 0.50 0.75 1.00

Optimality Gap

(a) Feasible Rewards

GACL

MECL

IRL-Base (Ours)

IRL-Plus (Ours)

IQM

Median

Mean

Optimality Gap

0.00 0.25 0.50 0.75 1.00

0.00 0.25 0.50 0.75 1.00

(b) Violation Rate

Liu et al. [2023a] to test whether this remains true with our implementation. The sub-optimal datasets are constructed to contain varying percentages of unsafe expert trajectories, i.e. expert trajectories with one or more constraint violations. For these experiments, we test the combination of all proposed modifications ( IRL-Plus ), i.e. L2 regularization on the expert rewards, separate critics and policy resets versus the two baselines MECL and GACL .

0

2500

5000

7500

GACL

MECL

IRL-Plus (Ours)

Feasible Rewards

0.00

0.25

0.50

0.75

1.00

GACL

MECL

IRL-Plus (Ours)

Violation Rate

(a) 20%

0

2500

5000

7500

Feasible Rewards

0.00

0.25

0.50

0.75

1.00

Violation Rate

(b) 50%

0

2500

5000

7500

Feasible Rewards

0.00

0.25

0.50

0.75

1.00

Violation Rate

(c) 80%

As shown in Figure 2, in all cases, our method performs much better than the prior IRL baseline method GACL , which does not achieve any feasible rewards. Our method also generally outperforms the Lagrangian-based method MECL though with notably high variance, so that the improved performance is only statistically significant in the 20% scenario. Overall, however, it is clear that the ICRL method MECL does not outperform IRL-Plus on a statistically significant basis.

5.2 Impact of Modifications

Here, we more closely examine the impact of the proposed modifications (Section 4.2) on constraint inference with IRL. Overall, while the impact of the modifications varies somewhat across environments, as can be seen in Figure 4, aggregated across all environments on a normalized basis, the combination of all modifications, i.e. L2 regularization, separate critics and policy resets ( IRL-Plus ) results in an increase in feasible rewards vs IRL-Base which is statistically significant in all statistics

0.00 0.25 0.50 0.75 1.00

0.00 0.25 0.50 0.75 1.00

0.00 0.25 0.50 0.75 1.00

IRL (IRL-Base)

IRL+L2

IRL+L2+PR

IRL+L2+SC

IRL+L2+PR+SC

(IRL-Plus)

IQM

0.00 0.25 0.50 0.75 1.00

Median

0.00 0.25 0.50 0.75 1.00

Mean

0.00 0.25 0.50 0.75 1.00

Optimality Gap

(a) Feasible Rewards

IRL (IRL-Base)

IRL+L2

IRL+L2+PR

IRL+L2+SC

IRL+L2+PR+SC

(IRL-Plus)

IQM

Median

Mean

Optimality Gap

0.00 0.25 0.50 0.75 1.00

0.00 0.25 0.50 0.75 1.00

(b) Violation Rate

except the median, with only a slight increase in the violation rate which is not statistically significant in any of the statistics.

L2 regularization versus clipping with reward normalization Overall, the most substantial increase in performance comes from the L2 regularization versus clipping with reward normalization. This modification improves feasible rewards by a statistically significant amount as measured by three of the four metrics, excepting the median. At the same time, it results in a small increase in violation rate, however this increase is not statistically significant for any of the four statistics. The addition of L2 regularization has the largest impact on Walker2D as can be seen in Figure 4.

Policy reset and separate critics Adding both separate critics and policy reset ( SC+PR ) slightly improves performance over IRL+L2 , across most statistics, in both feasible rewards and violation rate, though the impact is marginal and only statistically significant in some metrics. Interestingly, neither the separate critics modification nor the policy reset modification alone results in an improvement over IRL+L2 , suggesting it is the interaction of these two modifications that are beneficial. For example, separate critics without policy resets ( IRL+L2+SC ) slightly hurts performance in terms of feasible rewards with similar violation rate, and policy resets without separate critics ( IRL+L2+PR ) slightly increases violation rate without much improvement in feasible rewards, though we note that these effects are generally not statistically significant. Notably, when combining separate critics with policy resets we reset the last layer of only the constraint critic and not the reward critic. This allows the reward critic to learn the unchanging reward function without resets, while the constraint critic can adapt more quickly through resets to the changing constraint function, which may explain the performance boost observed when combining these two methods.

While the impact of separate critics and policy resets is marginal overall, in certain environments such as Inverted Pendulum and Half Cheetah , it does confer a meaningful improvement in feasible rewards, as can be seen in Figure 4, though with notably high variance across seeds.

6 Discussion and Conclusions

Overall, we have demonstrated that Lagrangian-based methods for constraint inference are theoretically equivalent to IRL for a broad constraint class. Moreover, these Lagrangian methods do not appear to confer performance benefits over simpler IRL methods, when IRL is implemented using MaxEnt IRL with constraint functions parameterized by neural networks with appropriate activation

0.00 0.25 0.50 0.75 1.00

0.00 0.25 0.50 0.75 1.00

GACL

MECL

IRL-Base

IRL-L2

IRL-L2+PR

IRL-L2+SC

IRL-L2+SC+PR

3000

6000

9000

Feasible Rewards

0.00

0.25

0.50

0.75

1.00

Violation Rate

(a) Half Cheetah

5000

10000 15000

Feasible Rewards

0.00

0.25

0.50

0.75

1.00

Violation Rate

(b) Ant

10

20

30

Feasible Rewards

0.00

0.25

0.50

0.75

1.00

Violation Rate

(c) Inv. Pendulum

800

1600

2400

Feasible Rewards

0.00

0.25

0.50

0.75

1.00

Violation Rate

(d) Walker

200

400

600

Feasible Rewards

0.00

0.25

0.50

0.75

1.00

Violation Rate

(e) Swimmer

functions. Further, with the addition of some simple-to-implement practical modifications to the IRL algorithm we can further improve performance of IRL over previous Lagrangian-based ICRL methods for constraint inference tasks.

Advantages of IRL over ICRL Besides performance improvements, using IRL for constraint inference confers several benefits to the field. IRL reduces the complexity of the optimization problem versus ICRL by removing a level in the nested optimization, making a simpler to implement solution that can more easily be applied to new domains. For example, we highlight that we did not have to perform any hyperparameter tuning, except on the constraint function learning rate, to achieve decent results in all environments. In contrast, previous ICRL methods use many environment-specific hyperparameters [Liu et al., 2023a]. Additionally, the use of IRL methods to solve constraint inference problems could facilitate applying the much broader field of research in IRL and its sub-fields to various extensions of constraint inference that have not yet been fully explored. These could include, in particular, offline constraint inference using offline IRL techniques [Yue et al., 2023].

Importantly, we emphasize that the constraint function learned through our modified IRL procedure is equivalent to the constraint function learned through Lagrangian-based ICRL methods, up to a scalar multiplicative constant, and hence can be treated as a constraint function in downstream transfer tasks. More specifically, though the constraint function is learned as an additive term in the reward of an unconstrained MDP, the learned function can be transferred as a constraint in a constrained MDP. Solving this constrained MDP will require scaling the learned constraint by a new Lagrangian term when optimizing for the new transfer task. Hence, there is no loss in the transferability of the constraint when performing constraint inference with IRL versus ICRL.

Limitations and Future Work While we show the feasibility of IRL for constraint inference as an alternative to ICRL, we note that there is substantial variance in the solutions produced by IRL on a per environment basis, as can be observed in Figure 4. Reducing this variance should be a priority of future work. Moreover, ongoing work is still needed to improve the performance of constraint inference across more challenging environments such as Inverted Pendulum .

Additionally, while our results indicate that our IRL-Plus method outperforms previous ICRL method MECL , we note that there are several algorithmic differences between our implementation and that of Liu et al. [2023a], such as the use of SAC vs PPO as the RL policy optimization algorithm. While we believe we have effectively demonstrated certain implementations of IRL can be competitive with ICRL we cannot conclude that IRL is strictly better than ICRL in all cases.

As mentioned, an exciting extension for future work would be to explore various subdomains of IRL in the context of constraint inference, particularly the offline IRL case, and to apply these techniques to real datasets.

Acknowledgments and Disclosure of Funding

We want to acknowledge funding support from NSERC, FRQNT and CIFAR, as well as compute support from Digital Research Alliance of Canada, Mila IDT, and NVidia.

References

Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 1, 2004.

Achiam. Spinning Up in Deep Reinforcement Learning. 2018.

Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems , 34:29304-29320, 2021.

Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems , 30, 2017.

Castaneda, H. Nishimura, R. T. McAllister, K. Sreenath, and A. Gaidon. In-distribution barrier functions: Self-supervised policy filters that avoid out-of-distribution states. In Learning for Dynamics and Control Conference , pages 286-299. PMLR, 2023.

Chou, D. Berenson, and N. Ozay. Learning constraints from demonstrations. In Algorithmic Foundations of Robotics XIII: Proceedings of the 13th Workshop on the Algorithmic Foundations of Robotics 13 , pages 228-245. Springer, 2020.

Dadashi, A. A. Taiga, N. Le Roux, D. Schuurmans, and M. G. Bellemare. The value function polytope in reinforcement learning. In International Conference on Machine Learning , pages 1486-1495. PMLR, 2019.

Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning , 110(9):2419-2468, 2021.

Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems , 34:4028-4039, 2021.

Gleave and S. Toyer. A primer on maximum causal entropy inverse reinforcement learning. arXiv preprint arXiv:2203.11409 , 2022.

Ha, P. Xu, Z. Tan, S. Levine, and J. Tan. Learning to walk in the real world with minimal human effort. arXiv preprint arXiv:2002.08550 , 2020.

Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016.

Kim, G. Swamy, Z. Liu, D. Zhao, S. Choudhury, and S. Z. Wu. Learning shared safety constraints from multi-task demonstrations. Advances in Neural Information Processing Systems , 36, 2023.

Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 2529, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=68n2s9ZJWF8 .

Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020.

Lindemann, A. Robey, L. Jiang, S. Das, S. Tu, and N. Matni. Learning robust output control barrier functions from safe expert demonstrations. IEEE Open Journal of Control Systems , 2024.

Liu, Y. Luo, A. Gaurav, K. Rezaee, and P. Poupart. Benchmarking constraint inference in inverse reinforcement learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023a. URL https://openreview.net/pdf?id=vINj_Hv9szL .

Liu, Z. Guo, H. Lin, Y. Yao, J. Zhu, Z. Cen, H. Hu, W. Yu, T. Zhang, J. Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303 , 2023b.

Lyle, Z. Zheng, E. Nikishin, B. A. Pires, R. Pascanu, and W. Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning , pages 23190-23211. PMLR, 2023.

Malik, U. Anwar, A. Aghasi, and A. Ahmed. Inverse constrained reinforcement learning. In International conference on machine learning , pages 7390-7399. PMLR, 2021.

Marot, A. Kelly, M. Naglic, V. Barbesant, J. Cremer, A. Stefanov, and J. Viebahn. Perspectives on future power system control centers for energy transition. Journal of Modern Power Systems and Clean Energy , 10(2):328-344, 2022. doi: 10.35833/MPCE.2021.000673.

L. McPherson, K. C. Stocking, and S. S. Sastry. Maximum likelihood constraint inference from stochastic demonstrations. In 2021 IEEE Conference on Control Technology and Applications (CCTA) , pages 1208-1213. IEEE, 2021.

Menner, P. Worsnop, and M. N. Zeilinger. Constrained inverse optimal control with application to a human manipulation task. IEEE Transactions on Control Systems Technology , 29(2):826-834, 2019.

Miryoosefi, K. Brantley, H. Daume III, M. Dudik, and R. E. Schapire. Reinforcement learning with convex constraints. Advances in Neural Information Processing Systems , 32, 2019.

Y. Ng, S. Russell, et al. Algorithms for inverse reinforcement learning. In Icml , volume 1, page 2, 2000.

Y. Ng, A. Coates, M. Diel, V. Ganapathi, J. Schulte, B. Tse, E. Berger, and E. Liang. Autonomous inverted helicopter flight via reinforcement learning. In Experimental robotics IX: The 9th international symposium on experimental robotics , pages 363-372. Springer, 2006.

Nikishin, M. Schwarzer, P. D'Oro, P.-L. Bacon, and A. Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning , pages 16828-16847. PMLR, 2022.

Pirotta, A. Tirinzoni, A. Touati, A. Lazaric, and Y . Ollivier. Fast imitation via behavior foundation models. In The Twelfth International Conference on Learning Representations , 2023.

A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems , 1, 1988.

Robey, H. Hu, L. Lindemann, H. Zhang, D. V. Dimarogonas, S. Tu, and N. Matni. Learning control barrier functions from expert demonstrations. In 2020 59th IEEE Conference on Decision and Control (CDC) , pages 3717-3724. IEEE, 2020.

Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

J. Russell and A. Zimdars. Q-decomposition for reinforcement learning agents. In Proceedings of the 20th international conference on machine learning (ICML-03) , pages 656-663, 2003.

Schaal. Learning from demonstration. Advances in neural information processing systems , 9, 1996.

R. R. Scobee and S. S. Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum? id=BJliakStvH .

Swamy, S. Choudhury, J. A. Bagnell, and S. Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning , pages 10022-10032. PMLR, 2021.

Swamy, D. Wu, S. Choudhury, D. Bagnell, and S. Wu. Inverse reinforcement learning without reinforcement learning. In International Conference on Machine Learning , pages 33299-33318. PMLR, 2023.

Weng, H. Chen, D. Yan, K. You, A. Duburcq, M. Zhang, Y. Su, H. Su, and J. Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research , 23(267):1-6, 2022. URL http://jmlr.org/papers/v23/21-1127.html .

Yue, G. Wang, W. Shao, Z. Zhang, S. Lin, J. Ren, and J. Zhang. Clare: Conservative model-based reward learning for offline inverse reinforcement learning. In International Conference on Learning Representations (ICLR) , 2023.

Zeng, C. Li, A. Garcia, and M. Hong. Maximum-likelihood inverse reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems , 35:10122-10135, 2022.

D. Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy . Carnegie Mellon University, 2010.

D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

A Experiment Details

A.1 Hyperparameters

All of our code is based on the Tianshou [Weng et al., 2022] and FSRL [Liu et al., 2023b] implementations of SAC and SAC-Lagrangian, respectively. Here we include all the hyperparameter configurations for our experiments. Any hyperparameters not listed here use the default hyperparameters in their respective libraries (Tianshou version 1.0.0 and FSRL version 0.1.0).

SAC Parameters

τ

γ

α

LR

α

optimizer

Critic hidden layers

Critic optimizer

Critic learning rate

Actor hidden layers

Actor learning rate

Actor optimizer n-step returns

Training Parameters

Max epochs

Batch size

Environment steps per epoch

Gradient steps per environment step

Test episodes per epoch

Episodes between gradient updates

Buffer size

1000

128

5000

1

5

1

1000000

The constraint function learning rate was tuned over two values (0.001, 0.0001) per environment with the following final parameters:

A.2 Computational Resources

Each run (consisting of five seeds) was trained on a node with a single GPU (varying GPU resources were used), 6 CPUs and 6GB of RAM per CPU. A single run across three seeds under this configuration took approximately 1.5 days to complete training.

B Proof of Theorem 4.1

We prove both claims individually.

0.005

0.99

0.0003

Adam

[256, 256]

Adam

0.001

[256, 256]

0.001

Adam

1

We begin by proving the first claim, that Opt ICRL (Π , F r , F c ) = Opt S -ICRL (Π , F r , F c ) when F c is a convex cone. Firstly, it is simple to show that Opt ICRL (Π , F r , F c ) ≥ Opt S -ICRL (Π , F r , F c ) ,

It remains to show that Opt ICRL (Π , F r , F c ) ≤ Opt S -ICRL (Π , F r , F c ) . We have

where the first inequality holds since λc ∈ F c whenever c ∈ F c by the hypothesis that F c is a convex cone. It follows that Opt ICRL (Π , F r , F c ) = Opt S -ICRL (Π , F r , F c ) . ■

B.2 Step 2: IRL Solves ICRL

We now prove the second claim. For any policy π , we define the successor features [Ng et al., 2000, Ziebart et al., 2008, Barreto et al., 2017] ψ π : S → R d according to

where A t ∼ π ( · | S t ) . Then, for any initial state distribution µ 0 and r ∈ F , we have J ( π, r ) = E s 0 ∼ µ [ ⟨ ψ π ( s 0 ) , w r ⟩ ] , where r ( x, a ) = ⟨ ϕ ( x, a ) , w r ⟩ -this holds by the linearity of expectation.

Now, we define Ψ = { E s 0 ∼ µ ψ π ( s 0 ) : π ∈ Π } ⊂ R d . Note that s ↦→ ψ π i ( s ) is equivalent to the value function under policy π for the reward function ϕ i , for any i ∈ [ d ] . Thus, since Π is assumed to be compact and ϕ is bounded, Dadashi et al. [2019] shows that the set V i = { ψ π i : π ∈ Π } is convex and compact for each i ∈ [ d ] . By continuity, V i = { E s 0 ∈ µ ψ π i ( s 0 ) : π ∈ Π } is convex compact. Therefore, it holds that as a product of convex and compact sets, Ψ = ⊗ d i =1 V i is convex and compact.

Next, suppose ( π ⋆ , c ⋆ -r ) realizes Opt IRL (Π , F r ) . Since Ψ is compact and convex and F is convex, we apply Sion's minimax theorem and observe that, for ψ E := E s 0 ∼ µ 0 ψ π E ( s 0 ) ,

Sion's minimax theorem

where the penultimate step holds because F , as a vector space, is a convex cone. Thus, we conclude that solving the IRL problem over the reward class F r and policy class Π solves the inverse constrained RL problem of interest. ■

C Additional Results

C.1 ICRL SAC Implementation Results

We attempted to reproduce the results of Liu et al. [2023a] using SAC for policy learning. We tested six settings for the Lagrangian and reward learning rates shown in Figure 5. All other hyperparameters were kept the same as Appendix A.1, except we ran each configuration for 3M timesteps. Notably, we had considerable difficulty reproducing the results of Liu et al. [2023a] using the SAC algorithm. While Kim et al. [2023] provided a SAC implementation of a similar algorithm, we note that their implementation added various additional components to the algorithm to improve convergence, including in particular, interleaving reward learning with behavior cloning. We did not use these techniques here.

C.2 Learning Curves

Here we provide the full learning curves for all ablations and all environments. An additional configuration, IRL-Base+BN is included which was tested but not included in the main results due to poor performance. This configuration added batch normalization to the output of the reward function to potentially help combat issues with vanishing gradients.

reward.lr=0.0001, lambda.lr=0.001

reward.lr=0.0001, lambda.lr=0.01

reward.lr=0.001, lambda.lr=0.001

reward.lr=0.001, lambda.lr=0.01

reward.lr=0.0001, lambda.lr=0.0001

reward.lr=0.001, lambda.lr=0.0001

0.5

1.0

1.5

2.0

2.5

3.0

1e6

0

2000

4000

6000

8000

10000

Feasible Reward

0.5

1.0

1.5

2.0

2.5

3.0

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Violation Rate

(a) Half Cheetah

0.5

1.0

1.5

2.0

2.5

3.0

1e6

0

5000

10000

15000

Feasible Reward

0.5

1.0

1.5

2.0

2.5

3.0

1e6

0.0

0.2

0.4

0.6

0.8

Violation Rate

(b) Ant

0.5

1.0

1.5

2.0

2.5

3.0

1e6

0

2

4

6

8

10

Feasible Reward

0.5

1.0

1.5

2.0

2.5

3.0

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Violation Rate

(c) Inverted Pendulum

0.5

1.0

1.5

2.0

2.5

3.0

1e6

200

400

600

800

1000

1200

Feasible Reward

0.5

1.0

1.5

2.0

2.5

3.0

1e6

0.0

0.1

0.2

0.3

0.4

Violation Rate

0.5

1.0

1.5

2.0

2.5

3.0

1e6

0

50

100

150

200

250

Feasible Reward

0.5

1.0

1.5

2.0

2.5

3.0

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Violation Rate

(e) Swimmer

IRL-L2+SC+PR

IRL-L2+SC

IRL-L2+PR

IRL-L2

IRL-Base

IRL-Base+BN

MECL

GACL

1

2

3

4

5

1e6

0

2000

4000

6000

8000

10000

Feasible Rewards

1

2

3

4

5

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Violation Rate

(a) Half Cheetah

1

2

3

4

5

1e6

0

5000

10000

15000

20000

Feasible Rewards

1

2

3

4

5

1e6

0.0

0.2

0.4

0.6

0.8

Violation Rate

(b) Ant

1

2

3

4

5

1e6

0

20

40

60

80

Feasible Rewards

1

2

3

4

5

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Violation Rate

(c) Pendulum

1

2

3

4

5

1e6

0

500

1000

1500

2000

Feasible Rewards

1

2

3

4

5

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Violation Rate

1

2

3

4

5

1e6

0

100

200

300

400

500

600

700

Feasible Rewards

1

2

3

4

5

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Violation Rate

(e) Swimmer

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim that simple IRL methods can perform as well or better than ICRL methods across environments. This is shown in our experiments comparing MECL to IRL (see Figure 1).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include a limitations section (see Section 6).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

We provide a complete and correct proof of our theoretical result in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provided full hyperparameter details for our experiments in Appendix A.1. Besides this, we reference exact implementations for each component of our algorithm (i.e. IRL from Zeng et al. [2022]) and we utilize standard and well-maintained libraries [Weng et al., 2022, Liu et al., 2023b] as the basis for our code. The datasets used are available from Liu et al. [2023a]. Finally, we provide a public Github for the code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We do not include code directly in the supplemental materials, however a link to the publicly available Github repository is provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provided the most relevant configuration details in Section 5 and full hyperparameter details for our experiments in Appendix A.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We include error bars in all figures (see Figures 1 to 4), computed across five seeds (and aggregated across five environments in the case of Figures 1 and 3) using bootstrapped 95% confidence intervals as recommended in Agarwal et al. [2021].

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide details of the computational resources used in Appendix A.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In Section 1 we discuss possible real-world applications for constraint inference such as in power grid control, that could confer societal benefits. We do not see any negative societal impacts that would come from our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release any data or models with high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The only assets we use in our experiments are the data provided by Liu et al. [2023a] which we cite extensively, including particularly in Section 5 to note that we are using these datasets in our experiments. We also cite the relevant code packages used to implement our experiments in Appendix A.1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not use crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not use human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a way to reduce the tri-level stucture of ICRL to bi-level, and uses solid experiment results to validate that this bi-level reformulation achieves better expirical results. The authors also intuitively explain that this is due to the fact that the tri-level optimization has a more complicated optimization landscape. In general, this paper clearly delivers its idea, and is easy to follow. However, the major weakness is that this simplification of tri-level to bi-level is trivial, and can be further exploited.\n\nSTRENGTHS:\nThis paper provides solid empirical results to evaluate its idea. This paper answers a fundamental question for ICRL, i.e., there is no fundamental difference between IRL and ICRL. In other words, we can use IRL algorithms to solve ICRL problems.\n\nWEAKNESSES:\nThe main weakness lies in the core contribution, i.e., the simplification of tri-level to bi-level formulation. This simplication comes from the observation that we can always learn a cost function $c'$ that captures both the dual variable $\\lambda$ and the original cost function $c$, i.e., $c'=\\lambda c$. Therefore, we can learn a single cost function $c'$ to replace $\\lambda c$.\n\nThis idea is correct, and I agree that learning a single cost function $c$ is enough. However, this idea is trivial and can be futher exploited. In fact, I have seen similar trick in an ICRL literature [1]. In reference [1], the authors also remove the dual variable $\\lambda$ and only learn a single cost function $c$. Indeed, the reference [1] does not highlight this modification as their novelty, so that it is totally fine for this paper to highlight this trick as a core contribution.\n\nHowever, this \"c'=\\lambda c\" idea is more like an engineering trick, and more contribution is needed to support this trick. For example, as mentioned in the paper, compared to the tri-level formulation, the bi-level reduction has a simpler optimization landscape and thus is expected to have better optimization result. It will be great if the authors can theoretically support this claim, i.e., theoretically prove that this bi-level formulation is easier to achieve better result than the tri-level formulation.\n\n[1] Liu \\& Zhu, \"Meta inverse constrained reinforcement learning: convergence guarantee and generalization analysis\".\n\nQUESTIONS:\nPlease see weakness.\n\nLIMITATIONS:\nThe limitation is discussed in the paper and is reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel inverse constraint learning approach that leverages inverse reinforcement learning (IRL). Prior work by Kim et al. and Malik et al. proposed a game-theoretic approach to the constraint learning problem, where the resulting optimization problem is a tri-level optimization problem involving the policy, the constraint function, and the Lagrange multiplier. The algorithm is an iterated optimization that alternates between constrained RL and constraint function learning. In contrast, the present work leverages the equivalence between the original problem and a simpler IRL problem, which was overlooked in prior work. This reduction removes the Lagrange multiplier from the decision variables and enables the use of an existing IRL algorithm for the purpose of simultaneous constraint learning and constrained policy optimization. Simulation results in the Mujoco environment suggest the feasibility of the IRL approach.\n\nSTRENGTHS:\n* The present work provides a key insight that the inverse constraint learning problem of Kim et al. is equivalent to inverse reinforcement learning, which was overlooked in the prior work. Although the equivalence requires an additional assumption that the set of constraint functions must be closed under multiplication by positive scalars, we can apply game-theoretic IRL algorithms to solve the constraint learning problem as long as we conform to the constraint function class.\n\nWEAKNESSES:\nAlthough the present work provides an interesting insight into constraint learning problems in RL, the paper in its current form possesses multiple issues.\n\n### Major Issues\nThe most concerning issue is the inconclusive and inconsistent nature of the analysis of experimental results, as detailed below.\n\n* The feasible rewards and the violation rate are two competing evaluation metrics employed in the paper, which are equally important considering the natural trade-off between performance and safety. Nevertheless, Figure 1 only focuses primarily on  the feasible rewards to compare different methods; most notably, it reports information on the violation rate only when it is worse than the proposed vanilla IRL method. Such a partial reporting can easily induce bias in the interpretation of the results, since it effectively ignores circumstances where the violation rate is improved for baselines. The authors should fully report both the feasible rewards and the violation rate, and investigate whether any method pareto-dominates the others in the two evaluation criteria. No definitive conclusion can be drawn without this analysis.\n\n* No detailed analysis of Figure 2 is presented in Section 5.1. In fact, the authors did not reference Figure 2 in the main body of Section 5, and the caption does not provide any analysis either. By looking at Figure 2, it is not clear whether any conclusion can be drawn about the relative performance between the proposed algorithms and the two baselines (i.e. MECL and GACL).\n\n* Speaking of the performance of MECL and GACL, It is not clear what “a best estimate” means in the footnote on page 6. In the first place, the authors should either reproduce the prior work with PPO or report the missing results as they are, with preference given to the former; they should never estimate missing values. If reproduction is impossible, I would contact the authors and ask for raw data.\n\n* Figure 3 is incomplete. Specifically, the left plot on Feasible Reward is missing GACL 80% and GACL 50%. Similarly, the right plot on Violation Rate is missing MECL 50%.\n\n* The paper examines whether various techniques for stabilizing constraint learning actually help the IRL approach, and provides some analysis in Figure 4 and Section 5.2. Unfortunately, its credibility is highly questionable given the small sample size (with only 3 seeds) and the fact that the error bars are largely overlapping in Figure 4. Rather than reporting standard deviation, the authors should report more appropriate uncertainty information, such as the standard error of the mean or a confidence interval. If time permits, please also conduct the experiment with more seeds, which may help separate the error bars and aid for more comprehensible results. (Note that the standard error reduces as the sample size is increased, as opposed to the standard deviation.)\n\n* The conclusion is partly self-conflicting. Specifically, in Section 5.2 the authors mention batch normalization (BN) and reward normalization (RN). They state that “we find these modifications are generally more harmful than beneficial. Including batch normalization, reward normalization or both, … tended to hurt performance over basic IRL in a majority of environments.” On the contrary, in Section 6 the authors write “certain combinations of additional simple regularization such as batch normalization and reward normalization can produce significantly better results in several environments.”\n\n\n### Minor Issues\n#### Problem Formulation\n* In equation (6), the Lagrange dual variable $\\lambda$ should always be the outer optimization variable. I believe that it should be max min, not min max (unless the minimax theorem holds.)\n\n* Equation (8) and its subsequent analysis are central to the development of this paper, and hence may be worth stated as a proposition or a theorem.\n\n#### Related Work\n* The authors seem to consider general Imitation Learning (IL) and IRL as two separate problems. A common view is that there are several categories in IL, of which behavioral cloning (BC) and IRL are among the most popular [1][2].\n\n* There is one reference that is missing on Page 3, line 98.\n\n* The authors state in Section 1 that “we would like to extract safety constraints from the data based on the expert behavior, which can then be used downstream to constrain task-specific learning.” Besides the RL approaches, there is a thread of prior work in learning-based control that considers this problem through the use of barrier functions and fully decoupling downstream task-specific learning from constraint learning (e.g. [3][4][5]), which is missing from the literature review.\n\n[1] Zare, Maryam, Parham M. Kebria, Abbas Khosravi, and Saeid Nahavandi. \"A survey of imitation learning: Algorithms, recent developments, and challenges.\" arXiv preprint arXiv:2309.02473 (2023).\n\n[2] Osa, Takayuki, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. \"An algorithmic perspective on imitation learning.\" Foundations and Trends® in Robotics 7, no. 1-2 (2018): 1-179.\n\n[3] Robey, Alexander, Haimin Hu, Lars Lindemann, Hanwen Zhang, Dimos V. Dimarogonas, Stephen Tu, and Nikolai Matni. \"Learning control barrier functions from expert demonstrations.\" In 2020 59th IEEE Conference on Decision and Control (CDC), pp. 3717-3724. IEEE, 2020.\n\n[4] Lindemann, Lars, Alexander Robey, Lejun Jiang, Satyajeet Das, Stephen Tu, and Nikolai Matni. \"Learning robust output control barrier functions from safe expert demonstrations.\" IEEE Open Journal of Control Systems (2024).\n\n[5] Castaneda, Fernando, Haruki Nishimura, Rowan Thomas McAllister, Koushil Sreenath, and Adrien Gaidon. \"In-distribution barrier functions: Self-supervised policy filters that avoid out-of-distribution states.\" In Learning for Dynamics and Control Conference, pp. 286-299. PMLR, 2023.\n\nQUESTIONS:\n* In equation (4), do we need the outer optimization in $\\lambda$ to be over non-negative real values, not the entire real line (because of the inequality constraint)?\n\n* In the IRL formulation, the class of constraint functions $\\mathcal{F}_c$ is more restrictive than the original formulation of Kim et al. (which is convex and compact). Can the authors comment on possible negative implications of this restriction?\n\n* I did not fully follow the reward scaling part in Section 4.2. I wonder if the authors can elaborate on what they meant by “the constraint function is learned independently of the reward scale and hence may be more robust to different reward scales.”\n\nLIMITATIONS:\n* The limitations are properly discussed in Section 6, but the inconclusive nature of the analysis can be possibly alleviated in the revision, as discussed above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explains how inverse constrained RL (ICRL) - the task of recovering (safety) constraints from expert behaviour respecting those constraints - is equivalent to (straight) inverse RL (IRL) - the task of recovering a reward function from expert behaviour approximately optimizing that reward. This allows the ICRL problem to be solved using the wide array of techniques available in IRL. The authors then go on to empirically show superior performance of using a basic IRL algorithm on the IRL formulation relative to specialized ICRL algorithms.\n\nSTRENGTHS:\n- The paper is rising an important point: ICRL has been arising as a subarea of ML in the past few years with its own methods. Showing that it's equivalent to an existing, more developed area of research (IRL), is certainly useful since it (1) unlocks the use of IRL methods in ICRL (2) puts into question the need for ICRL to exist as a separate subarea and certainly puts more burden on ICRL researchers to show that their methods indeed solve the problem better than vanilla IRL and why. The results from this paper suggest that currently that's not the case.\n- The paper is clear and easy to understand.\n- The paper is a prime illustration of the maxim that a machine learning conference paper should ideally make a single crisp point.\n- Experiments support the claims made in the paper.\n\nWEAKNESSES:\n- In some places, the paper is a bit too wordy and the same thing could be said as clearly with a shorter sentence (an LLM will surely be happy to provide suggestions, of which at least some may be good).\n- the statistical methodology is somewhat underwhelming in Section 5, but not a deal-breaker for me. Fig 1 could give confidence intervals on the mean. Similarly, it'd be useful to have a confidence interval on the mean in the other two figures and ideally across more seeds than 3 (since supposedly the hypothesis we're testing in is whether the mean is better than the baselines). Especially results in 5.2 are somewhat messy. Maybe estimating something like Shapley values may be helpful here? Also, normalizing the results and then aggregating across the environments could help paint a clearer picture (you could work with a normalized effect size relative to the IRL baseline as the main quantity). I definitely find mostly the final performance interesting, so I wouldn't be against moving the training curves into the appendix if you want to free up space.\n\nQUESTIONS:\nSuggestions for improvement:\n- There is often missing punctuation after equations (e.g. full stop at the end of line 150 or commas after equations 1 and 6). In general, equations are part of the sentence structure and should be punctuated as such (and don't necessarily need to be preceded by a colon).\n- Weakly held opinion: equations 5 and 6 would feel more natural to me in negated form, i.e. finding a policy that maximizes the return subject to reward that minimizes the return of the policy relative to the expert. (yes, I'm taking into consideration also that this then leads to equations 7,8). Like this I find them a bit confusing at first without explanation.\n- The colour-coding of Figure 1 seems a bit unfortunate and doesn't help me much in reading the figure - could e.g. positive values be shades of green and negative ones shades of red with white at zero (not fussy about particular colors, but something that creates more contrast)? Please add x axis labels to Figures 2,3,4. Not all of the horizontal lines in Fig 3 are visible (maybe add a slight offset to prevent perfect overlap?). I'd also prefer seeing confidence interval on the mean across more than 3 seeds, rather than standard deviation. \n\n\nMinor typos:\n- l.3 \"However\" -> \"however\"\n- l. 98 has a missing reference\n- l. 126 has double space\n\nLIMITATIONS:\nI think the paper has a clear scope corresponding to the scope of previous work and doesn't introduce significant limitations beyond that.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for learning the constraints from demonstrations. To achieve this goal they take note of the similarities between inverse reinforcement learning (IRL) and inverse constraint reinforcement learning (ICRL). The authors aim to reduce the tri-level optimization of constrained inverse reinforcement learning to a bi-level optimization using a special class of constraints.\n\nSTRENGTHS:\n* The main claim of the paper with regard to reducing the ICRL to IRL under certain classes of constraint functions is original and can be quite impactful.\n\nWEAKNESSES:\n* The authors' claim that the tri-level optimization can be reduced to a bi-level has a mathematical motivation but it not supported by a mathematical derivation. The authors aim to support the claim via experimental results but the results have high variance and are thus not fully convincing.\n* The paper's main claims are tested using methods developed by prior work (MECL, GACL). The specifics of the modifications done by the authors are somewhat unclear however. The authors mention using SAC instead of PPO for the IRL implementation of the prior method which they compare with. Later it is mentioned that their results are directly taken from the prior works due to lower performance of their SAC-based implementation. \n* The paper combines various modifications to prior work (batch and reward normalization, separate critics, policy reset) in order to examine their benefits and support the main claims. However due to large variance in results in limited locomotion environments, it is difficult to accept the results with high degree of certainty.\n\nQUESTIONS:\n* The clarity of the paper can be significantly improved by adding diagrams or pseudocode describing their specific modifications to the previous IRL methods.\n* Question mark instead of a citation on line 98 (Related Works -> Inverse Reinforcement learning).\n* Eq. (1) describes the IRL objective. In its current form it aims to find a reward function that minimizes the difference between the return of the expert and the current policy while maximizing this difference. The objective should be the opposite where the policy aims to reduce the difference between the return on the expert and the current policy and the reward should aim to maximize the difference between the return of the expert and the current policy.\n* Given that this paper focuses on learning policies that violate fewer constraints and given that the claims are supported mainly by experimental results, the experiments section should focus on a more diverse set of environments that require safety constraints or at least describe in more details the constraints present in the Mujoco environments used for the experiments.\n* If the tri-level optimization can be reduced to a bi-level IRL problem, can it then be further reduced to a single level optimization using methods such as Inverse Q-Learning [1] or IQ-Learn [2]? How does the constraint violation rate of the proposed method compare to such  algorithms?\n\n[1] Kalweit, G., Huegle, M., Werling, M., & Boedecker, J. (2020). Deep Inverse Q-learning with Constraints. Advances in Neural Information Processing Systems, 33, 14291-14302.\n[2] Garg, D., Chakraborty, S., Cundy, C., Song, J., & Ermon, S. (2021). IQ-Learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34, 4028-4039.\n\nLIMITATIONS:\n* High variance in the experimental results (acknowledged by the authors).\n* The number of constrained environments and the discussion on how these constraints are present in each environment is lacking.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}