Learning Shared Safety Constraints from Multi-task Demonstrations
Authors: Konwoo Kim
Gokul Swamy
Zuxin Liu
Ding Zhao
Sanjiban Choudhury
Steven Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: constraints
inverse reinforcement learning
safe reinforcement learning
TL;DR: We learn safety constraints from multi-task demonstrations.

Abstract: Regardless of the particular task we want to perform in an environment, there are often shared safety constraints we want our agents to respect. For example, regardless of whether it is making a sandwich or clearing the table, a kitchen robot should not break a plate. Manually specifying such a constraint can be both time-consuming and error-prone. We show how to learn constraints from expert demonstrations of safe task completion by extending inverse reinforcement learning (IRL) techniques to the space of constraints. Intuitively, we learn constraints that forbid highly rewarding behavior that the expert could have taken but chose not to. Unfortunately, the constraint learning problem is rather ill-posed and typically leads to overly conservative constraints that forbid all behavior that the expert did not take. We counter this by leveraging diverse demonstrations that naturally occur in multi-task setting to learn a tighter set of constraints. We validate our method with simulation experiments on high-dimensional continuous control tasks.

Learning Shared Safety Constraints from Multi-task Demonstrations
Konwoo Kim Carnegie Mellon University
Gokul Swamy" Carnegie Mellon University
Zuxin Liu Carnegie Mellon University
Ding Zhao Carnegie Mellon University
Sanjiban Choudhury
Zhiwei Steven Wu Carnegie Mellon University
Cornell University
Abstract
Regardless of the particular task we want them to perform in an environment, there are often shared safety constraints we want agents to respect. For example, regardless of whether it is making a sandwich or clearing the table, a kitchen robot should not break plate. Manually specifying such a constraint can be both lmneccosumingg and error-prone. We show how to learn constraints from expert demonstrations of safe task completion by extending inverse reinforcement learning (IRL) techniques to the space of constraints. Intuitively, we learn constraints that forbid highly rewarding behavior that the expert could have taken but chose not to. Unfortunately, the constraint learning problem is rather ill-posed and typically leads overly conservative constraints that forbid all behavior that the expert did not take. We counter this by leveraging diverse demonstrations that naturally occur in multi-task settings learn tighter set of constraints. We validate our method with simulation experiments on high-dimensional continuous control tasks.
Introduction
If a friend was in your kitchen and you told them to "make toast" or "clean the dishes," you would probably be rather surprised if they broke some of your plates during this process. The underlying safety constraint that forbids these kinds of behavior is both a) implicit and b) agnostic to the particular task they were asked to perform. Now, let's bring household robot into the equation, operating within your kitchen. How can we ensure that it adheres to these implicit safety constraints, regardless of its assigned tasks?
One approach might be write down specific constraints (e.g- joint torque limits) and pass them to the maaiiiinnmaking system of the robot. Unfortunately, more complex constraints like the ones consider above are both difficult to formalize mathematically and easy for an end-user to forget to specify (as they would be inherently understood by human helper). This problem is paralleled in the field of reinforcement learning (RL), where defining reward functions that lead to desirable behaviors for the learning agent is a recurring challenge [Hadfield-Menl challenging to handeraft the exact function one should be optimized to good driver. The standard solution to this sort of "reward design" problem to instead demonstrate the desired behavior of the agent and then extract a reward function that would incentivize such behavior. Such inverse reinforcement learning (IRL) techniques have found application in fields diverse robotics [Silver et al., 2010, Ratliff et al., 2009, Kolter et al., 2008, Ng et al., 2006, Zucker et al., 201 1], computer vision [Kitani al., 2012], and human-computer interaction [Ziebart al., 2008b, 2012]. Given the success of IRL techniques and the similarity between reward and constraint design, we propose
al., 2017]. For example, it is rather
Equal contribution. Correspondence gswamyRcmu. edu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

extending IRL techniques to the space of constraints. We term such techniques inverse constraint learning, or ICL for short.
More formally, we consider a setting in which we have access to demonstrations of an expert policy for a task, along with knowledge about the task's reward. This allows to look at the difference between the expert and reward-optimal policies for a task. Our first key insight is that the actions taken by the reward-optimal but not the expert policy are likely to be forbidden, allowing us to extract a constraint.
Unfortunately, the ICL problem is still rather ill-posed. Indeed, prior work in ICL will often learn overly conservative constraints that forbid all behavior the expert did not take [Scobee and Sastry, 2019,
et al., 2018, McPherson et al., 2021]. However, for tasks in shared environment with different rewards, there are often safety constraints that should be satisfied regardless of the task (e.g. a plate shouldn't be broken regardless of whether you're serving food on or cleaning up after a meal). Our second crucial insight is that we can leverage multi-task data provide more comprehensive demonstration coverage over the state space, helping our method avoid degenerate solutions.
More explicitly, the contributions of our work are three-fold.
1. We formalize the inverse constraint learning problem. We frame ICL as zero-sum game between a policy player and constraint player. The policy player attempts to maximize reward while satisfying a potential constraint, while the constraint player picks constraints that maximally penalize the learner relative to the expert. Intuitively, such a procedure recovers constraints that forbid high-reward behavior the expert did not take.
2. We develop a multi-task extension of inverse constraint learning. We derive a zero-sum game between set of policy players, each attempting to maximize task-specific reward, and constraint player that chooses a constraint that all policy players must satisfy. Because the constraint player looks aggregate learner and expert data, is less likely to select a degenerate solution.
3. We demonstrate the efficacy of our approach on various continuous control tasks. We show that with restricted function classes, are able to recover ground-truth constraints certain tasks. Even when using less interpretable function classes like deep networks, we can still ensure a match with expert safety and task performance. In the multi-task setting, we are able to identify constraints that a single-task learner would struggle to learn.
We begin with discussion of related work.
2 Related Work
Our work exists at the confluence of various research thrusts. We discuss each independently.
Inverse Reinforcement Learning. IRL [Zicbart et al., 2008a,b, 2012, Ho and Ermon, 2016] can be framed as two-player zero-sum game between policy player and a reward player [Swamy et al., 2021]. In most formulations of IRL,, a potential reward function is chosen in an outer loop, and the policy player optimizes it via RL in inner loop. Similar IRL, the constraint in formulation of ICL is chosen adversarially in an outer loop. However, in contrast to IRL, the inner loop of ICL is constrained reinforcement learning: the policy player tries to find the optimal policy that respects the constraint chosen in the outer loop.
Constrained Reinforcement Learning. Our approach involves repeated calls to constrained reinforcement learning (CRL) oracle [Garcıa and Fernández, 2015, Gu et al., 2022]. CRL aims to find reward-maan maximizing policy over a constrained set, often formulated optimization problem| [Altman, 1999, Xu et al., 2022]. Solving this problem via Frank-Woll methods is often unstable [Ray et al., 2019, Liang et al., 2018J. Various methods have been proposed to mitigate this instability, including variational techniques [Liu et al., 2022], imposing trust-region regularization [Achiam et al., 2017, Yang et al., 2020, Kim and Oh, 2022], optimistic game-solving algorithms [Moskovitz et al., 2023], and PID controller- based methods [Stooke et al., 2020]. In our practical implementations, use PiD-based methods for their relative simplicity.
constrained policy
Multi-task Inverse Reinforcement Learning. Prior work in IRL. has considered incorporating multi-task data [Xu et al., 2019, Yu et al., 2019, Gleave and Habryka, 2018]. We instead consider
2

---

setting in which we know task-specific rewards and are attempting to recover a shared component of the demonstrator's objective. Amin al. [2017] consider a similar setting but require the agent to be able to actively choose tasks or interactively query the expert, while our approach requires neither.
Inverse Constraint Learning. We are far from the first to consider the ICL, problem. Scobee and Sastry [2019], McPherson et al. [2021] extend the MaxEnt IRL algorithm of Ziebart et al. [2008a] to the ICL setting. We instead build upon the moment-matching framework of Swamy et al. [2021], allowing our theory to handle general reward functions instead of the linear reward functions MaxEnt IRL assumes. We are also able to provide performance and constraint satisfaction guarantees on the learned policy, unlike the aforementioned work. Furthermore, we consider the multi-task setting, addressing a key shortcoming of the prior work.
Perhaps the most similar paper to ours is the excellent work of Chou et al. [2020], who also consider the multi-task ICL setting but propose solution that requires several special solvers that depend on knowledge of the parametric family that a constraint falls into. In contrast, we provide a general algorithmic template that allows one apply whatever flexible function approximators (c.g. deep networks) and reinforcement learning algorithms (e.g. PPO) they desire. Chou et al.'s method also requires sampling uniformly over the set of trajectories that achieve higher reward than the expert, a task which is rather challenging to do on high-dimensional problems. In contrast, our method only requires the ability to solve standard RL problem. Theoretically, Chou et al. [2020] focus on constraint recovery, which we argue below is a goal that requires strong assumptions and is therefore red herring on realistic problems. This focus also prevents their theory from handling suboptimal experts. In contrast, we are able to provide rigorous guarantees on learned policy performance and safety, even when the expert is suboptimal. We include results Appendix C that show that our approach is far more performant.
In concurrent work, Lindner al. [2023] propose an elegant solution approach ICL: rather than learning constraint function, assume that any unseen behavior is unsafe and enforce constraints on the learner to play a convex combination of the demonstrated safe trajectories. The key benefit of this approach is that it doesn't 't require knowing the reward function the expert was optimizing. However, by foreing the learner to simply replay previous expert behavior, the learner cannot meaningfully generalize, and might therefore be extremely suboptimal on any new task. In contrast, we use the side information of a reasonable set of constraints to provide rigorous policy performance guarantees."
We now turn our attention to formalizing inverse constraint learning.
3 Formalizing Inverse Constraint Learning
We build up to our full method in several steps. We first describe the foundational algorithmic structures we build upon (inverse reinforcement learning and constrained reinforcement learning).
We then describe the single-task formulation before generalizing it to the multi-task setup.
We consider a finite-horizon Markov Decision Process (MDP) [Puterman, 2014] parameterized by (S, A, T, r, where S, A are the state and action spaces, T S A + A(S) is the transition operator, r: S A + [-1, 1] is the reward function, and T' is the horizon.
3.1 Prior Work: Inverse RL as Game Solving
In the inverse RL setup, we are given access trajectories generated by an expert policy T S - ((A), but do not know the reward function of the MDP. Our goal is to nevertheless learn a policy that performs as well as the expert's, ss no matter the true reward function.
We solve the IRL problem via equilibrium computation between policy player and an adversary that tries to pick out differences between expert and learner policies under potential reward functions Swamy et al. [2021]. More formally, we optimize over polices π S - ((A) E II and reward functions f : S x A + 1- E F,.. For simplicity, we assume that our strategy spaces (II and J,,) are convex and compact and that Jr, RE E II. We solve (i.e. compute an approximate Nash
We also note that, because we scale the learned constraint differently for each task, Lindner et al. [2023]'s impossibility result (Prop. 2) does not apply to our method, thereby elucidating why naive application of inverse RL on the aggregate data isn't 't sufficient for the problem we consider.
3

---

Algorithm CRL (Constrained Reinforcement Learning)
Input: Reward constraint c, learning rates 1/1:N, tolerance 6
Output: Trained policy π
Initialize 21 0 for i in 1...N do
my RL(r - -- Xzc) A + () -)]]
X -
end for Return Unif(TI:N).
Algorithm 2 ICL (Inverse Constraint Learning)
Input: Reward r. constraint class Fe, trajectories from T
Output: Learned constraint c
Initialize 61 Fe
for i in 1... N do
Ty, A3 CRL(r, C, 6 ((T:
// use any no-regret algo. to pick c, e.g. FTRL:
C++1 +- argmax 1/2 (J(,,,, - - R(c). end for Return best of C1.N on validation data.
equilibrium) of the two-player zero-sum game
min max J(H,)) - J), fElF
where [TT, f) = Be Et=o (+t, ar)] denotes the value of policy under reward function f.
3.2 Prior Work: Constrained Reinforcement Learning as Game Solving
In CRL, we are given access both the reward function and a constraint e S x A [-1, Our goal is to learn the highest reward policy that, over the horizon, has a low expected value under the constraint. More formally, we seek a solution to the optimization problem:
min - [mT r) s.t. J( c) S 6,
where ó some error tolerance. We can also formulate CRL game via forming the Lagrangian of the above optimization problem [Altman, 1999]:
min max J(T A(J(z,c) 5). xell X>0
Intuitively, the adversary updates the weight of the constraint term the policy player's reward function based on how in violation the leamer is.
3.3 Single-Task Inverse Constraint Learning
We are finally ready to formalize ICL. ICL, we are given access to the reward function, trajectories from the solution to CRL problem, and class of potential constraints F, in which we assume the ground-truth constraint c* lics. We assume that F. is convex and compact.
In the IRL setup, without strong assumptions on the dynamics of the underlying MDP and expert, it impossible guarantee recovery of the ground-truth reward. Often, the only reward function that actually makes the expert optimal is zero everywhere [Abbeel and Ng, 2004]. Instead, we attempt to find the reward function that maximally distinguishes the expert from an arbitrary other policy in our policy class via game-solving [Ziebart et al., 2008a, Ho and Ermon, 2016, Swamy et al., 2021]. Similarly, for ICL, exact constraint recovery can be challenging. For example, two constraints differ only on states the expert never visits, it is not clear how to break ties. We instead try to find a constraint that best separates the safe (but not necessarily optimal) ng from policies that achieve higher rewards.

---

ICL
J r]
Figure 1: A visual depiction of the optimization problem we're trying to solve in ICL. We attempt pick constraint that minimizes the value dif- ference over the expert policy a safe policy could have. The star corresponds to the output of CRL.
max M(r,r)
minimist over e
J(5,-0
More formally, we seek to solve the following constrained optimization problem.
min J( r) ((
(4)
=ell
s.t. max J[TE 50.
(5)
rEF:
Note that in contrast to the moment-matching problem we solve in imitation learning [Swamy et al., 2021], instead want be least safe the expert. This means that rather than having equality constraints, we have inequality constraints. Continuing, we can form the Lagrangian:
cEFe min max (TTE r) (TT A(max ( J [TE, c))
(6)
*ell A>0
max max min J J(TE, r Ac) - Ac). cEF: ASD #Ell
(7)
Notice that the form of the ICL game resembles a combination of the IRL. and CRL games. We describe the full game-solving procedure in Algorithm 2, where R(c) is an arbitrary strongly convex regularizer [McMahan, 2011] 1]. Effectively, we pick a constraint function in the same way we pick a reward function in IRL but CRL inner loop instead of RL step. Instead of a fixed constraint threshold, we set tolerance to the expert's constraint violation. Define
1 T
((c)
() E [-1,1]
(8)
as the per-round loss that the constraint player suffers their online decision problem. The best-in- hindsight comparator constraint is defined as
T
arginax cEFc
(().
(9)
We can then define the cumulative regret the learner suffers as
Reg(T) = E 4() ((())
(10)
and let e; E((2) f((cs). We prove the following theorem via standard machinery.
Theorem 3.1. Let C1:N be the iterates produced by Algorithm 2 and let 1 IN denote their time-averaged regret. Then, there exists a E C1:N such that π CRL(r, c, J(TT, c)) satisfies
N
J(T,C') - J[TT c") S ET and (TT 2 J(TE,T).
(11)
In words, by optimizing under the recovered constraint,
learn policy that (weakly) Pareto-
dominates the expert policy under c". We conclude by noting that because FTRL (Follow the Regularized Leader, McMahan [201 1J) is no-regret algorithm for linear losses like (8), have that limm-500 Reg(T) 0. This means that with enough iterations, the RHS of the above bound on ground-truth constraint violation will go O.

---

Algorithm 3 MT-ICL (Multi-task Inverse Constraint Learning)
Input: Rewards r I: constraint class Fe, trajectories from क k Output: Learned constraint
Set Set {o F vk [ J c) s 0)
Initialize 01 E Fe
for i in 1. ...... N do
for in n X CRL(1 c; 6
do
end for
// any no-regret algo. to pick c, e.g. FTRL: c)) R(c).
C++1 t- argmax eFF. TR (J(rt,))))
end for Return best of C1NN on validation data.
Figure 2: If we have sufficient diversity of expert policies, none of which are optimal along the reward vector, we can identify the hyperplane that separates the safe policies from the unsafe policies. The constraint (red, dashed) will be orthogonal to this hyperplane. For this example, because Ps. E IR? we need two expert policies.
PB
PiI
{p. : (p.,e") < 0)
3.4 Multi-task Inverse Constraint Learning
One of the potential failure modes of the single-task approach we outline above is that we could learn an overly conservative constraint, leading to poor task performance [Liu et al., 2023]. For example, imagine that we entropy-regularize our policy optimization [Ziebart et al., 2008a, Haarnoja et al., 2018], as is common practice, Assuming full policy class, the learner puts nonzero probability mass on all reachable states in the MDP. The constraint player is therefore incentivized to forbid all states the expert did not visit [Scobee and Sastry, 2019, McPherson et al., 2021]. Such a constraint would likely generalize poorly when combined with new reward function (F * r) it forbids all untaken rather than just unsafe behavior.
At heart, the issue with the single-task formulation lies in the potential for insufficient coverage of the state space within expert demonstrations. Therefore, it is natural explore multi-task extension to counteract this limitation. Let each task be defined by unique reward. We assume the dynamics and safety constraints are consistent across tasks. We observe samples the form (rk: { T)) This data allows us to define the multi-task variant of our previously described ICL game:
K
max min max cEf IIN EIR >0
Σ J(T -oo--xoo
Nc) - J(Z -Ve).
(12)
We describe how we solve this game in Algorithm 3, where R(c) is an arbitrary strongly convex regularizer [MeMahan, 2011]. In short, we alternate between solving K CRL problems and updating the constraint based on the data from all policies.
We now give two conditions under which generalization to reward functions is possible.
3.5 A (Strong) Geometric Condition for Identifiability
Consider for moment the linear programming (LP) formulation of reinforcement learning. We search over the space occupancy measures (px E A(S x A)) that satisfy the set of Bellman flow constraints [Sutton and Barto, 2018] and try to maximize the inner product with reward vector [|S|[A) We can write the CRL optimization problem (assuming d 0 for simplicity) LP

---

as well. Using Pn to denote the occupancy measures of all II,
max (P=: r) s.t. (Po. c²) =
P=Epa
We observe a (for simplicity, optimal) solution to such a problem for K rewards, begging the question of when that is enough to uniquely identify c². Recall that uniquely determine the equation of hyperplane in R need d linearly independent points. c* |ISÍAA,, so need [S|| [S||-A| expert policies. Furthermore, we need each of these points to lie on the constraint line and not on the boundary of the full polytope. Put differently, underlying constraint (i.c. プπ E s.t. J(TTE. r") < J( r")). Under these conditions, we can uniquely determine the hyperplane that separates safe from unsafe policies, to which the constraint vector is orthogonal. More formally,
need each distinct expert policy saturate the
Lemma 3.2. Lel TE 1:(8|AAl be distinct optimal expert policies such that a) Vi E [|S]A|], The relint( and b) no Pag can be generated by mixture of the other visitation distributions. Then, c* is the unique (up to scaling) nonzero vector in
Prt
Pa
Nul
(13)
P P11811A1-1
P
[8|AA|
We visualize this process for the [S)] = 2 case Fig. 2. Assuming we are able to recover c*, we guarantee that our learners will be able to act safely, regardless of the task they asked to do. However, the assumptions required to do so are quite strong: we are effectively asking for our expert policies form basis for the space of occupancy measures, which means must expert data for a large set of diverse tasks. Furthermore, we need the experts to be reward-optimal.
Identifiability (the goal of prior works like Chou et al. [2020], Amin et al. [2017]) is too strong goal as it requires us to estimate the value of the constraint everywhere the state-action space. we know the learner will only be incentivized to go to certain subset of states (as is often true in practice), we can guarantee safety without fully identifying c*. Therefore, we now consider how, by making distributional assumptions on how tasks are generated, can generalize to novel tasks.
3.6 A Statistical Condition for Generalization
Assume that tasks т are drawn i.i.d. from some P(r). Then, even we not see a wide enough diversity of expert policies to guarantee identifiability of the ground-truth constraint function, with enough samples, we can ensure we do well in expectation over tasks. For some constraint c, let us define
(c) E,- .c) - ((TE, c)],
(14)
where X", -- = CRL(TT, c) denote the solutions to the inner optimization problem. We begin by proving the following lemma.
Lemma 3.3. With
Fel
(2T)²
K 2 0 log
(15)
2²
samples, we have that with probability estimates of V (c) within absolute error.
2 1 6, we will be able to estimate all [Fel population
Note that we perform the above analysis for finite classes but one could easily extend it [Sriperumbudur al., 2009]. The takeaway from the above lemma is that we observe sufficient number of tasks, we can guarantee that we can estimate the population loss of all constraints, up to some tolerance.
Consider the learner being faced with the single task case, where it is clear how to set the cost limit passed to CRL, it is not clear how to do so for novel task. Hence, make the following assumption.
task they have
before test time. Unlike in
Assumption 3.4. We assume that E, ( g ct)] < 0. and that Vo E F,, 3r E IT s.t. J(T, c) S 0.
This (weak) assumption allows us to a) use cost limit for our CRL step and b) search over a subset of F. that the expert is safe under. Under this assumption, we are able to prove the following:
7

---

Theorem 3.5. Let C1:N be the iterates produced by Algorithm 3 with [(e, 8) chosen as in Lemma 3.3 N
and let ē N Σx 6; denote their time-averaged regret. Then, w.p. 2 1 6, there exists c C1:N such that π
(r) = CRL(T, c, 5 = =0) 0) satisfies
(p - - S ET 3eT and Emdd))
- r.) -ee..
In short, if we observe enough tasks, are able to learn constraint that, when optimized under, leads to policies that approximately Pareto-dominate those of the experts on average.
We now turn our attention to the practical implementation of these algorithms.
4 Practical Algorithm
We provide practical implementations of constrained reinforcement learning and inverse constraint learning and benchmark their performance on several continuous control tasks. We first describe the environments we our algorithms on. Then, we provide results showing that our algorithms learn policies that match expert performance and constraint violation. While it is hard to guarantee constraint recovery in theory, we show that we can recover the ground-truth constraint empirically we search over a restricted enough function class.
4.1 Tasks
We focus the ant environment from the PyBullet [Coumans and Bai, 2016] and MuJoCo [Todorov et al., 2012] benchmarks. The default reward function incentivizes progress along the positive æ direction. For our single-task experiments, we consider velocity and position constraint on top of this reward function.
1. Velocity Constraint: 19+11-0 oe < 0.75 where is the ant's position
2. Position Constraint: 0.5xr - V1 S where zt, Yt are the ant's coordinates
For our multi-task experiments, we build upon the D4RL [Fu et al., 2020] AntMaze benchmark. The default reward function incentivizes the agent to navigate fixed maze random goal position: exp(-11 Iguul 4:|12 2|2). We modify this environment such that the walls of the maze are permeable, but the agent incurs unit step-wise cost for passing through the maze walls.
Our expert policies are generated by running CRL with the ground-truth constraint. We use the Tianshou [Weng et al., 2022] implementation of PPO [Schulman et al., 2017] as our baseline policy optimizer. Classical Lagrangian methods exactly follow the gradient update shown in Algorithm 1, but they are susceptible oscillating learning dynamics and constraint- constraint-violating behavior during training. The PID Lagrangian method [Stooke et al., 2020] extends the naive gradient update of X with a proportional and derivative term to dampen oscillations and prevent cost overshooting. To reduce the amount of interaction required to solve the inner optimization problem, we warm-start our policy in each iteration by behavior cloning against the given expert demonstrations. We used single NVIDIA 3090 GPU for all experiments. Due to space constraints, we defer all other implementation details to Appendix B.
4.2 ICI. Results
We begin with results for the single-task problem, before continuing on to the multi-task setup.
4.3
Single-Tasse Task Continuous Control Results
As argued above, we expect a proper ICL implementation to learn policies that perform as well and are safe the expert. However, by restricting the class of constraints we consider, we can also investigate whether recovery of the ground- -round-truth constraint is possible. To this end, we consider a reduced-state version of our algorithm where the learned constraint takes a subset of the agent state as input. For the velocity constraint, the learned constraint a linear function of the velocity. while for the position and maze constraints, the learned constraint is linear function of the ant's position.
Using this constraint representation allows us to visualize the learned constraint over the course of training, as shown in Figure 3. We find that our ICL implementation is able to recover the constraint,

---

Nomen Corree
Figure 3: Over the course of training, the learned ICL constraint recovers the ground-truth constraints for the velocity and position tasks. The learned policy matches expert performance and constraint violation. Standard errors are computed across 3 seeds.
Figure 4: As ICL training progresses, the learned position constraint (red line) converges to the ground-truth constraint (blue line) and the policy learns to escape unsafe regions (red region).
as the learned constraint for both the velocity and position tasks converges to the ground-truth value. Our results further show that over the course of ICI. training, the learned policies match and exceed expert performance as their violations of the ground-truth constraint converge towards the expert's. Figure 4 provides direct depiction of the evolution of the learned constraint and policy. The convergence of the red and blue lines shows that the learned position constraint approaches the ground truth, and the policy's behavior approaches that the expert in response to this.
Figure 5: We re-conduct our position velocity constraint experiments using suboptimal experts to generate demonstrations. While, because of the ill ill-posedness of the problem, we do not exactly recover the ground truth constraint, we are able to use it to learn a policy that is higher performance than the expert while being just as safe. Standard errors computed across seeds.
To measure the robustness of our method to sub-optimal experts, we repeat the above experiments using demonstrations from an expert with i.i.d. Gaussian noise added to their actions at each timestep. We are still able to learn a safe and performant policy, which matches what our theory predicts.
4.4 Multi-Tast Continuous Control Results
We next consider an environment where, even with an appropriate constraint class, recovering the ground-truth constraint with single task isn't 't feasible due to the ill-posedness of the inverse constraint learning problem. Specifically, we use the umaze AntMaze environment from D4RL [Fu al., 2020], modified to have more complex maze structure. As seen in Figure 7, the goal of each task is to navigate through the maze from one of the starting positions (top/bottom left) to one of the grid cells in the rightmost column. We provide expert data for all 10 tasks the learner.
As we can see Figure 6, multi-task ICL is, within single iteration, able to learn policies that match expert performance and constraint violation across all tasks, all without ever interacting with the ground-truu truth maze. Over time, we are able to approximately recover the entire maze structure.
We visually compare several alternative strategies for using the multi-task demonstration data in the bottom row of Figure 7. The 0/1 values in the cells correspond to querying the deep constraint network from the last iteration of ICL on points from each of the grid cells and thresholding at some confidence. We see that a single-task network (d) learns spurious walls that would prevent the learner from completing more than half of the tasks. Furthermore, learning 10 separate classifiers and then aggregating their outputs (e) / (f) also fails to produce reasonable outputs. However, when we use data from all 10 tasks to train our multi-task constraint network (g) (h), we are able to approximately recover the walls of the maze. These results echo our preceding theoretical argument about the importance of multi-task data for learning constraints that generalize to future tasks.

---

Figure 6: We see that over ICL iterations, we are able to recover the ground-truth walls of the ant-maze, enabling the learner to match expert performance and constraint violations. Results for the second two plots are averaged across all 10 tasks. Standard emrors are computed across seeds.
MaliTaak Seap
(a)
(c)
(d)
(e)
(I)
(h)
Figure 7: We consider the problem of trying to learn the walls of custom maze (a) based on the AntMaze environment from D4RL [Fu et al., 2020]. We consider both a single-task (b) and multi-task (c) setup. We see that the single-task data is insufficient to learn an accurate constraint (d). Averaging taking the max over the constraints learned from the data for each of the ten goals (e)-(f) also doesn't work. However, if we use the data from all 10 tasks to learn the constraint (g)-(h), we are able to approximately recover the ground-trute -ound constraint with enough constraint learning iterations.
We release the code used for all of our experiments at httpp///////ttt/ttt//eppess http://hhh...cmmmoooo////ttttt
5
Discussion
In this work, we derive an algorithm for learning safety constraints from multi-task demonstrations. We show that by replacing the inner loop of inverse reinforcement learning with constrained policy optimization subroutine, we can learn constraints that guarantee learner safety on a single task. We then give statistical and geometric conditions under which can guarantee safety unseen tasks by planning under learned constraint. We validate our approach on several control tasks.
Limitations. In the future, would be interested in applying approach to real-world problems (e.g. offroad driving). Algorithmically, the CRL inner loop can be more computationally expensive than RI. loop- would be interested in speeding up CRL using expert demonstrations, perhaps by adopting the approach of Swamy et al. [2023]. We also ignore all finite-sample issues, which could potentially be addressed via data-augmentatton approaches like that of Swamy al. [2022].
6 Acknowledgements
We thank Drew Bagnell for edifying conversations on the relationship between ICL and IRL and Nan Jiang for connecting us to various references in the literature. We also thank an anonymous reviewer for pointing out that our method does not actually require the expert to be the optimal safe policy, a fact we did not fully appreciate beforehand. ZSW supported in part by the NSF FAI Award #1939606, Google Faculty Research Award, J.P. Morgan Faculty Award, a Facebook Research Award, an Okawa Foundation Research Grant, and Mozilla Research Grant. KK and GS are supported by GPU award from NVIDIA.
10

---

References
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, page 1, 2004.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on muchine learning, pages 22-31. PMLR, 2017.
Eitan Altman, Constrained Markov decision processes, volume 7. CRC press, 1999.
Kareem Amin, Nan Jiang, and Satinder Singh. Repeated inverse reinforcement learning. Advances in neural information processing systems, 30, 2017.
Glen Chou, Dmitry Berenson, and Necmiye Ozay. Learning constraints from demonstrations. In Algorithmic Foundations of Robotics XIII: Proceedings of the 13th Workshop the Algorithmic Foundations of Robotics 13, pages 228-245. Springer, 2020.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Adam Gleave and Oliver Habryka. Multi-task maximum entropy inverse reinforcement learning.
arXiv preprint arXiv: 088822 2018.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.1033:
2022.
Tuomas Haaroja,, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2.05905, 2018.
Dylan Hadfield Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning, 2016.
Dohyeong Kim and Songhwai Oh. Efficient off-policy safe reinforcement learning using trust region conditional value at risk. IEEE Robotics and Automation Letlers, 7(3):7644-7611, 2022.
Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting. In European conference on computer vision, pages 201-214. Springer, 2012.
J Zico Kolter, Mike P Rodgers, and Andrew Ng. control architecture for quadruped locomotion rough terrain. In 2008 IEEE International Conference Robotics and Automation, pages 811-818. IEEE, 2008
Qingkai Liang, Fanyu Que, and Eytan Modiano. Accelerated primal- policy optimization for safe reinforcement learning. arXiv preprint arXtv:1802.06488. 2018.
David Lindner, Xin Chen, Sebastian Tschiatschek, Katja Hofmann, and Andreas Krause. Learning safety constraints from demonstrations with unknown rewards. arXiv preprint arXiv:2305.1614; 2023.
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pages 13644-13668. PMLR, 2022.
Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. In International Conference on Learning Representations, 2023.

---

Brendan McMahan. Follow-the teewwwtherreeeeee regularized-leatee and mirror descent: Equivalence theorems and 11 regularization. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 525-533. JMLR Workshop and Conference Proceedings, 2011.
David McPherson, Kaylene Stocking, and Shankar Sastry. Maximum likelihood constraint inference from stochastic demonstrations. In 202/ IEEE Conference on Control Technology and Applications (CCTA), pages 1208-1213. IEEE, 2021.
Ted Moskovitz, Brendan O'Donoguue, Vivek Veeriah, Sebastian Flennerhag, Satinder Singh, and Tom Zahavy, Reload: Reinforcement learning with optimistic ascent-descent for last-iterate convergence in constrained mdps. arXiv preprint arXiv:2302.01225, 2023.
Andrew Ng. Adam Coates, Mark Diel, Varun Ganapathi, Jamic Schulte, Ben Tse, Eric Berger, and Eric Liang. Autonomous inverted helicopter flight via reinforcement learning. In Experimental robotics IX, pages 363-372. Springer, 2006.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Nathan D Ratliff, David Silver, and Andrew Bagnell. Learning search: Functional gradient
techniques for imitation learning, Autonomous Robots, 27(1):25-53, 2009.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Dexter RR Scobee and S Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. arXiv preprint
2019.
David Silver, Andrew Bagnell, and Anthony Stentz. Learning from demonstration for autonomous navigation in complex unstructured terrain. The International Journal of Robotics Research, 29 (12):1565-1592, 2010.
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG Lanekriet. On integral probability metries,) (phi-divergences and binary classification. arXiv preprint arXiv:0901. 2698, 2009.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety reinforcement learning by pid lagrangian methods. In International Conference Machine Learning, pages 9133 1133-9143. PMLR, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell, and Zhiwei Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap, 2021. URL https: //arxiv. org/ abaaabaa2/ 2212210022233
Gokul Swamy, Nived Rajaraman, Matthew Peng, Sanjiban Choudhury, Andrew Bagnell, Zhi- wei Steven Wu, Jiantao Jiao, and Kannan Ramchandran. Minimax optimal online imitation learning via replay estimation. arXiv preprint wrXiv:2205.15397 2022.
Gokul Swamy, Sanjiban Choudhury, Andrew Bagnell, and Zhiwei Steven Wu. Inverse reinforcement learning without reinforcement learning. arXiv preprint 1rXiv:2303.14623, 2023.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model -aaeed control. In 2012 IEFF/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
Marcell Bizquez-Chanlate Susmit Jha, Ashish Tiwari, Mark K Ho, and Sanjit Seshia. Learning task specifications from demonstrations. Advances in neural information processing systems, 31, 2018.
12

---

Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1-6, 2022. URL. http://tttmmltt org/ httpp/ppppss////e 21-1127.00 html.
Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, and Chelsea Finn. Learning prior over intent via meta-etttmiia mneta-inverse reinforcement learning. In International conference machine learning, pages 6962. PMLR, 2019.
Mengdi Xu, Zuxin Liu, Peide Huang, Wenhao Ding, Zhepeng Cen, Bo Li, and Ding Zhao. Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv preprint arXiv:2209.0002 2022.
Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-baag based constrained policy optimization. arXiv preprint arXiv:2010.03/52, 2020.
Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with probabilisstic context variables. Advances in neural information processing systems, 32, 2019.
Brian Ziebart, Anind Dey, and Andrew Bagnell. Probabilistic pointing target prediction via inverse optimal control. Proceedings of the 2012 ACM international conference on Intelligent User Interfaces, pages 1-10, 2012.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, II., USA, 2008a.
Brian D Ziebart, Andrew L Maas, Anind K Dey, and J Andrew Bagnell. Navigate like cabbie: Prob- abilistic reasoning from observed context-awars behavior. In Proceedings of the 10th international conference on Ubiquitous computing, pages 322-331, 2008b.
Matt Zucker, Nathan Ratifff, Martin Stolle, Joel Chestnutt, J Andrew Bagnell, Christopher G Atke- son, and James Kuffner. Optimization and learning for rough terrain legged locomotion. The International Journal Robotics Research, 1002)):755111, 2011.
13

---

A Proofs
A.1 Proof of Theorem 3.1
Proof. Let (c) denote the set of policies such that
J c) J(TR,)) 2 O.
(16)
First, note that Vc E Fes TE E [c(c). CRL is therefore trying to maximize reward over a set of policies that contains TE. Thus, the reward condition is trivially true. We therefore focus on the safety condition. First, we note that by the definition of regret,
N I(((TT))) -(TT, (J(m, - ((TE, N
N - s sT. (17)
1
N
This implies that
1 N
(J J(T, S ST,
(18)
è is the best-in-hindsighe constraint. We then note that J c.) - g (4) by the fact that my is produced via a CRL procedure, which means we can drop the former term from the above sum, giving us
1 -( ET.
(19)
N
Because this equation holds on average, there must at least one π E JiN for which it holds. Now, we recall that म. CRL(r, cj) to complete the proof.
A.2 Proof of Lemma 3.3
Proof. For a single e, standard Hoeffding bound tells that
-2K (2T)2
P( Σ Vi(c) - E(V() 2 S 2exp
(20)
where (c) denotes the value of the payoff using data from the kth task. We have [F constraints and want to within of the population mean uniformly. We can apply union bound to tell that w.p. at least
-2Ke )
1 - 2 F. exp( (2T)²
(21)
we will do so. we want to satisfy this condition with probability at least 6, simple algebra tells us that must draw
[Fo]
(2T)
K 0 log
(22)
samples.
A.3 Proof of Theorem 3.5
Proof. For each e F., define the set of safe policies as IT(c) { E [] S 0}. This set is non-empty by assumption. Define
" ( = 1 [()).
(23)
We prove each of the desired conditions independently.
Reward Condition. Let à C1.N. Recall that want prove that
(n-P())(((((")))) J(T r]] -2eT,
(24)
14

---

where x² CRL(r", "r",,,, à = 0). Observe that
g E [I(e) J(T ,r)
max J(T,T") J( r"). xel(ee)
(25)
Thus, if
0=[ 21-6,
(26)
we have that
E [JT'' -J(TE
(27)
- E, u(T (T, (() J ((TE,r'))
(28)
< [, u(,, [[] J( -
(29)
$ E, (-,,, 22T
(30)
$ -2T
(31)
We now prove that with K large enough, we can guarantee Eq. 26 holds true w.h.p. Define
= Ec
[K], Th E 11())).
(32)
We now argue that if
F 1
K K20
log
(33)
w.p. > 1 6, Eq. 26 holds true Ve E F.. This means that as long as we pick CA:N E F.. our desired condition will true. Note that this fewer than the number of samples we assumed the theorem statement.
For single constraint, Hoeffding bound tells us that
P(1 K 11(m [[))) E. [u(T, 2)]l s 2cxp 2cxpp-22K (-2Ke²).
(34)
Union bounding across Fe 2 Fe, we get that the probability that 3c E Fe s.t. Eq. 26 does not hold upper bounded by
2|F exp (-2K())
(35)
To have this quantity be 2 1 8, we need
Fr K20 log
(36)
e²
Safety Condition. We begin by considering the infinite sample setting. We therefore desire to prove
that
E-[J(T', - J(r c²) टT. ET.
(37)
Define the per-round loss of the constraint player as
1 (c)) = + [J e) - J(T f c)] E [-1,
(38)
the best-in-hindsiin comparator (c:),, and average regret as é = + Σn Proceeding similarly to the single-task case,
argmax ΣN (c(c), instantaneous regret e, (,())
1 N
ēT B-(J(TT,C)
J 0) - ((T),)))
(39)
N
1 N
Σ
J(TE:
(40)
We now argue that the second term the above sum must be non-positive. Consider an arbitrary task Then, because Tog E I1(c;) and CRL is optimizing over II(cs), this term must be negative. As is
15

---

negative task, it must be negative in expectation. Thus, we are free to drop the second term in the above expression which tells that
N ET Σ -
(41)
Because this equation holds average, there must be at least π E WINN for which it holds. Now, we recall that u.- = CRL(r" ()) to complete the infinite-sample proof.
We now consider the emror induced by only observing finite set of tasks. There two places finite-sample error enters: in estimating the value of (ç(c) and in estimating Fe.
By Lemma 3.3, the maximum error we can induce by estimating & from finite samples is upper bounded w.h.p by Thus, the extra error induced on the average regret is also bounded by Observe that our losses are scaled by 7 in comparison to difference of Js. Therefore, we need add an e7 to bound for the infinite-sample setting.
By our argument the reward section, T & [1((;) w.p. S When this is true, ( J q)) can be as big as 27. Thus, the term we dropped in Eq. 40 (V(c)) can be as big as 2cT instead of O. In the worst case, this adds an additional 2cT to our bound.
Combining both of the above, when we transition from the infinite sample setting to the finite sample setting, our bound degrades by 3eT.
16

---

B Experimental Details
An overview of the key experimental details is presented below. The exact code for reproducing our experiments is available https: //:g::/ghtttt cww.coo//////// http//ooook/mmic......om
B.1 CRL
We use the Tianshou [Weng et al., 2022] implementation of PPO [Schulman al., 2017] as our baseline policy optimizer. Classical Lagrangian methods exactly follow the gradient update shown Algorithm 1, but they are susceptible to oscillating learning dynamics and constraint-violating behavior during training. The PID Lagrangian method [Stooke et al., 2020] extends the naive gradient update of 2 with a proportional and derivative term to dampen oscillations and prevent cost overshooting.
We find that augmenting the policy state with the raw value of the constraint function be crucial for successful policy training. For CRL, this corresponds to the violation of the ground-trus -tuth constraint.
B.2 Expert Demonstrations
For all of our experiments, our experts CRL policies trained to satisfy ground truth constraint. The expert demonstrations we use in ICL are generated by rolling out these policies and collecting 20 trajectories. To simulate suboptimal experts, we generate noisy trajectories by adding zero- mean Gaussian noise (with standard deviation 0.7 0.5 for velocity position) to the policy's action at every timestep.
B.3 Single-Tass Task ICL.
Similar to CRL, we augment the policy state with the raw value of the constraint function. For ICL, this corresponds to the violation of the learned constraint.
To reduce the amount of interaction required solve the inner optimization problem, we warm-start our policy in each iteration by behavior cloning against the given expert demonstrations. We zero-out the augmented portion of the state when behavior cloning to avoid leaking constraint violation information from the expert.
When using CRL part of ICL, set the constraint threshold used in the Lagrangian update to be the expert's constraint violation. However, when starting with degenerate constraints, this can prevent policy optimization from learning at all the expert's 's violation under the constraint can be arbitrarily low. To circumvent this issue, we set the Lagrangian constraint threshold to use the expert's 's violation plus cost limit buffer, which we anneal over the course of training 0. This ensures that our learned policy satisfies the learned constraint as much as the expert does as desired.
Because ICL requires learning constraint, represent our constraints neural networks, mapping from the state space of our agent to bounded scalar in the range [0. 1]. To update this constraint, we solve the optimization problem using regression objective. Learner and expert constraint values are labeled with and 11 respectively, and we optimize a mean squared error loss. We found that weighting data equally (instead of via returned Lagrange multipliers) was sufficient for good performance and therefore utilize this simpler strategy.
For both CRL and ICL, we find that using log-activation top of the raw value of the constraint is an important detail for stable training.
B.4 Multi-Task ICL
For the multi-task maze setting, we consider 10 distinct tasks corresponding to unique goal locations, each with 2 starting locations.
Our solver combines waypoint planner with a low-level controller. Waypoints are calculated by discretizing the maze into a 10 by 10 grid and running Q-value iteration, while low-level actions are computed using CRL experts trained to move in single cardinal direction. To generate trajectory, follow sequence of waypoints from the planner using the controller.
17

---

For ICL, we visualize our learned constraints by uniformly sampling 100,000 points from every grid cell, averaging the constraint predictions, and thresholding limit of 0.5.
B.5 Hyperparameters
The key experimental hyperparameters are shown in Table 1. The exact configuration we use for our experiments is available at htttps://gitae config py.
Hyperparameter
Value
PPO Learning Rate
0.0003
PPO Value Loss Weight
0.25
PPO Epsilon Clip
0.2
PPO GAE Lambda
0.97
PPO Discount Factor
0.99
PPO Batch Size
512
PPO Hidden Sizes
[128, 128]
P-update Learning Rate
0.05
L-update Learning Rate
0.0005
D-update Learning Rate
0.1
Constraint Batch Size
4096
Constraint Learning Rate
0.05
Constraint Update Steps
250
Steps per Epoch
20000
CRL Velocity Epochs
50
CRL Position Epochs
100
ICL Expert Demonstrations
20
ICL Velocity Cost Limit
20
ICL Position Cost Limit
100
ICL Anneal Rate
10
ICL Velocity Outer Epochs
20
ICL Position Outer Epochs
10
ICL Velocity Epochs
10
ICL Position Epochs
50
Table 1: Experiment hyperparameters.
Comparison to Chou et al. [2020]
C.1 Experimental Setup
We attempt to faithfully implement the method of Chou et al. the problems consider baseline for single-task ICL. At a high level, their method requires steps:
1. For each expert trajectory. perform random search in trajectory space starting from the demo to try and compute the set of trajectories that are higher reward than the demo.
2. Solve a constrained optimization problem over a known parametric family that labels each expert trajectory with +1 and each learner trajectory with 0.
Note that in contrast to our method, Chou et al. perform a single constraint estimation, rather than an iterative procedure.
For Step 1), Chou et al. [2020] use hit-and-run sampling that ensures good coverage over this set of trajectories in the limit of infinite sampling. However, for tasks of the horizon (1000+) and state-space dimension (30) we consider, one would need rather large number of samples to ensure good coverage (i.e. e-net would require 30 samples).
18

---

As search in the space of trajectories is infeasible, we instead search in the space of policies, as is much more standard on long-horizon problems. We therefore initialize the policy with behavioral cloning (similar to starting from the expert trajectories), performing maximum entropy RI. (a search procedure with good coverage properties), and return the set of trajectories from the replay buffer that are of a higher reward than that of the expert.
For Step 2), Chou al. [2020] use a variety of mixed-integer LP solvers. essence however, they are trying to solve a classification problem. Of course, with their assumption of an optimal expert (or a "boundedlly suboptimal" expert with known sub-optimality gap), they can solve the strict feasibility problem. Because we assume that the expert is safe in expectation (rather than uniformly), it is natural to consider relaxation of this problem (i.e. a convex relaxation of the 0/1 loss).
We experimented with several such relaxations when developing our method and found the & relaxation (i.e. treating the problem regression problem with different targets for learner and expert demos) to work the best in practice. We therefore use the same relaxation for their method. For our single-task experiments,
clear how to specify the "parametric family" the constraint fits in (e.g. a linear threshold), so we believe the above is faithful implementation.
So, in short, we perform the same constraint learning procedure over different source of learner data.
C.2 Baseline Comparison Results
We implemented the above method on our velocity and position tasks and display the results (averaged over seeds) below.
Algo.
cc - cl (L)
- JJ (1) J(T,, - (TT,,(( (1)
ICL (iter I)
0.052
569.277
270.111
ICL (iter 10)
0.000
-72.149
133.59
Chou et al.
0.311
384.772
-1958.372
Table 2: Baseline Comparison (Position Constraint)
Algo.
|c" cc (l)
J(TE, - J(T, -( (1)
J(TE, (T,,"" (1)
ICL (iter 1)
0.249
12.267
-244.25
ICL (iter 10)
0.006
-46.771
-21.602
Chou al.
0.246
-28.315
-134.58
Table 3: Baseline Comparison (Velocity Constraint)
We see that our method performs better in terms of all performance criteria we evaluate under. As implemented, Chou et al.'s method appears to produce inaccurate constraint that leads to significant safety violation (the bottom right cell in both tables).
19


Reviews:
[{"review_id": "KCjOhJR9Ap", "reviewer": "Reviewer_giMg", "summary": "This paper addresses the challenge of learning safety constraints for agents in various tasks from expert demonstrations, instead of manually specifying them. The authors extend inverse reinforcement learning (IRL) techniques to the space of constraints, aiming to learn constraints that prevent highly rewarding behavior that the expert could have performed but avoided. However, the constraint learning problem is ill-posed and tends to result in overly conservative constraints. To mitigate this issue, the authors utilize diverse demonstrations from multi-task settings to learn a tighter set of constraints. The method is validated through simulation experiments on high-dimensional continuous control tasks.", "questions": "The author could better address the concerns mentioned in the weaknesses section. It is worth noting that the reviewer is not well-versed in this field and does not specialize in IRL or ICL problems. Thus, the overall rating may be subject to change based on feedback from other reviewers and the author's rebuttal.", "limitations": "Consider removing Section 4.2.\nA more intuitive explanation of Figure 2 would be helpful.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors develop a multi-task variant of inverse constraint learning, which consists of multiple policy players aiming to maximize task-specific rewards and a constraint player determining a single constraint that all policy players must follow. \n- The proposed approach reduces the chances of choosing degenerate solutions.\n- The inverse constraint learning problem is formulated as a zero-sum game involving a policy player, who seeks to maximize rewards while adhering to potential constraints, and a constraint player, who selects constraints that impose maximum penalties on the learner compared to the expert. \n- The proposed method demonstrates its effectiveness in various continuous control tasks. When applying restricted function classes, the technique can retrieve ground-truth constraints for certain tasks. ", "weaknesses": "The author argues that previous ICL problems have not been well-formulated, leading to solutions that only allow expert actions. However, it is not clear how the proposed method addresses this issue, aside from utilizing a greater number of demonstrations from a multi-task setting.\nIt would be interesting to determine whether the proposed method can work for tasks beyond maze locomotion, which lack complex interactions with the environment. For example, in manipulation tasks, safety constraints are more critical as the robot must interact with and alter the environment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough description of our work. Responding to the concerns raised:\n\nW1: Intuitively, if we see a more diverse set of behaviors from the expert by using multi-task data, our constraint learner will spuriously forbid fewer states. Importantly, it is not the greater number of demonstrations that matters for dealing with ill-posedness so much as it is the greater diversity of behavior. For example, building on the reviewer’s manipulation example, if we only see data from a single pick-and-place task, single-task ICL could recover a constraint that prevents the learner from placing an object anywhere else on the table. As long as these sort of “complex interactions with the environment” are encoded in the state of the agent (i.e. we are not in the partially observed setting), we believe our approach would be applicable.\n\nPlease also see our global response on experimental complexity.\n\nWhile we agree that focusing on manipulation specific applications would be interesting, our current experiments are equally high-dimensional (e.g. for our ant experiments, the state-space is 27 dimensional and the action space is 8 dimensional, compared to a 7 DoF manipulator) and we show results where are able to recover workspace constraints. More explicitly, given that we can learn a position constraint for a high-dimensional ant agent, we should also be able to learn constraints on the end effector of a manipulator). Similarly, if we are able to handle data that comes from the agent navigating to different parts of the maze, placing a block in different locations / goals should not be that different. Hence we see no conceptual reason why the algorithm would not scale to manipulation.\n\nFigure 2: We would be more than happy to add more description of this figure. At a high level, $\\rho_{\\Pi}$ refers to the space of occupancy measures (i.e. the set of state-action distributions of all policies in our policy class). The reason we chose this representation was that a constraint is simply a hyperplane in this space (the red line labeled $\\langle \\rho_{\\pi}, c^* \\rangle = 0$). To uniquely determine a line in $\\mathbb{R}^d$, we need $d$ points on the line – these are what the green dots / expert policies $\\pi_E^1$ and $\\pi_E^2$ correspond to. We need two policies as our visitation distributions are two-dimensional (e.g. a two-state MDP). The technical conditions in Lemma 3.2 are the more formal version of the above statements (e.g. relint meaning the points are on the line rather than on the boundary of the space)."}, {"review_id": "koHnbnWeb1", "reviewer": "Reviewer_NZBn", "summary": "Broadly, the paper address the challenge of safety constraints for robots, in particular the difficulty of handcrafting these. They propose learning safety constraints from expert demonstrations, particularly in a multi-task setting where each task reward is known and the safety constraints are task-agnostic.\n\nThey review related work on inverse RL (IRL), constrained RL (CRL), inverse constraint learning (ICL), and multi-task ICL. Compared to existing ICL approaches, they claim their approach is more general, provides more guarantees, and is simpler to implement.\n\nThey extensively formalize ICL, including explaining relevant approaches to IRL and CRL. As a brief recap, they compare ICL to IRL: IRL has an outer loop that searches over potential rewards, with an inner loop that uses RL to find an optimal policy. ICL has an outer loop that searches over potential constraints, with an inner loop that uses CRL to find a safe-optimal policy. In either case, the optimized policy is compared to the expert demonstrations to revise the learned reward or constraint.\n\nThey also explain a challenge with single-task ICL, namely that ICL may learn an overly-conservative safety constraint that simply forbids all states not visited by the expert demonstrations. This motivates their exploration of multi-task ICL, where the safety constraint is assumed to be shared among the tasks. Their primary novel contribution of the paper is an extension of a single-task ICL approach to multi-task: \"we alternate between solving K CRL problems and updating the constraint based on the data from all policies\". Using it, they prove that they can exactly identify a shared constraint given enough expert policies, under certain very strong assumptions. They also prove a weaker but more practical theorem: if they have enough tasks, they can learn a constraint that, if used to optimize a policy, leads to policies that don't exceed the constraint-violation of the expert demonstrations and do meet or exceed the reward of the expert demonstrations (within a tolerance).\n\nFor the single-task case, they evaluate their approach in the simulated ant environment in PyBullet and MuJoCo benchmarks. They choose arbitrary velocity and position constraints. They represent their learned constraints as neural networks, mapping from the state space of their agent to a bounded scalar in the range [0, 1]. They show that their approach can recover these ground-truth constraints, under certain simplifying assumptions (linear constraints; usage of a subset of the agent state as input).\n\nFor multi-task, they use the PointMaze environment from D4RL, with each task variant having one of two start positions and one of 10 goal positions. Using their approach, they are able to approximately learn the ground-truth constraints (namely the maze's blocked cells). They also show that naive approaches for combining 10 separate learned constraints from single-task ICL don't work well.", "questions": "Section 3.4 and 3.5 mention needing expert policies. However, the premise of your work is that you have access to expert demonstrations and reward functions, but not expert policies. Can you clarify?\n\nA suggestion for positioning this line of work: instead of \"safety constraints\", I might argue that a more broad (and thus more valuable) set of constraints to be learned from human demonstrations is \"task-agnostic human-behavior constraints\", which would encompass safety plus other conventions and preferences. E.g. coming back to the example in your introduction, if I ask my friend to make toast, I'd be surprised if they set a plate down upside-down or rinsed the bread under the faucet.\n\n>we consider a setting in which we have access to demonstrations of the optimal safe policy for a task, along with knowledge about the task’s reward\n\nCan you describe this setting in practical terms? It seems that your approach has stronger requirements, namely (1) access to not just the rewards received by the demonstrations, but access to the reward function itself, and (2) access to a simulator or other mechanism to train a policy using CRL. As a clarifying example: if you had access to expert trajectories of a real-world robot including full scene state and reward annotations, across multiple tasks, this alone would not be sufficient to learn safety constraints using your method, right?\n\n>In short, if we observe enough tasks, we are able to learn a constraint that, when optimized under, leads to policies that approximately Pareto-dominate that of the expert.\n\nCan you elaborate? What is the specific Pareto-dominance here? My understanding is: you can learn a constraint that, if used to optimize a policy, leads to policies that don't exceed the constraint-violation of the expert demonstrations and do meet or exceed the reward of the expert demonstrations (within a tolerance). However, I don't intuitively follow why this should be your goal for the learned safety constraint.\n\nFor Figure 2, can you offer an intuitive explanation? For example:\n\n= can you label the axes?\n\n= why is an expert policy a single point in this space?\n\n= what does the boundary of the polytope represent?\n\n= what does this mean? \"diversity of expert policies, none of which are optimal along the reward vector\"", "limitations": "No concerns here", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Their premise, that safety constraints are difficult to engineer and should be task-agnostic, is accurate and relevant to this domain.\n\nTheir approach is a straightforward extension of an existing single-task ICL approach to multi-task. It works well and matches the intuition that safety constraints should be task-agnostic.\n\nRigorous math in section 3 to formalize their work.", "weaknesses": "In section 3, it's not clear what is prior work and what is their novel contribution.\n\nIn 3.1, the authors copied several sentences verbatim from \"Inverse Reinforcement Learning without Reinforcement Learning\" (https://arxiv.org/pdf/2303.14623.pdf, section 3.1). The use of copied phrases like \"we solve\" suggest the authors, themselves, did this work.\n\nIn their single-task and multi-task results, they don't compare to any baselines, for example \"Learning constraints from demonstrations\" from Chou et al.\n\nFor readability, it would be helpful if they numbered references.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for an incredibly detailed summary of our work which evinces a very thorough reading of our work. \n\nW1: Sec. 3.1 and 3.2 are standard algorithmic techniques. We would be happy to add something like “Prior Work” to the subsection titles.\n\nW2: If we understand correctly, the reviewer is referring to lines 102-104 and 111-115. These are standard sets of definitions (i.e. 102-104 is the definition of an MDP, 111-115 is the standard description of game-theoretic IRL). Given we are describing prior work in this section, we chose to stick to the notation / assumptions (e.g. convexity / compactness of strategy spaces) that is common in the literature. We also note that there are differences (e.g. we search over stationary policies) from the particular paper the reviewer mentions.\n\nW3. Our lack of comparisons to baselines in single-task ICL is because there are limited prior methods for this problem that scale to continuous control tasks (e.g. McPherson et al. assume we can perform value iteration on a tabular MDP). We compare to several baselines in the multi-task setting (e.g. the average / max of the learned constraints) that show that using multi-task data is critical for successful constraint learning. See the global response for the connection to Chou et al.\n\nW4: Sure!\n\nQ1:  We apologize for any confusion: our methods definitely do not require a DAgger-style interactive expert.  We believe the reviewer is referring to terms like those that appear in Equation (12). This is mostly a notational issue: given access to trajectories from some $\\pi$, we can evaluate the value of the policy $J(\\pi, r)$ for an arbitrary reward function $r$ without query-access to the policy by simply re-labeling the demonstrations with the reward function of interest . $\\rho_{\\pi_E}$, the visitation distribution of the expert, once again does not actually require query access to the expert policy, so we can evaluate the expression in Equation (13).\n\nQ2: This is an interesting point that gets at the difference between rewards and constraints. We usually think of a reward term as something with a fixed weight across tasks that gently shapes behavior. In contrast, a constraint is a term in your cost function that can be arbitrarily scaled up until the learner is no longer violating it. We would think that conventions that can be sometimes violated (e.g. setting the plate upside down to let the bottom dry off) to fit more under the purview of rewards. That being said, it is an interesting question as to whether our method would end up learning conventions.\n\nQ3: In the inverse reinforcement learning literature, access to demonstrations and the environment is standard. However, once we move to the space of constraints, we require an additional piece of information: the expert’s reward. Intuitively, the reason knowledge of the reward function is important is that it allows us to distinguish between an action not being taken because it was a) unsafe and b) suboptimal. Without this piece of information, we would have to assume that *all* untaken actions were unsafe and would therefore likely recover an overly conservative constraint. We note that *all* prior published work on ICL that we are aware of also makes this assumption.\n\nQ4: Your understanding of what we meant by “Pareto dominate” is entirely correct. When thinking about learning constraints, one could either a) attempt to recover the ground-truth constraint or b) try to learn a constraint that allows the learner to act safely / performantly. As we discuss in Sec. 3.5, the former can be an unreasonably challenging goal for realistic problems as it requires a high diversity and number of expert policies.\n\nHowever, this is not a problem unique to ICL: it also shows up in inverse reinforcement learning. In IRL, there can be multiple reward functions that make the expert policy optimal (e.g. a constant reward of zero). Thus, we try to learn a reward function where, if we computed the optimal policy under it, we would be guaranteed to have a bounded sub-optimality with respect to the expert policy under the ground-truth reward function. In essence, this is what the game-solving procedure guarantees us.\n\nIn the ICL setting, we instead want to learn a constraint that explains the expert behavior (e.g. one that forbids highly rewarding but untaken behavior). The guarantees we get (that we learn a constraint that allows our learner to act safely / performantly) are analogous to the IRL guarantees and are in essence the strongest we could hope for without restrictive assumptions. This is part of what distinguishes our analysis from that of Chou et al., who focus on constraint recovery guarantees.\n\nQ5: We would be more than happy to add more description of this figure. At a high level, $\\rho_{\\Pi}$ refers to the space of occupancy measures (i.e. the set of state-action distributions of all policies in our policy class). The reason we chose this representation was that a constraint is simply a hyperplane in this space (the red line labeled $\\langle \\rho_{\\pi}, c^* \\rangle = 0$). To uniquely determine a line in $\\mathbb{R}^d$, we need $d$ points on the line – these are what the green dots / expert policies $\\pi_E^1$ and $\\pi_E^2$ correspond to. We need two policies as our visitation distributions are two-dimensional (e.g. a two-state MDP). The technical conditions in Lemma 3.2 are the more formal version of the above statements (e.g. relint meaning the points are on the line rather than on the boundary of the space). Each corner of the polytope represents a policy, while the boundaries represent a convex combination of two policies. “Diversity of expert policies” means distinct policies and “not optimal along the reward vector” refers to their being a policy that performs better than the expert under the reward function (otherwise, we don’t need to solve a constrained problem at all and therefore the expert policy is not useful for extracting a constraint)."}, {"review_id": "RMuTkNEhAG", "reviewer": "Reviewer_3RKE", "summary": "This paper proposes to use inverse constraint learning (ICL) in multi-task scenarios. It uses game solving to describe the ICL problem, and then illustrates the learning algorithm under the single-task and multi-task problems. The authors give a theoretical analysis to explain under what conditions the learned reward can be generalized. The author shows the correctness of the proposed algorithm in both single-task and multi-task control experiments.", "questions": "- Can the author supplement the experimental results on more complex robotics control tasks? For example, consider such a manipulation task as follow. We hope that the force or speed of the robot arm will not be too large while picking up an object. Can the algorithm learn such complex safety-constraints?\n\n- The authors' experiment in multi-task scenarios essentially improves the diversity of samples through randomization. We generally don't think this is a real multi-task RL scenario because all tasks are navigation. Can the author explore diverse but related manipulation tasks (like push and pull an object) in the open-source taskset (such as [meta-world](https://meta-world.github.io/)) to show the effectiveness of the proposed method?", "limitations": "It is good that the paper focuses on the safety problems of reinforcement learning for robotics. However I suggest that the authors use more realistic robotics tasks (at least in simulation) that really need to consider safety-constraint to verify the method. Regardless of the comparison with related works or the complexity of the task, the experiments in the paper are not solid enough to support good motivation and theoretical results.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation of this paper is clear and reasonable to extract common safety-constraint from multi-task.\n- The paper considers whether the learned reward and constraint can be generalized theoretically, and gives two conditions and related proofs.", "weaknesses": "- The paper did not give a comparison with the ICL algorithms mentioned in the related work, and did not show the advantages of the algorithm in both learning or safety concerns.\n- The experimental settings of the paper is too simple, and only consider two constraints that do not require complex approachs to learn. It makes the machine learning method shows no advantages on human priors.\n- The authors did not give the results of robotics control tasks (both in simulation and real-world) that really need safety-constraints, such as robotic manipulation with human.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their appreciation of our theoretically-grounded approach. Responding to the concerns raised:\n\nW1: Please see our global response for the relationship between our work and that of Chou et al.\n\nW2: We’re not exactly sure what a “human prior” is here, could the reviewer clarify?\n\nW3/Q1/Q2: If we understand correctly, all three of these bullet points are in essence asking the same question: “what evidence do you have that the proposed method can scale to tasks like manipulation around people?\n\n**We would ask the reviewer to first read our global response on experiment complexity.**\n\nAt a high level, our focus in this paper is not robotic manipulation specifically, but on learning with constraints more abstractly. Thus, we would argue that a lack of experiments in this particular domain is not a sufficient reason to reject our paper.\n\nAs we point out in our global response, much of the prior work in inverse constraint learning focuses on tabular problems, while most of the work in constrained reinforcement learning focuses on relatively low dimensional tasks. So, we would argue that compared to the prior art, we are considering strictly harder settings.\n\nWhile we agree that focusing on manipulation specific applications would be interesting, our current experiments are equally high-dimensional (e.g. for our ant experiments, the state-space is 27 dimensional and the action space is 8 dimensional, compared to a 7 DoF manipulator) and we show results where are able to recover workspace constraints. More explicitly, given that we can learn a position constraint for a high-dimensional ant agent, we should also be able to learn constraints on the end effector of a manipulator). Similarly, if we are able to handle data that comes from the agent navigating to different parts of the maze, placing a block in different locations / goals should not be that different. Hence we see no conceptual reason why the algorithm would not scale to manipulation."}, {"review_id": "GPDWXcHNju", "reviewer": "Reviewer_doVy", "summary": "The paper proposes a novel inverse constraint learning (ICL) approach that leverages constrained reinforcement learning (CRL) and a game-theoretic formulation. Specifically, the paper shows that the game-theoretic view of inverse reinforcement learning (IRL) naturally extends to ICL, by forming the Lagrangian of CRL. The resulting single-task ICL algorithm is shown to Pareto-dominate the expert policy in terms of its expected cumulative reward and the constraint violation. However, the recovered constraint may be too conservative and generalize poorly under a new task. To address this issue, the framework is extended to incorporate expert demonstrations under multiple tasks that share the same constraint.The paper shows that the multi-task ICL framework is expected to approximately Pareto-dominate the expert policies. Simulation results suggest the efficacy of the approach in continuous and discrete MDPs.", "questions": "* What is $c_i^k$ in Algorithm 3? Shouldn’t it be just $c_i$ since the constraint is shared across all the $K$ tasks?\n\n* The order of minimization and maximization is swapped between (6) and (7). Does the equality hold because of the standard minimax theorem in game theory? If so, are there any conditions imposed on the form of $J$ in order for the equality to hold?", "limitations": "* The paper assumes access to demonstrations of the optimal safe policy for a given task. While the availability of safe demonstrations is a reasonable assumption,optimal demonstrations may be too demanding to ask for a human demonstrator in real-world settings [2]. It would be quite interesting to see how the proposed ICL framework handles suboptimal demonstrations; in theory, the objective function of equation (4) does not seem to require the optimality of the expert under the given reward function $r$.\n\n[2] Xu, Haoran, Xianyuan Zhan, Honglei Yin, and Huiling Qin. \"Discriminator-weighted offline imitation learning from suboptimal demonstrations.\" In International Conference on Machine Learning, pp. 24725-24742. PMLR, 2022.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Unlike prior ICL methods, the proposed framework is built on the game-theoretic approach. This brings in multiple benefits: 1) the form of the reward function can remain general; 2) conditions on approximate Pareto dominance can be derived, which gives theoretical justification of the proposed approach; 3) the resulting algorithm is relatively straightforward to implement.\n\n* Figures 1 and 2 give a nice visual representation of the proposed approach. Algorithm 1, 2, and 3 are presented in a consistent and organized way that clarifies the differences among CRL, single-task ICL, and multi-task ICL.", "weaknesses": "* The most concerning point is that the paper provides no comparison to prior work in the experiment. Some important insights of the proposed approach is shared by the seminal work of [1]. Specifically, both approaches 1) observe that access to reward functions enables constraint learning by looking at the differences between safe demonstrations and unsafe optimal demonstrations; 2) can leverage safe demonstrations from multiple tasks. To strengthen the contribution of the paper, it should include qualitative and quantitative comparison to [1] in Section 4 (where applicable), to clarify the practical advantages of the proposed framework. This might involve evaluating the ICL algorithm in more realistic problems than the toy task of maze navigation, since the authors claim better scalability of their approach compared to [1] in realistic problems (in Section 2). In addition, the paper should give a more detailed description of [1] to better highlight the similarities and differences.\n\n* There are multiple places in the paper where a variable or an acronym is used without definition: 1) $c^*$ in Theorem 3.1; 2) “FTRL” in line 153. Also, some figures in Section 4 are confusing and hard to interpret. Specifically, I believe that the blue cells in Figure 5(b) and (c) represent start and goal cells. However, the same color is used in other sub-figures to represent the constraints. It is recommended to use different colors to represent the start and the goals so that readers can make sense of Figure 5 as a whole. For further points concerning the presentation of the paper, please see the questions below.\n\n[1] Chou, Glen, Dmitry Berenson, and Necmiye Ozay. \"Learning constraints from demonstrations with grid and parametric representations.\" The International Journal of Robotics Research 40, no. 10-11 (2021): 1255-1283.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer found our figures / algorithm descriptions clear and that our algorithm seemed straightforward to implement – we think this bodes well for its application to real-world problems. Responding to the concerts raised:\n\n\nW1: Please see our global response for the relationship between our work and that of Chou et al. We would be more than happy to add the above discussion to our work (perhaps in the appendix as it is rather lengthy?). Please also see our global response on experimental complexity.\n\nW2: $c^*$ is the ground-truth constraint, as defined on line 127 – that being said, we would be happy to add a reminder of this point to the theorem statement. FTRL stands for follow the regularized leader, a standard no-regret algorithm – we apologize for not explaining this acronym and would be happy to add in a description. While we did label the start/goal and maze squares differently, we can see why their shared color would be confusing and would be happy to use a different color.\n\nQ1: You are totally right, we will update that in our next draft.\n\nQ2: Yes, this is allowed because of the minimax theorem. In the setup of the problem, we assumed the conditions for Sion’s minimax theorem to hold (convex / compact strategy spaces). Even more simply, in the finite policy / constraint class setting, we are solving a matrix game for which von Neumann’s original minimax theorem holds.\n\nL1: This is an excellent point we didn’t fully appreciate until the reviewer pointed it out – thanks! We don’t actually require the expert to be the optimal policy, only that they are as safe as we want our agent to be. Our Thms. 3.1 / 3.5 clearly allows for the learner to return a policy that out-performs that of the expert. We would be more than happy to re-word any claims to the contrary in our current draft. Please see our global response for experimental evidence that our method is able to take advantage of suboptimal demonstrations.\n"}, {"review_id": "VqIEtOQAQr", "reviewer": "Reviewer_Ray8", "summary": "The authors propose a novel method for constraint learning from expert demonstrations by an optimal CRL policy. The idea behind the algorithm is to frame the problem as a zero-sum game between a policy player and a constraint player, using inspiration from a two-player zero-sum game expression of IRL and CRL. With the reward function available, expert demonstrations, and a class of potential constraints containing the ground truth constraint, the authors propose to minimize the cost to go subject to being at least as safe as the expert, i.e. to obtain a larger cost than the expert policy on all constraints in the set. \n\nThe authors derive no regret convergence bounds for the learned constraints and deal with degeneracy issues by extending the problem to the multi-task framework with asymptotic guarantees. They present simulation results on high dimensional robotics tasks.", "questions": "- How do the authors expect their solution to scale for systems where the state is much higher in dimension than the simple examples presented. For example do they think that their method could be a direction to learn constraints from sequences of images in applications such as autonomous driving or flight?\n\n- (Optional) out of curiosity, I am wondering if the authors have any insight on how much more challenging it would be to consider constraints in multi-agent environments?", "limitations": "- The authors mention the lack of real world experiments as a limitation, I could not agree more but would go even further in suggesting that the proposed simulations are insufficient to help readers fully grasp the potential of the method. The paper would warrant a higher rating should that aspect be improved.\n\n- I am awaiting the authors response to better understand whether the method would actually be practical to use for real life applications and robotics problems of interest.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Very sound paper, and very clean and ingenuous framing of the problem as a game. I really enjoyed reading the paper as the work elegantly composes concepts from IRL, CRL, GT and no regret learning. I also see it as being of big significance for a very important problem, that of learning constraints from demonstrations. Overall very solid work.", "weaknesses": "- Although the algorithmic contribution is very strong, it seems that the practical algorithm section of the paper is much weaker and feels rushed. The single task results seem very simplistic while the multi-task maze example is way too briefly discussed. There is one sentence describing the threshold after which correct recovery of the constraints occurs, an aspect that would deserve a more extensive presentation and analysis.\n- Along that line, figure 5 deserve to be reworked for clarity, it mixes too many things in together with little readability.\n- There is no discussion on the computation costs of running the algorithm and on its data requirements. The authors only mention succinctly that CRL is slower than RL (by how much?), but it would be useful to know how many demonstrations one needs for typical robotics scenarios and how much compute the recovery of constraints needs.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their kind words about our work. Responding in order to the concerns raised:\n\nW1: Because of the overly conservative constraints single-task ICL produces on some problems (e.g. a maze), our main goal with the single-task experiments was to show that this is not always the case, if one is able to restrict the class of constraints appropriately. Specifically, on both of the high-dimensional continuous control problems we consider, we are able to recover the ground-truth constraint. In response to your concern about the multi-task experiments, as we mention in the global response, we performed experiments on a more complex problem: recovering the walls of the ant-maze (rather than the point maze).\n\nW2: We agree and would be more than happy to split up Figure 5 into multiple plots, given an additional page of space.\n\nW3: While our sample complexity analysis does provide an answer to the question of how many tasks we might need to see to recover a good constraint, we agree that we could add more details about how many demonstrations we used for our practical experiments. For all experiments, we used 20 demos from the expert per task. Once we’ve optimized the policy, learning a constraint is just a classification problem and is therefore of limited computational expense. The computational complexity of constrained RL vs. regular RL depends a lot on how related the constraint and the reward function are to each other (as if they are highly correlated, we are likely to need more Lagrange multiplier updates / policy optimization steps). It is therefore difficult for us to make a global statement.\n\nQ1: We would expect that learning constraints in a high-dimensional input space like images would be fundamentally difficult (for all methods) as different views of the same scene could correspond to the same level of constraint violation. However, for problems like self-driving, it is common to perform policy learning on top of a fixed multi-modal scene encoder rather than end-to-end learning. We would expect our approaches to scale quite well to these relatively low-dimensional representations.\nQ2: In theory, it shouldn’t be any more challenging to have multi-agent environments. We were actually thinking of adding in some experiments of this flavor but, because solving multi-agent RL problems can be challenging in and of itself, chose not to as we did not want to introduce additional complexities.\n\nL 1/2: We believe our method could be easily applied on top of a pre-existing planning stack. One direction we are currently working with some collaborators on is the problem of learning costmaps for off-road driving. There, one could imagine using our method to, by comparing human drivers and the output of a standard planner, learn the set of obstacles in the scene the ATV should avoid. Of course, anything with real robots takes more than a week to implement, but we do believe our method is applicable to these sorts of problems.\n"}]

Author rebuttal:
We thank all reviewers for their carefully considered feedback.

**Limited Experiments:**

We would like to begin by noting that, in comparison to many of the standard benchmarks in safe RL, our considered tasks are higher dimensional (e.g. our ant-based tasks are higher dimensional than every task in the standard Safety Gym benchmark) and we are successfully solving them without knowing the ground-truth constraint. Furthermore, much of the prior work in ICL has performed experiments purely on gridword problems [Vazquez-Chanlatte et al., Scobee and Shastry et al., McPherson et al.] or used simple constraint function classes like axis-aligned rectangles [Chou et al.], while we perform experiments with deep neural networks on continuous control problems.

That being said, we agree with the reviews that it would only make our paper stronger if our multi-task experiments were of same dimensionality as our single-task experiments and therefore implemented our algorithm on the D4RL AntMaze task (rather than the PointMaze task we included in our initial draft). On this significantly harder problem, we were able to recover all of the walls correctly (within a single iteration) and therefore learn policies that match expert performance and safety. We also performed multiple iterations of ICL to show that our learned constraint is stable (and therefore does not require the early stopping that single-task ICL approaches need). We believe this is strong evidence that our method is able to scale to complex, high-dimensional, multi-task problems. Furthermore, we note that our multi-task method is the only method which correctly recovers the ground truth constraint compared to 
a variety of baselines (e.g. the max, mean, or individual single-task constraints) which are overly conservative, highlighting how our focus on the multi-task setup has produced a more practical method than the prior, single-task art. See Figures 5/6 in PDF for full results.

**Relationship to the work of Chou et al.:**

As we pointed out in our related work section, our problem setup is quite similar to the setup of Chou et al., in that we assume access to both safe demonstrations and task rewards. However, algorithmically, we believe our techniques are more general. Chou et al. use variants of hit-and-run sampling to generate low-cost, safe trajectories (a non-standard algorithm for non-convex control problems), while we make no assumptions about the particular constrained RL method one uses. This allows us to learn deep feedback control policies, rather than open-loop trajectories. Their “gridded” constraint learning methods require the use of integer / mixed-integer program solvers while their “parametric” constraint learning methods require restrictive constraint classes (e.g. “axis-aligned hyper-rectangles”) or solving mixed integer feasibility programs. In contrast, we only require the ability to solve a classification problem, making it clear how to learn deep constraint networks using our method. This is why we described our approach as being more “likely to scale to realistic problems” in our writeup: we can leverage flexible function approximators and modern deep reinforcement learning algorithms. As Chou et al. wrote in Sec. 8.4 of their paper, “To scale to high-dimensional constraint spaces, we assume a known, relatively simple parameterization, which is in general not the case for real constraints.” We also note that, given the nonstandard constrained policy optimization and constraint learning methods Chou et al. propose using and the fact that they did not release their code, it would be quite an undertaking for us to re-implement their methods on the problems we consider.

We are able to prove policy performance / safety guarantees for policies learned via our recovered constraint, while we were unable to find the analogous guarantees in the work of Chou et al. In essence, the theory of Chou et al. focuses on constraint recovery which, as we discuss in Sec 3.5, requires a very large number of diverse demonstrations, outside of problems with the relatively simple constraint structures they consider. On a more technical level, we do not require the restrictive Lipschitzness assumptions they make in their analysis section (which again reflects the fact that they are focused on the impossibly high goal of perfect constraint recovery). Also, in theory, Chou et al. require “boundedly suboptimal” demonstrations, while we, as a reviewer points out, do not.

In short, we believe that our writeup captures what is essential to the multi-task inverse constraint learning problem, and allows the reader to pick the particular set of solvers that are most effective on the particular instance they are interested in. Furthermore, we are also able to provide strong guarantees on the performance / safety of the policy optimized under the learned constraint, while Chou et al. seem to focus on a goal that is somewhat of a red herring on realistic problems.


**Expert Optimality Requirement:**

An insight we did not fully appreciate until it was brought to our attention by a reviewer was that nowhere in our formalism do we actually require the expert to be the (soft) optimal policy under the ground truth constraint, as all prior works we know of in the ICL domain [McPherson et al., Chou et al.] do. Of course, if the expert data is extremely suboptimal, it is hard for any method to infer a reasonable constraint. However, in contrast to prior work, our theory does not break down even in this case (as we focus on policy performance / safety rather than perfect constraint recovery) and still provides rigorous guarantees.

To empirically validate this point, we redid our single-task experiments using an expert that is far from optimal on the task. Using this data, we are able to learn a constraint that induces a policy that is as safe and significantly more performant than the expert. See Figures 7/8 PDF for full results.

Meta-review: The reviewers generally found the technical contribution strong, but had some concerns about the strength of the experimental evaluation, including the absence of comparison to prior work, absence of real-world experiments, and the limited complexity of the experimental settings. The authors addressed the former concern with additional experiments. The authors responded to the latter concern by pointing to prior work that solved problems of lower complexity and including additional experiments on a more challenging task that fit the studied setting of multi-task demonstrations, which showed their method performing well. The reviewers generally were satisfied by these responses.
Decision: Accept (poster)