Title: Learning Shared Safety Constraints from Multi-task Demonstrations
Venue: NeurIPS 2023
Accepted: yes
Keywords: constraints

inverse reinforcement learning

safe reinforcement learning
Abstract: Regardless of the particular task we want to perform in an environment, there are often shared safety constraints we want our agents to respect. For example, regardless of whether it is making a sandwich or clearing the table, a kitchen robot should not break a plate. Manually specifying such a constraint can be both time-consuming and error-prone. We show how to learn constraints from expert demonstrations of safe task completion by extending inverse reinforcement learning (IRL) techniques to the space of constraints. Intuitively, we learn constraints that forbid highly rewarding behavior that the expert could have taken but chose not to. Unfortunately, the constraint learning problem is rather ill-posed and typically leads to overly conservative constraints that forbid all behavior that the expert did not take. We counter this by leveraging diverse demonstrations that naturally occur in multi-task setting to learn a tighter set of constraints. We validate our method with simulation experiments on high-dimensional continuous control tasks.

Learning Shared Safety Constraints from Multi-task Demonstrations

Abstract

Regardless of the particular task we want them to perform in an environment, there are often shared safety constraints we want our agents to respect. For example, regardless of whether it is making a sandwich or clearing the table, a kitchen robot should not break a plate. Manually specifying such a constraint can be both time-consuming and error-prone. We show how to learn constraints from expert demonstrations of safe task completion by extending inverse reinforcement learning (IRL) techniques to the space of constraints. Intuitively, we learn constraints that forbid highly rewarding behavior that the expert could have taken but chose not to. Unfortunately, the constraint learning problem is rather ill-posed and typically leads to overly conservative constraints that forbid all behavior that the expert did not take. We counter this by leveraging diverse demonstrations that naturally occur in multi-task settings to learn a tighter set of constraints. We validate our method with simulation experiments on high-dimensional continuous control tasks.

1 Introduction

If a friend was in your kitchen and you told them to 'make toast" or 'clean the dishes," you would probably be rather surprised if they broke some of your plates during this process. The underlying safety constraint that forbids these kinds of behavior is both a) implicit and b) agnostic to the particular task they were asked to perform. Now, let's bring a household robot into the equation, operating within your kitchen. How can we ensure that it adheres to these implicit safety constraints, regardless of its assigned tasks?

One approach might be to write down specific constraints (e.g. joint torque limits) and pass them to the decision-making system of the robot. Unfortunately, more complex constraints like the ones we consider above are both difficult to formalize mathematically and easy for an end-user to forget to specify (as they would be inherently understood by a human helper). This problem is paralleled in the field of reinforcement learning (RL), where defining reward functions that lead to desirable behaviors for the learning agent is a recurring challenge [Hadfield-Menell et al., 2017]. For example, it is rather challenging to handcraft the exact function one should be optimized to be a good driver. The standard solution to this sort of 'reward design" problem is to instead demonstrate the desired behavior of the agent and then extract a reward function that would incentivize such behavior. Such inverse reinforcement learning (IRL) techniques have found application in fields as diverse as robotics [Silver et al., 2010, Ratliff et al., 2009, Kolter et al., 2008, Ng et al., 2006, Zucker et al., 2011], computer vision [Kitani et al., 2012], and human-computer interaction [Ziebart et al., 2008b, 2012]. Given the success of IRL techniques and the similarity between reward and constraint design, we propose

extending IRL techniques to the space of constraints. We term such techniques inverse constraint learning , or ICL for short.

More formally, we consider a setting in which we have access to demonstrations of an expert policy for a task, along with knowledge about the task's reward. This allows us to look at the difference between the expert and reward-optimal policies for a task. Our first key insight is that the actions taken by the reward-optimal but not the expert policy are likely to be forbidden, allowing us to extract a constraint.

Unfortunately, the ICL problem is still rather ill-posed. Indeed, prior work in ICL will often learn overly conservative constraints that forbid all behavior the expert did not take [Scobee and Sastry, 2019, Vazquez-Chanlatte et al., 2018, McPherson et al., 2021]. However, for tasks in a shared environment with different rewards, there are often safety constraints that should be satisfied regardless of the task (e.g. a plate shouldn't be broken regardless of whether you're serving food on it or cleaning up after a meal). Our second crucial insight is that we can leverage multi-task data to provide more comprehensive demonstration coverage over the state space, helping our method avoid degenerate solutions.

More explicitly, the contributions of our work are three-fold.

We formalize the inverse constraint learning problem. We frame ICL as a zero-sum game between a policy player and a constraint player. The policy player attempts to maximize reward while satisfying a potential constraint, while the constraint player picks constraints that maximally penalize the learner relative to the expert. Intuitively, such a procedure recovers constraints that forbid high-reward behavior the expert did not take.

We develop a multi-task extension of inverse constraint learning. We derive a zero-sum game between a set of policy players, each attempting to maximize a task-specific reward, and a constraint player that chooses a constraint that all policy players must satisfy. Because the constraint player looks at aggregate learner and expert data, it is less likely to select a degenerate solution.

We demonstrate the efficacy of our approach on various continuous control tasks. We show that with restricted function classes, we are able to recover ground-truth constraints on certain tasks. Even when using less interpretable function classes like deep networks, we can still ensure a match with expert safety and task performance. In the multi-task setting, we are able to identify constraints that a single-task learner would struggle to learn.

We begin with a discussion of related work.

2 Related Work

Inverse Reinforcement Learning. IRL [Ziebart et al., 2008a,b, 2012, Ho and Ermon, 2016] can be framed as a two-player zero-sum game between a policy player and a reward player [Swamy et al., 2021]. In most formulations of IRL, a potential reward function is chosen in an outer loop, and the policy player optimizes it via RL in an inner loop. Similar to IRL, the constraint in our formulation of ICL is chosen adversarially in an outer loop. However, in contrast to IRL, the inner loop of ICL is constrained reinforcement learning: the policy player tries to find the optimal policy that respects the constraint chosen in the outer loop.

Constrained Reinforcement Learning. Our approach involves repeated calls to a constrained reinforcement learning (CRL) oracle [Garcıa and Fernández, 2015, Gu et al., 2022]. CRL aims to find a reward-maximizing policy over a constrained set, often formulated as a constrained policy optimization problem [Altman, 1999, Xu et al., 2022]. Solving this problem via Frank-Wolfe methods is often unstable [Ray et al., 2019, Liang et al., 2018]. Various methods have been proposed to mitigate this instability, including variational techniques [Liu et al., 2022], imposing trust-region regularization [Achiam et al., 2017, Yang et al., 2020, Kim and Oh, 2022], optimistic game-solving algorithms [Moskovitz et al., 2023], and PID controller-based methods [Stooke et al., 2020]. In our practical implementations, we use PID-based methods for their relative simplicity.

Multi-task Inverse Reinforcement Learning. Prior work in IRL has considered incorporating multi-task data [Xu et al., 2019, Yu et al., 2019, Gleave and Habryka, 2018]. We instead consider a

setting in which we know task-specific rewards and are attempting to recover a shared component of the demonstrator's objective. Amin et al. [2017] consider a similar setting but require the agent to be able to actively choose tasks or interactively query the expert, while our approach requires neither.

Inverse Constraint Learning. We are far from the first to consider the ICL problem. Scobee and Sastry [2019], McPherson et al. [2021] extend the MaxEnt IRL algorithm of Ziebart et al. [2008a] to the ICL setting. We instead build upon the moment-matching framework of Swamy et al. [2021], allowing our theory to handle general reward functions instead of the linear reward functions MaxEnt IRL assumes. We are also able to provide performance and constraint satisfaction guarantees on the learned policy, unlike the aforementioned work. Furthermore, we consider the multi-task setting, addressing a key shortcoming of the prior work.

Perhaps the most similar paper to ours is the excellent work of Chou et al. [2020], who also consider the multi-task ICL setting but propose a solution that requires several special solvers that depend on knowledge of the parametric family that a constraint falls into. In contrast, we provide a general algorithmic template that allows one to apply whatever flexible function approximators (e.g. deep networks) and reinforcement learning algorithms (e.g. PPO) they desire. Chou et al.'s method also requires sampling uniformly over the set of trajectories that achieve a higher reward than the expert, a task which is rather challenging to do on high-dimensional problems. In contrast, our method only requires the ability to solve a standard RL problem. Theoretically, Chou et al. [2020] focus on constraint recovery, which we argue below is a goal that requires strong assumptions and is therefore a red herring on realistic problems. This focus also prevents their theory from handling suboptimal experts. In contrast, we are able to provide rigorous guarantees on learned policy performance and safety, even when the expert is suboptimal. We include results in Appendix C that show that our approach is far more performant.

In concurrent work, Lindner et al. [2023] propose an elegant solution approach to ICL: rather than learning a constraint function, assume that any unseen behavior is unsafe and enforce constraints on the learner to play a convex combination of the demonstrated safe trajectories. The key benefit of this approach is that it doesn't require knowing the reward function the expert was optimizing. However, by forcing the learner to simply replay previous expert behavior, the learner cannot meaningfully generalize, and might therefore be extremely suboptimal on any new task. In contrast, we use the side information of a reasonable set of constraints to provide rigorous policy performance guarantees. 2

We now turn our attention to formalizing inverse constraint learning.

3 Formalizing Inverse Constraint Learning

We build up to our full method in several steps. We first describe the foundational algorithmic structures we build upon (inverse reinforcement learning and constrained reinforcement learning). We then describe the single-task formulation before generalizing it to the multi-task setup.

We consider a finite-horizon Markov Decision Process (MDP) [Puterman, 2014] parameterized by ⟨S , A , T , r, T ⟩ where S , A are the state and action spaces, T : S × A → ∆( S ) is the transition operator, r : S × A → [ -1 , 1] is the reward function, and T is the horizon.

3.1 Prior Work: Inverse RL as Game Solving

In the inverse RL setup, we are given access trajectories generated by an expert policy π E : S → ∆( A ) , but do not know the reward function of the MDP. Our goal is to nevertheless learn a policy that performs as well as the expert's, no matter the true reward function.

We solve the IRL problem via equilibrium computation between a policy player and an adversary that tries to pick out differences between expert and learner policies under potential reward functions Swamy et al. [2021]. More formally, we optimize over polices π : S → ∆( A ) ∈ Π and reward functions f : S × A → [ -1 , 1] ∈ F r . For simplicity, we assume that our strategy spaces ( Π and F r ) are convex and compact and that r ∈ F r , π E ∈ Π . We solve (i.e. compute an approximate Nash

Algorithm 1 CRL (Constrained Reinforcement Learning)

Reward r , constraint c , learning rates η , tolerance δ

Algorithm 2 ICL (Inverse Constraint Learning)

Return best of c 1: N on validation data.

equilibrium) of the two-player zero-sum game

where J ( π, f ) = E ξ ∼ π [ ∑ T t =0 f ( s t , a t )] denotes the value of policy π under reward function f .

3.2 Prior Work: Constrained Reinforcement Learning as Game Solving

In CRL, we are given access to both the reward function and a constraint c : S × A → [ -1 , 1] . Our goal is to learn the highest reward policy that, over the horizon, has a low expected value under the constraint. More formally, we seek a solution to the optimization problem:

where δ is some error tolerance. We can also formulate CRL as a game via forming the Lagrangian of the above optimization problem [Altman, 1999]:

Intuitively, the adversary updates the weight of the constraint term in the policy player's reward function based on how in violation the learner is.

3.3 Single-Task Inverse Constraint Learning

We are finally ready to formalize ICL. In ICL, we are given access to the reward function, trajectories from the solution to a CRL problem, and a class of potential constraints F c in which we assume the ground-truth constraint c ∗ lies. We assume that F c is convex and compact.

In the IRL setup, without strong assumptions on the dynamics of the underlying MDP and expert, it is impossible to guarantee recovery of the ground-truth reward. Often, the only reward function that actually makes the expert optimal is zero everywhere [Abbeel and Ng, 2004]. Instead, we attempt to find the reward function that maximally distinguishes the expert from an arbitrary other policy in our policy class via game-solving [Ziebart et al., 2008a, Ho and Ermon, 2016, Swamy et al., 2021]. Similarly, for ICL, exact constraint recovery can be challenging. For example, if two constraints differ only on states the expert never visits, it is not clear how to break ties. We instead try to find a constraint that best separates the safe (but not necessarily optimal) π E from policies that achieve higher rewards.

ICL

minimize

over c

More formally, we seek to solve the following constrained optimization problem.

Note that in contrast to the moment-matching problem we solve in imitation learning [Swamy et al., 2021], we instead want to be at least as safe as the expert. This means that rather than having equality constraints, we have inequality constraints. Continuing, we can form the Lagrangian:

Notice that the form of the ICL game resembles a combination of the IRL and CRL games. We describe the full game-solving procedure in Algorithm 2, where R ( c ) is an arbitrary strongly convex regularizer [McMahan, 2011]. Effectively, we pick a constraint function in the same way we pick a reward function in IRL but run a CRL inner loop instead of an RL step. Instead of a fixed constraint threshold, we set tolerance δ to the expert's constraint violation. Define

as the per-round loss that the constraint player suffers in their online decision problem. The best-inhindsight comparator constraint is defined as

We can then define the cumulative regret the learner suffers as

and let ϵ i = ℓ i (ˆ c ) -ℓ i ( c i ) . We prove the following theorem via standard machinery.

Theorem 3.1. Let c 1: N be the iterates produced by Algorithm 2 and let ¯ ϵ = 1 N ∑ N i ϵ i denote their time-averaged regret. Then, there exists a c ∈ c 1: N such that π = CRL ( r, c, δ = J ( π E , c )) satisfies

In words, by optimizing under the recovered constraint, we can learn a policy that (weakly) Paretodominates the expert policy under c ∗ . We conclude by noting that because FTRL (Follow the Regularized Leader, McMahan [2011]) is a no-regret algorithm for linear losses like (8), we have that lim T →∞ Reg ( T ) T = 0 . This means that with enough iterations, the RHS of the above bound on ground-truth constraint violation will go to 0.

3.4 Multi-task Inverse Constraint Learning

One of the potential failure modes of the single-task approach we outline above is that we could learn an overly conservative constraint, leading to poor task performance [Liu et al., 2023]. For example, imagine that we entropy-regularize our policy optimization [Ziebart et al., 2008a, Haarnoja et al., 2018], as is common practice. Assuming a full policy class, the learner puts nonzero probability mass on all reachable states in the MDP. The constraint player is therefore incentivized to forbid all states the expert did not visit [Scobee and Sastry, 2019, McPherson et al., 2021]. Such a constraint would likely generalize poorly when combined with a new reward function ( ˜ r = r ) as it forbids all untaken rather than just unsafe behavior.

̸

At heart, the issue with the single-task formulation lies in the potential for insufficient coverage of the state space within expert demonstrations. Therefore, it is natural to explore a multi-task extension to counteract this limitation. Let each task be defined by a unique reward. We assume the dynamics and safety constraints are consistent across tasks. We observe K samples of the form ( r k , { ξ ∼ π k E } ) . This data allows us to define the multi-task variant of our previously described ICL game:

We describe how we solve this game in Algorithm 3, where R ( c ) is an arbitrary strongly convex regularizer [McMahan, 2011]. In short, we alternate between solving K CRL problems and updating the constraint based on the data from all policies.

We now give two conditions under which generalization to new reward functions is possible.

3.5 A (Strong) Geometric Condition for Identifiability

Consider for a moment the linear programming (LP) formulation of reinforcement learning. We search over the space of occupancy measures ( ρ π ∈ ∆( S × A ) ) that satisfy the set of Bellman flow constraints [Sutton and Barto, 2018] and try to maximize the inner product with reward vector r ∈ R |S||A| . We can write the CRL optimization problem (assuming δ = 0 for simplicity) as an LP

as well. Using ρ Π to denote the occupancy measures of all π ∈ Π ,

We observe a (for simplicity, optimal) solution to such a problem for K rewards, begging the question of when that is enough to uniquely identify c ∗ . Recall that to uniquely determine the equation of a hyperplane in R d , we need d linearly independent points. c ∗ ∈ R |S||A| , so we need |S||A| expert policies. Furthermore, we need each of these points to lie on the constraint line and not on the boundary of the full polytope. Put differently, we need each distinct expert policy to saturate the underlying constraint (i.e. ∃ π ∈ Π s.t. J ( π k E , r k ) < J ( π k , r k ) ). Under these conditions, we can uniquely determine the hyperplane that separates safe from unsafe policies, to which the constraint vector is orthogonal. More formally,

Lemma 3.2. Let π 1: |S||A| E be distinct optimal expert policies such that a) ∀ i ∈ [ |S||A| ] , π i E ∈ relint ( ρ Π ) and b) no ρ π i E can be generated by a mixture of the other visitation distributions. Then, c ∗ is the unique (up to scaling) nonzero vector in

We visualize this process for the |S||A| = 2 case in Fig. 2. Assuming we are able to recover c ∗ , we can guarantee that our learners will be able to act safely, regardless of the task they are asked to do. However, the assumptions required to do so are quite strong: we are effectively asking for our expert policies to form a basis for the space of occupancy measures, which means we must see expert data for a large set of diverse tasks. Furthermore, we need the experts to be reward-optimal.

Identifiability (the goal of prior works like Chou et al. [2020], Amin et al. [2017]) is too strong a goal as it requires us to estimate the value of the constraint everywhere in the state-action space. If we know the learner will only be incentivized to go to a certain subset of states (as is often true in practice), we can guarantee safety without fully identifying c ∗ . Therefore, we now consider how, by making distributional assumptions on how tasks are generated, we can generalize to novel tasks.

3.6 A Statistical Condition for Generalization

Assume that tasks τ are drawn i.i.d. from some P ( τ ) . Then, even if we do not see a wide enough diversity of expert policies to guarantee identifiability of the ground-truth constraint function, with enough samples, we can ensure we do well in expectation over tasks. For some constraint c , let us define where λ τ , π τ = CRL ( r τ , c ) denote the solutions to the inner optimization problem. We begin by proving the following lemma.

Lemma 3.3. With

samples, we have that with probability ≥ 1 -δ , we will be able to estimate all |F c | population estimates of V ( c ) within ϵ absolute error.

Note that we perform the above analysis for finite classes but one could easily extend it [Sriperumbudur et al., 2009]. The takeaway from the above lemma is that if we observe a sufficient number of tasks, we can guarantee that we can estimate the population loss of all constraints, up to some tolerance.

Consider the learner being faced with a new task they have never seen before at test time. Unlike in the single task case, where it is clear how to set the cost limit passed to CRL, it is not clear how to do so for a novel task. Hence, we make the following assumption.

Assumption 3.4. We assume that E τ [ J ( π τ E , c ∗ )] ≤ 0 , and that ∀ c ∈ F c , ∃ π ∈ Π s.t. J ( π, c ) ≤ 0 .

This (weak) assumption allows us to a) use a cost limit of 0 for our CRL step and b) search over a subset of F c that the expert is safe under. Under this assumption, we are able to prove the following:

Theorem 3.5. Let c 1: N be the iterates produced by Algorithm 3 with K ( ϵ, δ ) chosen as in Lemma 3.3 and let ¯ ϵ = 1 N ∑ N i ϵ i denote their time-averaged regret. Then, w.p. ≥ 1 -δ , there exists a c ∈ c 1: N such that π ( r ) = CRL ( r, c, δ = 0) satisfies

In short, if we observe enough tasks, we are able to learn a constraint that, when optimized under, leads to policies that approximately Pareto-dominate those of the experts on average.

We now turn our attention to the practical implementation of these algorithms.

4 Practical Algorithm

We provide practical implementations of constrained reinforcement learning and inverse constraint learning and benchmark their performance on several continuous control tasks. We first describe the environments we test our algorithms on. Then, we provide results showing that our algorithms learn policies that match expert performance and constraint violation. While it is hard to guarantee constraint recovery in theory, we show that we can recover the ground-truth constraint empirically if we search over a restricted enough function class.

4.1 Tasks

We focus on the ant environment from the PyBullet [Coumans and Bai, 2016] and MuJoCo [Todorov et al., 2012] benchmarks. The default reward function incentivizes progress along the positive x direction. For our single-task experiments, we consider a velocity and position constraint on top of this reward function.

Velocity Constraint: ∥ q t +1 -q t ∥ 2 dt ≤ 0 . 75 where q t is the ant's position

Position Constraint: 0 . 5 x t -y t ≤ 0 where x t , y t are the ant's coordinates

For our multi-task experiments, we build upon the D4RL [Fu et al., 2020] AntMaze benchmark. The default reward function incentivizes the agent to navigate a fixed maze to a random goal position: exp( -∥ q goal -q t ∥ 2 ) . We modify this environment such that the walls of the maze are permeable, but the agent incurs a unit step-wise cost for passing through the maze walls.

Our expert policies are generated by running CRL with the ground-truth constraint. We use the Tianshou [Weng et al., 2022] implementation of PPO [Schulman et al., 2017] as our baseline policy optimizer. Classical Lagrangian methods exactly follow the gradient update shown in Algorithm 1, but they are susceptible to oscillating learning dynamics and constraint-violating behavior during training. The PID Lagrangian method [Stooke et al., 2020] extends the naive gradient update of λ i with a proportional and derivative term to dampen oscillations and prevent cost overshooting. To reduce the amount of interaction required to solve the inner optimization problem, we warm-start our policy in each iteration by behavior cloning against the given expert demonstrations. We used a single NVIDIA 3090 GPU for all experiments. Due to space constraints, we defer all other implementation details to Appendix B.

4.2 ICL Results

We begin with results for the single-task problem, before continuing on to the multi-task setup.

4.3 Single-Task Continuous Control Results

As argued above, we expect a proper ICL implementation to learn policies that perform as well and are as safe as the expert. However, by restricting the class of constraints we consider, we can also investigate whether recovery of the ground-truth constraint is possible. To this end, we consider a reduced-state version of our algorithm where the learned constraint takes a subset of the agent state as input. For the velocity constraint, the learned constraint is a linear function of the velocity, while for the position and maze constraints, the learned constraint is a linear function of the ant's position.

Using this constraint representation allows us to visualize the learned constraint over the course of training, as shown in Figure 3. We find that our ICL implementation is able to recover the constraint,

0

5

10

15

ICL

Iteration

0.5

0.6

0.7

0.8

Velocity

Velocity Constraint

c

i

c

∗

0

5

10

15

ICL

Iteration

1200

1400

1600

J

(

·

,

r

)

Velocity Constraint

π

π

E

0

5

10

15

ICL

Iteration

-

200

0

200

J

(

·

,

c

∗

)

Velocity Constraint

π

π

E

0

2

4

6

8

ICL

Iteration

0.46

0.48

0.50

Slope

Position Constraint

c

i

c

∗

0

2

4

6

8

ICL

Iteration

1250

1500

1750

2000

2250

J

(

·

,

r

)

Position Constraint

π

π

E

0

2

4

6

8

ICL

Iteration

-

400

-

200

0

200

J

(

·

,

c

∗

)

Position Constraint

π

π

E

as the learned constraint for both the velocity and position tasks converges to the ground-truth value. Our results further show that over the course of ICL training, the learned policies match and exceed expert performance as their violations of the ground-truth constraint converge towards the expert's. Figure 4 provides a direct depiction of the evolution of the learned constraint and policy. The convergence of the red and blue lines shows that the learned position constraint approaches the ground truth, and the policy's behavior approaches that of the expert in response to this.

0

2

4

6

8

ICL

Iteration

0.46

0.48

0.50

0.52

Slope

Position Constraint

c

i

c

∗

0

2

4

6

8

ICL

Iteration

1600

1800

2000

J

(

·

,

r

)

Position Constraint

π

π

E

0

2

4

6

8

ICL

Iteration

-

750

-

500

-

250

0

J

(

·

,

c

∗

)

Position Constraint

π

π

E

0

5

10

15

ICL

Iteration

0.2

0.4

0.6

0.8

Velocity

Velocity Constraint

c

i

c

∗

0

5

10

15

ICL

Iteration

500

1000

1500

J

(

·

,

r

)

Velocity Constraint

π

π

E

0

5

10

15

ICL

Iteration

-

600

-

400

-

200

0

200

J

(

·

,

c

∗

)

Velocity Constraint

π

π

E

To measure the robustness of our method to sub-optimal experts, we repeat the above experiments using demonstrations from an expert with i.i.d. Gaussian noise added to their actions at each timestep. We are still able to learn a safe and performant policy, which matches what our theory predicts.

4.4 Multi-Task Continuous Control Results

We next consider an environment where, even with an appropriate constraint class, recovering the ground-truth constraint with a single task isn't feasible due to the ill-posedness of the inverse constraint learning problem. Specifically, we use the umaze AntMaze environment from D4RL [Fu et al., 2020], modified to have a more complex maze structure. As seen in Figure 7, the goal of each task is to navigate through the maze from one of the starting positions (top/bottom left) to one of the grid cells in the rightmost column. We provide expert data for all 10 tasks to the learner.

As we can see in Figure 6, multi-task ICL is, within a single iteration, able to learn policies that match expert performance and constraint violation across all tasks, all without ever interacting with the ground-truth maze . Over time, we are able to approximately recover the entire maze structure.

We visually compare several alternative strategies for using the multi-task demonstration data in the bottom row of Figure 7. The 0/1 values in the cells correspond to querying the deep constraint network from the last iteration of ICL on points from each of the grid cells and thresholding at some confidence. We see that a single-task network (d) learns spurious walls that would prevent the learner from completing more than half of the tasks. Furthermore, learning 10 separate classifiers and then aggregating their outputs (e) / (f) also fails to produce reasonable outputs. However, when we use data from all 10 tasks to train our multi-task constraint network (g) / (h) , we are able to approximately recover the walls of the maze. These results echo our preceding theoretical argument about the importance of multi-task data for learning constraints that generalize to future tasks.

0

1

2

3

4

ICL

Iteration

0.90

0.95

1.00

Maze IoU

c

i

c

∗

0

1

2

3

4

ICL

Iteration

14.0

14.5

15.0

15.5

J

(

·

,

r

)

π

π

E

0

1

2

3

4

ICL

Iteration

0

100

200

300

J

(

·

,

c

∗

)

π

π

E

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

1

1

1

0

0

0

1

1

1

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

Ground-Truth Constraint

(a)

S

G

S

Single-Task ICL Setup

(b)

S

G

G

G

G

G

G

G

G

G

S

G

Multi-Task ICL Setup

(c)

0

0

1

1

1

1

1

1

1

1

0

0

1

1

1

1

1

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

1

1

1

1

0

0

1

1

1

0

1

1

1

1

0

0

0

0

0

0

1

1

1

1

0

0

0

0

1

1

1

1

1

1

0

0

0

0

0

0

1

1

1

1

0

0

0

0

0

0

1

1

1

1

Single-Task ICL Constraint

(d)

0

0

1

1

1

1

1

1

1

1

0

0

1

1

1

1

1

1

1

1

0

0

1

1

1

1

0

0

1

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

1

1

1

0

0

0

1

1

1

0

1

1

1

0

0

0

0

0

0

0

1

1

1

1

0

0

0

0

0

0

1

1

1

1

0

0

0

0

0

0

1

1

1

1

0

0

0

0

0

0

1

1

1

1

Average of Single-Task ICL Constraints

(e)

0

0

1

1

1

1

1

1

1

1

0

0

1

1

1

1

1

1

1

1

1

0

1

1

1

1

1

1

1

1

1

0

1

1

1

0

0

0

0

1

1

0

1

1

1

1

1

1

1

1

1

0

1

1

1

0

1

1

1

1

0

0

0

0

0

0

1

1

1

1

0

0

0

0

1

1

1

1

1

1

0

0

0

0

0

1

1

1

1

1

0

0

0

1

1

1

1

1

1

1

Max of Single-Task ICL Constraints

(f)

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

1

1

1

0

0

0

1

1

1

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

0

0

0

0

0

0

1

1

1

0

Multi-Task ICL Constraint

(g)

(h)

We release the code we used for all of our experiments at https://github.com/konwook/mticl .

5 Discussion

In this work, we derive an algorithm for learning safety constraints from multi-task demonstrations. We show that by replacing the inner loop of inverse reinforcement learning with a constrained policy optimization subroutine, we can learn constraints that guarantee learner safety on a single task. We then give statistical and geometric conditions under which we can guarantee safety on unseen tasks by planning under a learned constraint. We validate our approach on several control tasks.

Limitations. In the future, we would be interested in applying our approach to real-world problems (e.g. offroad driving). Algorithmically, the CRL inner loop can be more computationally expensive than an RL loop - we would be interested in speeding up CRL using expert demonstrations, perhaps by adopting the approach of Swamy et al. [2023]. We also ignore all finite-sample issues, which could potentially be addressed via data-augmentation approaches like that of Swamy et al. [2022].

References

Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 1, 2004.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

Eitan Altman. Constrained Markov decision processes , volume 7. CRC press, 1999.

Kareem Amin, Nan Jiang, and Satinder Singh. Repeated inverse reinforcement learning. Advances in neural information processing systems , 30, 2017.

Glen Chou, Dmitry Berenson, and Necmiye Ozay. Learning constraints from demonstrations. In Algorithmic Foundations of Robotics XIII: Proceedings of the 13th Workshop on the Algorithmic Foundations of Robotics 13 , pages 228-245. Springer, 2020.

Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480, 2015.

Adam Gleave and Oliver Habryka. Multi-task maximum entropy inverse reinforcement learning. arXiv preprint arXiv:1805.08882 , 2018.

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330 , 2022.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems , 30, 2017.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning, 2016.

Dohyeong Kim and Songhwai Oh. Efficient off-policy safe reinforcement learning using trust region conditional value at risk. IEEE Robotics and Automation Letters , 7(3):7644-7651, 2022.

Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting. In European conference on computer vision , pages 201-214. Springer, 2012.

J Zico Kolter, Mike P Rodgers, and Andrew Y Ng. A control architecture for quadruped locomotion over rough terrain. In 2008 IEEE International Conference on Robotics and Automation , pages 811-818. IEEE, 2008.

Qingkai Liang, Fanyu Que, and Eytan Modiano. Accelerated primal-dual policy optimization for safe reinforcement learning. arXiv preprint arXiv:1802.06480 , 2018.

David Lindner, Xin Chen, Sebastian Tschiatschek, Katja Hofmann, and Andreas Krause. Learning safety constraints from demonstrations with unknown rewards. arXiv preprint arXiv:2305.16147 , 2023.

Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning , pages 13644-13668. PMLR, 2022.

Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. In International Conference on Learning Representations , 2023.

Brendan McMahan. Follow-the-regularized-leader and mirror descent: Equivalence theorems and l1 regularization. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 525-533. JMLR Workshop and Conference Proceedings, 2011.

David L McPherson, Kaylene C Stocking, and S Shankar Sastry. Maximum likelihood constraint inference from stochastic demonstrations. In 2021 IEEE Conference on Control Technology and Applications (CCTA) , pages 1208-1213. IEEE, 2021.

Ted Moskovitz, Brendan O'Donoghue, Vivek Veeriah, Sebastian Flennerhag, Satinder Singh, and Tom Zahavy. Reload: Reinforcement learning with optimistic ascent-descent for last-iterate convergence in constrained mdps. arXiv preprint arXiv:2302.01275 , 2023.

Andrew Y Ng, Adam Coates, Mark Diel, Varun Ganapathi, Jamie Schulte, Ben Tse, Eric Berger, and Eric Liang. Autonomous inverted helicopter flight via reinforcement learning. In Experimental robotics IX , pages 363-372. Springer, 2006.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Nathan D Ratliff, David Silver, and J Andrew Bagnell. Learning to search: Functional gradient techniques for imitation learning. Autonomous Robots , 27(1):25-53, 2009.

Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7, 2019.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Dexter RR Scobee and S Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. arXiv preprint arXiv:1909.05477 , 2019.

David Silver, J Andrew Bagnell, and Anthony Stentz. Learning from demonstration for autonomous navigation in complex unstructured terrain. The International Journal of Robotics Research , 29 (12):1565-1592, 2010.

Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG Lanckriet. On integral probability metrics, \ phi-divergences and binary classification. arXiv preprint arXiv:0901.2698 , 2009.

Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning , pages 9133-9143. PMLR, 2020.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . 2018.

Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell, and Zhiwei Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap, 2021. URL https: //arxiv.org/abs/2103.03236 .

Gokul Swamy, Nived Rajaraman, Matthew Peng, Sanjiban Choudhury, J Andrew Bagnell, Zhiwei Steven Wu, Jiantao Jiao, and Kannan Ramchandran. Minimax optimal online imitation learning via replay estimation. arXiv preprint arXiv:2205.15397 , 2022.

Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Zhiwei Steven Wu. Inverse reinforcement learning without reinforcement learning. arXiv preprint arXiv:2303.14623 , 2023.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Marcell Vazquez-Chanlatte, Susmit Jha, Ashish Tiwari, Mark K Ho, and Sanjit Seshia. Learning task specifications from demonstrations. Advances in neural information processing systems , 31, 2018.

Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research , 23(267):1-6, 2022. URL http://jmlr.org/papers/v23/ 21-1127.html .

Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, and Chelsea Finn. Learning a prior over intent via meta-inverse reinforcement learning. In International conference on machine learning , pages 6952-6962. PMLR, 2019.

Mengdi Xu, Zuxin Liu, Peide Huang, Wenhao Ding, Zhepeng Cen, Bo Li, and Ding Zhao. Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv preprint arXiv:2209.08025 , 2022.

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152 , 2020.

Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with probabilistic context variables. Advances in neural information processing systems , 32, 2019.

Brian Ziebart, Anind Dey, and J Andrew Bagnell. Probabilistic pointing target prediction via inverse optimal control. In Proceedings of the 2012 ACM international conference on Intelligent User Interfaces , pages 1-10, 2012.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008a.

Brian D Ziebart, Andrew L Maas, Anind K Dey, and J Andrew Bagnell. Navigate like a cabbie: Probabilistic reasoning from observed context-aware behavior. In Proceedings of the 10th international conference on Ubiquitous computing , pages 322-331, 2008b.

Matt Zucker, Nathan Ratliff, Martin Stolle, Joel Chestnutt, J Andrew Bagnell, Christopher G Atkeson, and James Kuffner. Optimization and learning for rough terrain legged locomotion. The International Journal of Robotics Research , 30(2):175-191, 2011.

A Proofs

A.1 Proof of Theorem 3.1

Proof. Let Π( c ) denote the set of policies such that

First, note that ∀ c ∈ F c , π E ∈ Π( c ) . CRL is therefore trying to maximize reward over a set of policies that contains π E . Thus, the reward condition is trivially true. We therefore focus on the safety condition. First, we note that by the definition of regret,

This implies that

as ˆ c is the best-in-hindsight constraint. We then note that J ( π i , c i ) -J ( π E , c i ) ≤ 0 by the fact that π i is produced via a CRL procedure, which means we can drop the former term from the above sum, giving us

Because this equation holds on average, there must be at least one π ∈ π 1: N for which it holds. Now, we recall that π i = CRL ( r, c i ) to complete the proof.

A.2 Proof of Lemma 3.3

Proof. For a single c , a standard Hoeffding bound tells us that

where V k ( c ) denotes the value of the payoff using data from the k th task. We have |F c | constraints and want to be within ϵ of the population mean uniformly. We can apply a union bound to tell us that w.p. at least

we will do so. If we want to satisfy this condition with probability at least 1 -δ , simple algebra tells us that we must draw samples.

A.3 Proof of Theorem 3.5

Proof. For each c ∈ F c , define the set of safe policies as Π( c ) = { π ∈ Π | J ( π, c ) ≤ 0 } . This set is non-empty by assumption. Define

✶ We prove each of the desired conditions independently.

Reward Condition. Let ˆ c ∈ c 1: N . Recall that we want to prove that

where π τ = CRL ( r τ , ˆ c, δ = 0) . Observe that

Thus, if we have that

We now prove that with K large enough, we can guarantee Eq. 26 holds true w.h.p. Define

w.p. ≥ 1 -δ , Eq. 26 holds true ∀ c ∈ ˜ F c . This means that as long as we pick c 1: N ∈ ˜ F c , our desired condition will be true. Note that this is fewer than the number of samples we assumed in the theorem statement.

For a single constraint, a Hoeffding bound tells us that

Union bounding across F c ⊇ ˜ F c , we get that the probability that ∃ c ∈ ˜ F c s.t. Eq. 26 does not hold is upper bounded by

To have this quantity be ≥ 1 -δ , we need

Safety Condition. We begin by considering the infinite sample setting. We therefore desire to prove that

Define the per-round loss of the constraint player as

the best-in-hindsight comparator as ˆ c = argmax c ∈F c ∑ N i ℓ i ( c ) , instantaneous regret as ϵ i = ℓ i (ˆ c ) -ℓ i ( c i ) , and average regret as ¯ ϵ = 1 N ∑ N i ϵ i . Proceeding similarly to the single-task case,

We now argue that the second term in the above sum must be non-positive. Consider an arbitrary task τ . Then, because π τ E ∈ Π( c i ) and CRL is optimizing over Π( c i ) , this term must be negative. As it is

We now argue that if

negative per-task, it must be negative in expectation. Thus, we are free to drop the second term in the above expression which tells us that

Because this equation holds on average, there must be at least one π ∈ π 1: N for which it holds. Now, we recall that π τ i = CRL ( r τ , c i ) to complete the infinite-sample proof.

We now consider the error induced by only observing a finite set of tasks. There are two places finite-sample error enters: in estimating the value of ℓ i ( c ) and in estimating ˜ F c .

By Lemma 3.3, the maximum error we can induce by estimating ℓ i from finite samples is upper bounded w.h.p by ϵ . Thus, the extra error induced on the average regret is also bounded by ϵ . Observe that our losses are scaled by 1 T in comparison to difference of J s. Therefore, we need to add an ϵT to our bound for the infinite-sample setting.

By our argument in the reward section, π τ E / ∈ Π( c i ) w.p. ≤ ϵ . When this is true, J ( π τ i , c i ) -J ( π τ E , c i ) can be as big as 2 T . Thus, the term we dropped in Eq. 40 ( V ( c i ) ) can be as big as 2 ϵT instead of 0. In the worst case, this adds an additional 2 ϵT to our bound.

Combining both of the above, when we transition from the infinite sample setting to the finite sample setting, our bound degrades by 3 ϵT .

B Experimental Details

An overview of the key experimental details is presented below. The exact code for reproducing our experiments is available at https://github.com/konwook/mticl .

B.1 CRL

We use the Tianshou [Weng et al., 2022] implementation of PPO [Schulman et al., 2017] as our baseline policy optimizer. Classical Lagrangian methods exactly follow the gradient update shown in Algorithm 1, but they are susceptible to oscillating learning dynamics and constraint-violating behavior during training. The PID Lagrangian method [Stooke et al., 2020] extends the naive gradient update of λ i with a proportional and derivative term to dampen oscillations and prevent cost overshooting.

We find that augmenting the policy state with the raw value of the constraint function to be crucial for successful policy training. For CRL, this corresponds to the violation of the ground-truth constraint.

B.2 Expert Demonstrations

For all of our experiments, our experts are CRL policies trained to satisfy a ground truth constraint. The expert demonstrations we use in ICL are generated by rolling out these policies and collecting 20 trajectories. To simulate suboptimal experts, we generate noisy trajectories by adding zero-mean Gaussian noise (with standard deviation 0.7 / 0.5 for velocity / position) to the policy's action at every timestep.

B.3 Single-Task ICL

Similar to CRL, we augment the policy state with the raw value of the constraint function. For ICL, this corresponds to the violation of the learned constraint.

To reduce the amount of interaction required to solve the inner optimization problem, we warm-start our policy in each iteration by behavior cloning against the given expert demonstrations. We zero-out the augmented portion of the state when behavior cloning to avoid leaking constraint violation information from the expert.

When using CRL as part of ICL, we set the constraint threshold used in the Lagrangian update to be the expert's constraint violation. However, when starting with degenerate constraints, this can prevent policy optimization from learning at all as the expert's violation under the constraint can be arbitrarily low. To circumvent this issue, we set the Lagrangian constraint threshold to use the expert's violation plus a cost limit buffer, which we anneal over the course of training to 0. This ensures that our learned policy satisfies the learned constraint as much as the expert does as desired.

Because ICL requires learning a constraint, we represent our constraints as neural networks, mapping from the state space of our agent to a bounded scalar in the range [0 , 1] . To update this constraint, we solve the optimization problem using a regression objective. Learner and expert constraint values are labeled with 1 and -1 respectively, and we optimize a mean squared error loss. We found that weighting data equally (instead of via returned Lagrange multipliers) was sufficient for good performance and therefore utilize this simpler strategy.

For both CRL and ICL, we find that using a log-activation on top of the raw value of the constraint is an important detail for stable training.

B.4 Multi-Task ICL

For the multi-task maze setting, we consider 10 distinct tasks corresponding to unique goal locations, each with 2 starting locations.

Our solver combines a waypoint planner with a low-level controller. Waypoints are calculated by discretizing the maze into a 10 by 10 grid and running Q-value iteration, while low-level actions are computed using CRL experts trained to move in a single cardinal direction. To generate a trajectory, we follow a sequence of waypoints from the planner using the controller.

For ICL, we visualize our learned constraints by uniformly sampling 100,000 points from every grid cell, averaging the constraint predictions, and thresholding at a limit of 0.5.

B.5 Hyperparameters

The key experimental hyperparameters are shown in Table 1. The exact configuration we use for our experiments is available at https://github.com/konwook/mticl/blob/main/mticl/utils/ config.py .

C Comparison to Chou et al. [2020]

C.1 Experimental Setup

We attempt to faithfully implement the method of Chou et al. on the problems we consider as a baseline for single-task ICL. At a high level, their method requires 2 steps:

For each expert trajectory, perform a random search in trajectory space starting from the demo to try and compute the set of trajectories that are higher reward than the demo.

Solve a constrained optimization problem over a known parametric family that labels each expert trajectory with +1 and each learner trajectory with 0.

Note that in contrast to our method, Chou et al. perform a single constraint estimation, rather than an iterative procedure.

For Step 1), Chou et al. [2020] use hit-and-run sampling that ensures good coverage over this set of trajectories in the limit of infinite sampling. However, for tasks of the horizon (1000+) and state-space dimension (30) we consider, one would need a rather large number of samples to ensure good coverage (i.e. an ϵ -net would require 30 1000 ϵ samples).

As search in the space of trajectories is infeasible, we instead search in the space of policies, as is much more standard on long-horizon problems. We therefore initialize the policy with behavioral cloning (similar to starting from the expert trajectories), performing maximum entropy RL (a search procedure with good coverage properties), and return the set of trajectories from the replay buffer that are of a higher reward than that of the expert.

For Step 2), Chou et al. [2020] use a variety of mixed-integer LP solvers. In essence however, they are trying to solve a classification problem. Of course, with their assumption of an optimal expert (or a "boundedly suboptimal" expert with a known sub-optimality gap), they can solve the strict feasibility problem. Because we assume that the expert is safe in expectation (rather than uniformly), it is natural to consider a relaxation of this problem (i.e. a convex relaxation of the 0/1 loss).

We experimented with several such relaxations when developing our method and found the ℓ 2 relaxation (i.e. treating the problem as a regression problem with different targets for learner and expert demos) to work the best in practice. We therefore use the same relaxation for their method. For our single-task experiments, it is clear how to specify the "parametric family" the constraint fits in (e.g. a linear threshold), so we believe the above is a faithful implementation.

So, in short, we perform the same constraint learning procedure over a different source of learner data.

C.2 Baseline Comparison Results

Weimplemented the above method on our velocity and position tasks and display the results (averaged over 3 seeds) below.

We see that our method performs better in terms of all 3 performance criteria we evaluate under. As implemented, Chou et al.'s method appears to produce an inaccurate constraint that leads to a significant safety violation (the bottom right cell in both tables).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel method for constraint learning from expert demonstrations by an optimal CRL policy. The idea behind the algorithm is to frame the problem as a zero-sum game between a policy player and a constraint player, using inspiration from a two-player zero-sum game expression of IRL and CRL. With the reward function available, expert demonstrations, and a class of potential constraints containing the ground truth constraint, the authors propose to minimize the cost to go subject to being at least as safe as the expert, i.e. to obtain a larger cost than the expert policy on all constraints in the set. \n\nThe authors derive no regret convergence bounds for the learned constraints and deal with degeneracy issues by extending the problem to the multi-task framework with asymptotic guarantees. They present simulation results on high dimensional robotics tasks.\n\nSTRENGTHS:\nVery sound paper, and very clean and ingenuous framing of the problem as a game. I really enjoyed reading the paper as the work elegantly composes concepts from IRL, CRL, GT and no regret learning. I also see it as being of big significance for a very important problem, that of learning constraints from demonstrations. Overall very solid work.\n\nWEAKNESSES:\n- Although the algorithmic contribution is very strong, it seems that the practical algorithm section of the paper is much weaker and feels rushed. The single task results seem very simplistic while the multi-task maze example is way too briefly discussed. There is one sentence describing the threshold after which correct recovery of the constraints occurs, an aspect that would deserve a more extensive presentation and analysis.\n- Along that line, figure 5 deserve to be reworked for clarity, it mixes too many things in together with little readability.\n- There is no discussion on the computation costs of running the algorithm and on its data requirements. The authors only mention succinctly that CRL is slower than RL (by how much?), but it would be useful to know how many demonstrations one needs for typical robotics scenarios and how much compute the recovery of constraints needs.\n\nQUESTIONS:\n- How do the authors expect their solution to scale for systems where the state is much higher in dimension than the simple examples presented. For example do they think that their method could be a direction to learn constraints from sequences of images in applications such as autonomous driving or flight?\n\n- (Optional) out of curiosity, I am wondering if the authors have any insight on how much more challenging it would be to consider constraints in multi-agent environments?\n\nLIMITATIONS:\n- The authors mention the lack of real world experiments as a limitation, I could not agree more but would go even further in suggesting that the proposed simulations are insufficient to help readers fully grasp the potential of the method. The paper would warrant a higher rating should that aspect be improved.\n\n- I am awaiting the authors response to better understand whether the method would actually be practical to use for real life applications and robotics problems of interest.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel inverse constraint learning (ICL) approach that leverages constrained reinforcement learning (CRL) and a game-theoretic formulation. Specifically, the paper shows that the game-theoretic view of inverse reinforcement learning (IRL) naturally extends to ICL, by forming the Lagrangian of CRL. The resulting single-task ICL algorithm is shown to Pareto-dominate the expert policy in terms of its expected cumulative reward and the constraint violation. However, the recovered constraint may be too conservative and generalize poorly under a new task. To address this issue, the framework is extended to incorporate expert demonstrations under multiple tasks that share the same constraint.The paper shows that the multi-task ICL framework is expected to approximately Pareto-dominate the expert policies. Simulation results suggest the efficacy of the approach in continuous and discrete MDPs.\n\nSTRENGTHS:\n* Unlike prior ICL methods, the proposed framework is built on the game-theoretic approach. This brings in multiple benefits: 1) the form of the reward function can remain general; 2) conditions on approximate Pareto dominance can be derived, which gives theoretical justification of the proposed approach; 3) the resulting algorithm is relatively straightforward to implement.\n\n* Figures 1 and 2 give a nice visual representation of the proposed approach. Algorithm 1, 2, and 3 are presented in a consistent and organized way that clarifies the differences among CRL, single-task ICL, and multi-task ICL.\n\nWEAKNESSES:\n* The most concerning point is that the paper provides no comparison to prior work in the experiment. Some important insights of the proposed approach is shared by the seminal work of [1]. Specifically, both approaches 1) observe that access to reward functions enables constraint learning by looking at the differences between safe demonstrations and unsafe optimal demonstrations; 2) can leverage safe demonstrations from multiple tasks. To strengthen the contribution of the paper, it should include qualitative and quantitative comparison to [1] in Section 4 (where applicable), to clarify the practical advantages of the proposed framework. This might involve evaluating the ICL algorithm in more realistic problems than the toy task of maze navigation, since the authors claim better scalability of their approach compared to [1] in realistic problems (in Section 2). In addition, the paper should give a more detailed description of [1] to better highlight the similarities and differences.\n\n* There are multiple places in the paper where a variable or an acronym is used without definition: 1) $c^*$ in Theorem 3.1; 2) “FTRL” in line 153. Also, some figures in Section 4 are confusing and hard to interpret. Specifically, I believe that the blue cells in Figure 5(b) and (c) represent start and goal cells. However, the same color is used in other sub-figures to represent the constraints. It is recommended to use different colors to represent the start and the goals so that readers can make sense of Figure 5 as a whole. For further points concerning the presentation of the paper, please see the questions below.\n\n[1] Chou, Glen, Dmitry Berenson, and Necmiye Ozay. \"Learning constraints from demonstrations with grid and parametric representations.\" The International Journal of Robotics Research 40, no. 10-11 (2021): 1255-1283.\n\nQUESTIONS:\n* What is $c_i^k$ in Algorithm 3? Shouldn’t it be just $c_i$ since the constraint is shared across all the $K$ tasks?\n\n* The order of minimization and maximization is swapped between (6) and (7). Does the equality hold because of the standard minimax theorem in game theory? If so, are there any conditions imposed on the form of $J$ in order for the equality to hold?\n\nLIMITATIONS:\n* The paper assumes access to demonstrations of the optimal safe policy for a given task. While the availability of safe demonstrations is a reasonable assumption,optimal demonstrations may be too demanding to ask for a human demonstrator in real-world settings [2]. It would be quite interesting to see how the proposed ICL framework handles suboptimal demonstrations; in theory, the objective function of equation (4) does not seem to require the optimality of the expert under the given reward function $r$.\n\n[2] Xu, Haoran, Xianyuan Zhan, Honglei Yin, and Huiling Qin. \"Discriminator-weighted offline imitation learning from suboptimal demonstrations.\" In International Conference on Machine Learning, pp. 24725-24742. PMLR, 2022.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to use inverse constraint learning (ICL) in multi-task scenarios. It uses game solving to describe the ICL problem, and then illustrates the learning algorithm under the single-task and multi-task problems. The authors give a theoretical analysis to explain under what conditions the learned reward can be generalized. The author shows the correctness of the proposed algorithm in both single-task and multi-task control experiments.\n\nSTRENGTHS:\n- The motivation of this paper is clear and reasonable to extract common safety-constraint from multi-task.\n- The paper considers whether the learned reward and constraint can be generalized theoretically, and gives two conditions and related proofs.\n\nWEAKNESSES:\n- The paper did not give a comparison with the ICL algorithms mentioned in the related work, and did not show the advantages of the algorithm in both learning or safety concerns.\n- The experimental settings of the paper is too simple, and only consider two constraints that do not require complex approachs to learn. It makes the machine learning method shows no advantages on human priors.\n- The authors did not give the results of robotics control tasks (both in simulation and real-world) that really need safety-constraints, such as robotic manipulation with human.\n\nQUESTIONS:\n- Can the author supplement the experimental results on more complex robotics control tasks? For example, consider such a manipulation task as follow. We hope that the force or speed of the robot arm will not be too large while picking up an object. Can the algorithm learn such complex safety-constraints?\n\n- The authors' experiment in multi-task scenarios essentially improves the diversity of samples through randomization. We generally don't think this is a real multi-task RL scenario because all tasks are navigation. Can the author explore diverse but related manipulation tasks (like push and pull an object) in the open-source taskset (such as [meta-world](https://meta-world.github.io/)) to show the effectiveness of the proposed method?\n\nLIMITATIONS:\nIt is good that the paper focuses on the safety problems of reinforcement learning for robotics. However I suggest that the authors use more realistic robotics tasks (at least in simulation) that really need to consider safety-constraint to verify the method. Regardless of the comparison with related works or the complexity of the task, the experiments in the paper are not solid enough to support good motivation and theoretical results.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBroadly, the paper address the challenge of safety constraints for robots, in particular the difficulty of handcrafting these. They propose learning safety constraints from expert demonstrations, particularly in a multi-task setting where each task reward is known and the safety constraints are task-agnostic.\n\nThey review related work on inverse RL (IRL), constrained RL (CRL), inverse constraint learning (ICL), and multi-task ICL. Compared to existing ICL approaches, they claim their approach is more general, provides more guarantees, and is simpler to implement.\n\nThey extensively formalize ICL, including explaining relevant approaches to IRL and CRL. As a brief recap, they compare ICL to IRL: IRL has an outer loop that searches over potential rewards, with an inner loop that uses RL to find an optimal policy. ICL has an outer loop that searches over potential constraints, with an inner loop that uses CRL to find a safe-optimal policy. In either case, the optimized policy is compared to the expert demonstrations to revise the learned reward or constraint.\n\nThey also explain a challenge with single-task ICL, namely that ICL may learn an overly-conservative safety constraint that simply forbids all states not visited by the expert demonstrations. This motivates their exploration of multi-task ICL, where the safety constraint is assumed to be shared among the tasks. Their primary novel contribution of the paper is an extension of a single-task ICL approach to multi-task: \"we alternate between solving K CRL problems and updating the constraint based on the data from all policies\". Using it, they prove that they can exactly identify a shared constraint given enough expert policies, under certain very strong assumptions. They also prove a weaker but more practical theorem: if they have enough tasks, they can learn a constraint that, if used to optimize a policy, leads to policies that don't exceed the constraint-violation of the expert demonstrations and do meet or exceed the reward of the expert demonstrations (within a tolerance).\n\nFor the single-task case, they evaluate their approach in the simulated ant environment in PyBullet and MuJoCo benchmarks. They choose arbitrary velocity and position constraints. They represent their learned constraints as neural networks, mapping from the state space of their agent to a bounded scalar in the range [0, 1]. They show that their approach can recover these ground-truth constraints, under certain simplifying assumptions (linear constraints; usage of a subset of the agent state as input).\n\nFor multi-task, they use the PointMaze environment from D4RL, with each task variant having one of two start positions and one of 10 goal positions. Using their approach, they are able to approximately learn the ground-truth constraints (namely the maze's blocked cells). They also show that naive approaches for combining 10 separate learned constraints from single-task ICL don't work well.\n\nSTRENGTHS:\nTheir premise, that safety constraints are difficult to engineer and should be task-agnostic, is accurate and relevant to this domain.\n\nTheir approach is a straightforward extension of an existing single-task ICL approach to multi-task. It works well and matches the intuition that safety constraints should be task-agnostic.\n\nRigorous math in section 3 to formalize their work.\n\nWEAKNESSES:\nIn section 3, it's not clear what is prior work and what is their novel contribution.\n\nIn 3.1, the authors copied several sentences verbatim from \"Inverse Reinforcement Learning without Reinforcement Learning\" (https://arxiv.org/pdf/2303.14623.pdf, section 3.1). The use of copied phrases like \"we solve\" suggest the authors, themselves, did this work.\n\nIn their single-task and multi-task results, they don't compare to any baselines, for example \"Learning constraints from demonstrations\" from Chou et al.\n\nFor readability, it would be helpful if they numbered references.\n\nQUESTIONS:\nSection 3.4 and 3.5 mention needing expert policies. However, the premise of your work is that you have access to expert demonstrations and reward functions, but not expert policies. Can you clarify?\n\nA suggestion for positioning this line of work: instead of \"safety constraints\", I might argue that a more broad (and thus more valuable) set of constraints to be learned from human demonstrations is \"task-agnostic human-behavior constraints\", which would encompass safety plus other conventions and preferences. E.g. coming back to the example in your introduction, if I ask my friend to make toast, I'd be surprised if they set a plate down upside-down or rinsed the bread under the faucet.\n\n>we consider a setting in which we have access to demonstrations of the optimal safe policy for a task, along with knowledge about the task’s reward\n\nCan you describe this setting in practical terms? It seems that your approach has stronger requirements, namely (1) access to not just the rewards received by the demonstrations, but access to the reward function itself, and (2) access to a simulator or other mechanism to train a policy using CRL. As a clarifying example: if you had access to expert trajectories of a real-world robot including full scene state and reward annotations, across multiple tasks, this alone would not be sufficient to learn safety constraints using your method, right?\n\n>In short, if we observe enough tasks, we are able to learn a constraint that, when optimized under, leads to policies that approximately Pareto-dominate that of the expert.\n\nCan you elaborate? What is the specific Pareto-dominance here? My understanding is: you can learn a constraint that, if used to optimize a policy, leads to policies that don't exceed the constraint-violation of the expert demonstrations and do meet or exceed the reward of the expert demonstrations (within a tolerance). However, I don't intuitively follow why this should be your goal for the learned safety constraint.\n\nFor Figure 2, can you offer an intuitive explanation? For example:\n\n= can you label the axes?\n\n= why is an expert policy a single point in this space?\n\n= what does the boundary of the polytope represent?\n\n= what does this mean? \"diversity of expert policies, none of which are optimal along the reward vector\"\n\nLIMITATIONS:\nNo concerns here", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of learning safety constraints for agents in various tasks from expert demonstrations, instead of manually specifying them. The authors extend inverse reinforcement learning (IRL) techniques to the space of constraints, aiming to learn constraints that prevent highly rewarding behavior that the expert could have performed but avoided. However, the constraint learning problem is ill-posed and tends to result in overly conservative constraints. To mitigate this issue, the authors utilize diverse demonstrations from multi-task settings to learn a tighter set of constraints. The method is validated through simulation experiments on high-dimensional continuous control tasks.\n\nSTRENGTHS:\n- The authors develop a multi-task variant of inverse constraint learning, which consists of multiple policy players aiming to maximize task-specific rewards and a constraint player determining a single constraint that all policy players must follow. \n- The proposed approach reduces the chances of choosing degenerate solutions.\n- The inverse constraint learning problem is formulated as a zero-sum game involving a policy player, who seeks to maximize rewards while adhering to potential constraints, and a constraint player, who selects constraints that impose maximum penalties on the learner compared to the expert. \n- The proposed method demonstrates its effectiveness in various continuous control tasks. When applying restricted function classes, the technique can retrieve ground-truth constraints for certain tasks.\n\nWEAKNESSES:\nThe author argues that previous ICL problems have not been well-formulated, leading to solutions that only allow expert actions. However, it is not clear how the proposed method addresses this issue, aside from utilizing a greater number of demonstrations from a multi-task setting.\nIt would be interesting to determine whether the proposed method can work for tasks beyond maze locomotion, which lack complex interactions with the environment. For example, in manipulation tasks, safety constraints are more critical as the robot must interact with and alter the environment.\n\nQUESTIONS:\nThe author could better address the concerns mentioned in the weaknesses section. It is worth noting that the reviewer is not well-versed in this field and does not specialize in IRL or ICL problems. Thus, the overall rating may be subject to change based on feedback from other reviewers and the author's rebuttal.\n\nLIMITATIONS:\nConsider removing Section 4.2.\nA more intuitive explanation of Figure 2 would be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}