SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Konwoo Kim*
Gokul Swamy*
Zuxin Liu
Ding Zhao
Regardless of the particular task we want them to perform in an environment, there are often shared safety constraints we want our agents to respect. For example, regardless of whether it is making a sandwich or clearing the table, a kitchen robot should not break a plate. Manually specifying such a constraint can be both time-consuming and error-prone. We show how to learn constraints from expert demonstrations of safe task completion by extending inverse reinforcement learning (IRL) techniques to the space of constraints. Intuitively, we learn constraints that forbid highly rewarding behavior that the expert could have taken but chose not to. Unfortunately, the constraint learning problem is rather ill-posed and typically leads to overly conservative constraints that forbid all behavior that the expert did not take. We counter this by leveraging diverse demonstrations that naturally occur in multi-task settings to learn a tighter set of constraints. We validate our method with simulation experiments on high-dimensional continuous control tasks.
If a friend was in your kitchen and you told them to "make toast" or "clean the dishes," you would probably be rather surprised if they broke some of your plates during this process. The underlying safety constraint that forbids these kinds of behavior is both  $a$ ) implicit and  $b$ ) agnostic to the particular task they were asked to perform. Now, let's bring a household robot into the equation, operating within your kitchen. How can we ensure that it adheres to these implicit safety constraints, regardless of its assigned tasks?
One approach might be to write down specific constraints (e.g. joint torque limits) and pass them to the decision-making system of the robot. Unfortunately, more complex constraints like the ones we consider above are both difficult to formalize mathematically and easy for an end-user to forget to specify (as they would be inherently understood by a human helper). This problem is paralleled in the field of reinforcement learning (RL), where defining reward functions that lead to desirable behaviors for the learning agent is a recurring challenge [Hadfield-Menell et al., 2017]. For example, it is rather challenging to handcraft the exact function one should be optimized to be a good driver. The standard solution to this sort of "reward design" problem is to instead demonstrate the desired behavior of the agent and then extract a reward function that would incentivize such behavior. Such inverse reinforcement learning (IRL) techniques have found application in fields as diverse as robotics [Silver et al., 2010, Ratliff et al., 2009, Kolter et al., 2008, Ng et al., 2006, Zucker et al., 2011], computer vision [Kitani et al., 2012], and human-computer interaction [Ziebart et al., 2008b, 2012]. Given the success of IRL techniques and the similarity between reward and constraint design, we propose
*Equal contribution. Correspondence to gswamy@cmu.edu.
extending IRL techniques to the space of constraints. We term such techniques inverse constraint learning, or ICL for short.
More formally, we consider a setting in which we have access to demonstrations of an expert policy for a task, along with knowledge about the task's reward. This allows us to look at the difference between the expert and reward-optimal policies for a task. Our first key insight is that the actions taken by the reward-optimal but not the expert policy are likely to be forbidden, allowing us to extract a constraint.
Unfortunately, the ICL problem is still rather ill-posed. Indeed, prior work in ICL will often learn overly conservative constraints that forbid all behavior the expert did not take [Scobee and Sastry, 2019, Vazquez-Chanlatte et al., 2018, McPherson et al., 2021]. However, for tasks in a shared environment with different rewards, there are often safety constraints that should be satisfied regardless of the task (e.g. a plate shouldn't be broken regardless of whether you're serving food on it or cleaning up after a meal). Our second crucial insight is that we can leverage multi-task data to provide more comprehensive demonstration coverage over the state space, helping our method avoid degenerate solutions.
More explicitly, the contributions of our work are three-fold.
1. We formalize the inverse constraint learning problem. We frame ICL as a zero-sum game between a policy player and a constraint player. The policy player attempts to maximize reward while satisfying a potential constraint, while the constraint player picks constraints that maximally penalize the learner relative to the expert. Intuitively, such a procedure recovers constraints that forbid high-reward behavior the expert did not take.
2. We develop a multi-task extension of inverse constraint learning. We derive a zero-sum game between a set of policy players, each attempting to maximize a task-specific reward, and a constraint player that chooses a constraint that all policy players must satisfy. Because the constraint player looks at aggregate learner and expert data, it is less likely to select a degenerate solution.
3. We demonstrate the efficacy of our approach on various continuous control tasks. We show that with restricted function classes, we are able to recover ground-truth constraints on certain tasks. Even when using less interpretable function classes like deep networks, we can still ensure a match with expert safety and task performance. In the multi-task setting, we are able to identify constraints that a single-task learner would struggle to learn.
We begin with a discussion of related work.
Our work exists at the confluence of various research thrusts. We discuss each independently.
Inverse Reinforcement Learning. IRL [Ziebart et al., 2008a,b, 2012, Ho and Ermon, 2016] can be framed as a two-player zero-sum game between a policy player and a reward player [Swamy et al., 2021]. In most formulations of IRL, a potential reward function is chosen in an outer loop, and the policy player optimizes it via RL in an inner loop. Similar to IRL, the constraint in our formulation of ICL is chosen adversarially in an outer loop. However, in contrast to IRL, the inner loop of ICL is constrained reinforcement learning: the policy player tries to find the optimal policy that respects the constraint chosen in the outer loop.
Constrained Reinforcement Learning. Our approach involves repeated calls to a constrained reinforcement learning (CRL) oracle [Garcia and Fernandez, 2015, Gu et al., 2022]. CRL aims to find a reward-maximizing policy over a constrained set, often formulated as a constrained policy optimization problem [Altman, 1999, Xu et al., 2022]. Solving this problem via Frank-Wolfe methods is often unstable [Ray et al., 2019, Liang et al., 2018]. Various methods have been proposed to mitigate this instability, including variational techniques [Liu et al., 2022], imposing trust-region regularization [Achiam et al., 2017, Yang et al., 2020, Kim and Oh, 2022], optimistic game-solving algorithms [Moskovitz et al., 2023], and PID controller-based methods [Stooke et al., 2020]. In our practical implementations, we use PID-based methods for their relative simplicity.
Multi-task Inverse Reinforcement Learning. Prior work in IRL has considered incorporating multi-task data [Xu et al., 2019, Yu et al., 2019, Gleave and Habryka, 2018]. We instead consider a
setting in which we know task-specific rewards and are attempting to recover a shared component of the demonstrator's objective. Amin et al. [2017] consider a similar setting but require the agent to be able to actively choose tasks or interactively query the expert, while our approach requires neither.
Inverse Constraint Learning. We are far from the first to consider the ICL problem. Scobee and Sastry [2019], McPherson et al. [2021] extend the MaxEnt IRL algorithm of Ziebart et al. [2008a] to the ICL setting. We instead build upon the moment-matching framework of Swamy et al. [2021], allowing our theory to handle general reward functions instead of the linear reward functions MaxEnt IRL assumes. We are also able to provide performance and constraint satisfaction guarantees on the learned policy, unlike the aforementioned work. Furthermore, we consider the multi-task setting, addressing a key shortcoming of the prior work.
Perhaps the most similar paper to ours is the excellent work of Chou et al. [2020], who also consider the multi-task ICL setting but propose a solution that requires several special solvers that depend on knowledge of the parametric family that a constraint falls into. In contrast, we provide a general algorithmic template that allows one to apply whatever flexible function approximators (e.g. deep networks) and reinforcement learning algorithms (e.g. PPO) they desire. Chou et al.'s method also requires sampling uniformly over the set of trajectories that achieve a higher reward than the expert, a task which is rather challenging to do on high-dimensional problems. In contrast, our method only requires the ability to solve a standard RL problem. Theoretically, Chou et al. [2020] focus on constraint recovery, which we argue below is a goal that requires strong assumptions and is therefore a red herring on realistic problems. This focus also prevents their theory from handling suboptimal experts. In contrast, we are able to provide rigorous guarantees on learned policy performance and safety, even when the expert is suboptimal. We include results in Appendix C that show that our approach is far more performant.
In concurrent work, Lindner et al. [2023] propose an elegant solution approach to ICL: rather than learning a constraint function, assume that any unseen behavior is unsafe and enforce constraints on the learner to play a convex combination of the demonstrated safe trajectories. The key benefit of this approach is that it doesn't require knowing the reward function the expert was optimizing. However, by forcing the learner to simply replay previous expert behavior, the learner cannot meaningfully generalize, and might therefore be extremely suboptimal on any new task. In contrast, we use the side information of a reasonable set of constraints to provide rigorous policy performance guarantees.
We now turn our attention to formalizing inverse constraint learning.
3 Formalizing Inverse Constraint Learning
We build up to our full method in several steps. We first describe the foundational algorithmic structures we build upon (inverse reinforcement learning and constrained reinforcement learning). We then describe the single-task formulation before generalizing it to the multi-task setup.
We consider a finite-horizon Markov Decision Process (MDP) [Puterman, 2014] parameterized by  $\langle S, \mathcal{A}, \mathcal{T}, r, T \rangle$  where  $S, \mathcal{A}$  are the state and action spaces,  $\mathcal{T}: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$  is the transition operator,  $r: \mathcal{S} \times \mathcal{A} \to [-1, 1]$  is the reward function, and  $T$  is the horizon.
3.1 Prior Work: Inverse RL as Game Solving
In the inverse RL setup, we are given access trajectories generated by an expert policy  $\pi^E: S \to \Delta(\mathcal{A})$ , but do not know the reward function of the MDP. Our goal is to nevertheless learn a policy that performs as well as the expert's, no matter the true reward function.
We solve the IRL problem via equilibrium computation between a policy player and an adversary that tries to pick out differences between expert and learner policies under potential reward functions Swamy et al. [2021]. More formally, we optimize over polices  $\pi : S \to \Delta(\mathcal{A}) \in \Pi$  and reward functions  $f: S \times \mathcal{A} \to [-1, 1] \in \mathcal{F}_r$ . For simplicity, we assume that our strategy spaces  $(\Pi$  and  $\mathcal{F}_r)$  are convex and compact and that  $r \in \mathcal{F}_r$ ,  $\pi_E \in \Pi$ . We solve (i.e. compute an approximate Nash
2We also note that, because we scale the learned constraint differently for each task, Lindner et al. [2023]'s impossibility result (Prop. 2) does not apply to our method, thereby elucidating why a naive application of inverse RL on the aggregate data isn't sufficient for the problem we consider.
Algorithm 1 CRL (Constrained Reinforcement Learning)
Input: Reward  $r$  constraint  $c$  , learning rates  $\eta_{1:N}$  , tolerance  $\delta$    
Output: Trained policy  $\pi$    
Initialize  $\lambda_1 = 0$    
for  $i$  in  $1\dots N$  do   
 $\pi_i\gets \mathrm{RL}(r = r - \lambda_i c)$ $\lambda_{i}\gets [\lambda_{i} + \eta_{i}(J(\pi_{i},c) - \delta)]^{+}$    
end for   
Return Unif  $(\pi_{1:N})$
Algorithm 2 ICL (Inverse Constraint Learning)
Input: Reward  $r$ , constraint class  $\mathcal{F}_c$ , trajectories from  $\pi_E$   
Output: Learned constraint  $c$   
Initialize  $c_1 \in \mathcal{F}_c$   
for  $i$  in  $1 \ldots N$  do  
     $\pi_i, \lambda_i \gets \mathrm{CRL}(r, c_i, \delta = J(\pi_E, c_i))$   
    // use any no-regret algo. to pick  $c$ , e.g. FTRL:  
     $c_{i+1} \gets \operatorname{argmax}_{c \in \mathcal{F}_c} \frac{1}{T} \sum_j^i (J(\pi_j, c) - J(\pi_E, c)) - R(c)$ .  
end for  
Return best of  $c_{1:N}$  on validation data.
equilibrium) of the two-player zero-sum game
$$
\min  _ {\pi \in \Pi} \max  _ {f \in \mathcal {F} _ {r}} J (\pi , f) - J (\pi_ {E}, f), \tag {1}
$$
where  $J(\pi, f) = \mathbb{E}_{\xi \sim \pi}[\sum_{t=0}^{T} f(s_t, a_t)]$  denotes the value of policy  $\pi$  under reward function  $f$ .
3.2 Prior Work: Constrained Reinforcement Learning as Game Solving
In CRL, we are given access to both the reward function and a constraint  $c: \mathcal{S} \times \mathcal{A} \to [-1, 1]$ . Our goal is to learn the highest reward policy that, over the horizon, has a low expected value under the constraint. More formally, we seek a solution to the optimization problem:
$$
\min  _ {\pi \in \Pi} - J (\pi , r) \text {s . t .} J (\pi , c) \leq \delta , \tag {2}
$$
where  $\delta$  is some error tolerance. We can also formulate CRL as a game via forming the Lagrangian of the above optimization problem [Altman, 1999]:
$$
\min  _ {\pi \in \Pi} \max  _ {\lambda > 0} - J (\pi , r) + \lambda (J (\pi , c) - \delta). \tag {3}
$$
Intuitively, the adversary updates the weight of the constraint term in the policy player's reward function based on how in violation the learner is.
3.3 Single-Task Inverse Constraint Learning
We are finally ready to formalize ICL. In ICL, we are given access to the reward function, trajectories from the solution to a CRL problem, and a class of potential constraints  $\mathcal{F}_c$  in which we assume the ground-truth constraint  $c^*$  lies. We assume that  $\mathcal{F}_c$  is convex and compact.
In the IRL setup, without strong assumptions on the dynamics of the underlying MDP and expert, it is impossible to guarantee recovery of the ground-truth reward. Often, the only reward function that actually makes the expert optimal is zero everywhere [Abbeel and Ng, 2004]. Instead, we attempt to find the reward function that maximally distinguishes the expert from an arbitrary other policy in our policy class via game-solving [Ziebart et al., 2008a, Ho and Ermon, 2016, Swamy et al., 2021]. Similarly, for ICL, exact constraint recovery can be challenging. For example, if two constraints differ only on states the expert never visits, it is not clear how to break ties. We instead try to find a constraint that best separates the safe (but not necessarily optimal)  $\pi_E$  from policies that achieve higher rewards.
Figure 1: A visual depiction of the optimization problem we're trying to solve in ICL. We attempt to pick a constraint that minimizes the value difference over the expert policy a safe policy could have. The star corresponds to the output of CRL.
More formally, we seek to solve the following constrained optimization problem.
$$
\min  _ {\pi \in \Pi} J \left(\pi_ {E}, r\right) - J (\pi , r) \tag {4}
$$
$$
\text {s . t .} \max  _ {c \in \mathcal {F} _ {c}} J (\pi , c) - J (\pi_ {E}, c) \leq 0. \tag {5}
$$
Note that in contrast to the moment-matching problem we solve in imitation learning [Swamy et al., 2021], we instead want to be at least as safe as the expert. This means that rather than having equality constraints, we have inequality constraints. Continuing, we can form the Lagrangian:
$$
\min  _ {\pi \in \Pi} \max  _ {\lambda > 0} J \left(\pi_ {E}, r\right) - J (\pi , r) + \lambda \left(\max  _ {c \in \mathcal {F} _ {c}} J (\pi , c) - J \left(\pi_ {E}, c\right)\right) \tag {6}
$$
$$
= \max  _ {c \in \mathcal {F} _ {c}} \max  _ {\lambda > 0} \min  _ {\pi \in \Pi} J \left(\pi_ {E}, r - \lambda c\right) - J (\pi , r - \lambda c). \tag {7}
$$
Notice that the form of the ICL game resembles a combination of the IRL and CRL games. We describe the full game-solving procedure in Algorithm 2, where  $R(c)$  is an arbitrary strongly convex regularizer [McMahan, 2011]. Effectively, we pick a constraint function in the same way we pick a reward function in IRL but run a CRL inner loop instead of an RL step. Instead of a fixed constraint threshold, we set tolerance  $\delta$  to the expert's constraint violation. Define
$$
\ell_ {i} (c) = \frac {1}{T} \left(J \left(\pi_ {i}, c\right) - J \left(\pi_ {E}, c\right)\right) \in [ - 1, 1 ] \tag {8}
$$
as the per-round loss that the constraint player suffers in their online decision problem. The best-in-hindsight comparator constraint is defined as
$$
\hat {c} = \underset {c \in \mathcal {F} _ {c}} {\operatorname {a r g m a x}} \sum_ {i} ^ {T} \ell_ {i} (c). \tag {9}
$$
We can then define the cumulative regret the learner suffers as
$$
\operatorname {R e g} (T) = \sum_ {i} ^ {T} \ell_ {i} (\hat {c}) - \sum_ {i} ^ {T} \ell_ {i} \left(c _ {i}\right), \tag {10}
$$
and let  $\epsilon_{i} = \ell_{i}(\hat{c}) - \ell_{i}(c_{i})$ . We prove the following theorem via standard machinery.
Theorem 3.1. Let  $c_{1:N}$  be the iterates produced by Algorithm 2 and let  $\bar{\epsilon} = \frac{1}{N}\sum_{i}^{N}\epsilon_{i}$  denote their time-averaged regret. Then, there exists a  $c\in c_{1:N}$  such that  $\pi = CRL(r,c,\delta = J(\pi_E,c))$  satisfies
$$
J \left(\pi , c ^ {*}\right) - J \left(\pi_ {E}, c ^ {*}\right) \leq \bar {\epsilon} T a n d J (\pi , r) \geq J \left(\pi_ {E}, r\right). \tag {11}
$$
In words, by optimizing under the recovered constraint, we can learn a policy that (weakly) Pareto-dominates the expert policy under  $c^*$ . We conclude by noting that because FTRL (Follow the Regularized Leader, McMahan [2011]) is a no-regret algorithm for linear losses like (8), we have that  $\lim_{T\to \infty}\frac{\operatorname{Reg}(T)}{T} = 0$ . This means that with enough iterations, the RHS of the above bound on ground-truth constraint violation will go to 0.
Algorithm 3 MT-ICL (Multi-task Inverse Constraint Learning)
Input: Rewards  $r^{1:K}$ , constraint class  $\mathcal{F}_c$ , trajectories from  $\pi_E^{1:K}$   
Output: Learned constraint  $c$   
Set  $\widetilde{\mathcal{F}}_c = \{c \in \mathcal{F}_c | \forall k \in [K], J(\pi_E^k, c) \leq 0\}$   
Initialize  $c_1 \in \widetilde{\mathcal{F}}_c$   
for  $i$  in  $1 \ldots N$  do  
for  $k$  in  $1 \ldots K$  do  
 $\pi_i^k, \lambda_i^k \gets \mathrm{CRL}(r^k, c_i, \delta = 0)$   
end for  
// use any no-regret algo. to pick c, e.g. FTRL:  
 $c_{i+1} \gets \operatorname{argmax}_{c \in \widetilde{\mathcal{F}}_c} \frac{1}{TK} \sum_j^i \sum_k^K (J(\pi_j^k, c) - J(\pi_E^k, c)) - R(c)$ .  
end for  
Return best of  $c_{1:N}$  on validation data.
Figure 2: If we have a sufficient diversity of expert policies, none of which are optimal along the reward vector, we can identify the hyperplane that separates the safe policies from the unsafe policies. The constraint (red, dashed) will be orthogonal to this hyperplane. For this example, because  $\rho_{\pi} \in \mathbb{R}^2$ , we need two expert policies.
3.4 Multi-task Inverse Constraint Learning
One of the potential failure modes of the single-task approach we outline above is that we could learn an overly conservative constraint, leading to poor task performance [Liu et al., 2023]. For example, imagine that we entropy-regularize our policy optimization [Ziebart et al., 2008a, Haarnoja et al., 2018], as is common practice. Assuming a full policy class, the learner puts nonzero probability mass on all reachable states in the MDP. The constraint player is therefore incentivized to forbid all states the expert did not visit [Scobee and Sastry, 2019, McPherson et al., 2021]. Such a constraint would likely generalize poorly when combined with a new reward function  $(\tilde{r} \neq r)$  as it forbids all untaken rather than just unsafe behavior.
At heart, the issue with the single-task formulation lies in the potential for insufficient coverage of the state space within expert demonstrations. Therefore, it is natural to explore a multi-task extension to counteract this limitation. Let each task be defined by a unique reward. We assume the dynamics and safety constraints are consistent across tasks. We observe  $K$  samples of the form  $(r_k,\{\xi \sim \pi_E^k\})$ . This data allows us to define the multi-task variant of our previously described ICL game:
$$
\max  _ {c \in \mathcal {F} _ {c}} \min  _ {\pi^ {1: K} \in \Pi} \max  _ {\lambda^ {1: K} > 0} \sum_ {i} ^ {K} J \left(\pi_ {E} ^ {i}, r ^ {i} - \lambda^ {i} c\right) - J \left(\pi^ {i}, r ^ {i} - \lambda^ {i} c\right). \tag {12}
$$
We describe how we solve this game in Algorithm 3, where  $R(c)$  is an arbitrary strongly convex regularizer [McMahan, 2011]. In short, we alternate between solving  $K$  CRL problems and updating the constraint based on the data from all policies.
We now give two conditions under which generalization to new reward functions is possible.
3.5 A (Strong) Geometric Condition for Identifiability
Consider for a moment the linear programming (LP) formulation of reinforcement learning. We search over the space of occupancy measures  $(\rho_{\pi} \in \Delta(\mathcal{S} \times \mathcal{A}))$  that satisfy the set of Bellman flow constraints [Sutton and Barto, 2018] and try to maximize the inner product with reward vector  $r \in \mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$ . We can write the CRL optimization problem (assuming  $\delta = 0$  for simplicity) as an LP
as well. Using  $\rho_{\Pi}$  to denote the occupancy measures of all  $\pi \in \Pi$
$$
\max  _ {\rho_ {\pi} \in \rho_ {\Pi}} \left\langle \rho_ {\pi}, r \right\rangle \text {s . t .} \left\langle \rho_ {\pi}, c ^ {*} \right\rangle \leq 0.
$$
We observe a (for simplicity, optimal) solution to such a problem for  $K$  rewards, begging the question of when that is enough to uniquely identify  $c^*$ . Recall that to uniquely determine the equation of a hyperplane in  $\mathbb{R}^d$ , we need  $d$  linearly independent points.  $c^* \in \mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$ , so we need  $|\mathcal{S}||\mathcal{A}|$  expert policies. Furthermore, we need each of these points to lie on the constraint line and not on the boundary of the full polytope. Put differently, we need each distinct expert policy to saturate the underlying constraint (i.e.  $\exists \pi \in \Pi$  s.t.  $J(\pi_E^k, r^k) < J(\pi^k, r^k)$ ). Under these conditions, we can uniquely determine the hyperplane that separates safe from unsafe policies, to which the constraint vector is orthogonal. More formally,
Lemma 3.2. Let  $\pi_E^{1:|\mathcal{S}||\mathcal{A}|}$  be distinct optimal expert policies such that  $a)\forall i\in [[\mathcal{S}||\mathcal{A}]]$ $\pi_E^i\in$  relint(  $\rho_{\Pi}$  ) and  $b)$  no  $\rho_{\pi_E^i}$  can be generated by a mixture of the other visitation distributions. Then,  $c^*$  is the unique (up to scaling) nonzero vector in
$$
N u l \left(\left[ \begin{array}{c} \rho_ {\pi_ {E} ^ {1}} - \rho_ {\pi_ {E} ^ {2}} \\ \vdots \\ \rho_ {\pi_ {E} ^ {| S | | \mathcal {A} | - 1}} - \rho_ {\pi_ {E} ^ {| S | | \mathcal {A} |}} \end{array} \right]\right). \tag {13}
$$