Learning Soft Constraints From Constrained Expert Demonstrations
Authors: Ashish Gaurav
Kasra Rezaee
Guiliang Liu
Pascal Poupart
Venue: ICLR 2023 notable top 25% (2023)
Keywords: inverse reinforcement learning
constraint learning

Abstract: Inverse reinforcement learning (IRL) methods assume that the expert data is generated by an agent optimizing some reward function. However, in many settings, the agent may optimize a reward function subject to some constraints, where the constraints induce behaviors that may be otherwise difficult to express with just a reward function. We consider the setting where the reward function is given, and the constraints are unknown, and propose a method that is able to recover these constraints satisfactorily from the expert data. While previous work has focused on recovering hard constraints, our method can recover cumulative soft constraints that the agent satisfies on average per episode. In IRL fashion, our method solves this problem by adjusting the constraint function iteratively through a constrained optimization procedure, until the agent behavior matches the expert behavior. We demonstrate our approach on synthetic environments, robotics environments and real world highway driving scenarios.

Published as a conference paper at ICLR 2023
LEARNING SOFT CONSTRAINTS FROM CONSTRAINED EXPERT DEMONSTRATIONS
Ashish Gaurav Kasra Rezaee', Guiliang Liu", Pascal Poupart!2 a5gaurav@uwaterrl oo. ca, kasra. raaaa.rezzeeeee rezaee@huawei. ca,
Hiugiilinng@cun edu. en, spoupart@uwaterllnr
'Cheriton School of Computer Science, University of Waterloo, Canada
Vector Institute, Toronto, Canada
Huawwi Technologies Canada "School of Data Science, The Chinese University of Hong Kong, Shenzhen
ABSTRACT
Inverse reinforcement learning (IRL) methods assume that the expert data is gen- erated by agent optimizing some reward function. However, in many settings, the agent may optimize a reward function subject to some constraints, where the constraints induce behaviors that may be otherwise difficult to express with just a reward function. We consider the setting where the reward function is given, and the constraints are unknown, and propose method that is able to recover these constraints satisfactorily from the expert data. While previous work has focused on recovering hard constraints, our method can recover cumulative soft constraints that the agent satisfies on average per episode. In IRL fashion, our method solves this problem by adjusting the constraint function iteratively through constrained optimization procedure, until the agent behavior matches the expert behavior. We demonstrate our approach on synthetic environments, robotics environments and real world highway driving scenarios.
INTRODUCTION
Inverse reinforcement learning (IRL.) (Ng et al., 2000 Russell 1998) refers to the problem of learning a reward function given observed optimal or near optimal behavior. However, in many setups, expert actions may result from a policy that inherently optimizes a reward function subject to certain constraints. While IRL methods are able to learn reward function that explains the expert demonstrations well, many tasks also require knowing constraints. Constraints can often provide a more interpretable representation of behavior than just the reward function (Chou al. (2018). In fact, constraints can represent safety requirements more strictly than reward functions, and therefore especially useful in safety-critical applications (Chou al. 2021 Scobee & Sastry 2019).
Inverse constraint learning (ICL) may therefore be defined as the process of extracting the constraint function(s) associated with the given optimal (or near optimal) expert data, where we assume that the reward function is available. Notably, some prior work (Chou et al. (2018) 2020; 2021 Scobee &
Sastry 2019, Malik al. 202T) has tackled this problem by learning hard constraints (i.e., fuucctions that indicate which state action-pan pairs are allowed).
We propose a novel method for ICL (for simplicity, our method is also called ICL) that learns cumulative soft constraints from expert demonstrations while assuming that the reward function is known. The difference between hard constraints and soft constraints can be illustrated as follows. Suppose in an environment, we need to obey the constraint "do not use more than 3 units of energy". As a hard constraint, we typically wish to ensure that this constraint is always satisfied for any individual trajectory. The difference between this "hard" constraint and proposed "soft" constraints that soft constraints are not necessarily satisfied in every trajectory, but rather only satisfied in expectation. This is equivalent to the specification "on average across all trajectories, do not use more than 3 units of energy". In the case of soft constraints, there may be certain trajectories when the constraint is violated, but in expectation, is satisfied.

---

Published as a conference paper at ICLR 2023
To formulate method, we adopt the framework of constrained Markov decision processes (CMDP) (Altman 1999). where an agent seeks to maximize expected cumulative rewards subject to constraints on the expected cumulative value of constraint functions. While previous work in constrained RL focuses on finding an optimal policy that respects known constraints, we seek to learn the constraints based expert demonstrations. We adopt approach similar to IRI., but the goal is to learn the constraint functions instead of the reward function.
Contributions. Our contributions can be summarized follows: (a) We propose novel formulation and method for ICL. Our approach works with any state-action spaces (including continuous state- action spaces) and can learn arbitrary constraint functions represented by flexible neural networks. To the best of our knowledge, our method is the first to learn cumulative soft constraints (such constraints can take into account noise in sensor measurements and possible violations in expert demonstrations) bounded in expectation as in constrained MDPs. (b) We demonstrate our approach by learning constraint functions in various synthetic environments, robotics environments and real world highway driving scenarios.
The paper is structured as follows. Section provides some background about IRL and ICL. Section summarizes previous work about constraint learning. Section describes our new technique to learn cumulative soft constraints from expert demonstrations. Section 3 demonstrates the approach for synthetic environments and discusses the results (more results are provided in Appendix BB. Finally, Section concludes by discussing limitations and future work.
BACKGROUND
Markov Decision Process (MDP). An MDP defined as a tuple (S,A,p,J4, 7), where state space, A is the action space, p(-|s, a) are the transition probabilities over the next states given the current state s and current action a, r S x + the reward function, μ S + [0, 1] is the initial state distribution and y is the discount factor. The behavior of agent in this MDP can be represented by a stochastic policy π S A - [0. 1]. which is a mapping from a state to a probability distribution over actions. constrained MDP augments the MDP structure contain constraint function e : S x A + IR and an episodic constraint threshold 3.
Reinforcement learning and Constrained RL. The objective of any standard RL procedure (control) is to obtain a policy that maximizes the (infinite horizon) expected long term discounted reward
(Sutton & Barto 2018):
arg max EE (p))) (1s op)
Σ 2'r(s,, an) =: JG
Similarly, in constrained RL, additionally the expectation of cumulative constraint functions c; must not exceed associated thresholds Be:
arg max (r) such that (G) S B
π
For simplicity, in this work we consider constrained RL with only one constraint function.
Inverse reinforcement learning (IRL) and inverse constraint learning (ICL). IRL performs the inverse operation of reinforcement learning, that is, given access dataset D = (T,)8=1
{[(s), (2)))11)) 1. 1 sampled using optimal or near optimal policy x", the goal is to obtain reward function r that best explains the dataset. By "best explanation", we mean that if we perform the RL procedure using then the obtained policy captures the behavior demonstrated in D as closely as possible. In the same way, given access to a dataset D (just like in IRI.), which sampled using optimal near optimal policy '²" (respecting some constraints G; and maximizing some known reward r), the goal of ICL to obtain the constraint functions e; that best explain the dataset, that is, if we perform the constrained RL procedure using r, ciVi, then the obtained policy captures the behaviour demonstrated D.
Setup. Similar to prior work (Chou et al. 2020), we learn only the constraints, but not the reward function. Essentially, it is difficult to say whether a demonstrated behaviour is obeying a constraint, or maximizing a reward, or doing both. So, for simplicity, assume the (nominal) reward is given, and
2

---

Published as a conference paper at ICLR 2023
just need to learn (single) constraint function. Without loss of generality, we fix the threshold to predetermined value and learn only a constraint function
RELATED WORK
Several works in the inverse RL literature consider the framework of constrained MDPs (Kalweit (et al., 2020; Fischer et al., [021 Ding & Xue 2022). However, those works focus on learning the reward function while assuming that the constraints are known. In contrast, we focus on learning constraints while assuming that the reward function is known.
Initial work in constraint learning focused on learning instantancous constraints with respect to states and actions. When states and actions are discrete, constraint sets can be learned to distinguish
feasible acttonaction pairs from infeasible ones (Chou al.. 2018 Scobee & Sastry 2019 McPherson et al. 2021 Park et al. 2020). In continuous domains, various types of constraint functions are learned to infer the boundaries of infeasible state-action pairs. This includes geometrie space
constraints (Armesto et al.) 2017 Pérez-D' Arpino & Shah, 2017). task space equality constraints (Lin)
al. 2013 2017), convex constraints (Menner et al. 2021) and parametric non-convex constraint functions (Chou et al., 2220; Malik et al.) (002)). Several other works learn local trajectory-baser
constraints with respect
single trajectory (Calinon & Billard 2008 Pais et al. 2013 Li &
Berenson
2016
Mehr et al., 2016 Ye & Alterovitz (2017).
The vast majority of previous work considers hard constraints, which fine in deterministic domains. However, in stochastic domains, the probability of violating a constraint can rarely be reduced to zero, making hard constraints inadequate. To that effect, Glazier et al. (2021) learn probabilistic constraints that hold with high probability in expert demonstrations. This approach extends the framework of maximum entropy inverse reinforcement learning to learn a constraint value that treated negative reward added to the given reward function. Since the probability of trajectory is proportional to the exponential of the rewards, this has the effect of reducing the probability of trajectories with high constraint values. Several other works also extend the inverse entropy RL framework to learn constraints that reduce some transition probabilities to 0 when a constraint violated, but this amounts to hard constraints again (Scobee & Sastry 2019 Park et al. 2020 McPherson et al.) [202]] Malik et al. (021). In another line of work, Bayesian approaches (Chou et al. al. (021)) have also been proposed learn a distribution over constraints due to the unidentifiability and uncertainty of the true underlying constraints. With the exception of (Papadımitriou et al. 202T), existing techniques that learn probabilistic constraints are restricted to discrete states and actions. Furthermore, the probabilistic constraints that are learned do not correspond to the soft constraints of constrained MDPs. Hence, we fill this gap by proposing a first technique that learns soft cumulative constraint functions bounded in expectations (as in constrained MDPs) for stochastic domains with any state-action spaces.
2021; [02]]Papadimitrion
APPROACH
4.1 INVERSE CONSTRAINT LEARNING: TWO PHASES
We propose an approach to do ICL, that is, given a reward r and demonstrations D, this strategy obtains constraint function such that when r, c used in the constrained RL procedure (any algorithm that optimizes Equation (2). the obtained policy x² explains the behavior in D. Our approach is based on the template of IRL. where typically alternate between policy optimization phase and a reward adjustment phase (Arora & Doshi (021). We first describe a theoretical procedure that captures the essence of our approach and then later adapt it into a practical algorithm. The theoretical approach starts with an empty set of policies (i.e., II = 0) and then grows this set of
Mathematically equivalent constraints can be obtained by multiplying the constraint function and the threshold by the same value. Therefore there is no loss in fixing 3 leam canonical constraint within the set of equivalent constraints, assuming that the leareed constraint function can take arbitrary values.
3

---

Published as a conference paper at ICLR 2023
policies by alternating between two optimization procedures until convergence:
POLICY OPTIMIZATION: Tt" == arg max (r) such that (c) s 8 and II + IIU [
(3)
CONSTRAINT ADJUSTMENT: c* =- arg max min J (c) such that J (c) < 8
(4)
sell
The optimization procedure in Equation (3 performs forward constrained RL (see Section
Background and Equation (2) for the definition of constrained RL and the notation J to find an optimal policy n" given reward function and a constraint function This optimal policy is then added to the set of policies II. This is followed by the optimization procedure in Equation (4) which adjusts the constraint function e to increase the constraint values of the policies in II while keeping the constraint value of the expert policy TH bounded by B. This is achieved by maximizing the accumulated constraint value for the most feasible (lowest total constraint value) policy in II. This selectively increases the constraint function until the most feasible optimal policies become infeasible. At each iteration of those two optimization procedures a new policy is found but its constraint value will be increased past 3 unless it corresponds to the expert policy (which is enforced by the constraint in Equation ) . By doing so, this approach will converge to the expert policy (or an equivalent policy when multiple policies can generate the same trajectories) as shown in Theorem []] Intuitively, this happens because all policies and trajectories except the expert's ' are made infeasible in the long run.
Theorem 1. Assuming there is unique policy x= that achieves J (r). the alternation of optimiza- tion procedures in Equation (3) and Equation 4 converges to a set of policies II such that the last policy x" added to II is equivalent to the expert policy nt" F. in the sense that "²' and x generate the (For proof, see Appendix (A)
same trajectories.
In practice, there several challenges in implementing the optimization procedures proposed in Equation (3) and Equation (4). First, we do not have the expert policy TE, but rather trajectories generated based on the expert policy. Also, the set II of policies can grow to become very large before convergence is achieved. Furthermore, convergence may not occur or may occur prematurely due to numerical issues and whether the policy space contains the expert policy. The optimization procedures include constraints and one of them requires min-max optimization. Therefore, we approximate the theoretical approach in Equation (3) and Equation (4) into the practical approach described in Algorithm|] !
As first step, we replace the optimization procedure Equation (4) by simpler optimization de- scribed in Equation (Sp. More precisely, we replace the max-min optimization (which is continuous- discrete optimization problem) of the constraint values of the policies in 11 by a maximization of the constraint value of the mixture Tii of policies in 11. In this case, the mixture Tmix is a collection of optimal policies where each policy has a weight, which is used in the computation of Jm (c) in Equation (5). The details of this computation are stated in Algorithm Overall, changing the objec- tive avoids challenging max-min optimization, but we lose the theoretical guarantee of convergence to policy equivalent the expert policy. However, find in our experiments that the algorithm still converges empirically.
== arg max J. (c) such that JC) S 3
(5)
Maximizing the constraint values of mixture of policies II usually tends increase the constraint values for all policies in II most of the time, and when policy's constraint value is not increased beyond 3 it will usually be policy close the expert policy.
4.2 SOLVING THE CONSTRAINED OPTIMIZATIONS THROUGH THE PENALTY METHOD
The constrained optimization problems Equation (3) Equation 4 Equation 5 described in the preceding subsection belong to the following general class of optimization problems (here, J,g are potentially non-linear and non-convex):
min f(y) such that (v) < 0
(6)
While many existing constrained RL algorithms formulate constrained optimization from Lagrangian perspective (Borkar 2005 Bhatnagar & Lakshmanan 2012 Tessler et al.) 2018 [Bohez et al.) 2019)
min-max problem that can be handled by gradient ascent-descent type algorithms, Lagrangian
4

---

Published as a conference paper at ICLR 2023
formulations are still challenging in terms of empirical convergence, notably suffering from oscillatory
behaviour (demonstrated for Equation (5) in Appendix (B.e)
Therefore,
less commonly used optimization framework, namely, the penalty method (Bertsekas) 2014; Donti et al. 2020) which converts a constrained problem into an unconstrained problem with non differentiable RELU term. Roughly, the approach starts by instantiating y = yo. and then two (sub) procedures are repeated for a few steps each until convergence: (a) first a feasible solution found by repeatedly modifying y in the direction of -Vg(y) until g (note that this is equivalent to ReeL (g()),, also called feasibility projection, then (b) soft loss is optimized that simultancously minimizes fyy while trying to keep y within the feasible region. This soft loss is as follows: (here, À is a hyperparameter):
min soft (v) == f(y) ARELU(g(Y))
As an alternative to Lagrangian based approaches, the penalty method has the advantage of simpler algorithmic implementation. Further, as we will demonstrate later in our experiments, it also performs well empirically.
Note that the choice of À is crucial here. In fact, plan to use the penalty method to optimize both Equation (3) and Equation (5). using different for each case. small À means that in the soft loss optimization subprocedure (Equation (7)), the gradient update of (eIUUggV)) is miniscule in comparison to the gradient update of fty) because of which may not stay within the feasible region during the soft loss optimization. In this case. the feasibility projection subprocedure is important to ensure the overall requirement of feasibility is met (even if exact feasibility cannot be guaranteed). Conversely, large À means that minimizing the soft loss is likely to ensure feasibility, and the feasibility projection subprocedure may therefore be omitted. This forms the basis for our proposed optimization procedures (see Algorithms ann 3 for the exact steps of these procedures). Namely, we perform forward constrained RL (Equation (3) using À = 0 with the feasibility projection subprocedure, and constraint adjustment (Equation (4) Equation (5) using a moderate/high À and
without the feasibility projection subprocedure.
should be noted that out the two procedures represented by Equation 3 and Equation 55 the constrained RL procedure can still be replaced by any equivalent algorithm that solves Equation efficiently. Even in that case, the primary optimization objectives remain the same as expressed in Equation 33. Equation (4) and Equation (5). However, preferred method for solving Equation is using a penalty formulation as described.
4.3 IMPROVING CONSTRAINT ADJUSTMENT
Figure Given expert (green) and agent (red) behaviour for an arbitrary Gridworld environment (first figure), the proposed algorithm decreases the constraint value for the expert behaviour (second figure) and increases the constraint value for the agent behaviour (third figure). Constraint learning is slower if there is overlap between the two behaviours since there exist some states for which the algorithm tries to simultaneously increase and decrease the constraint value. Note that dark gray represents higher constraint value, light gray medium constraint value and white represents lower constraint value.
We can further improve the constraint adjustment subprocedure by noticing that an overlap in expert and agent data used in computing the terms J (c) and _ (c) respectively in Equation (5) hampers convergence learning constraints. We can see this in the following way. We know that constraint adjustment (Equation (4), Equation (5) is finding the decision boundary between expert and agent trajectories (see Figure for an illustrationn. Consider the soft loss objective for Equation (5)
min (cf == -- Jmii (c) (RELU(JEE( 33)
5

---

Published as a conference paper at ICLR 2023
Here, the expert data is used to compute J (c), and the agent data is used to compute Jamen (c). Depending whether _ (c) 3 I or not, the ReLU term vanishes in soft (c). There are two cases. (I) If J - I O, then is already feasible, that is, for this value of the average constraint value across expert trajectories less than or equal to 3. there are expert (or expert-like) trajectories in agent data (used to compute - (c)). then will then end up increasing the constraint value across these expert trajectories (on average), which is not desirable since it will lead to becoming more infeasible. This will lead requiring more iterations converge in learning the constraint function c. (IT) If @ (c) - 3 > O, then there is a nonzero ReLU term in (-sot((e) (c). there are some expert or expert-like trajectories in agent data (and subsequently -Jffmiz (c)), and we take the gradient of (c), we will get two contrasting gradient terms trying to increase and decrease the constraint value across the same expert (or expert-like) trajectories. The gradient update associated with the ReI.U term is required since we want e to become feasible, but having expert (or expert-like) trajectories in -i (c) diminishes the effect of the ReLU term and then more iterations are needed for convergence of constraint function C.
To improve empirical convergence, we propose two reweightings: (a) we reweight the policies in Tmix (line 6 in Algorithm 3) so that policies dissimilar to the expert policy are favoured more in the calculation of - J meic (c), and (b) we reweight the individual trajectories in the expectation
-i (c) (line 8 in Algorithm B) to ensure that there is less or negligible weight associated with the expert expert- like trajectories. We can perform both these reweightings using a density estimator (specifically, RealNVP flow (Dinh al. ) ). The idea is to learn the density of expert or expert- like state attteaaction pairs and compute the negative log- log-probability (NLP) of any given trajectory's s state-action pairs at test time to determine if it is expert or expert-like, or not. Practically, this is estimated by computing the mean and std. deviation of the NLP of expert state-action pairs, and then at test time, checking if the NLP of the given state state-action pairs is within one std. deviation of the mean not. Please refer to lines 1-3 in Algorithm for the exact pseudocode.
Our complete algorithm is provided in Algorithm I As mentioned in Section 4.1 the algorithm repeatedly optimizes the objectives defined in Equation B and Equation (5) lines 6 and 7. The procedures for these optimizations are provided in Algorithmssaang as described in Section 4.2 Further, Algorithm Santthhm@essse describes constraint adjustment using normalizing flow as described in this subsection. Inputs and oulputs to each algorithm have also been specified.
which use the penalty method
Algorithm LNNEERSE-COOSTTRAANLLLEEEE
hyper-parameters: number of ICL iterations n, tolerance
input: expert dataset D = (TEEED == (((s); atsssiiiir
1: initialize normalizing flow f
2: optimize likelihood of f on expert state action data: max, SUMes))erree((0spp(...)
3: initialize constraint function c (parameterized by ()
4: for 1 n do
5: initialize policy ny (parameterized by 0;)
6: perform π. CONSTRAINED- RR (L, c)
7:
perform c == CONSTRAINT ADTUTRRAITTTA D, DALLAALLZEDDACCUAALLL-DIIIIIIIID
break 8:
D,.) Se
> See Section 5 for normalized accrual dissimilarity metric
9: end for
output: learned constraint function e (neural network with sigmoid output),
learned most recent policy म,
5 EXPERIMENTS
Environments. We conduct several experiments on the following environments: (a) Gridworld (A, B), which are 7x7 gridworld environments, (b) CartPole (MR or Move Right, Mid) which are variants of the CartPole environment from OpenAl Gym (Brockman et al. 2016), (c) Highway driving environment based on the HighD dataset (Krajewski et al. 2018), (d) Mujoco robotics environments (Ant- Constrained, HalfCheetah-Constie Constrained) and (c) Highway lane change environment based on the ExiD dataset (Moers et al.) 2022). Due to space requirements, we provide further details on all these environments (figures and explanation) in Appendix C.1

---

Published as a conference paper at ICLR 2023
Algorithm 2 CONSTRAINED-RI
hyper-parameters: learning rates n. n2, constraint threshold B, constrained RL epochs m input: policy mi parameterized by 0, constraint function e
1: for J S do
2: correct x; to be feasible: (iterate) 0, t 0, - Ve. (c)
3:
optimize expected discounted reward: 0; + 0, - 72 Vo, PPO-LOSS(T)
Proximal Policy Optimization (Schulman et al. 2017
4: end for output: learned policy i
Algorithm 3 CONSTRAINT-ADIUSMMA
byper-parameters: learning 73. penalty wt. X, constraint threshold B3 constraint adjustment epochs
input: policies Tis constraint function c, trained normalizing flow
expert dataset D = (TTTED == {{(s), (4))ssss))ssss
given: (T( == SUM ((s,) at)),
SAMPLE, (II, p) which generates (D] trajectories T = {(se. at) aellisesm) where for each
T, we choose π E II with prob. p(z),, then, 81 ~ u(.),, ar ~ (T 8041 ~ p(118,,0000
1: Pg (.) lo a.)
2: dg STD- (E.DDEEvs.o)e.rreD logg a))
3: w(T) := MEAN( (1 1o a) ME + σE)) trajectory dissimilarity w.r.t expert
4: construct policy dataset D.,, = SAMPLE, {m:). {1})
5: ú; = MEANFED=, w((r)
unnormalized policy weights
6: construct agent dataset DA SAMPLE, (II 1is P(T,) w
ppolicy reweighting
7: for 1 S S do
constraint function adjustment
8: compute J mur (c) SUM A SUM w(T)
(((T)=((T)
b trajectory reweighting
9: compute _ (c) == MEANFED(("(T)
compute soft loss (c) J mi (c) (C)
optimize constraint function Φ Φ 173 (oft(())
12:
end for output: constraint function e
Baselines and metrics. We use baselines compare against method: (a) GAIL-Constraint, which is based on Generative adversarial imitation learning method (Ho & Ermon 2016), and (b) Inverse constrained reinforcement learning (ICRL) (Malik et al., 202T) which is recent method that can learn arbitrary neural network constraints. In the absence of any other neural network constraint learning technique, only those two relevant baselines are compared to empirically. We provide further details in Appendix C.2 Next, we define two metrics for our experiments: (a) Constraint Mean Squared Error (CMSE), which is the mean squared error between the true constraint and the recovered constraint, and (b) Normalized Acerual Dissimilarity (NAD), which is dissimilarity measure computed between the expert and agent accruals (state-action visitations of policy/dataset) Further details can be found in Appendix
Finally, we use constraint neural network with sigmoid output, so that we can also interpret the outputs as safeness values (this is able to capture the true constraints arbitrarily closely, justified in Appendix (CI0). Our results are reported in Tables T 2 [ The average recovered constraint functions and accruals provided in Appendix D For the highway datasets, results reported in Figures 2 and [1 The hyperparameter configuration (e.g., choice of X), training strategy and training time statistics are elaborated in Appendix C
5.1 RESULTS FOR SYNTHETIC EXPERIMENTS
Results and discussion for the other environments be found in Appendix B.6
B.7 and
B.8
7

---

Published as a conference paper at ICLR 2023
Table 1: Constraint Mean Squared Error (Mean Std. Deviation across 5 seeds)
Algorithml, Environment->
Gridworld (A)
Gridworld (B)
CartPole (MR)
CartPole (Mid)
GAIL--Conttrint
0.31 = 0.01
0.25 = 0.01
0.12 # 0.03
0.25 + 0.02
ICRL
0.11 = 0.02
0.21 = 0.04
0.21 = 0.16
0.27 + 0.03
ICL (ours)
0.08 = 0.01
0.04 = 0.01
0.02 + 0.00
0.08 0.05
Table 2: Normalized Acerual Dissimilarity (Mean + Std. Deviation across 5 seeds)
Algorithml, Environment->
Gridworld (A)
Gridworld (B)
CartPole (MR)
CariPole (Mid)
GAII-Constraint
1.76 0.25
1.29 0.07
1.80 + 0.24
7.23 3.88
ICRL
1.73 # 0.47
2.15 = 0.92
12.32 # 0..88
13.21 # 1.81
ICL (ours)
0.36 + 0.10
1.26 = 0.62
1.63 + 0.89
3.04 + 1.93
Conter-to-cantee (m)
Center-to-ceeter (m)
folpee-l-eeeerr (m)
190
46
44
15
Agent velocity (m/s)
Agent velocity (m/s)
Agent velocity (m/s)
GAlL-Constraint
ICRL
ICL (B = 0.1) (ours)
Figure 2: Constraint functions (averaged across 5 seeds) recovered for the HighD highway driving environment. X-axis is the agent (ego car) velocity v (ms-)) )) Y-axis is the gap from the ego car to the vehicle (center-to-center distance) in front (m). Red points are (discretized) binary expert accruals, i.e. the states that the agent has been in. Both the baselines, GAII.-Constraint and ICRL assign high (white) constraint value to more expert states compared to our method, ICL, which overlaps less with the expert states. Ideally, is expected that the constraint value should be less for expert states, since they acceptable states. Further discussion is provided in Appendix B.7
Accuracy and sharpness of learned constraints. While our (practical) method is not guaranteed known problem (Chou et al..
to produce the true constraint function (constraint unidentifiability 2020)), empirically, we find that our method is still able to learn constraint functions that strongly resemble the true constraint function, as can be seen by our low CMSE scores (Table D. Other methods, e.g., GAIL-Constraint can find the correct constraint function for all environments except CartPole (Mid), however, the recovered constraint is more diffused throughout the state action space (see Appendix for training plots). In contrast, recovered constraint is pretty sharp, even without a regularizer.
Ability to learn complex constraints. From the training plots, find that the ICRL method is able to find the correct constraint function only for CartPole (MR) and to a less acceptable degree for Gridworld (A). This surprising as ICRL should be able to theoretically learn any arbitrary constraint function (note that we used the settings in (Malik et al.) 2021) except for common hyperparameters) and expect it to perform better than MaIn--onsrrr Our justification for this is two-fold. One, the authors of ICRL have only demonstrated their approach with simple single-propositin and for more complex settings, ICRL may not be able to perform as well. Second, ICRL may require more careful hyperparameter tuning for each constraint function setting, even with the same environment, depending on the constraint. On the other hand, our method is able to learn these relatively complex constraints satisfactorily with relatively fewer hyperparameters.
constraints,
Similarity of learned policy to expert policy. We find a strong resemblance between the accruals recovered by our method and the expert, as can be seen by our low NAD scores (Table In these scores, there some variability (e.g., in Cartpole (MR) and Gridworld (B) environments) which is expected, since the policies stochastic. For baselines, GAIL-Constrain accruals are similar the expert accruals except for CartPole (Mid) environment, where is also unable to learn the correct constraint function. Overall, this indicates that GAIL. is able correctly imitate the constrained
8

---

Published as a conference paper at ICLR 2023
expert across most environments, as would expect. On the other hand, ICRL accruals worse than GAIL, indicating that it is unable to satisfactorily imitate the constrained expert, even on the environments for which it is able to generate somewhat satisfactory constraint function. Training plots are provided in Appendix [
5.2 ABLATIONS AND QUESTIONS
Lagrangian formulations are quite popular in the literature, however, they are inherently min-max problems that admit gradient ascent-descent solutions which are prone to oscillatory behavior, as we observe in Appendix B Thus, we prefer the simpler framework of penalty method optimization which works well empirically,
Use of penalty method vs. Lagrangian. As mentioned in Section 4.2
Stochastic dynamics. Our algorithm can also be applied to stochastic environments, which we demonstrate in Appendix B.3 with slippery Gridworld environments. Stochasticity would lead to noisier demonstrations, and we find that little noise in the demonstrations helps in the quality of the recovered constraint, but too much noise can hamper the constraint learning.
Quality of learned policy w.r.t. imitation learning. We reiterate that our objective is to learn constraints, while the goal of imitation learning techniques (e.g.. behavior cloning) to directly learn policy that mimics the expert without inferring any reward or constraint function. Even then, once the constraints have been learned, they can be used with the given reward to learn a constrained policy, Depending on factors like quality of expert data etc., ICL can learn a better or worse policy compared to an imitation learning method. We not have definitive evidence to conclude that either is true.
Entropy regularized approaches and overfitting. Maximum entropy techniques (e.g.. (Scobee & Sastry 2019; Malik et al. (021]) have been previously proposed the literature. Entropy maximiza- tion indeed help reducing overfitting, but they require completely different mathematical setup. Our method doesn't 't belong to this family of approaches, but we hope to incorporate these strategies in the future reap the mentioned benefits. We still note that empirically, our approach doesn't seem
seem to suffer from any overfitting issues.
Use of normalizing flow and reweighting. We justify our choice of using a normalizing flow reweigh policies and trajectories (Algorithm 3) in Appendix [.2]] We find that ICL with reweighting performs better than ICL without reweighting, although the improvement minor.
6 CONCLUSION, LIMITATIONS AND FUTURE WORK
This paper introduces new technique to learn (soft) constraints from expert demonstrations, While several techniques have been proposed to learn constraints stochastic domains, they adapt the maximum entropy inverse RL framework to constraint learning (Scobee & Sastry| [219 Park et al.,
2020, McPherson et al. 2021 Glazier et al.) 2021 Malik et al. (021). We propose the first approach that recover soft cumulative constraints common in constrained MDPs.
Limitations. First, the proposed technique assumes that the reward function is known and can only learn a single constraint function. The challenge with simultaneously learning the reward function as well as multiple constraints is that there can be many equivalent configurations of reward and constraint functions that can generate the same trajectories (unidentifiability (Ng al. 2000)). Second, it is possible that expert demonstrations are sub-optimal. In that case, we hypothesize that our method would learn alternative constraints such that the provided demonstrations become optimal for these constraints. Finally, our approach requires several outer iterations of forward CRL and constraint adjustment, and as a result, it should in principle require more training time than the existing baselines.
Future work. First, for learning rewards and/or multiple constraints, we need a way to specify preference for specific combinations of reward and constraint functions. Second, it may be more desirable to express constraints not through expectations, but through probabilistic notions, e.g., a cumulative soft constraint which holds (i.e., is < (8) with probability p. Finally, we hope to extend our work to handle demonstrations from multiple experts and incorporate entropy regularization into framework for its benefits.

---

Published as a conference paper at ICLR 2023
6.1 ACKNOWLEDGEMENTS
Resources used in preparing this research at the University of Waterloo were provided by Huawei Canada, the province of Ontario and the government of Canada through CIFAR and companies sponsoring the Vector Institute.
REFERENCES
Eitan Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
Leopoldo Armesto, Jorren Bosga, Vladimir Ivan, and Sethu Vijayakumar, Efficient learning of constraints and generic null space policies. In 2017 IEEE International Conference Robotics and Automation (ICRA), pp. 1520-1526. IEEE, 2017.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
Dimitri P Bertsekas. Constrained optimization and Lagrange multiplier methods. Academic press, 2014.
Shalabh Bhatnagar and K Lakshmanan. An online actor-critic algorithm with function approximation for constrained markov decision processes. Journal Optimization Theory and Applications, 153 (3):688-7000 2012.
Steven Bohez, Abbas Abdolmaleki, Michael Neunert, Jonas Buchli, Nicolas Heess, and Raia Hadsell. Value constrained model-free continuous control. arXiv preprint arXiv:1902.04622 2019.
Vivek S Borkar. An actor-critic algorithm for constrained markov decision processes. Systems & control letters, 54(3):207-2133 2005.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Sylvain Calinon and Aude Billard. A probabilistic programming by demonstration framework handling constraints in joint space and task space. In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 367-372, 2008. doi: 10.1109/IROS SWW.//RROS.008..... 4650593.
Glen Chou, Dmitry Berenson, and Necmiye Ozay. Learning constraints from demonstrations. In International Workshop on the Algorithmic Foundations of Roboties, pp. 228 245. Springer, 2018.
Glen Chou, Necmiye Ozay, and Dmitry Berenson. Learning parametric constraints in high dimensions from demonstrations. In Conference on Robot Learning, pp. 121 1-1230. PMLR, 2020.
Glen Chou, Dmitry Berenson, and Necmiye Ozay. Uncertainty-awe aware constraint learning for adaptive safe motion planning from demonstrations. In Conference Robot Learning, pp. 1612-1661 1639. PMLR, 2021.
Fan Ding and Yeiang Xue. X-men: Guaranteed xor-maximum entropy constrained inverse reinforce- ment learning. 2022. URL http:///axxvv.rrooo httt:::://axxlvooor///////////// 2001..///22.11122222 11842
Laurent Dinh, Jascha -ock-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv 2016.
preprint
Priya L Donti, David Rolnick, and Zico Kolter. De3: A learning method for optimization with hard constraints. In International Conference on Learning Representations, 2020.
Johannes Fischer, Christoph Eyberg, Moritz Werling, and Martin Lauer. Sampling based inverse rein- forcement learning algorithms with safety constraints. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 791-798, 2021. doi: 10.109//ROS55166.22277 202I. 9636672.
10

---

Published as a conference paper at ICLR 2023
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalic T.H. Gayraud, Hicham Janati, Alain Rakotomamoniy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021. URL http: http://ww....rppppes////////////
451 html
Arie Glazier, Andrea Loreggia, Nicholas Mattei, Taher Rahgooy, Francesca Rossi, and K Brent Ven- able. Making human-like trade-offs in constrained environments by learning from demonstrations. arXiv preprint arXiv:2
2021.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Gabriel Kalweit, Maria Huegle, Moritz Werling, and Joschka Boedecker. Deep inverse q-learning with constraints. Advances in Neural Information Processing Systems, 33, 2020.
Robert Krajewski, Julian Bock, Laurent Kloeker, and Lutz Eckstein. The highd dataset: A drone dataset of naturalistic vehicle trajectories german highways for validation of highly automated driving systems. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 21 2118-2125, 2018, doi: 10. 101.1///S...221885595550
Jaeyoung Lee, Aravind Balakrishnan, Ashish Gaurav, Krzysztof Czarnecki, and Sean Sedwards. W ise ove: A framework investigate safe deep reinforcement learning for autonomous driving. In International Conference on Quantitative Evaluation of Systems, pp. 350-354. Springer, 2019.
Changshuo Li and Dmitry Berenson. Learning object orientation constraints and guiding constraints for narrow passages from one demonstration. In International symposium on experimental robotics, pp. 197-210. Springer, 2016
Hsiu-Chin Lin, Matthew Howard, and Sethu Vijayakumar. Learning null space projections. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 2613-2619. IEEE, 2015.
Hsiu-Chin Lin, Prabhakar Ray, and Matthew Howard. Learning task constraints in operational space formulation. In 2017 IEBE International Conference on Robotics and Automation (ICRA), pp. 309-315. IEBE, 2017.
Yiren Lu. Irl-imitation. http::://giitubb..om//..bbuuu wwtiiirlmmmmtttiiiooo 2019.
Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. International Conference on Machine Learning, pp. 7390-7399. PMLR, 2021.
David L McPherson, Kaylene C Stocking, and S Shankar Sastry. Maximum likelihood constraint inference from stochastic demonstrations. In 2021 IEEE Conference on Control Technology and Applications (CCTA), pp. 1208-1211 IEEE, 2021.
Negar Mehr, Roberto Horowitz, and Anca D Dragan. Inferring and assisting with constraints in shared autonomy. In 2016 IEEE 55th Conference Decision and Control (CDC), pp. 6689-6699 6696.
IEEE, 2016.
Mareel Menner, Peter Worsnop, and Melanie N. Zeilinger. Constrained inverse optimal control with application to human manipulation task. IEEE Transactions on Control Systems Technology, 29 (2)8826-834, 2021.
doi: 10.1 10.1119//TT.221999955555
Tobias Moers, Lennart Vater, Robert Krajewski, Julian Bock, Adrian Zlocki, and Lutz Eckstein. The exid dataset: A real-world trajectory dataset of highly interactive highway scenarios in germany. In 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 958-964, 2022, doi: 1099/ 2022.9827305.
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. Icml, volume 1, pp. 2, 2000.
11

---

Published as a conference paper at ICLR 2023
Lucia Pais, Keisuke Umezawa, Yoshihiko Nakamura, and Aude Billard. Learning robot skills through motion segmentation and constraints extraction. In HRI Workshop on Collaborative Manipulation. pp. 5. Citeseer, 2013.
Dimitris Papadimitriou, Usman Anwar, and Daniel S Brown. Bayesian inverse constrained rein- forcement learning, In Workshop on Safe and Robust Control of Uncertain Systems (NeurIPS), 2021.
Dachyung Park, Michael Noseworthy, Rohan Paul, Subhro Roy, and Nicholas Roy. Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 1005 1014. PMLR, 30 Oct-01 Nov 2020. URL https: http:///ppceeeeeg....oo mlr
wttp:///pppa....... html
Claudia Pérez-D' Arpino and Julie A Shah. C-learn: Learning geometric constraints from demonstra- tions for multi-step manipulation in shared autonomy. In 2017 IEEE International Conference Robotics and Automation (ICRA), pp. 4058 48884406..000 IEEE, 2017.
Alex Ray, Joshua Achiam, and Dario Amodei, Benchmarking Safe Exploration in Deep Reinforce- ment Learning. 2019.
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual conference Computational learning theory, pp. 101-103, 1998.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint urXiv:1707.088:
2017.
Dexter RR Scobee and S Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. International Conference on Learning Representations, 2019.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv: wttl::1805.. 1074, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, 5026-5033. IEEE, 2012.
Gu Ye and Ron Alterovitz. Demonstration Guided Motion Planning, pp. 291-307, Springer Interna- tional Publishing, Cham, 2017. ISBN 978-3-319-29366-99 doi: 10. 10.1097/7 URL https:: /wi//doiin// org 1007000..00000000 -19--88-----99-3--1 9_
12

---

Published as a conference paper at ICLR 2023
APPENDIX
A PROOF OF THEOREM
We give proof by contradiction under assumptions. The 3 assumptions are:
(i) The reward function is non-negative.
The space of policies is finite.
(iii There a unique policy T that achieves J (r). In other words no other policy achieves the same expected cumulative reward as T=
Note that the first assumption not restrictive since we can always shift reward function by adding sufficiently positive constant the reward of all state-action pairs such that the new reward function always non-negative. This new reward function is equivalent to the original one since it does not change the value ordering of policies.
We argue that the second assumption is reasonable in practice. Even we consider what appears to be continuous policy space because the parameterization of the policy space is continuous, in practice, parameters have finite precision and therefore the space of policies is necessarily finite (albeit possibly very large).
Based on assumption (ii), it should be clear that the alternation of the optimization procedures in Equation (3) and Equation (4) will converge fixed point. Since there are finitely many policies, and we add new policy to II at each iteration (until we reach the fixed point), in the worst case, the alternation will terminate once all policies have been added to
When the algorithm converges to fixed point, let nt' be the optimal policy found by policy opti- mization Equation ( and c be the optimal constraint funetion found by constraint adjustment in Equation (4) According to Equation
(3). we know that J (c*) B. Similarly, we show that the objective in Equation (4) is less than or equal to B:
max min (c) subject to JH (c) S
(8)
e wEll
min J=((")
since e* is the optimal solution of equation
(9)
8
since nt" E II and J (c') S
(10)
Next, if we assume that x² is not equivalent to x F, then show that the objective in Equation 4 is greater than B.
max min (c) such that J (c) s 3
(11)
where we choose č(s, a) = r(s, a)
2 min (c)
J (r)
(12)
=min JT (r)
by substituting è by its definition
well
(13)
J (r)B/J5 (r) 8
since (I(r) J (r) Vi II according to equation 3
and J (r) * J (r) by assumption (iii)
(14)
The key step in the above derivation is Equation 12 where we choose a specific constraint function è Intuitively, this constraint function is selected to parallel the reward function while making sure that J (c) < 3. We know that all policies in II achieve higher expected cumulative rewards than x F This follows from the fact that the optimization procedure in Equation B finds optimal policy x² at each iteration and this optimal policy is not equivalent to T based on our earlier assumption. So the policies in IT must earn higher expected cumulative rewards than zt E So if we select constraint
13

---

Published as a conference paper at ICLR 2023
function parallel to the reward function then the policies in II will have constraint values greater than the constraint value of x and therefore they should be ruled out in the constrained policy optimization Equation 3).
Since the incqualities contradict the inequalities
we conclude that x² must be equivalent
to π
B ADDITIONAL DISCUSSION
B.1 CONSTRAINED PPO
For the forward constrained RL procedure used in synthetic and HighD environments, we use modification of the PPO algorithm (Schulman al. 2017) that approximately ensures constraint feasibility. This modification is based on the penalty method mentioned in Section[4.2 This modified PPO works in two steps. First, we update the policy parameters ensure constraint feasibility. Then, we update the policy parameters to optimize the PPO objective. If the policy π is parameterized by 0, then these two steps are:
FEASIBILITY PROJECTION: O - 0 (e) B)
PPO UPDATE: 0 t 0 - 7/2 VoPPO-LLOS((T)
The gradient for the feasibility projection step can be computed just like policy gradient:
(c) =
c(ss, az)
(15)
t=to
V VoJ,(c) = (100)) 1e
Σ log)VolooTT(lls)
(16)
(O - > [o
(17)
The PPO-LOSS(T) term is computed over minibatch as follows (note that A(st, az) is an advantage estimate). This is the same as the original paper (Schulman et al. 2017), except for entropy term.
(18)
π (a: 84; Ooid)
PPO-LOSS(T) :- -E, min(r,, clip(re, 1 EPPO, 1 (ppo)) Xena (T)) (19)
This gradient can be computed through automatic differentiation.
Overall, do not have RELU term in the PPO update, since its gradient would interfere with the PPO update and cause difficulty in learning. Instead, the feasibility projection step tries to ensure feasibility of the solution before every PPO update. While this doesn't 't guarantee that feasibility will hold during and after the PPO update, we see empirically that feasibility approximately holds.
The fcasibility projection step is comparable projection step, since it projects the solution into the feasibility region, although not necessarily the closest feasible point.
Our results reported in Figures
B.2 ABLATION: EFFECT POLICY MIXTURE AND REWEIGHTING
To assess the effect of having a policy mixture and reweighting, we conduct experiments with two more variants of ICL on all the synthetic environments:
ICL wihout mixture policy or reweighting (use the policy learned in constrained RL step instead of Tmiz to do constraint adjustment step)
ICL with mixture policy but with reweighting (use uniform weights)
We find that ICL without a mixture policy or reweighting is unable perform well other methods with Gridworld (A) environment, Empirically, it is
Our results are reported in Figure 5
14

---

Published as a conference paper at ICLR 2023
Avg. Roward 1
Avg. Reward
Arg. Rewar
A Reward
Epoch
Epoch
Epoch
Epoch
Gridworld (A) (3 0.99)
Gridworld (B) (3 0.99)
CartPole (MR)
CartPole (Mid)
(3 50)
(B 30)
Figure 3: Avg reward per PPO epoch for the expert (1 seed). As seen, the modified PPO is able to learn good policy that achieves a high reward.
Avg. Violation
Avg. Violation
Avg. Violation
Avg. Violation
Epoch
Epoch
Epoch
Epoch
Gridworld (A) (3 = 0.99)
Gridworld (B) (B = 0.99)
CartPole (MR) (3 50)
CartPole (Mid) (33 = 30)
Figure 4: Avg constraint violations per PPO epoch for the expert (1 seed). As seen, the modified PPO is able to learn good policy that reduces the constraint violations as the training progresses.
prone to divergence and cyclic behavior. This divergent behavior can be observed in the Gridworld (A) case. ICL with mixture policy but with no reweighting unable converge as fast as ICL with mixture policy and reweighting in the Gridworld (B) and CartPole (Mid) environments. For the CartPole (MR) environment, all the methods converge fairly quickly and have almost the same performance. We also note that there is significant variance in the result for ICL on the CartPole (Mid) environment. Despite this, overall, ICL with mixture policy and reweighting performs better than the other variants (or close to the best). Thus, we report this variant (with mixture policy and reweighting) in Table and Table
421
422
CMSSB
CMSE
CMSE
CMSE
4.5
iteration
iteratinn (i)
ittrrtttion
iteration (i)
Gridworld (A) (3 0.99)
Gridworld (B) (3 0.99)
CartPole (MR) (3 50)
CartPole (Mid)
(3 30)
Figure 5: CMSE (Y-axis) ICL Iteration (X-axis), In each iteration, we do one complete procedure of constrained RL and one complete procedure of constraint function adjustment.
B.3 ENVIRONMENT STOCHASTICITY
To assess the effect of environment stochasticity on the performance of our algorithm, we conduct experiments with stochastic variant of Gridworld (A) environment. More specifically, we apply the
15

---

Published as a conference paper at ICLR 2023
Table Constraint Mean Squared Error
Algorithm Environment
Palip 0.1 0.02 # 0.00
Pstip 0.3
Pstip 0.5
ICL
0.03 = 0.01
0.08 = 0.00
Table 4: Normalized Accrual Dissimilarity
AlgorithmJ, Environment->
Palip = 0.1
Pslip 0.3
Pstip 0.5
ICL
0.45 + 0.10
1.13 0.37
1.29 + 0.20
Reported metrics (Mean + Std. Deviation across 5 seeds) for the experiments stochastic Gridworld (A) environments.
ICL algorithm (Algorithm 1) the Gridworld (A) environment, with varying values of the transition probability. A transition probability of implies deterministic transition to the next state. A transition probability Pstiy implies stochastic transition, such that the agent can enter the intended next state with this probability, and go to any other random direction with overall probability Patip- Our results are reported in Tables 3 andd and in Figure
Y-cooditzate
Y-oondiraate
Yoouddiaate
X-coordinate
X-caordinate
X-coordinate
X-coordinate
ICL
ICL
ICL
ICL
(Pslip 0)
(Palip 0.1)
(Pslip 0.3)
(Pslip 0.5)
Figure 6: Average constraint function value (averaged across training seeds) for stochastic Gridworld (A) environment, demonstrating the effect of changing Pslip-
Empirically, we observe that the increase in environment stochasticity leads to decrease in the performance of the constrained RL algorithm. Additionally, with a higher stochasticity, there is more noise in the demonstrations. When we increase the stochasticity slightly (going from deterministic environment to Pstip = 0.1), we find that the recovered constraint function has lower error (see Figure 6). This happens because with a little stochasticity, there is more exploration in terms of states visited, which leads to a more accurate constraint function since the algorithm tries to avoid more states. However, with more stochasticity (Palip 0.5), the demonstrations themselves become more noisy, and hence the algorithm recovers a worse constraint function than in the case with little stochasticity (see Table 3). Finally, we also find that aceruals of the learned policy also have more error w.r.t. provided demonstrations as the stochasticity increases (see Table 4).
B.4 COMPARISON W.R.T. T. A LAGRANGIAN IMPLEMENTATION OF EQUATION 5
Equation 5 can also be represented using a min-max optimization:
min max Tiii (c) X(J
(20)
is possible to perform this optimization instead of the penalty based strategy proposed in this paper. In terms of implementation, this would amount an alternating gradient ascent descent procedure, where we first ascent on c, followed by descent on A, and we repeat these steps until convergence. However, is the with min-max optimizations, this is prone training instability and oscillatory behavior. We demonstrate this with the Gridworld (A) environment. In particular, we can see this oscillatory behavior in the plot for the Lagrange multiplier À (Figure second subfigure). We also
16

---

Published as a conference paper at ICLR 2023
plot the expert satisfaction % for the Lagrangian and non Lagrangian implementation (ICL) (see third and fourth subfigure in Figure 7. The expert satisfaction is computed as the percentage of expert demonstrations satisfying the constraint using the current constraint function, during training. As the plots show, the Lagrangian implementation is prone to oscillatory behavior. Initially it does achieve high satisfaction, but time, the satisfaction degrades. This does not happen in the ICL implementation. This is our primary rationale for using a penalty based method for Equation (
In terms of the recovered constraint function, we find that the Lagrangian implementation is still able to achieve a noisy yet reasonable constraint function (see the first subfigure in Figure 7p. Therefore, the Lagrangian approach indeed correct in principle and can used recover a constraint function. However, in terms of other training metrics, the Lagrangian approach does not perform well. Our proposed method able to mitigate this.
Lambda
exper__satifaction
exsert_ioiissattoo
Y-cocrUmute
04
X-coerdinate
300 150
Avg. constraint value (5 seeds) Gridworld (A)
Lagrange multiplier À Expert satisfaction over Expert satisfaction over for Lagrangian implementation
time for Lagrangian implementation
time for ICI.
Figure 7: The first figure shows the avg. constraint function (averaged across 5 seeds) for the Lagrangian implementation. The second figure shows the value of the Lagrange multiplier during training for seed (X-axis denotes the adjustment iteration). The last two figures show the value of the expert satisfaction (%) for the Lagrangian implementation and the regular implementation (ICL).
B.5 EFFECT OF CONSTRAINT THRESHOLLD
We also assess the effect of 3 with the Gridworld (A, B) environments. We do not assess the effect of on the CartPole (MR, Mid) environments since our claim can be easily verified by just observing the learned costs for the Gridworld (A, B) environments. Our claim is as follows. With higher B, learned constraint function should have higher CMSE and more state-action pairs with higher constraint values. This is because since 3 is higher, the agent is allowed to visit more high constraint value state-action pairs, as the constraint threshold more relaxed. To verify this claim, we run experiments the Gridworld (A, B) environments. Specifically, in addition to B 0.99 (which corresponds to the results Tables [ and 2). we also run experiments with B = 2.99, 5.99. Our results reported in Figures 9 From the figures, it is apparent that claim correct.
Y-oordansee
Veoordantte
Ysoordiaaate
Yeoordinate
X-coerdinate
X-coedinate
X-coeriiate
X-coordinate
True constraint function
ICL (3 0.99)
ICL 2.99)
ICL (8 5.99)
Figure 8: Average constraint function value (averaged across 5 training seeds) for Gridworld (A) environment, demonstrating the effect changing B.
17

---

Published as a conference paper at ICLR 2023
Y-ocerdintte
V-cood dinatte
Y-soerdimnte
Vrooddinate
X-coordinate
X-caordinate
X-coerdinaee
X-caordinate
True constraint function
ICL
ICL (B = 2.99)
ICL (B = 5.99)
(5 = 0.99)
Figure 9: Average constraint function value (averaged across 5 training seeds) for Gridworld (B) environment, demonstrating the effect of changing B.
B.6 MUJoCO EXPERIMENTS
We perform robotics experiments using the Mujoco (Todorov et al.) 2012) environment setups described in the ICRL paper (Malik al. 202T) (Ant and HalfCheetah environments). While there are four Mujoco environments in the ICRL paper, two of them use implicit constraints (Point and Ant-Broken environments). As we learning constraint functions, use explicit constraints. To reduce training time, we use ICL with (highly optimized) PPO-Lagrange (Ray et al. 2019) instead of ICL with PPO-Penalty (Appendix B.I. principle, any procedure that can effciently optimize Equation 33) can be used for the policy optimization part of ICL, as mentioned in Section 4.2) The Mujoco environment setups are described Appendix C.1 and the hyperparameter configuration for PPO-Lagrange is described in Table 11 Our results are reported in Tables and
consider the two setups which
Table 5: Constraint Mean Squared Error (Mean + std. deviation across 5 seeds)
AlgorithmJ, Environment
Ant- Constrained
HalfCheetah-Constrrance
GAIL-Constraint
0.17 # 0.04
0.20 + 0.03
ICRL
0.41 0.00
0.35 0.17
ICL (ours)
0.07 = 0.00
0.05 = 0.00
Table 6: Normalized Accrual Dissimilarity (Mean + std. deviation across 5 seeds)
Algorithm.J, Environment->
Ant-Constrained
HalfCheetal-Consttrinee
8.02 2.84
14.38 + 2.36
ICRL
9.50 = 2.84
7.50 : 4.97
ICL (ours)
6.84 + 1.29
10.16 + 7.49
We observe that ICL able find the constraint more accurately compared GAIL-Constraint and ICRL, as can be seen from the low CMSE scores in Table (recovered constraint function plots in Figures 24 and 26). With respect the accruals, our method achieves the lowest NAD score for Ant- Lnt-Constrained environment (accrual plot in Figure 25 However, the NAD score achieved by ICL not the lowest for HalfCheetah-Comme Constrained environment, and instead, ICRL achieves lower score (Table 6). In fact, ICRI succeeds at matching the accruals (low NAD score) for this environment better than ICI., although finds a hard constraint.
This can be explained by visually inspecting the accruals for ICL and ICRL for the HalfCheetah- Constrained environment (Figure 27) More precisely, while both these accruals are shifted slightly to the right, which is appropriate for an agent trying to move right, ICRL has lower accrual mass for 2 < 0 and a larger accrual mass for 2 2, which indicates that it is able to find a better constrained optimal policy for this specific environment, compared to ICL (also Figure [O). However, do have a better constraint function (lower CMSE), and thus a constrained RL procedure that could find the optimal constrained policy in this specific setting could alleviate this issue.
18

---

Published as a conference paper at ICLR 2023
Avg. Epiisodic Reward
Avg. Episodic Reward
Avg. Cost
Avg. Cost
14
Training timesteps
Training timesteps
Training timesteps
Training timesteps
ICRL episodic reward ICRL average cost
ICL episodic reward
ICL average cost
Figure 10: Training plots for 2M timesteps for ICRL and ICL.
For the Ant-Constrained environment, ICRL completely fails recover reasonable constraint (Figure 24). This is probably due to the challenging nature of the constraint. GAll-Constraint performs moderately for these environments, while method find the (nearly) correct constraint.
B.7 RESULTS FOR REAL WORLD EXPERIMENT (HIGHD)
We report the results for the HighD driving environment Figure 2 Overall, all the methods are able to find constraint function that shows that as the agent's velocity increases, the center-to-center distance that must maintained between the cars also must increase. The time gap that must be maintained can be calculated by dividing the requisite center-to-centee distance by the agent's velocity. For all the methods, this time gap is approximately between 3-4.5 seconds. More precisely, at = 20ms -1 = 72kmh the gap is roughly between 60-80 m (equivalently, 3-4 seconds) for all methods, while close
40ms -1 144kmh" the gap between 125-150 150 (equivalently 3-4 seconds) for our method, and 160-180 m (equivalently 4-4.5 seconds) for GAlL-Constrain and ICRL. Note that our method the only one which doesn't assign high constraint value to large center-to-cente gaps (no white areas in the black regions in Figure 2 The possible justification for this is that the other methods are unable to explicitly ensure that expert trajectories are assigned low constraint value, while method able to do so through the constraint adjustment step. Moreover, our method has fewer accrual points (red) in the white (high constraint value) region compared to the baselines.
B.8 RESULTS FOR REAL WORLD EXPERIMENT (ExID)
DM
Signed distance (m))
Signed distance (m)
Signed distance (m)
344
16
-220
Lateral velocity (m/s)
Lateral velocity (un/s)
Lateral velocity (in's)
GAIL-Constraint
ICRL
ICL (3 5) (ours)
Constraint functions (averaged across 5 seeds) recovered for the ExiD highway lane change environment. White indicates a constraint value of 1, while black represents a constraint value of 0. X-axis is the agent (ego car) lateral velocity action (ms-1), Y-axis is the signed distance (m) to the center line the target lane. Red points are (discretized) binary expert accruals, i.e. and ICRL are unable
Figure 11:
the states/actions that the agent has been in. Both basclines, to find satisfactory constraint function for this task. On the other hand, our method, ICL, finds reasonable constraint function.
19

---

Published as a conference paper at ICLR 2023
We also conduct experiments with the ExiD environment (environment setup is described in Appendix The objective is to discover a constraint on the lateral velocity and signed distance to the target lane for doing a lane change in either direction. Note that just like the HighD environment, the real world constraint function is not known for this setting. The recovered constraint functions are reported in Figure and the accruals are reported in Figure 29
C.I
Overall, we find that only ICL is able to find satisfactory constraint function for this setting. GAIL- Constraint finds diffused constraint that overlaps the expert state-action pairs (and hence disallows them), while ICRL is unable to find a reasonable constraint. We can interpret the constraint plot of ICL as follows. Since the top-right and bottom-left quadrants (u 0, d and v S 0, S 0) are high in constraint value (white). they are relatively unsafe/prohibited. This is in line with expectation, since depending on the sign of the distance to the target lane, movement in one direction should be allowed and other should be disallowed (some risky behaviour is permitted since this is soft constraint). More specifically, to reduce the magnitude of d, we must apply a control input e of opposite sign. Further, the shape of the constraint is also not exactly square/rectangular. That is, it constrains movement to be in one direction when far away (e.g., for d = 10, v S -1), and allows get closer the target lane (e.g., for 2.5, any is allowed).
movement in both directions
C TRAINING SETUP AND EXPERIMENTAL DETAILS
All PPO variants mentioned in this paper are single-process for simplicity. For the tables, abbreviate the Gridworld (A, B), CartPole (MR, Mid), Ant-Constrained, HallCheetal--Constaanee environments as GA, GB, CMR, CMid, Ant and HC.
C.I ENVIRONMENTS
We use the following environments in this work:
Gridworld (A, B) are 7x7 gridworld environments (adapted from yrlu's repository (Lu
2019)). The start states, true constraint function and the goal state are indicated in Figure 12 The action space consists of 8 discrete actions including 4 nominal directions and 4 diagonal directions.
CartPole (MR, Mid) (MR means Move Right) are variants of CartPole environment from OpenAI Gym (Brockman et al. 2016) where the objective is to balance pole for as long as possible (maximum episode length is 200). The start regions and the true constraint function are indicated in Figure 12 Both variants may start in region of high constraint value, and the objective is to move to a region of low constraint value and balance the pole there, while the constraint function is being learned.
HighD environment:
We construct an environment using R 100 trajectories of length < 1000 from the HighD highway driving dataset (Krajewski et al. 2018) (see Figure 13). The environment is adapted from the Wise-Move framework (Lee et al. 2019). For each trajectory, the agent starts on straight road on the left side of the highway, and the objective is to reach the right side of the highway without colliding into any longitudinally moving cars. The state features consist of 7 ego features (x, y, speed, acceleration, rate of change of steering angle, steering angle and the heading angle) and 7 relevant predicates, propositions, and processed features (distance to the vehicle in front, processed distance, whether the ego is stopped, whether the ego has reached goal state, whether time limit has been exceeded, whether the ego is within road boundaries, and if ego has collided with another vehicle). The action space consists of single continuous action, i.c., acceleration. Note that for this environment, we do not have the true constraint function. Instead, we aim to learn a constraint function that is able capture the relationship between an agent's velocity and the distance to the car in front.
environments are Mujoco robotics envi- ronments used in (Malik et al.) 202T) (see Figure I4]. One difference in our experiments is the use of z 2 -1 constraint rather than z 2 3 constraint used in the original paper,
ant-Constrained,
WWACCheeaah-onnttraiie
20

---

Published as a conference paper at ICLR 2023
L
Y-coerdintte
Y-coordinste
Coustriiu vallee
Coustraiut volue
X-coordinate
X-caordinate
Pole position
Pole position
Gridworld (A)
Gridworld (B)
CartPole (MR)
CartPole (Mid)
Figure 12: Environments used in the experiments. White regions indicate constraint value of 1. Light green (or yellow) regions correspond to start states. Dark green regions correspond to the goal state. Gridworld environments have discrete states and discrete actions. Cartpole environments have continuous states and discrete actions. For the CartPole environments, the constraints represent desired behaviour. For CartPole (MR), the pole must stay in 2 0.2 and for CartPole (Mid), the pole is not allowed to go right (a = 1) for 2 and not allowed to go left (a 0) for x S -1.
t-000
Figure 13: HighD driving dataset environment. Leftmost white region indicates start region and rightmost white region indicates end region. Ego car is in blue, other are in red. This environment has both discrete and continuous states, and a continuous action.
which is more challenging than the original setting. More concretely, the agent must stay in z [-1, oo) and cannot go backwards, otherwise will violate the constraint.
Similar the HighD environment, we construct an environment using = 1000 trajectories of varying lengths (< 1000) from 5 different scenarios of ExiD highway driving dataset (Moers et al., 2022). To do so, we select trajectories where ego performs a lane change such that there are other vehicles around the ego. On every reset, the environment initializes to the starting location of one of the 1000 trajectories. The goal is to perform lane change to reach the target lane, which can be either to the left or right. The state consists of the signed distance (m) to the target lane and the action a continuous lateral velocity (ms-)) that can be directly controlled by the agent.
ExiD environment:
The environments are more clearly illustrated in Figures 12 and
C.2 DESCRIPTION OF BASELINE METHODS
We use the following two baselines to compare against our method:
Generative adversarial imitation learning (Ho & Ermon 2016 is an imitation learning method that can used to learn a policy that mimics the expert policy. The discriminator can be considered as a local reward function that incentivizes the agent to mimic the expert. We assume that the agent maximizing the reward r(s, a) == ro a) log(1 c(s, a)) where is the given true reward, and then the log term corresponds to the GAIL's discriminator. Note that when c(s, a) = 0, the discriminator reward is 0, and when c(s, a) 1, the discriminator reward tends to -o0.
GAII-CConstraint
Inverse constrained reinforcement learning (ICRL) Malik et al. 2021 is recent method that able learn arbitrary Markovian neural network constraint functions, however, can only handle hard constraints.
For both these methods, we use a similar training regime as adopted by (Malik et al. 2021) however, try to keep the constraint function architecture fixed across all experiments (baselines and our method). Note that we did not include the method in Scobee and Sastry (Scobee Sastry] (2019) as is unable to handle continuous state spaces. Similarly, did not include the binary constraint
21

---

Published as a conference paper at ICLR 2023
Ant-Constrained
HalfCheetah-Constie
Figure 14: Mujoco environments used in the experiments. These environments have continuous states
and continuous actions. Figures taken from ICRL paper (Malik et al. 2021). Instead the original constraint 2 -3, consider more challenging constraint
Real world map for the shown scenario.
Start (cyan) and target (green) lanes for the lane change in this scenario.
Simplified environment (agent starts from left- most point)
Simplified environment (agent moves accord- ing to chosen action)
Figure 15:
ExiD dataset lane change environment (one of multiple scenarios). The state space contains the signed distance to the target lane, and the action space contains the lateral velocity. There no other vehicles, only the ego car. The objective is to find constraint the lateral velocity (action) that an agent must maintain for any certain signed distance from the target lane.
22

---

Published as a conference paper at ICLR 2023
(BC) method described in (Malik et al. 2021) GC and ICRL empirically perform better than BC, according to their experiments.
C.3 METRICS
We define metrics for our experiments.
Constraint Mean Squared Error (CMSE) is computed as the mean squared error between the true constraint function and recovered constraint function on a uniformly discretized state-action space for the respective environment.
Normalized Accrual Dissimilarity (NAD) is computed as follows. Given policy learned by the method, we compute an agent dataset of trajectories. Then, the accrual (state-action visitation frequency) is computed for both the agent dataset and the expert dataset over a uniformly discretized state-action space, which the same as the one used for CMSE. Finally, the accruals are normalized to sum to 1, and the Wasserstein distance is computed between the accruals.
More concretely, the NAD metric is computed follows. Given the policy learned by any method (ICL or baseline), we sample a dataset 7 of agent trajectories (note that [Dx] [D], where D is the expert dataset):
D == SAMPLE, (II (=),p= (1}), SAMPLE is defined in Algorithm 1
Then, the accrual (state-action visitation frequency) computed for both the agent dataset and the expert dataset uniformly discretized state-action action space (this depends the constraint function inputs). For the various environments, these are:
Gridworld (A, B): {(a, gll,y, E (0.,,.... 6}}
CartPole (MR, Mid): (zr,a)lz 224.4 223, ... ,24), a (0, 1})
(0,1,...,80))) (here, is in ms and g is in HighD: {(t,g) ()c.g)) E {0. 1... 40},9 10,1,... .808)
pixels, where pixel 2.5m)
Ant-Constrained, SelrCheetah-Consraimie {z|2 E {-5, 4.9, ... ,5}
ExiD: {(d, v)ld {-10, -9.5, ... ,10,, {-2.5, 2.4, 2.5}} (here d is in and u in ms-1)
Finally, the accruals are normalized to sum to 1, and the Wasserstein distance computed between the accruals. If the environment is computed and summed across actions. Otherwise, the 2D distance is computed. We use the Python Optimal Transport library (Flamary et al. 2021). specifically the emd2 function achieve this.
Hat.-onttraine/HllChheeeaaa Soaahh-Cootttaiieed the ID distance
C.4 COMMON HYPERPARAMETERS
Hyperparameters common to all the experiments are listed in Table
Conttrainnd continuous action space environ- ments, we use TANH activation for the policy, which outputs the mean and std. deviation of learned Gaussian action distribution.
Since
The inputs to the constraint functions are as follows:
Gridworld (A, B): x, y coordinates of the agent's position in the 7x7 grid
CartPole (MR, Mid): ir position of the cart, and the discrete action (0/1 corresponding
left or right)
HighD: velocity (ms 1has- of the ego, and the distance of the ego to the front vehicle in pixels (1 pixel R= 2.5 m)
Ant-Constrained, HallCheetah-Comssie Constrained: z-coordinate of the agent
ExiD: signed distance (m) the ego to the center line of the target lane, and the lateral velocity action (ms-1) of the ego
23

---

Published as a conference paper at ICLR 2023
Table 7: Common Hyperparameters
Hyperparameter
Value(s)
PPO learning rate (722
5 x 10
Constraint function learning rate (7a)
5 X 10-4
Constraint function hidden layers
64+RELU, 64+RELU
Constraint function final activation
SIGMOID 64+RELU, 64+RELU
PPO policy layers PPO value fn. layers
64+RELU, 64+RELU
Minibatch size
64
PPO clip parameter (CPPO)
0.1
PPO entropy coefficient (Aent) PPO updates per epoch
0.01
25
No. trajectories for any dataset Training seeds
50
1, 2, 3, 4,
The environment maximum horizons (episodes are terminated after these many steps) are as follows:
Gridworld (A, 50 steps
CartPole (MR, Mid): 200 steps
HighD: 1000 steps
Ant-Constrained: 500 steps
1000 steps
ExiD: 1000 steps
C.5 HYPERPARAMETERS FOR GAIL-CONSTRAINT
Hyperparameters specific the GAIL-Constraint method listed in Table This method was adapted from Malik et al. Malik et al., (02T) and they performed 1M timesteps of training on all environments. To be that this method performs expected, did 2M timesteps of training (except for the HighD environment where we do 0.4M timesteps of training, since was computationally expensive to simulate).
Table GAIL-Constraint Hyperparameters
Environment
Hyperparameter
GA
GB
CMR
CMid
HighD
Discount factor (7) PPO steps per epoch PPO value fn. loss coefficient
1.0
1.0
0.99
0.99
0.99
2000
2000
2000
2000
2000
0.5
0.5
0.5
0.5
0.5
PPO gradient clip value
0.5
0.5
0.5
0.5
0.5
PPO total steps
2M
2M
2M
2M
0.4M
Environment
Hyperparameter
Ant
HC
ExiD
Discount factor (7)
0.99
0.99
0.99
PPO steps per epoch
4000
4000
1000
PPO value fin loss coefficient
0.5
0.5
0.5
PPO gradient clip value
0.5
0.5
0.5
PPO total steps
3.5M
3.5M
3.5M
C.6 HYPERPARAMETERS FOR ICRL
Similar to GAIL-Constre Constrann,,, we performed 2M timesteps of training on all environments except HighD, where performed 0.4M timesteps of training. Hyperparameters are listed in Table ICRL.
24

---

Published as a conference paper at ICLR 2023
did not perform expected two out of the four synthetic environments, that is, Gridworld (B) and the CartPole (Mid) environments, and may require further environment specific hyperparameter tuning. Nevertheless,
these parameters since they (somewhat) worked on Gridworld (A) and CartPole (MR) environments. Note that for a fair comparison, we tried to ensure that for some of method, ICI. (these parameters number of rollouts,
the parameters, the same value batch size, learning rate, 7, constraint function architecture etc.).
Table ICRL Hyperparameters
Environment CMR
Hyperparameter
GA
GB
CMid
HighD
Discount factor (7)
1.0
1.0
0.99
0.99
0.99
PPO steps per epoch PPO value fnn loss coefficient
2000
2000
2000
2000
2000
0.5
0.5
0.5
0.5
0.5
Constraint value fn. loss coefficient
0.5
0.5
0.5
0.5
0.5
PPO gradient clip value
0.5
0.5
0.5
0.5
0.5
Eval episodes
100
100
100
100
100
ICRL Iterations
200
200
200
200
40
Forward PPO timesteps
10000
10000
10000
10000
10000
PPO Budget
0
0
0
0
0
Penalty initial value (v)
0.1
0.1
0.1
0.1
Penalty learning rate
0.01
0.01
0.01
0.01
0.01
Per step importance sampling
Yes
Yes
Yes
Yes
Yes
Forward KL (Old/New)
10
10
10
10
10
Backward KL (New/Old)
2.5
2.5
2.5
2.5
2.5
Backward iterations
5
5
5
5
5
Constraint network reg. coefficient
0.6
0.6
0.6
0.6
0.6
Environment
Hyperparameter
Ant
HC
ExiD
Discount factor (r)
0.99
0.99
0.99
PPO steps per epoch PPO value fn. loss coefficient
4000
4000
1000
0.5
0.5
0.5
Constraint value fn. loss coefficient
0.5
0.5
0.5
PPO gradient clip value
0.5
0.5
0.5
Eval episodes
100
100
100
ICRL Iterations
125
125
250
Forward PPO timesteps
10000
10000
10000
PPO Budget
0
0
0
Penalty initial value (v)
0.1
0.1
0.1
Penalty learning rate
0.01
0.01
0.01
Per step importance sampling
Yes
Yes
Yes
Forward KL (Old/New)
10
10
10
Backward KL (New/Old)
2.5
2.5
2.5
Backward iterations
5
5
5
Constraint network reg, coefficient
0.6
0.6
0.6
C.7 HYPERPARAMETERS FOR ICL (OUR METHOD)
The hyperparameters for our method are listed in Table 10 Our PPO implementation differs from the baselines (the baseline code was adapted from Malik al. Malik et al. 2021) official repository) as it does fixed number of episodes per epoch rather than doing fixed number of environment steps per epoch. We adjusted the number PPO epochs in ICL so that each PPO procedure is roughly the same length (or less) the ICRL/GAIL PPO procedure. Even then, the entire ICL training process requires lot more steps than the GAIL-Constrain/ICRR baselines, since every iteration requires a complete PPO procedure. We have mentioned this in conclusions section.
For the Gridworld (A, B) environments, we chose 3 = 0.99 to disallow any state of constraint value 1. For the CartPole (MR, Mid) environments, the cartpole may start in high constraint value region.
25

---

Published as a conference paper at ICLR 2023
To allow it to navigate low constraint value region, chose high B. We chose , by doing hyperparameter search on (30,50) for the CartPole environments.
For some environments, forward constrained RL procedure. This implementation (provided by OpenAl) is highly optimized for constrained RL tasks, especially roboties tasks. We use this variant of ICL for Mujoco and ExiD experiments. For this implementation, we mostly use the default parameters provided by |aa al. 2019). A list of relevant hyperparameters are provided in Table IT] Certain hyperparameters have not been altered (e.g., learning rate, training iterations etc.). PPO epochs for the intermediate CRL procedures have also been kept 50 (as in the source code), which is sufficient to achieve decent performance for these constrained environments, 3 was chosen by doing hyperparameter search on (5, 15] for the Mujoco and ExiD environments.
variant of ICL that uses PPO-Lagrange (Ray al. 2019) the
To choose A, we tried values 1.5, 15, 150 and found that 1.5 was insufficient for the optimization. We know that À the RELU multiplier term the soft loss objective, and cannot be low value for constraint adjustment procedure. Hence, we chose value of 15 which was appropriate for our experiments.
Table 10: ICL Hyperparameters
Environment)
Hyperparameter
GA
GB
CMR
CMid
HighD
Discount factor (7)
1.0
1.0
0.99
0.99
0.99
ICL Iterations (n)
10
10
3
Correction learning rate (m))
2.5
10-5
2.5
10-3
2.5
10-5
2.5
10-5
2.5
10-3
PPO episodes per epoch
20
20
20
20
20
8
0.99
0.99
50
30
0.1
Constraint update epochs (e)
20
20
20
20
25
PPO epochs (m)
500
500
300
300
50
Soft loss coefficient À
15
15
15
15
15
Table 11: ICL Hyperparameters (with PPO-I Lagrange from Ray et al. (019
Environment
Hyperparameter
Ant
HC
ExiD
Discount factor (n)
0.99
0.99
0.99
GAE Lambda (AGAE)
0.97
0.97
0.97
ICL Iterations (n)
5
5
5
PPO steps per epoch
4000
4000
1000
B
5
15
5
Constraint update epochs (e)
25
25
PPO epochs (m)
50
50
50
C.8 TRAINING TIME STATISTICS
Training time statistics are provided in Table 12 For each environment and method, we report the average training time hours and minutes, averaged across seeds. In principle, one forward constrained RL procedure takes t time and one constraint adjustment procedure takes i' time, we expect GAIL-Constraint and ICRL to take at least 2t time, since we run these procedures at least twice as long as one forward procedure (longer for Mujoco and ExiD environments, but still a constant times t). With n iterations ICL, ICL should take at least nt( + t') time which depends on f, t', d',n n. Empirically, since Penalty) differs from the baselines, we observe that ICL is actually comparable to the baselines in overall runtime. We also note that the training times may depend CPU/GPU load.
10 and the implementation of forward constrained RL (PPO- OPPO-Lagrange/PPO.
26

---

Published as a conference paper at ICLR 2023
Table 12: Average training times (hours/minutes), averaged across 5 seeds. PPOPen refers to ICL implementation with PPO-Penalty, and PPOLag refers to the ICL implementation with PPO-Lagrange (Ray al. 2019).
Average training time
Environment Method->
GAIL ConstConntr
ICRL
ICL (n, type)
GA
h m
10 m
16 (10, PPOPen)
GB
7 13 m
10 37
h 20 (1O, PPOPen)
CMR
7h 39 m
15 56 m
9 (10, PPOPen)
CMid
8 0 m
16 m
7 40 m (10, PPOPen)
Highl)
4h m
3h
5 (3, PPOPen)
Ant
2 8 m
6h
6h 25 m (5, PPOLag)
HC
h 32 m
78
9 m (5, PPOLag)
Exil
1 52 m
32
2 22 (5, PPOLag)
C.9 EXPERT DATASET GENERATION PROCESS
For synthetic experiments (Gridworld and CartPole) and robotics experiments (Mujoco), we simply perform constrained RL for sufficient number of epochs until convergence. Afterwards, we generate trajectories using the learned policy, ensuring that the expected discounted constraint value across the expert dataset is < 3 for the specific environment.
For the HighD dataset, the data collection process is described in Krajewski et al. (2018). For this dataset, we choose one of the multiple scenarios in this dataset and use s 100 trajectories from this scenario that go from the start region to the end region. For the ExiD dataset, the data collection process is similarly described in Moers et al. (202)). For this dataset, we randomly choose 5 scenarios, and filter tracks get 1000 trajectories in each of which vehicle performs single lane change, with no vehicle in the target lane.
C.10 CONSTRAINT FUNCTION ARCHITECTURE CHOICE
We also further assess whether the chosen constraint function architecture sufficient capture the true constraint arbitrarily closely. To perform this assessment, we consider the following constraint function architectures:
architectures where the number of hidden nodes vary:
(A1): two hidden layers with 32 nodes each and RELU activation, followed by sigmoid activation for the output
(B): hidden layers with 64 nodes each and RELU activation, followed by sigmoid activation for the output; this is the architecture we use for all the ICL. and baseline experiments
(Ci): hidden layers with 128 nodes each and RELU activation, followed by sigmoid activation for the output
architectures where the number of hidden layers vary:
(A2): one hidden layers with 64 nodes and RELU activation, followed by sigmoid
activation for the output
(B): two hidden layers with 64 nodes each and RELU activation, followed by sigmoid activation for the output; this the architecture we use for all the ICL and baseline
experiments
(C2): three hidden layers with 64 nodes each and RELU activation, followed by sigmoid activation for the output
Given particular constraint function architecture, we train to minimize enror w.r.t. the known true constraint function, using supervised learning. This produces a lower bound on the CMSE for that setting. Intuitively, architecture with more representational power will lead to lower value of this [3]and
lower bound. Our results are reported in Tables T Land[[ [4] As can be seen from the tables, having too few hidden nodes layers is not sufficient arbitrarily approximate the constraint function.
27

---

Published as a conference paper at ICLR 2023
Table 13: CMSE achieved by various constraint function architectures (varying number of hidden nodes), which can be treated as lower bounds for CMSE in the requisite settings.
Architecture
Environment], CMSE->
A
B
GA
0.03 0.01
0.00 = 0.00
0.00 # 0.00
GB
0.02 = 0.01
0.00 0.00
0.00 + 0.00
CMR
0.00 0.00
0.00
CMid
0.01 = 0.00
0.00
0.00 = 0.00
Ant
0.00 = 0.00
0.00 0.00
0.00 # 0.00
HC
0.00 0.00
0.00 0.00
0.00 0.00
Table 14: CMSE achieved by various constraint function architectures (varying number of hidden layers), which can be treated lower bounds for CMSE in the requisite settings.
Architecture
Environment CMSE-
A2
C2
GA
0.06 0.00
0.00
0.00
0.00 # 0.00
GB
0.09 # 0.01
0.00 = 0.00
0.00 + 0.00
CMR
0.01 = 0.00
0.00
0.00 0.00
CMid
0.02 0.00
0.00
0.00 # 0.00
Ant
0.01 # 0.00
0.00
0.00 # 0.00
HC
0.01 = 0.00
0.00 0.00
0.00 + 0.00
D TRAINING PLOTS
For completeness, we provide the training plots for the recovered constraint functions (Figures 16
18
29 Note that we have reported the costs for the HighD/ExiD environments in the main paper and in the Appendix (Figures
20
22
24 26) and their respective accruals (Figures 17 19 21
23
25
I1I.
Please note that we have omitted the individual reward and constraint value plots, since there aren't t any interesting insights. Typically, the reward goes up over time and the constraint value goes down over time, similar to Figures 3 and
Y-coorditate
Y-uordinatee
Y-cuordicate
Yeuurdinate
X-coordinate
X-coordinate
X-coerdinate
X-coordinate
True constraint fn
CAIl-Constraint
ICRL
ICL (8 0.99)
Figure 16: Average constraint function value (averaged across training seeds) for Gridworld (A) environment.
28

---

Published as a conference paper at ICLR 2023
Y-corduuat
Yeoorduam dustte
Yeourdiaate
X-coordinate
X-coordinate
X-coordinate
Expert accruals
GAIL-Constraint
ICRL
ICL (3 = 0.99)
Figure 17: Average normalized accruals (averaged across 5 training seeds) for Gridworld (A) environment.
Y-coordiaste
Yeoordieate
Yeoendizatee
Feoordirate
X-coordinate
X-coordinate
X-coerdinate
X-coordinate
True constraint fn.
GAIL-Constrain
ICRL
ICL (B 0.99)
Figure 18: Average constraint function value (averaged across training seeds) for Gridworld (B) environment.
Y-coordiatee
Y-oordinate
Y-wvordiztee
Yeoordinate dinatte
X-coordinate
X-cocrdinate
X-coerdinate
X-cocrdinate
Expert accruals
GAIL-Constraint
ICRL
ICL ( 0.99)
Figure 19: Average normalized accruals (averaged across 5 training seeds) for Gridworld (B) environment.
Cocsenint value
Consemiot value
Coustint value 2
Conseriot value
Pole position
position
Pole position
position
True constraint fn.
GAlL-Constraint
ICRL
ICL (3 = 50)
Figure 20: Average constraint function value (averaged across 5 training seeds) for CartPole (MR) environment. Blue line represents the constraint value for = O (going left) and the orange line represents the constraint value for a = (going right).
29

---

Published as a conference paper at ICLR 2023
Accmal valiic
Accmal value
Accmal valic
Accmal value
Pole position
Polo position
Polo position
Pole posin.on
Expert accruals
GAIL-Constraint
ICRL
ICL (3 50)
Figure 21: Average normalized accruals (averaged across 5 training seeds) for CartPole (MR) environment.
Cocstraine valoe
Constraint valuee
Constraiat valie
Constraiot valoe
Pole positioa
Pole position
Polo position
Pole position
True constraint fn.
GAIL-Constraint
ICRL
ICL (3 30)
Figure 22: Average constraint function value (averaged across 5 training seeds) for CartPole (Mid) environment. Blue line represents the constraint value for a = O (going left) and the orange line represents the constraint value for (going right).
Accual valoe
Accrual value
Accuaa valce
Accrual
Pole position
Pole position
Pole position
Pole position
Expert accruals
GAIL-Constraint
ICRL
ICL (3 30)
Figure 23: Average normalized accruals (averaged across training seeds) for CartPole (Mid) environment.
Constraint valie
Constraint valoe
Constraint value
W
Constraint
z-coordimlle
2-coordinale
z-coordinate
z-corddiull
True constraint fn
MAll-Constraint
ICRL
ICL (8 5)
Figure 24: Average constraint function value (averaged across training seeds) for Ant-Constrained environment.
30

---

Published as a conference paper at ICLR 2023
Accrmal vaine
mmmm M
Accrual value
Accrual
z-coordinate
z-coerdinate
z-coordinate
Expert accruals
ICRL
ICL (3 5)
Figure 25: Average normalized accruals (averaged across 5 training seeds) for Ant-Constraine environment.
34-
Constraint valuee
Constraini valloe
Constraint
Constraint
2-coordinate
2-coordinate
2-coordinate
z-coordinate
True constraint fn.
GAIL-Constraint
ICRL
ICL (8 5)
Figure 26: Average constraint function value (averaged across 5 training seeds) for HalfCheetah- Constrained environment.
Mhy
Accrual value
Accmial value
Accrual
z=coordimale
2-coordinate
z-coordimale
Expert accruals
GAIL-Constraint
ICRL
ICL (3 5)
Figure 27: Average normalized accruals (averaged across training seeds) for HalfCheetah-
Constrained environment.
Center to cosier (m)
Center (ta)
Conter center (co)
Agent velecity (m/9)
Agont velocity (m/8)
Agent velocity (m/s)
Agent velocity (m//)
Expert accruals
GAIL-Constraint
ICRL
ICL (8 = 0.1)
Figure 28: Average normalized binary accruals (averaged across 5 training seeds) for HighD en- vironment. The X-axis and Y-axis are ego velocity in ms-1 and gap to the front vehicle in m respectively.
31

---

Published as a conference paper at ICLR 2023
Signed distance (m0)
Signed distunce (10)
Sigred distancc (1n)
Sigmed
velocity (m/))
Laterl velocity (m/))
Lateral velocity (m19)
Lateral velocity
Expert accruals
GAIl-Constraint
ICRL
ICL (3 = 5)
Figure 29: Average normalized accruals (averaged across 5 training seeds) for ExiD lane change environment. The X-axis and Y-axis are lateral velocity in ms= and signed distance to the the center line of the target lane d in m respectively.
32


Reviews:
[{"review_id": "o5rkADQXJ3", "reviewer": "Reviewer_EuC1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for learning soft constraints in sequential decision making problems that can be modeled as MDPs, under the assumption that the entire MDP model is known (both transition probabilities and the reward function), but a constraint on the state-action space exists that is known only to a decision maker who can provide optimal decisions that respect this constraint, and the objective is to recover the constraint from the observed decisions of the expert. Empirical evaluation on several domains demonstrates that the proposed method is far superior to estimating the constraint than two existing methods. ", "summary_of_the_review": "The paper proposes a novel method for recovering soft constraints in sequential decision problems, where the MDP model is known, and expert demonstrations are available under the unknown constraint. The empirical evaluation demonstrates that the method has much lower MSE in recovering the unknown constraint that two existing baseline methods.", "strength_and_weaknesses": "The paper proposes a principled way of addressing the problem, borrowing from the computational principle and machinery of inverse reinforcement learning (IRL) methods, and the empirical evaluation appears to be convincing. The limitation that there is only one constraint is probably not significant, as this constraint could potentially be an arbitrarily complex function on state/action pairs. (Unlike many IRL algorithms, where the learned reward function is linear in the vector of parameters.) It appears that the chosen parametric form of the constraint function is a neural net with two hidden layers of 64 units each, ReLU activations, and final output with a sigmoid activation function (Table 5). Why this form, and is it really sufficient to represent arbitrary functions? What about the constraints in the chosen experimental environments, how well does it represent its constraints? Maybe learning the constraints directly from the known functions by means of supervised learning could answer this question, and also effectively provide a lower bound on the MSE that this method can provide?  \n\nAnother thing I was not clear about is the effect of the initial form of the constraint (that is, c in Equation 3 on the very first iteration of the algorithm). Was it completely random in the computational experiments, and does its value matter? In other words, does the algorithm always converge to a global minimum, or there are local minima?\n\nSome minor typos:\nP.1: \"are therefore are\" -> \"therefore are\"\nP.3: \"in an other line of work\" -> \"in another line of work\"\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "hnbmQDCOO8", "reviewer": "Reviewer_5hKt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of inverse constraint learning, whose goal is to learn the unknown (soft) constraints that the expert demonstrations obey in a given expert dataset, assuming that the reward function is known. The authors propose a novel method named ICL, which learns a constraint function in a way that the total constraint is bounded for the expert demonstrations but maximized for agent behaviors. They demonstrate that their proposed method outperforms two baselines on three environments.", "summary_of_the_review": "While their proposed method is well-motivated and sensible, I cannot recommend acceptance in its current form due to its limited experimental evaluation. I would be happy to raise my score if the authors address my concerns above.\n\n-------------(11/16 Update)-------------\n\nI raised my score from 5 to 6 as the response addressed my initial concerns.", "strength_and_weaknesses": "**[Strengths]**\n- The paper is well-written and easy to understand.\n- The proposed approach is theoretically well-motivated and sensible.\n- The experiments include ablation studies.\n\n**[Weakness]**\n- The main weakness of the paper is its limited empirical evaluation. The authors tested their method only on two toy environments (GridWorld and CartPole) and the HighD environment, which also seems relatively simple as it has a 1D action space (and presumably a 2D state space). Given the complexity of the method, it would be desirable to include more extensive comparisons on diverse, more challenging domains (e.g., constrained MuJoCo environments as in the ICRL paper).\n\n**[Questions]**\n- Is the constraint function $c$ in the paper a per-step function (i.e., $c(s_t, a_t)$) or a per-trajectory function ($c(\\tau)$)? If the former is the case, there should be infinitely many ways to assign per-step constraints to the expert trajectories because the only constraint is that their sum should be bounded by $\\beta$. How does the method deal with this?\n- I didn't understand the CartPole figures in Fig. 10. What do the \"a=0\" and \"a=1\" lines mean? What are the axes?\n- How is the state space defined in the HighD environment? How does the agent observe the cars in front of it? How are the expert data collected?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "DgAhthx-qU", "reviewer": "Reviewer_8Rs7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel framework to learn soft constraints for ICL. The paper gives a theoretical formulation with proof of convergence, and then provides a practical algorithm with relaxation on the theoretical formulation and several techniques. Experiments on synthetic and real-world environments show improvements of the proposed method compared to baselines. Some ablation studies are provided to verify the design choices of the algorithm.", "summary_of_the_review": "I think the paper provides an interesting framework with a practical algorithm. Each part of the paper is 'good' but not 'great' with the weaknesses mentioned above. ", "strength_and_weaknesses": "Strength:\nThe paper provides a novel framework with a theoretical formulation and a practical algorithm with reasonable relaxations. It also provides empirical results to show the improvements.\n\nWeaknesses:\n1. For the definition of 'soft' constraint, as described in future work, I also feel that the constraint through expectation is not as useful as constraint through probability for stochastic environments. Can the author give more explanation of where the current setting could be applied in real life?\n\n2. It would be great if the author could explain more about how this 'soft' constraint is guiding the algorithm design. \n\n2. The experiment results seems not strong enough to me. There is only one real-work environment and the setting is also relatively simple. It would be good to provide more complex experiment setups (e.g. robotics manipulations).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2yqgnCWAtkf", "reviewer": "Reviewer_3D7X", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed an inverse RL algorithm for recovering soft constraints from expert trajectories. The algorithm takes a nominal reward function, and expert trajectories as input, and outputs a recovered constraint function that best matches the behavior of the expert trajectories. The proposed algorithm interleaves between two steps: 1) solving a constrained RL problem with the latest constraint function, and 2) adjusting the constraint function to maximize the constraint value among all the previously obtained policies, subject to that expert trajectories satisfying the constraint. A theorem is provided to prove the convergence to the expert policy and a practical algorithm is provided that shows empirically successful results on three constraint learning tasks. The proposed method shows better performance in terms of constraint satisfaction as well as similarity to expert trajectories than baseline methods on the selected tasks.", "summary_of_the_review": "My high-level concerns can be seen in the weaknesses section. One more detailed comment: the baseline method ICRL was applied to several more difficult continuous control tasks. It would more convincing if the proposed method was compared on similar tasks with the baselines.\n\nAlso, it is mentioned that the proposed method requires more training time than the baseline methods but I didn't find any numbers (I might have missed it). Could you provide some more details about how much slower it is?\n\nIn general the proposed method seems reasonable and attempts to solve an important problem. If the concerns above could be addressed it can be a good contribution to the ML community.\n\n\n==============\npost rebuttal: I have updated the score from 5 to 6 after reading other reviews and the authors' rebuttal.", "strength_and_weaknesses": "Strengths:\nThe problem of learning constraints is an important one and a solution to this problem can be impactful\nThe theoretical framework is helpful in gaining insight of the algorithm as well as providing theoretical justifications for the proposed algorithm\nThe proposed method shows decent performance on the selected tasks\n\nWeaknesses:\nThe requirement of known reward function and single constraint is understandable but could limit the applicability of the algorithm to real-world problems.\nThe tasks used in the work are relatively simple.\nThe proposed computational algorithm seems quite complex with various tricks, such as the reweighting scheme, the two-stage constrained optimization with different weights, etc. This makes it potentially difficult to be applied to a wider variety of tasks, especially ones that are notably more complex.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%