Benchmarking Constraint Inference in Inverse Reinforcement Learning
Authors: Guiliang Liu
Yudong Luo
Ashish Gaurav
Kasra Rezaee
Pascal Poupart
Venue: ICLR 2023 poster (2023)
Keywords: Inverse Reinforcement Learning
Constrained Reinforcement Learning
Variational Bayesian Inference

Abstract: When deploying Reinforcement Learning (RL) agents into a physical system, we must ensure that these agents are well aware of the underlying constraints. In many real-world problems, however, the constraints are often hard to specify mathematically and unknown to the RL agents. To tackle these issues, Inverse Constrained Reinforcement Learning (ICRL) empirically estimates constraints from expert demonstrations. As an emerging research topic, ICRL does not have common benchmarks, and previous works tested algorithms under hand-crafted environments with manually-generated expert demonstrations. In this paper, we construct an ICRL benchmark in the context of RL application domains, including robot control, and autonomous driving. For each environment, we design relevant constraints and train expert agents to generate demonstration data. Besides, unlike existing baselines that learn a deterministic constraint, we propose a variational ICRL method to model a posterior distribution of candidate constraints. We conduct extensive experiments on these algorithms under our benchmark and show how they can facilitate studying important research challenges for ICRL. The benchmark, including the instructions for reproducing ICRL algorithms, is available at https://github.com/Guiliang/ICRL-benchmarks-public.

Published as a conference paper at ICLR 2023
BENCHMARKING CONSTRAINT INFERENCE IN INVERSE REINFORCEMENT LEARNING
Guiliang Liu Liu Yudong Luo Ashish Gaurav Kasra Rezaee', Pascal Poupart 'The Chinese University of Hong Kong, Shenzhen, Universitty of Waterloo, Vector Institute, "Huawei liuguiliang@euhi: edu. en, yudong. luo@uwaterloo. ca, ashish. www.sss.geuraveuttrrro.ooo ca, kasra. rezaee@huawei. com, www.pprtewwteerel...oo
ABSTRACT
When deploying Reinforcement Learning (RI.) agents into physical system, we must ensure that these agents are well aware the underlying constraints. In many real-world problems, however, the constraints mathematically and unknown to the RL agents. To tackle these issues, Inverse Constrained Reinforcement Learning (ICRL) empirically estimates constraints from expert demonstrations. As an emerging research topic, ICRL does not have common benchmarks, and previous works tested algorithms under environments with canuuall-geeerateel expert demonstrations. In this paper, we construct an ICRL benchmark the context of RI. application domains, including robot control, and autonomous driving. For each environment, design relevant constraints and train expert agents to generate demonstration data. Besides, unlike existing baselines that learn "point estimate" constraint, propose variational ICRL method to model a posterior distribution of candidate constraints. We conduct extensive experiments show how they can facilitate studying important research challenges for ICRL. The benchmark, including the instructions for reproducing ICRL algorithms, is available at
often hard specify
these algorithms under our benchmark and
com/ http/GGillanggggg/ MaaalliinnnnnLR-Leie Sww.RL--eechaarr.o. b
INTRODUCTION
Constrained Reinforcement Learning (CRL) typically learns policy under some known or predefined constraints (Liu al. 2021). This setting, however, not realistic in many real-world problems since it is difficutt to specify the exact constraints that agent should follow, especially when these constraints are time-varyigg such information may not be completely revealed to the agent. For example, human drivers tend to determine an implicit speed limit and a minimum gap to other cars based on the traffic conditions, rules of the road, weather, and social norms. To derive driving policy that matches human performance, an autonomous agent needs to infer these constraints from expert demonstrations.
context-dependent, and inherent to experts' own experience. Further,
An important approach recovering the underlying straints is Inverse Constrained Reinforcement Learning (ICRL) MMalik al.) (021). ICRL infers constraint function to approximate constraints respected by expert demonstrations. This is often done by alternating between updating an imitating policy and a constraint function. Figure summarizes the main procedure ICRL. As an emerging research topic. ICRL does not have common datasets and benchmarks for evaluation. Existing valida- tion methods heavily depend on the safe-Gym (Ray et al.) 2019) environments. Utilizing these environments has some important drawbacks: 1) These environ- ments are designed for control instead of constraint inference. To fill this gap, previous works often pick some environments and add external constraints to them. Striving for simplicity, many of the selected environments
Step: Poliy Upate (hT, + Mtts)
Sampling Trajectory:
Constraint Function: fo(brv))
Interations
Days
Inverse Step: Constraint Upate (#) Φvx2)
Expert Trajectory: D.
Figure 1: The flowchart of ICRL.
deterministic with discretized state and action spaces (Scobee & Sastry 2020) McPherson et al.) 2021 Glazier et al., 2021 Papadimitriou et al.) 202T; Gaurav et al. (022)). Generalizing model performance in these simple environments practical applications is difficult.

---

Published as a conference paper at ICLR 2023
2) ICRL algorithms require expert demonstrations respecting the added constraints while general RL environments do not include such data, and thus previous works often manually generate the expert data. However, without carefully fine-tuning the generator, it is often unclear how the quality of expert trajectories influences the performance of ICRI. algorithms.
In this paper, propose benchmark for evaluating ICRL algorithms. This benchmark includes a rich collection of testbeds, including virtual, realistic, and discretized environments. The virtual environments are based on MuJoCo (Todorov et al. 2012), but we update some of these robot control tasks by adding location constraints and modifying dynamic functions. The realistic environments are constructed based on highway vehicle tracking dataset (Krajewski et al. 2018). so the environments can suitably reflect what happens in a realistic driving scenario, where we consider constraints about car velocities and distances. The discretized environments are based on grid-worlds for visualizing the recovered constraints (see Appendix [B. To generate the demonstration dataset for these environments, we expand the Proximal Policy Optimization (PPO) (Schulman et al.) 2017) and policy iteration (Sutton& & Barto 2018) methods by incorporating ground-truth constraints into the optimization with Lagrange multipliers. We empirically demonstrate the performance of the expert models trained by these methods and show the approach to generating expert demonstrations.
For ease of comparison, our benchmark includes ICRL baselines. Existing baselines learn a constraint function that is most likely differentiate expert trajectories from the generated ones. However, this point estimate (i.e., single constraint estimate) may be inaccurate. On the other hand, a more conceptually-satin saatsfffyng method is accounting for all possibilities of the learned constraint by modeling its posterior distribution. To extend this Bayesian approach to solve the task in our benchmark, we propose a Variational Inverse Constrained Reinforcement Learning (VICRL) algorithm that can efficiently infer constraints from the envir invironment with a high-dimensional and continuous state space.
Besides the above regular evaluations, our benchmark can facilitate answering a series of important research questions by studying how well ICRL algorithms perform 1) when the expert demonstrations may violate constraints (Section 4.3) 2) under stochastic environments (Section 4.4) 3) under envi-
ronments with multiple constraints (Section 5.2) and 4) when recovering the exact least constraining
constraint (Appendix B.2).
2 BACKGROUND
In this section, we introduce Inverse Constrained Reinforcement Learning (ICRL) that alternatively solves both forward Constrained Reinforcement Learning problem (CRL) and an inverse constraint inference problem (see Figure JT)
2.1 CONSTRAINED REINFORCEMENT LEARNING
Constrained Reinforcement Learning (CRL) is based on Constrained Markov Decision Processes (CMDPs) M which can be defined by tuple (S, A., PR. PT, {PCc. <4) Iri y, I') where: 1) and A denote the space of states and actions. 2) Pr(s' (s, a) and p& (r)s, a) define the transition and reward distributions. 3) pe, (c] s, a) denotes stochastic constraint function with an associated bound 6, where indicates the index of a constraint, and the cost e E [0, co). 4) E 10. is the discount factor and T is the planning horizon. Based on CMDPs, we define trajectory T = [so, do; ..., aT- sT] and p(T) = p(so) II=- π commonly consider the following optimization problems.
(9t+1 1t,, ar). To learn a policy under CMDPs, CRL agents
Cumulative Constraints. We consider a CRL problem that finds a policy π to maximize expected discounted rewards under a set of cumulative soft constraints:
1 H((T) s.t. Fee, PT Σ Eta(ona sa vi [0, (1)
arg max Epp, PT 7
Ett,
3
where H(T) denotes the policy entropy weighted by J. This formulation is useful given an infinite horizon (T = oo), where the constraints consist of bounds on the expectation of cumulative constraint values. In practice, we commonly use this setting define soft constraints since the agent can recover from an undesirable movement (corresponding to a high cost cı(s,, oz)) as long as the discounted additive cost is smaller than the threshold (et).
2

---

Published as a conference paper at ICLR 2023
Trajectory-based Constraints. An alternative approach is directly defining constraints the sampled trajectories without relying on the discounted factor:
T
arg max Epp.PP PT
Σ
7Tr + H((T) s.t. Err(pr [c((T] S. vi [0,
(2)
Depending on how we define the trajectory cost c(T), the trajectory constraint can be more restrictive than the cumulative constraint. For example, inspired by Malik et al. (2021), we define c(T) 1 ((s.a))rr (88, a) where o(s, a) indicates the probability that performing action under a state is safe (i.e., within the support of the distribution of expert demonstration). Compared to the above additive cost, this factored cost imposes a stricter requirement on the safety of each state-action pair in trajectory (i.e., if 3(5, a) 77,0(5,0 [) + O, then (s.,a) o(.) 0 and thus c(T) - 1).
2.2 INVERSE CONSTRAINT INFERENCE
In practice, instead of observing the constraint signals, we often have access to expert demonstrations that follow the underlying constraints. Under this setting, the agent must recover the constraint models from the dataset. This is a challenging task since there might be various equivalent combinations of reward distributions and constraints that can explain the same expert demonstrations (Ziebart et al.) 2008). To guarantee the identifiability, ICRL algorithms generally assume that rewards are observable, and the goal is to recover the minimum constraint set that best explains the expert data (Scobee & Sastry 2020). This is the key difference with Inverse Reinforcement Learning (IRL), which aims to learn rewards from an unconstrained MDP.
Maximum Entropy Constraint Inference. Existing ICRL works commonly follow the Maximum Entropy framework. The likelihood function represented as follow (Malik al., 2021):
N
p(Dl = (Z ) exp exp [)(r(0)] 1 M (7)
where 1) N denotes the number of trajectories in the demonstration dataset De. 2) the normalizing 1 (r)dT, and 3) the indicator 1 (r(1) can be defined by (()()))=
term ZM = exp [r(r)) M
IT_ 6e and (s at) defines to what extent the trajectory 7(1) is feasible, which can substitute the indicator in Equation (3. and thus we define:
N
T
log (p(D = Σ (1) + log II
Too [o)
N Nlog exp(r(+)) II
(0 )
(4)
1=1
=
==
We can update the parameters 0 of the feasibility function o by computing the gradient of this likelihood function:
T logloo
Vo log (p(D)()) =
logloa(s ot [²)]] - NN me
(Br, ac]]]
(5)
t=0
t=0
oxplr(F))o(F)
where i is sampled based on executing policy (+) (T))) This is a max- when
imum entropy policy that can maximize cumulative rewards subject (T) (note that (o(s,a)
E(r.)er (s(s,a) policy by constrained maximum entropy RL according objective (2) In this sense, ICRL can be formulated as a bi-level optimization problem that iteratively updates the upper-level objective ( for policy optimization and the lower-level objective ( for constraint learning until convergence (I
- Φr as defined above). In practice, we can learn this
matches the expert policy).
EVALUATION METHODS
In this section, we introduce our approach to evaluating the ICRL algorithms. To quantify the performance of ICRL algorithms, the benchmark must be capable of determining whether the learned constraint is correct. However, the true constraints satisfied by real-world agents are often unavailable, for example, the exact constraints satisfied by human drivers are unknown, and we thus cannot use real-world datasets as expert demonstrations. To solve these issues, ICRL benchmark enables incorporating external constraints into the environments and generates expert demonstrations satisfying these constraints. Based on the dataset, design evaluation metrics and bascline models for comparing ICRL algorithms under our benchmark.
3

---

Published as a conference paper at ICLR 2023
3.1 DEMONSTRATION GENERATION
To generate the dataset, we train a PPO-Lagrange (PPO-Lag) under the CMDP with the known constraints (Table and Table 3 by performing the following steps:
Training Expert Agent. We train expert agents by assuming the ground-truth constraints are unknown under different environments (introduced in Appendix [ Section 4 Section [5). The cost function c* (st, 02) returns if the constraint is violated when the agent performs de in the state 8, otherwise 0. In the environments (in Section 4 and Section [ with continuous state and action spaces, we train the expert agent by utilizing the Proximal Policy Optimization Lagrange (PPO-Lag) method in Algorithm [ In the environment with discrete action and state space, we learn the expert policy with the Policy Iteration Lagrange (PI-Lag) method in Algorithm 2 The empirical results (Figure [ and Figure 6) show that PI-Lag and PPO-Lag can achieve satisfactory performance given the ground-truth constraint function.
Generating a Dataset with Expert Agents. We initialize D. = (0} and run the trained expert agents in the testing environments. While running, we monitor whether the ground-truth constraints are violated until the game ends. If yes. we mark this trajectory as infeasible, otherwise, we record the corresponding trajectory: D. D. U {Te]. We repeat this process until the demonstration dataset has enough trajectories. To understand how 'D. influences constraint inference, our benchmark enables studying the option of including these infeasible trajectories in the expert dataset (Section 4.3). Note there is no guarantee the trajectories in De are optimal in terms of maximizing the rewards. For more details, please check Appendix E Our experiment (Section 4.2) shows ICRL algorithms can outperform PPO-Lag under some easier environments.
3.2 BASELINES
For ease of comparison, our benchmark contains the following state-of-the-ar baselines:
Binary Classifier Constraint Learning (BC2L) build a binary classifier to differentiate expert trajectories from the generated ones to solve the constraint learning problem and utilizes PPO-Lag PI-Lag (Algorithms|]] ( to optimize the policy given the learned constraint. BC2L is independens of the maximum entropy framework, which often induces loss of identifiability in the learned constraint models.
Generative Adversarial Constraint Learning (GACL) follows the design of Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016), where C(s, a) assigns 0 to violating state-action pairs and 1 to satisfying ones. In order include the learned constraints into the policy update, we construct a new reward r'(s, , a) = r(-) + log (<(-)) In this way, GAIL enforces hard constraints by directly punishing the rewards on the violating states or actions through assigning them -o penalties (without relying any constrained optimization technique).
Maximum Entropy Constraint Learning (MECL) is based on the maximum entropy IRL frame- work (Ziebart et al.) (208). with which Scobee & Sastry (2020) proposed an algorithm to search for constraints that most increase the likelihood observing expert demonstrations. This algorithm focused on discrete state spaces only. A following work (Malik et al., (202)) expanded MECL to continuous states and actions. MECL utilizes PPO-Lag (or PI-Lag in discrete environments) to optimize the policy given the learned constraint.
Variational Inverse Constrained Reinforcement Learning (VICRL) is also based on the maximum entropy IRL framework (Ziebart et al. 2008), but instead of learning "point estimate" cost function, we propose inferring the distribution of constraint for capturing the epistemic uncertainty in the demonstration dataset. To achieve this goal, VICRL infers the distribution of a feasibility variable T so that ((0) 8, a) measures to what extent an action a should be allowed in a particular state { The instance & define soft constraint given by: Cp(s, a) 1 & where p(-1s, a). Since + is continuous variable with range [0, 1]. we parameterize p( s, a) by a Beta distribution:
((s.a) a) p(ols. s. a) Beta(a, (3 where (a, S log[1 + exp(f(s.
(6)
here f implemented by a multi-layer network with 2-dimensional outputs (for a and (3). In practice, the true posterior ((o)D,. is intractable for nigh-dimensional input spaces, so VICRL learns
We use uppercase letter and lowercase letter define random variable and instance of this variable.
4

---

Published as a conference paper at ICLR 2023
an approximate posterior ((((De) by minimizing Dki [(dd)e))ll))) maximizing an Evidence Lower Bound (ELBo):
This is equivalent to [ (ddDe)))lp))
Bg log [ogg(Delo)]
DHi
where the log-likelihood term log papp((pp (e)o) follows Equation 3 and the major challenge is to de- fine the KL divergence. Striving for the ease of computing mini mini-batch gradients, we approximate Dia [(p))))lp(( with E(s.))ED DM g( s, a) ())) Since both the posterior and the prior are Beta distributed, we define the KL divergence by following the Dirichlet JoE et al. (2020):
T(aa + + log (T(a)))(3))
I'a+ + 9)
Du ((d)s)
= llog
((a)
(8)
+ (a-
aa) a²) (ba() v/aa + m3] + (B - 88)
(9) (
3]
where 1) [a", ,38] and [a, 3] are parameters from the prior and 2) the posterior functions and I and d denote the gamma and the digamma functions. Note that the goal of ICRL is to infer the least constraining constraint for explaining expert behaviors (see Section 2.2 To achieve this, previous methods often use a regularizer 11[1 <(T)) Malik et al. (2021) ) for punishing the scale of constraints, whereas our KL-divergence extends by further regularizing the variances of constraints.
3.3 EXPERIMENT SETTING
Running Setting, Following Malik et al. (2021). we evaluate the quality of recovered constraint by checking if the corresponding imitation policy maximize the cumulative rewards with minimum violation rate for the ground-truth constraints. We repeat each experiment with different random seeds, according which we report the mean + standard deviation (std) results for each studied baseline and environment. For the details of model parameters and random seeds, please see Appendix C.3
Evaluation Metric. To be consistent with the goal of ICRL, our benchmark uses the following evaluation metrics to evaluate the tasks 1) constraint violation rate quantifies the probability with which a policy violates constraint in trajectory. 2) Feasible Cumulative Rewards computes the total number of rewards that the agent collects before violating any constraint.
4 VIRTUAL ENVIRONMENT
An important application of RL robotic control, and our virtual benchmark mainly studies the robot control task with location constraint. In practice, this type of constraint captures the locations of obstacles the environment. For example, the agent observes that none the expert agents visited some places. Then it is reasonable to infer that these locations must be unsafe, which can be represented by constraints. Although the real-world tasks might require more complicated constraints, benchmark, the first benchmark for ICRL,, could serve as stepping stone for these tasks.
4.1 ENVIRONMENT SETTINGS
We implement our virtual environments by utilizing MuloCo Todorov et al. (2012). a virtual simulator suited to robotic control tasks. To extend MuJoCo for constraint inference, modify the MuJoCo enviromments by incorporating some predefined constraints into each environment and adjusting some reward terms. Table summarizes the environment settings (see Appendix C.I for more details). The virtual environments have different robotic control environments simulate by MuJoCo. We add constraints on the X X-corrdinate of these robots: 1) For the environments where is relatively easier for the robot to move backward rather than forward (c.g.. Half-Chectae Ant, and Walker), our constraints bound the robot the forward direction (the -coordinnate having positive values), 2) For the environments where moving forward is easier (e.g.. Swimmer), the constraints bound the robot in the backward direction (the X-coordinate having negative values). these environments, the rewards are determined by the distance that robot moves between two continuous time steps, so the robot is likely to violate the constraints in order to maximize the magnitude of total rewards (see analysis below). To increase difficulty, we include Biased Pendulum environment that has larger reward on the left side. We nevertheless enforce a constraint to prevent the agent to go too far on the left side. The agent must resist the influence of high rewards and stay in safe regions.
5

---

Published as a conference paper at ICLR 2023
Table 1: The virtual and realistic environments in benchmark.
Type
Name
Dynamics
Obs.
Dim.
Act.
Dim.
Constraints
Blocked Half-cheetah
Deterministie
18
6
X X-Cordiinate 2 -3
Blocked Ant
Deterministic
113
X-Coordinate 2
Virtual
Biased Pendulumn
Deterministic
X-Coordinate > -0.015
Blocked Walker
Deterministic
18 6
X- X-Coordinate 2 -3
Blocked Swimmer
Deterministic
10
X-Coordinate - 0.5
The significance of added Constraints. The thresholds of the constraints in Table are determined experimentally to ensure that these constraints "matter" for solving the control problems. This shown in Figure D.I in the appendix: 1) without knowing the constraints, a PPO agent tends to violate these constrants in order to collect more rewards within a limited number of time steps. 2) When inform the agent of the ground-truti constraints (with the Lagrange method in Section 3. the PPO-Lag agent learns how to stay in the safe region, but the scale of cumulative rewards is likely be compromised. Based on these observations, we can evaluate whether the ICRL algorithms have learned a satisfying constraint function by checking whether the corresponding RL agent can gather more rewards by performing feasible actions under the safe states.
Hicket Hilh Cheocash
Barked Are
Bisond Perchdurm
Blocbdd Waller
Bbc bec Seirame
Dontsanteaaatenn Pate
0.3
00-
GACL
04
02
2.0
.V
191
2006
4000
6090
5000
12000
12300
20000
43000 5000 SXC
10090 28000
30000 40002
2530
5000
7500
33000
1665
scceed Chalchhetthh
14300
Bocked Art
Slined Pendalarnn
Bloced Ral
Blocked Saitener
Fousble Rowards
2207
10300
SeCE
GACL
DC2-
6300
GAC
GACL
GACL
300
GACL
DC2L
500
2022
4230
1010
5602
1200
1990
200C #0:0 MM023 (CS9)
1240 29c6 kicc 78020
16.0
1503
Figure The constraint violation rate (top) and feasible rewards (i.c., the rewards from the trajectories without constraint violation, bottom) during training. From left to right, the environments are Blocked Half-cheetah, Blocked Ant, Bias Pendulum, Blocked Walker, and Blocked Swimmer.
Table 2: Testing performance. We report the average feasible rewards and the constraint violation rate in 100 runs. Check Appendix D.3 for the complete tsdd results. (L) indicates that score is statistically greater (smaller) than the score achieved by VICRL with p-value < 0.05 according to the Wilcoxon signed-rank test. (Table D.I reports the p values.).
Blocked Cheetab
Blocked Ant
Biased Pendulum
Blocked
Blocked
HighD
HighD
Walker
Swimmer
Speed
Distance
GACL.
3.48F+33
7.21F+3
8.50F-1,
2.84F+1
5.78E+21
-1.33B+1 J
.1.70F+1
Feasible
BC2L
8.70E42
1,20E+4
5.73E+0
4,87E-1
1.41E+2.
-2.93E-1
3,84E+0
Rewards
MECI
3.02F+3
8.55F+33
1002+00
1,27
6.37F+1
9.67F-1
2.15H+
VICRL
3.81E+3
1.37E+4
6.64E+0
9,34E+1
1.914+2
-8 8.999-1
4.60E+0
Constraint
GACL.
0%
0%
100% 1
0%
42% 1
14%
19% J
Violation
58%-
f
33%
33%
MECL
40%
0%
73%
19%
88% 1
31%
41% 1
Rate
VICRL
0%
39%
7%
59%
24%
31%
4.2 CONSTRAINT RECOVERY IN THE VIRTUAL ENVIRONMENT
Figure and Table 2 show the training curves and the corresponding testing performance in each virtual environment. Compared to other baseline models, we find VICRL generally performs better with lower constraint violation rates and larger cumulative rewards. This is because VICRL captures the uncertainty constraints by modeling their distributions and requiring the agent to satisfy all sampled constraints, which facilitates conservative imitation policy. Although MECL and GACL outperform VICRL in the Blocked Walker and the Blocked Swimmer environments, respectively, none of these algorithms perform consistently better than the others. Figure D.5 visualizes the constraints learned by VICRL for a closer analysis.

---

Published as a conference paper at ICLR 2023
4.3 CONSTRAINT RECOVERY FROM VIOLATING DEMONSTRATIONS
We use our virtual environment to study "How well do the algorithms perform when the expert demonstrations may violate the true underlying constraim?? Under the definition of ICRL problems, violation indicates that expert trajectories contain state-action pairs that do not satisfy the ground-trutt constraint. The existence of violating expert trajectories is a crucial challenge for ICRL since in practice the expert data is noisy and there is no guarantee that all trajectories strictly follow underlying constraints. Our benchmark provides testbed to study how the scale of violation influences the performance of ICRL baselines. To achieve this, we perform random actions during expert data generation so that the generated expert trajectories contain infeasible state-action pairs that violate ground-truth constraints.
Figure 3 shows the performance in- cluding the constraint violation rate (top row) and the feasible rewards (bottom row). We find the constraint violation rate (top row) increases sig- nificantly and the feasible rewards de- crease as the scale of violation in- creases in the expert dataset, espe- cially for GACL and BC2L, whose performance is particularly vulnerable to violating trajectories. Among the studied baselines, MECL is the most robust to expert violation, although its performance drops significantly when the violation rate reaches 80%. How to design an ICLR algorithm that is robust to expert violation remains challenge for future work.
Makey Demro - Impartectttettt. 0.2
Na Demso - rmportect DJ5
Noby Deroo trperfee: 0.6
Faneers en Wenstin Dar
REZX
MEOL VICRA PPO Jag
MECL
86
PPO
WCD
66
PCC,
WIRI
02
Nobly Derro- Impertect 02
Noaay Demo Impartect 0.5
Malry Demo Imsprrestttteeecccc. DJ3
493
RC2
RC2L
MLLL
3000
MECL
WCRL
VCCL
VICLL
MMO Agg
PPCLAN
1000
1000
Figure 3: Model performance in the Blocked CheftCheetahh environment. From left to right, the percentages of trajec- tories containing violating state-action pairs are 20%, 50%, and 80%. Check Figure i. Appendix for all results.
4.4 CONSTRAINT RECOVERY FROM STOCHASTIC ENVIRONMENTS
Our virtual environment answer the question "How well do ICRL algorithms perform in stochas- tic environments?" To achieve this, we modify the MuloCo environ- ments by adding noise to the tran- sition functions at each step such that p( 0z)
help
Stechettc les 218,
Stechattc Nome AlB 1F-2)
Qocheett Noba Ato, 15-1)
SACL
GACL
BCN
DCL
BCZL
MECL
MECL
PROUA0
64
0.2
2005
2000
2000
[(s, (e) + 7. where 7) ~ o)). Under this design, our benchmark enables study- ing how the scale of stochasticity in- fluences model performance by con- trolling the level of added noise. Fig- ure showss the results. We find ICRL models are generally robust to addi- tive Gaussian noises in environment dynamics until they reach a threshold (e.g., (O, 0.1)). Another intriguing finding is that the constraint inference
Stachentic Netur ATO 3C
Stochantic 100
Stechaatic No
CACL
CACL
SACL
BCCL
BCAL
DICL
NECL
MECL
worl
2003
1100
1000
4000
6000
2003
4000
6002
200C
5000
Figure 4: Model performance in the Blocked Half- Chef--hhettah environment. From left to right, the transition function has the noises (0, 0.001). (0, 0.01), and N( 0.1). Check Figure D.7 in Appendix for results in all environments.
methods (MECL and B2CL) can benefit from a proper scale of random noise since these noisy signals induce stricter constraint functions and thus a lower constraint violation rate.
5 REALISTIC ENVIRONMENT
Our realistic environment defines a highway driving task. This HighD environment examines if the agent can drive safely the ego car to the destination by following the constraints learned from human
7

---

Published as a conference paper at ICLR 2023
drivers' trajectories (see Figure 5). In practice, many of these constraints are based on driving context and human experience. For example, human drivers tend to keep larger distances from trucks and drive slower on crowded roads. Adding these constraints to an auto-driving system can facilitate more natural policy that resembles human preferences.
Observation Radius
Destination
6-000
Figure 5: The Highway Driving (HighD) environment. The ego car is in blue, other cars are in red. The ego car can only observe the things within the region around (marked by blue). The goal is to drive the ego car to the destination (in yellow) without going off-road, colliding with other cars, or violating time limits and other constraints (e.g., speed and distance other vehicles).
Table 3: The constraints for realistic environments.
Type
Name
Dynamics
Obs. Dim.
Act. Dim.
Constraints
HighD Velocity Constraint
Stochastic
76
2
Car Velocity I 40 m/s Car Distance 2 m
Realistic
HighD Distance Constraint Stochastic
76
2
Environment Settings. This environment is constructed by utilizing the HighD dataset (Krajewski al. (018)). Within each recording, HighD contains information about the static background (e.g.. the shape and the length of highways), the vehicles, and their trajectories. We break these recordings into 3,041 scenarios so that each scenario contains less than 1,000 time steps. To create the RL environment, we randomly select a scenario and an ego car for control in this scenario. The game context, which is constructed by following the background and the trajectories of other vehicles, reflects the driving environment in real life. To further imitate what autonomous vehicles can observe on the open road, we ensure the observed features in our environment are commonly used for autonomous driving (c.g.. Speed and distances nearby vehicles). These features reflect only partial information about the game context. To collect these features, we utilize the features collector from mainly study
Commonroad RL (Wang al. 202T In this Highl) environment, constraint and a car distance constraint (see 3 to ensure the ego car can drive at safe speed and keep proper distance from other vehicles. Section 5.2 further studies an environment having both of these constraints.
Speed
Note that the HighD environment stochastic since 1) Human drivers might behave differently under the same context depending on the road conditions and their driving preferences. The population of drivers induces underlying transition dynamics that are stochastic. The trajectories in the HighD dataset are essentially samples from these stochastic transition dynamics. 2) Each time an environment reset (either the game ends or the step limit is reached), randomly picks scenario with a set of driving trajectories. This is equivalent to sampling from the aforementioned transition dynamics.
The significance Constraints. We show the difference in performance between a PPO-Lag agent (Section 3.1) that knows the ground-truth constraints and a PPO agent without knowing the constraints. Figure 6 reports the violation rate of the speed constraint (top left) and the distance constraint (top right). The bottom graphs report the cumulative rewards in both settings. We find 1) the PPO agent tends to violate the constraints in order to get more rewards and 2) the PPO-Lag agent abandons some of these rewards in order satisfy the constraints. Their gap demonstrates the significance of these constraints. Appendix [.6 explains why these constraints are ideal by comparing them with other candidate constraint thresholds.
5.1 CONSTRAINT RECOVERY IN THE REALISTIC ENVIRONMENT
Figure 7 shows the training curves and Table 2 shows the testing performance. Among the studied methods, VICRL achieves a low constraint violation rate with a satisfying number of rewards. Although GACL has the lowest violation rate, is at the cost of significantly degrading the controlling performance, which demonstrates that directly augmenting rewards with penalties (induced by constraints) can yield a control policy with much lower value. Appendix Dopendix(..4illuuus D.4 the causes of failures by showing the collision rate, time-out rate, and rof- rate. To illustrate how well the constraint is captured by the experimented algorithms, our plots include the upper bound of rewards
8

---

Published as a conference paper at ICLR 2023
-910 Ohiorss Centrer
Hay D.y.CCammmcccco
GRCL
GRC
3.3
0.1
0.0
3000
Caladde
BC3L
WORLS
170
2003
00.00
2000
2003
2200
200
2002
Figure 6: Model performance in the HighD envi- ronment with the speed (left) and distance (right) feasible rewards (bottom) with the speed (left) constraint.
Figure 7: The constraint violation rate (top) and
and distance (right) constraints.
and the performance of the PPO-Lag agent (trained under the true constraints). It shows that there is sufficient space for future improvement under our benchmark.
5.2 MULTIPLE CONSTRAINTS RECOVERY
We consider the research question "How well do ICRI. algorithms work in terms of recovering multiple constraints? ?" Unlike the previously studied environments that include only one constraint, we extend the HighD environment to include both the speed and the distance constraints. To achieve this, we generate an expert dataset with the agent that considers both constraints by following 3.1 and test ICRL algorithms by using this dataset.
15.00 Velidry EMCra Contteend
Hi.00 Viludly DaCr Centee
1400 Viludyy DAEN DWT.II De
1400 VMLu-iy CANE Darettale
OMt
OUT
0541
DC3.
DCS.
CED Ouard
Doand
Doand
Dound
1200
1520
Figure 8: Model Performance in an environment with the speed and distance constraints. From left to right, we report speed and distance constraint violation rates, feasible rewards, and success rates.
Figure 8 shows the results. Compared to the performance of its single-constraint counterparts (in Figure 7. the rewards collected by the imitation policy are reduced significantly, although the constraint violation rate remains uninfluenced.
CONCLUSION
In this work, introduced a benchmark, including robot control environments and highway driving environments, for evaluating ICRL algorithms. Each environment is aligned with a demonstration dataset generated by expert agents. To extend the Bayesian approach to constraint inference, we pro- posed VICRL to learn a distribution of constraints. The empirical evaluation showed the performance of ICRL algorithms under our benchmark.
ACKNOWLEDGEMENTS
Resources used in preparing this research at the University of Waterloo were provided by Huawei Canada, the province of Ontario and the government Canada through CIFAR and companies sponsoring the Vector Institute. Guiliang Liu's research was in part supported by the Start-up Fund UDF01002911 the Chinese University of Hong Kong, Shenzhen. We would like to thank Guanren Qiao for providing valuable feedback for the experiments.

---

Published as a conference paper at ICLR 2023
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Picter Abbeel. Constrained policy optimization. In International Conference on Machine Learning (ICML), volume 70, pp. 22 31. PMLR, 2017.
Leopoldo Armesto, Jorren Bosga, Vladimir Ivan, and Sethu Vijayakumar. Efficient learning of constraints and generic null space policies. In 2017 IEEE International Conference Robotics and Automation (ICRA), pp. 1520-1526, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. International Conference Machine Learning, volume 97, pp. 783-792, 2019a.
Daniel S. Brown, Wonjoon Goo, and Scott Niekum. Better-hhandemmonsraai imitation learning via automatiically-rnkees demonstrations. In Conference on Robot Learning (CoRI.), volume 100, pp- 330-359, 2019b.
Sylvain Calinon and Aude Billard. A probabilistic programming by demonstration framework handling constraints in joint space and task space. 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 367-372. IEEE, 2008
Letian Chen, Rohan R. Paleja, and Matthew C. Gombolay. Learning from suboptimal demonstration via self-supervieed reward regression. In Conference on Robot Learning (CoRL.), volume 155 of Proceedings of Machine Learning Research, pp. 1262-1277, 2020.
Glen Chou, Dmitry Berenson, and Necmiye Ozay. Learning constraints from demonstrations. In Workshop on the Algorithmic Foundations of Robotics, WAFR 2018, volume 14, pp. 228-245. Springer, 2018.
Glen Chou, Necmiye Ozay, and Dmitry Berenson. Learning parametric constraints in high dimensions from demonstrations. In Conference on Robot Learning (CoRL), volume 100, pp. 1211-1230, 2019.
Glen Chou, Dmitry Berenson, and Necmiye Ozay. Uncertainty-aware constraint learning for adaptive safe motion planning from demonstrations. In Conference on Robot Learning (CoRL., volume 155, pp. 1612-1639, 2020.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Mohammad Ghavamzadeh, and Edgar A. Duéñez- Guzmán. Lyapunov-based safe policy optimization for continuous control. CoRR, abs/1901.10003 2019.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In Neural Information Processing Systems (NeurIPS), pp. 2021.
Ashish Gaurav, Kasra Rezaee, Guiliang Liu, and Pascal Poupart. Learning soft constraints from constrained expert demonstrations, CoRR, abs/2206.01311, 2022.
Aric Glazier, Andrea Loreggia, Nicholas Mattci, Taher Rahgooy, Francesca Rossi, and Kristen Brent Venable. Making human-like trade-off's in constrained environments by learning from demonstra- tions. CoRR, abs/2109.1101s 2021.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Neural Information Processing Systems (Neurips). pp. 4573,, 2016.
Vinamra Jain, Prashant Doshi, and Bikramjit Banerjee. Model-free IRL using maximum likelihood estimation. In AAAI Conference on Artificial Intelligence, pp. 3951-3958, 2019.
Weonyoung Joo, Wonsung Lee, Sungrae Park, and II-Chul Moon. Dirichlet variational autoencoder. Pattern Recognit., 107:107514, 2020.
10

---

Claudia Pérez-D' Arpino and Julie A. Shah. C-LEARN: learning geometric constraints from demon- strations for multi-step manipulation in shared autonomy. In 2017 IEEE International Conference Published as a conference paper at ICLR 2023
Iya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off off-policy distribution matching. In International Conference on Learning Representations (ICLR). OpenReview.net, 2020.
Robert Krajewski. Julian Bock, Laurent Kloeker, and Lutz Eckstein. The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems. In 2018 21 International Conference on Intelligent Transportation Systems (ITSC), pp. 2118-2125, 2018. doi: 10.1109/TTSC.2288555550
Donghun Lee, Srivatsan Srinivasan, and Finale Doshi Velez. Truly batch apprenticeship learning with deep successor features. In International Joint Conference on Artificial Intelligence (IJCAI),
pp. 5909-5915, 2019.
Changshuo Li and Dmitry Berenson. Learning object orientation constraints and guiding constraints for narrow passages from one demonstration. In International Symposium on Experimental Robotics (ISER), volume I, pp. 197-210, 2016.
Hsiu-Chin Lin, Matthew Howard, and Sethu Vijayakumar. Learning null space projections. In IEEE International Conference Robotics and Automation (ICRA), pp. 2613-2619. IEEE, 2015.
Hsiu-Chit Chin Lin, Prabhakar Ray, and Matthew Howard. Learning task constraints in operational space formulation. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 309-315. IEEE, 2017.
Yongshuai Liu, Jiaxin Ding, and Xin Liu. IPO: interior-point policy optimization under constraints. In AAAI Conference on Artificial Intelligence, pp. 4940-4947, 2020.
Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints in model-free rein- forcement learning: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pp. 4508 4510. ijcai.org, 2021.
Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139, pp. 7390-7399, 2021.
David Livingston McPherson, Kaylene C. Stocking, and Shankar Sastry. Maximum likelihood constraint inference from stochastic demonstrations, In IEEE Conference Control Technology and Applications, (CCTA), pp. 1208-1213, 2021.
Negar Mehr, Roberto Horowitz, and Anca D. Dragan. Inferring and assisting with constraints in shared autonomy. In IEEE Conference on Decision and Control (CDC), pp. 6689-6696. IEEE, 2016.
Marcel Menner, Peter Worsnop, and Mclanie N. Zeilinger. Constrained inverse optimal control with application to a human manipulation task. IEEE Trans. Control. Syst. Technol., 29(2):826-834 2021.
Lucia Pais, Keisuke Umezawa, Yoshihiko Nakamura, and Aude Billard. Learning robot skills through motion segmentation and constraints extraction. In HRI Workshop on Collaborative Manipulation, pp. Citeseer, 2013.
Dimitris Papadimitriou, Usman Anwar, and Daniel S Brown. Bayesian inverse constrained rein- forcement learning. In Workshop on Safe and Robust Control of Uncertain Systems (NeurIPS), 2021.
Dachyung Park, Michael Noseworthy, Rohan Paul, Subhro Roy, and Nicholas Roy. Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning. In Conference on Robot Learning (CoRI.), volume 100, pp. 1005-1014, 2019.
Robotics and Automation, ICRA, pp. 4058 4065. IEFF, 2017.
11

---

Published as a conference paper at ICLR 2023
Alex Ray. Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXi::1910.01700: 7:1, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707 .63347, 2017.
Dexter Scobee and S. Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. In 8th International Conference on Learning Representations, (ICLR). OpenReview.net, 2020.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, and Masashi Sugiyama. Variational imitation learning with diverse-quality demonstrations. In International Conference on Machine Learning (ICML), volume 119, pp. 9407-9417, 2020.
Voot Tangkaratt, Nontawat Charoenphakdee, and Masashi Sugiyama. Robust imitation learning from noisy demonstrations. Artificial Intelligence and Statistics (AISTATS), volume 130, pp. 298-306, 2021.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations ICLR. OpenReview.net, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pp. 5026-5033. IEEE, 2012.
Xiao Wang, Hanna Krasowski, and Matthias Althoff. Commonroad-rl: A configurable reinforcement learning environment for motion planning of autonomous vehicles. In 24th IEEE International Intelligent Transportation Systems Conference, ITSC 2021, Indianapolis, IN, USA, September 19-22, 2021, pp. 466-472. IEEE, 2021.
Yueh-Hua WU, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In International Conference on Machine Learning (ICML), volume 97, pp. 6818-6827, 2019.
Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection- based constrained policy optimization. In International Conference on Learning Representations (ICLR), 2020.
Teu
Gu Ye and Ron Alterovitz. Demonstration--uuiee motion planning. In Robotics Research The 15th International Symposium ISRR, volume 100, pp. 291-307, 2011.
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference Artificial Intelligence, pp. 1433-1438. AAAI Press, 2008
12

---

Published as a conference paper at ICLR 2023
A RELATED WORK
In this section, we introduce the previous works that are most related to our research.
Inferring Constraints from Demonstrations. Previous works commonly inferred constraints to identify whether an action is allowed or a state is safe. Among these works, (Chou et al., 2018 Scobee & Sastry 2020 McPherson al., 2021 Park et al. 2019) are based on the discrete state-action space and constructed constraint sets to distinguish feasible state-action pairs from infeasible ones. Regarding continuous domains, the goal is to infer the boundaries between feasible and infeasible state-action pairs: (Lin et al., 2013 2017 Armesto et al. 2017) estimated a constraint matrix from observations based the projection of its null-space matrix. (Pérez-I) Arpino & Shah] (017)) learned geometric constraints by constructing a knowledge base from demonstration. (Menner et al. 2021) proposed to construct constraint sets that correspond the convex hull of all observed data.
(Malik et al. 2021] Gaurav et al. 2022) approximated constraints by learning neural functions from demonstrations. Some previous works (Calinon Billard [008 Ye & Alterovitz 2011] Pais
et al. 2013 Li & Berenson 2016; Mehr et al. (016) focused on learning local trajectory-basee
constraints from single trajectory. These works focus on inferring a single candidate constraint while some recent works learn distribution over constraints, for example, (Glazier et al. (202)) learned a constraint distribution by assuming the environment constraint follows a logistic distribution. (Chou et al. 2020 Papadimitriou et al., (02II) utilized Bayesian approach to update their belief over constraints, but these methods are restricted to discrete state spaces or toy environments like grid-worlds.
Testing Environments for ICRL. To the best of our knowledge, there is no common benchmark for ICRL., and thus previous works often define their own environments for evaluation, including 1) Worlds are the most popular environments due to their simplicity and interpretability. Previous
works (Scobee & Sastry 2020 McPPerson al., 2021 Papadimitriou al., [021 Glazier et al.,
2021 Gaurav et al. 2022) added some obstacles to a grid map and examined whether their algorithms can locate these obstacles by observing expert demonstrations. However, it is difficult to generalize the model performance in these grid worlds to real applications with high-dimensional and continuous state spaces. 2) Robotie Applications have been used as test beds for constraint inference, for example, the manipulation of robot arms (Park et al., 2019 Menner et al. 2021 Armesto et al. 2017 Pérezz D' A'Arpino Shah 2017). quadrotors (Chou et al. 201 2020), and humanoid robot hands (Lin jtt 2017). However, there is consistent type of robot for comparison, and the corresponding equipment is not commonly available. A recent work (Malik et al., (021) used a robotic simulator by adding some pre-defined constraints into the simulated environments. Our virtual environments use similar setting, but we cover more control tasks and include a detailed study of the environments and the added constraints. 3) Kafety-Gym (Ray al. 2079) is one of the most similar benchmarks to work. However, Safety Gym is designed for validating forward policy-updating algorithms given some constraints, whereas our benchmark is designed for the inverse constraint-infernnce problem.
B DISCRETE ENVIRONMENTS
Our benchmark includes a Grid-World World environment, which has a discrete state and action space. Although migrating the model performance to real-world applications is difficult, FriddW are commonly studied RL environments where we can visualize the recovered constraints and the trajectories generated by agents. Our benchmark uses Grid-Worke to answer the question "How well do the ICRL algorithms perform in terms of recovering the exact least constraining constraint?"
B.1 ENVIRONMENT SETTINGS
Our benchmark constructs a map of size 7 7 and four different constraint maps (top row Fig- ure B.I for testing the baseline methods. For benchmarking ICRL algorithms, each environment is accompanied by a demonstration dataset of expert trajectories generated with the PI-Lag algorithm 2 (see Section 3.1). Note that to be compatible with previous work that studied ments Scooee & Sastry| (2020), we replace the policy gradient algorithm in the baseline algorithms with policy iteration for solving discretized control problems.
environ-
B.2 EXPERIMENT RESULTS
13

---

Published as a conference paper at ICLR 2023
To study how well the ICLR algo- rithms perform terms of recovering the exact least constraining constraint, we visualize the ground truth con- straint map and the constraint maps recovered by ICLR baselines in Fig- ure B.1 (For other metrics, please find the constraint violation rate and feasible cumulative rewards in Fig- ure D.3 and the generated trajecto- ries Figure D.2) ). We find the dif- ference between the added constraint (top row Figure B.I) and the recov- ered constraint is significant, although most algorithms (BC2L, MECL, and VICRL) learn policy that matches well the policy of an expert agent. In settings, the size of the recovered constraint set is larger than the ground- truth constraint (i.e., constraint learn- ing is too conservative). While base- lines including MECL, and VICRL in- tegrated regularization about the size of the constraint set into their loss, the results show that the impact of this regularization limited, and there is plenty of room for improvement.
VTS 2M
Figure B.I: The recovered constraints under 4 settings (from the left to right columns). From the second to the last row, the experimented methods are GACL, BC2L, MECL, and VICRL. Blue and red mark the starting and target locations.
C MORE IMPLEMENTATION
AND ENVIRONMENT DETAILS
C.1 MORE INFORMATION ABOUT THE VIRTUAL ENVIRONMENTS
Figure C.I: Mujoco environments. From left right, the environments are Half-cheetah, Ant, Inverted Pendulum, Walker and Swimmer.
Our virtual environments are based on Mujoco (see Figure C.I We provide more details about the virtual environments as follows:
Blocked Half- Cheetahhe The agent controls a robot with two legs. The reward is determined by the distance walks between the current and the previous time step and a penalty over the magnitude of the input action. The game ends when maximum time step (1000) is reached. We define constraint that blocks the region with X-coordinate < -3, so the robot only allowed to move in the region with X-coordinate between -3 and co.
Blocked Ant. The agent controls robot with four legs. The rewards are determined by the distance to the origin and a healthy bonus that encourages the robot to stay balanced. The game ends when a maximum time step (500) is reached. Similar to the Blocked half-Cheetah environment, we define a constraint that blocks the region with X-coordinate -3, so the robot is only allowed to move in the region with X-coordinate between -3 and
14

---

Published as a conference paper at ICLR 2023
Biased Pendulum. Similar to the Gym CartPole (Brockman al. 2016), the agent's goal is to balance a pole on a cart. The game ends when the pole falls or a maximum time step (100) is reached. At each step, the environment provides reward of 0.1 if the X-coordinate 2 0 and a reward of 1 if the X-coordinate v -0.01. The reward monotonically increases from 0. to when 00.01 < X-coordinate < O. We define a constraint that blocks the region with X-coordinate < -0.015, so the reward incentivizes the cart to move left, but the constraint prevents it from moving too far. If the agent can detect the ground-truth constraint threshold, it will drive the cart to move into the region with X-coordinate between -0.0 -0.015 and -0.01 and stay balanced there.
Blocked Walker. The agent controls robot with two legs and learns how to make the robot walk. The reward determined by the distance it walks between the current and the previous time step and penalty over the magnitude of the input action (this is following the original Walker2d environment). The game ends when the robot loses its balance or reaches a maximum time step (500). Similar to the Blocked Half-Cheetae and Blocked Ant environment, we constrain the region with X-coordinate < -3, so the robot is only allowed to move in the region with X-coordinate between -3 and co.
The agent controls a robot with two rotors (connecting three segments) and learns how to move. The reward is determined by the distance it walks between the current and the previous time step and penalty over the magnitude of the input action. The game ends when the robot reaches a maximum time step (500). Unlike the Blocked Half-Cheetah and Blocked Ant environment, it is easier for the Swimmer robot to move ahead than move back, and thus constrain the region with X-coordinate 2 0.5, so the robot is only allowed to move in the region with X-coordinate between oo and 0.5.
Blocked Swimmer.
C.2 MORE ALGORITHM
We show the PI-Lag in Algorithm
Algorithm 1: Proximal Policy Optimization Lagrange (PPO-Lag)
Input:
Constraint function f" constraint threshold c, Lagrange multiplier A, rollout rounds B, update rounds IC, loss parameters & and &z, clipping parameter w, imitation policy Te. value functions V, and Va:
Initialize state 80 from CMDP and the roll-out dataset Drout
for b= 1,2, B do
Perform Monte-Carlo roll-out with the policy wo in the environment;
Collect trajectories πи = [ ap, Tor Co...., Sy, ar, Pr, orl where o = f (sy, a;);
Calculate reward advantages A,, total rewards Re, constraint advantages A& and total costs
Ct from the trajectory:
Add samples to the dataset Droil = =Drott U (80, ar ro, Ar, Re, cr A CYY
end for
.C do Sample a data point Sx, an,, Tw, Are Ri,, Ccy Ace Cn. from the dataset Drout Calculate the clipping loss
Cxy
LCLIP- min r(a sx) (Ár + xAA)) clip( I(awlan)
Raldlan Bx), w, () + xAE)
Ratalax
Calculate the value function loss LVF = V1 (sx) Rx.13 1/3 + [[V] Cx
Update policy parameters 0 by minimizing the loss: - CCCLPP + E. {TH(
end
Update the Lagrange multiplier X by minimizing the loss L.>: A A/E re (Åo) - c):
C.3 HYPER-PARAMETERS
We published our benchmarks, including the configurations of the environments and the mod- els at jactps:/ ///::th/gt README. file for more details. We provide brief summary of the hyper-parameters.
see the
15

---

Published as a conference paper at ICLR 2023
Algorithm 2: Policy Iteration Lagrange(PI-Lagg
Input:
Constraint function f" Lagrange multiplier rollout rounds B, update rounds K, loss parameters & and &2. imitation policy no;
Initialize state 80 from CMDP and the roll-out dataset Droit: Initialize Values (s)) R and r(s) A(ss for all E S; while not converge; do
// Policy evaluation.
for 8 SESSE SESor do
V(s) D( (())(r Ae +>V(6') where of = (8r);
end
end
while not converge; do
// Policy update.
for s E S do
π T(() arg maxa En, p(s',rls, Ac +s)(s)) where of = f (8t):
end
end
while not converge: do
// Lagrange multiplier update.
for b 1, 1, do
Collect trajectories ть = [80, ag, co...., .r, or, cr] where o = f (st): Calculate total costs C₁ from the trajectory from Tb:
Add samples to the dataset Droit Droul" U {st, ae Ct)t=1i
end
Update the Lagrange multiplier X by minimizing the loss LA: A[ED,on (Ct) e):
end
In order to develop fair comparison among ICRL algorithms, use the same setting for all algorithms.
In the virtual environments, layer to 64, and 3) the number of hidden layers for the policy function, the value function, and the cost function constraint model, by following some previous work (Malik et al. (222]) and their implementation. The random seeds of virtual environments 123, 321, 456, 654, and 666.
1) the batch size of PPO-Lag 64, 2) the size of the hidden
We decide the other parameters, including the learning rate of both PPO-Lag and
In the realistic environments, we set 1) the batch size of the constraint model to 1000, 2) the size of the hidden layer 64 and 3) the number of hidden layers for the policy function, the value function and the cost function to 3. We decide the other parameters, including the learning rate of both PPO-Lag and constraint model, by following CommonRoad RL. (Wang et al. 2021 and their implementation. During our experiment, we received plenty of help from their forum We will acknowledge their help the formal version of this paper. The random seeds of realistic environments are 123, 321, and 666.
C.4 EXPERIMENTAL EQUIPMENT AND INFRASTRUCTURES
We run the experiment on a cluster operated by the Slurm workload manager. The cluster has multiple kinds of GPUs, including Tesla T4 with 16 GB memory. Tesla P100 with 12 GB memory, and RTX 6000 with 24 GB memory. We used machines with 12 GB of memory for training the ICRL models. The number of running nodes is and the number of CPUs requested per task is 16. Given the aforementioned resources, running one seed in the virtual environments and the realistic environments takes 2-4 hours and 10-12 hours respectively.
16

---

Published as a conference paper at ICLR 2023
C.5 COMPUTATIONAL COMPLEXITY
We provide a brief analysis of the computational complexity. The ICRL algorithms, including GACL,, MECI,, BC2I,, and VICRI., use iterative updating paradigm and thus their computational complexities are similar. Let K denote the number of iterations. Within each iteration, the algorithms update both the imitation policy and the constraint model. Let M denote the number of episodes that the PPO-Lag algorithm runs in the environments. Let N denote the number of sampling and expert trajectories. Let denote the maximum length of each trajectory. During training, we use mini-batch gradient descent. Let B denote the batch size, and then the computational complexity is O(KL(M + B).
C.6 EXPLORING OTHER CONSTRAINTS IN THE REALISTIC ENVIRONMENTS
The constraint thresholds in our environments are determined empirically according to the perfor- mance (constraint violation rate and rewards) of the PPO agent and the PPO-Lag agent. To support this claim, we show the performance of other thresholds and analyze why they are sub-optimal in terms of validating ICRL algorithms.
0.00
port
0.00
0.75
PED
1000
2000
2000
4000
1000
2000
2000
4000
2000
3000
4000
3000
3000
2000
2000
4000
2000
"ogoo relocity coneraien telodhycceeee)
Hig relectt Conerare teloctycis
Higto dieanon CondIISEE Bpponcee-9at
Highl diernce Constrsie oppareeseaar
16
10
1972
1912
1912
3200
3200
3000
Figure C.2: From left to right, the constraint violation rate (top) and rewards (bottom) of the PPO and PPO-Lag agents in the HighD environments with constraints 1) Ego Car Velocity < 30 m/s, 2) Ego Car Velocity < 35 m/s, 3) Car Distance >> 40 m, and 4) Car Distance > 60 m.
We have explored the option of using a 30m/s velocity constraint (The first column on the left in Figure Fi2) and 35m/s velocity constraint (The second column on the left in Figure C.2). Ideally, these constraints should be closer to the realistic speed limit in most countries. However, the HighD dataset comes from German highways where there no speed limit. Moreover, when building the environment, the ego car is accompanied by an initial speed calculated from the dataset. We observed that the initial speed is already higher than the speed limit (e.g.. 35m/s) in many scenarios. and thus the violation rate will always be in these scenarios, leaving opportunity for improving the policy, This explains why the corresponding violation rates are high for the PPO and the PPO-Lag agents.
We also explored the option of using a 40m distance constraint (third column in Figure 60m distance constraint (fourth column in Figure C.2). Ideally, these constraints should be more consistent with the 2-second gap recommendation (the average speed is around 30m/s in HighD, so the recommended gap is 2*30m/s=60m), but we find the controlling performance of the PPO-Lag agents very limited, which shows the agent cannot even develop satisfying control policy when knowing the ground -oound-truthhaoo constraints. This is because the ego car learns to frequently go oof-road in order maintain the large gap.
+ and
17

---

Published as a conference paper at ICLR 2023
D MORE EXPERIMENTAL RESULTS
D.1 ADDITIONAL EXPERIMENTAL RESULTS IN THE VIRTUAL ENVIRONMENTS
Figure D.1 shows the additional experimental results in the virtual environment.
Rockod Shefchhothhh
Blockd
Blided Nonda
Blcket Swireee
Constarr-ooattn Rate
03
PR0
HDD
PPQ
PD PT0
PPg_0g
pno_ig
35000
Coledd
Caleds
Eloket Sheotsh
Slaced hondskamn
Mloked
Swirerer
$300
6030
200C
4330
43
890
3300
FFD
1ag
IF0
1ag
1ag
PnO
kgg
PPC
Figure D.1: The constraint violation rate (top) and rewards (bottom). Environments from left to right: Blocked Half-cheetah, Blocked Ant, Biased Pendulum, Blocked Walker, and Blocked Swimmer.
D.2 ADDITIONAL EXPERIMENTAL RESULTS IN THE DISCRETE ENVIRONMENTS
Figure D.2 and Figure
D.3 show the additional experimental results in the discrete environment.
D.3 THE COMPLETE RESULTS FOR TESTING PERFORMANCE
Table D.2, Table D.2 Table performance.
D.3 Table D.4 and Table show the complete results for the testing
Table D.I: The p Values of Wilcoxon signed-rank test across 100 runs. We repeat the test for the models trained by one random seed (we have a total of random seeds) and report the averaged p values.
Blocked Half-
Blocked
Biased
Blocked Walker
Blocked
HighD
HighD
Cheetah
Ant
Pendulum
Swimmer
Velocity
Distance
GACL
1.17E-2
1.23E-06
2.46E-14
3.90E-18
3.42E-4
8.33E-5
3,28E-05
Feasible
BC2L.
1.40--15
2.98E-05
4.89E-4
2.34E-8
9.98E-10
5.75E-1
3.43E-2
Rewards
MECL
4.07E-2
3.72E-08
2.46E-14
2.94E-2
7.27E-3
3.40E-1
4.86E-2
Constraint
GACI.
1.57E-1
6.13H-2
1.71E-18
7.68F-2
1,48H-4
2.81H-1
1.53H-2
Violation
BC2L
7.93E-2
8.17E-1
1.848-06
1.21E-7
8.79B-13
3.65B-2
5.54B-2
Rate
MECL
1.52B-23
5.20E-1
1.718-18
9.98E-2
1.55B-4
1.72B-2
7.16B-3
Table D.2: Testing performance in the virtual environments. We report the feasible rewards (i.e., the rewards from the trajectories without constraint violation) computed with 50 runs.
Half-cheetah
Blocked Ant
Biased Pendulum
Blocked Walker
Blocked Swimmer
GACL
3477.53 1 416.54
7213.62 1 993.12
0.85 + 0.02
28.35 = 0.77
578.27 148.16
BC2L.
870.09 499.03
11956.26 t 1980.88
5.73 1: 5.60
48.73 1 4.18
141.82 + 152.14
MBCL
3024.88
1364.59
8546.19 1 1262.03
1.02 1 1.63
126.76 52.21
63.66 1 107.95
VICRL
3805.72 = 511.66
13670.32 + 1.89
6.64 + 4.45
93.40 + 93.97
191.11 + 154.57
D.4 COMPLEMENTARY RESULTS IN THE REALISTIC ENVIRONMENT
Figure repoots the average velocity, collision rate, off-road rate, time-out rate and goal-reaching rate during training. We find the oof-road rate of GACL is significantly higher than other methods.
18

---

Published as a conference paper at ICLR 2023
Ground-Truth
Ground-Truth
Ground-Truth
Ground-Truth
5
5
5
4-
3
3-
3-
3
2
211 1-
2
251 Txt
2
021
12 Tx
2
15
1
0
0
0
0
421
3
2
2
2
Iteration-10 10
Iteration-11
Iteration-10
Ittratiin-10
6
6
6
4
4
4
3
3
3
3
2-
2
2
1
201
311
1
2m1
331
1
2m
331
27
271
0
0
0
2
2
2
2
Iteration-10
Iteration-10
Iteration-10
Iteration-10
5
5
5-
5
4
4
4
4
3
3
3
3
2
211
2
8n1
2
0m Tx
2
on Tr
0
01 Th
0
0
40
0
m
0
3
5
2
4
2
4
0
2
4
Iteration-10
Iteration-10
Iteration-10
Iteration-5
5-
5
5-
5
3
3
3
3
2-
3:1
351
2-
121
2
151
1
211
1
211
1
271
1
20
0
0
0
0
Iteration-10
Iteration-10
Iteration-10
Iteration-10
6-
6
6
4-
4
4
4
2
2
5n
2
2
02
1
211
711
1
301
1
351
1
27
30
0
2
3
5
6
0
2
4
6
0
2
4
0
2
4
Figure D.2: The trajectories generated by different agents in the discrete environments.
Table D.3: Testing performance the virtual environments. report the constraint violation rate computed with 50 runs.
Half-cheetah
Blocked Ant
Biased Pendulum
Blocked Walker
Blocked Swimmer
GACL
0.0 0.0
0.0 + 0.0
1.0 0.0
0.0 0.0
0.42 0.23
BC21.
0.47 - 0.24
0.0 4 0.0
0.58 4 0.23
0.0 0.0
0.84 4 0.14
MECL
0.40
0.0 - 0.0
0.73 = 0.17
0.19 0.17
0.88 + 0.12
VICRL
0.0 0.0
0.02 i 0.02
0.39 0.22
0.07 0.07
0.59 0.23
19

---

Published as a conference paper at ICLR 2023
Gird Word Sacting
Gid Wordd Setting
Sird Wordd Setting
Gid World Settin 4
2,5
-.0
-.0
1.0
SC2L
BCZL
BC2L
BC2L
SECL
0.8
MECL
0.81
MECL
0.8
MECL
MICRL
VICAL
0.6
VICKL
VICRL
PhiLag
20
90
20
Epteade
Gird World Setting
Gird World Settingg
GPd World Setting
Gird World Setting
2.4
2.0
1.0
2.0
SC2L
BC2L
BC2L
BC2L
MECCL
0.8
MECL
0.5
MECL
0.8
MECL
V.C.L
VICLL
06
VICRL
0.6
VICAL
0.5
23
180
20
150
Eptoods
Figure D.3: The constraint violation rate (top) and feasible rewards (i.e., the rewards from the
trajectories without constraint violation, bottom) during training.
Table D.4: Testing performance in the realistic environments. We report the feasible rewards (i.e., the rewards from the trajectories without constraint violation) computed with 50 runs.
HighD Velocity Constraint
HighD Distance Constraint
GACT.
-19.13 + 2.99
-17.02 + 3,31
BC2L
-0.29 + 11.18
3.84 + 11.28
MECL
0.97 1 11.48
2.15 10.45
VICRI.
-0.90 11.80
4.60 11.71
Table D.5: Testing performance in the realistic environments. We report the constraint violation rate computed with 50 runs.
HighD Velocity Constraint
HighD Distance Constraint
GACI.
0.14 + 0.09
0.19 + 0.11
BC2L
0.33 + 0.15
0.33 = 0.15
MECL
0.31 0.15
0.41 0.17
VICRL
0.24 1 0.12
0.31 1 0.15
20

---

Published as a conference paper at ICLR 2023
explains why GACL cannot achieve satisfying performance. Another main limitation of current baselines is their incapability of preventing the collision events, especially under the car distance constraints.
1920 wibectt Ceatres
1925 Zerence Conatrate
DNC)
DNC
EC2L
PPO kug
Opacce
Optoccs
Hig Condrreet
DC3L
DC3L
EC3L
FTC
19g
7920 Nereene Conatrate
We0D Constran
R30
06:1
6E21
6.0
EC3L
PRO
FnO
FTC
0.0
0.0
Optodd
Optccc
Optocds
1gf
1 Dorance Conatraie
12h0 Velicoo Dtssance Conatrent
$C.11
PECL
00
0.0
0.0
Gpionds
High
strane
Dkizancs Contralls
BC2L
RC2L
PECC
IFCC
6.0
1500
Figure D.4: The average velocity (first column), collision rate (second column), off road rate (third column), time out rate (fourth column) and goal reaching rate (last column) during training. From left to right, the environments are HighD with the velocity constraints, the distance constraints and both of these constraints.
D.5 CONSTRAINT VISUALIZATION
Figure D.5 visualizes the learned constraints with 1) partial dependency plots (red curve, on the left) accompanied by the samples from the constraint distribution (blue points, i.c., Po(cls, a), marked by cost) and 2) histograms (blue, the right) showing the number of states with specific feature value (e.g., position) during testing. The x-axis of these plots show the features where the ground-truth constraints are defined (this message is hidden from the agents during training). In order to understand how well the constraints are captured, we can compare these plots with the definition of ground truth constraints in Table and Table
21

---

Published as a conference paper at ICLR 2023
130
Figure D.5: Visualization the learned constraints by pair of plots for VICRL-RS (left column) and (right column) VICRL-VaR. Each pair includes 1) partial dependency plots (red curve, the left) accompanied by the samples from the constraint distribution (blue points, i.e., Pe(cls, a), marked by cost) and histograms (blue, the right) showing the number of states with a specific feature value (e.g., x position) during testing. From top to bottom, the testing environments are Blocked Saff-Cheetah, Blocked Ant, Biased Pendulum, Blocked Walker and Blocked Swimmer.
22

---

Published as a conference paper at ICLR 2023
D.6 CONSTRAINT RECOVERY FROM VIOLATING DEMONSTRATIONS IN OTHER FOUR ENVIRONMENTS
Figure D. The text the result of the Half-Cheetah environment, and the following is the result of the other four environments. From top to bottom is Blocked Ant, Blocked Walker, Blocked Swimmer and Biased Pendulum.
D.7 CONSTRAINT RECOVERY FROM STOCHASTIC ENVIRONMENT IN OTHER FOUR ENVIRONMENTS
Figure D.7 The text the result of the Half-Cheetah environment, and the following is the result of the other four environments. From top to bottom is Blocked Ant, Blocked Walker, Blocked Swimmer and Biased Pendulum respectively.
E LIMITATIONS, CHALLENGES AND OPEN QUESTIONS
We introduce limitations and challenges in ICRL, as well as open questions for future work.
Constraint Violation. The imitation policies ICRL agents are updated with RCPO (Tessler et al. 2019), but Lagrange relaxation methods are sensitive to the initialization of the Lagrange multipliers and the learning rate. There is no guarantee that the imitation policies can consistently satisfy the given constraints (Liu al. 202I). As result, even when learned constraint function matches the cround-truth constraint, the learned policy may not match the expert policy, causing significant variation in training and sub-optimal model convergence. If we replace the Lagrange relaxation with Constrained Policy Optimization (CPO) (Achiam et al. 2017 Chow et al., [2019] Yang et al. 2020 Liu et al. 2020), ICRL may not finish training within reasonable amount of time since CPO is computationally more expensive. How to design an efficient policy learning method that matches ICRL's iterative updating paradigm will be an important future direction.
Unrealistic Assumptions about Expert Demonstrations. ICRL algorithms typically assume that the expert demonstrations are optimal in terms of satisfying the constraints and maximizing rewards. There is no guarantee that these assumptions hold in practice since many expert agents (e.g.. humans) do not always strive for optimality and constraint satisfaction. Previous works (Brown et al., [019a]bb
Wu et al. 2019 Chen et al. 2020 Tangkaratt et al. 2020 2021), introduced IRL approaches to learn rewards from sub-optimal demonstrations, but how to extend these methods to constraint inference unclear. A promising direction is model soft constraints that assume that expert agents only follow the constraints with a certain probability.
Insufficient Constraint Diversity. ICRL can potentially recover complex constraints, but bench- mark mainly considers linear constraints as the ground-truth constraints (although this information is hidden from the agent). Despite this simplification, our benchmark is still very challenging: ICRL agent must identify relevant features (e.g., velocity in x and coordinates) among all input features (78 in total) and recover the exact constraint threshold (e.g., 40 m/s). For future work, we will explore nonlinear constraints and constraints on high-dimensional input spaces (e.g., pixels).
Online versus Offline ICRL. ICRL algorithms commonly learn an imitation policy by interacting with the environment. The online training nevertheless contradicts with the setting of many realistic applications where only the demonstration data instead of the environment is available. Given the recent progress in offline IRL (Jain et al. 2019 Lee et al. 2019] Kostrikov et al. 2020 Garg et al. 2021), extending ICRL the offline training setting will be an important future direction.
F SOCIETAL IMPACT
Positive Societal Impacts The ability to discover what can be done and what cannot be done is an important function of modern AI systems, especially for systems that have frequent interactions with humans (e.g.. house keeping robots and smart home systems). As important stepping stone towards the design of effective systems, constraint models can help develop human -iinann-- AI systems and facilitate their deployments in real applications.
23

---

Published as a conference paper at ICLR 2023
Naley Dema Imperfect 0.2
Noley Demo Imperfect 0.5
Nossy Dema Imparfect 0.8
5000
1000
5000
BCZL
BC2L
BC2L
4000
4000-
MECL
MECL VICAL
NECL
Rawards
2000
2000
2000
1000
1000
10000
13000
10000
5000
120000
Noisy Demo Imperfect 0.2
Noisy Demo - Imperfect 0.5
Noisy Demo - Imperfect 0.8
1.0
10-
BC2L
BC2L
BC2L
MECL
MECL
MECL
Corainar
0.4
0.4
0.4
10000
15000
5000
10000
15000
5000
10000
15000
Noisy Demo Imperfect 0.2
Nolay Demo - Imperfect 0.5
Noisy Demo Imperfect 0.8
400
BC2L
BC2L
BC2L
GACL
GACL
300
MECL
300
MECL
MECL
250
VICRL
250
250
150
150
100
100
100
0
10000
20000
30000
40000
10000
20000
30000
40000
10000
20000
30000
40000
Noisy Demo Imperfect 0.2
Noisy Demo Imperfect 0.5
Noisy Demo - Imperfect 0.8
BC2L
1.5
BC2L
BC2L
GACL
0.8
MECL
0.8
NECL
0.8
MECL
VICRL
0.6
0.5
11.
Oziattrr
0.4
0.4
0.4
10000
20000
30000
40000
0
10000
20060
30000
40000
10000
20000
30000
40000
Nolay Demo
Impertect 0.2
Noley Demo Impertect 0.5
Noisy Demo Impertect 0.8
500
300
BUZL
GACL
GACL
GACL
MECL
MECL
MECL
VICRL
300
300
200
200
2000
4000
8000
5000 10000
2000
4000
6000
8000 10000
2000 10000
4000 5000
5000
Epsode
Episode
Sosodde
Nakky Dema Imporfact 0.2
Noley Demo Imporfect 0.5
Naiky Damo Imporfect D.8
BC2L GACL MECL
GACL
GACL MECL
AgpOOA
Consenent VondtnenVaaattnooo Pate
MECL
0.8
VICRL
VICRI
0.5
0.4
0.4
0.4
0.2
0.7
0.2
0.0
0.5
0.0
2000
4006
ROO0
8008 10000
0
2000
0000
5000
BUDO
10000
2000
4000
BOOBI
8O00
16800
Ep Audde
Episode
24

---

Published as a conference paper at ICLR 2023
Naiky Damo Impertect 0.2
Noley Demo Imporfoct 0.5
Noiey Demo Imperfect 0.8
50
50
50
8C.2L
BCZL
BC2L
40
MECL VICRL
MECL
MECL
Faastbee Ruwards
VICRL
20
20
20
30
ABOCO
50000
600U0
BOUGO
40000
SE000
Noisy Demo - Imperfect 0.2
Noisy Demo Imperfect 0.5
Noisy Demo - Imperfect 0.8
1.0
1.0
BC2L
BC2L
BC2L
0.8
NECL
MECL
NECL
0.5
Constreent
0.4
0.4
0.4
0.2
6OR00 BOCO0
20000
40000
BO000
00080
BOODO
Figure D.6: From left to right, the percentages of trajectories containing violating state-action pairs 20%, 50%, and 80%. The environment from top bottom is Blocked Ant, Blocked Walker, Blocked Swimmer and Biased Pendulum. Feasible rewards(top) and constraint violation rate(bottom) metrics during training.
Negative Societal Impacts Possible real-world applications of constraint models include au- tonomous driving systems. Since constraint models are often represented by black-box deep models, there no guarantee that the models are trustworthy and interpretable. When an autonomous vehicle is involved into accident, it is difficult to identify the cause of this accident, which might cause loss of confidence in autonomous systems while negatively impacting society.
25

---

Published as a conference paper at ICLR 2023
Stochestit Noise w1..101.... -01))
Stochestic Noise ATO. 12-21
Stochestic Nobe M1. -3-3
14000
14000
14030
BC2L
BC2L
BC2L
12000
12000
GACL
12000
Rawweds
VICRL
MECL
10000
MECL
MECL
4000
8000
0830
4000
4000
4000
2000
2000
2000
5000
10000
15000
5000
15000
5000
10000
15000
Stochastic Noise AED, 1E
Stachastic Noise .70..1
Stochastic Noise ATO, 1E
1.0
1.0
BC2L
BC2L
BC2L
GACL
GACL
GACL
MECL
MECL
MECL
VICRL
0.6
0.5-
0.6
Corainar
0.4
0.4
0.4
0.2
10000
15000
0.0
5000
10000
15000
5000
10000
15000
Stochastic Naise ArDD, 1F-
Stochastic Noiso N10,1F (F-)
Stochastic Nake AED, 1F
500
500
BC2L
BC2L
BC2L
GACL
GACL
GACL
MECL
MECL
MECL VICAL
VICRL
VICRL
200
300
300
200
200
200
0
10000
20000
30000
40000
10000
20000
30000
40000
10000
20000
30000
40000
Stachastic Nalse ATO, 1E (1)
Stachastic Noise .70., 2)
Stochastic Noise ATO, 1E
1.0
BC2L
BCZL
BC2L
GACL
GACL MECL VICRL
GACL
MECL
018
MECL
0.6
0.010
0.4
0.005
10000
20000
30000
40000
0
10000
20060
30000
40000
10000
20000
30000
40000
Stochastic Noise AT0,, 16-1)
Stochastic Noise N70. 16
Stochastic Noise AEO, 1F
500
GACL
GACL
GACL
BC2L
BC2L
BC2L
MECL
MECL
VICRL
VICRL
VICAL
.00
300
300
200
200
0
2000
4000 5000
5000 10000
2000
4000
6000
8000
10000
2000
4000 5000
5000
10000
Epsode
spioode
Sossode
Stochestic Noise ATO. -11
Stochastic Noise .NO 16
Stochastic Noise
15
GACL BC2U MECL VICKL 106
BC2L
0.5
BC2L
CondtnenVaaattoo Pate
MECL
MECL
VICRL
VICRL
0.6
0.5
0.4
0.0
7000
4006
ROOD
8008 30000
0,0 0
2000
4000
5000
BODO
10000
0.0
2000
4000 BOO0I 8000 36800 withe
Ep Audee
Episode
26

---

Published as a conference paper at ICLR 2023
Stochastic Noise AEO, 15
Stochastic Nolso 11
Stochastic Noise AT0, 1F
14
BC2L
14
BC2L
14
GACL
GACL
GACL
MECL
MECL
MECL
10
VICRL
10
30-
VICRL
20000
40000
60000
80000
20000
40000
60000
80000
20000
40000
50000
30000
Stochestic Noise AEO, 1E 11
1.0
Stochastic Noise A10. 1E-
Stochestic Noise AEO, 1E 11
BC2L
GACL
GACL
GACL
MECL
VICRL
VICRL
VICRL
0.6
0.5
11.
0.4
0.0
20000
40000
63089
B0000
0.9
0
20000
40000
50000
50009
0.0
20090
40000
6000U
B0U00
Episode
Figure D.7: From left to right, the transition function has the noises // 0.001). 0.01), and (0,0.1). 0.1). The enviromment from top bottom Blocked Ant, Blocked Walker, Blocked Swimmer and Biased Pendulum. We use feasible rewards(top) and constraint violation rate(bottom) as the two metries of the experiment.
27


Reviews:
[{"review_id": "yCT7R7i4us", "reviewer": "Reviewer_cYaH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors provide a new benchmark for constrained inverse reinforcement learning, including modified mujoco environments, as well as a self-driving-car inspired environment.  The authors additionally propose a Bayesian algorithm for solving problems in this class they call Variational Inverse Constrained Reinforcement Learning (VICRL).", "summary_of_the_review": "A solid submission, including a new algorithmic technique and a new benchmark.  I have some reservations about lasting impact, but these are not show-stopping reservations.", "strength_and_weaknesses": "Strengths: The benchmark seems well motivated, and the experiments provided by the authors seem sufficient to establish the shape of the problem, but also to gesture at headroom for improvement.\n\nWeaknesses: It's unclear to me how long this will remain a relevant benchmark.  The MuJoCo suite is a bit dated now, and the community has tended to overfit to it as a reference simulator.  A benchmark like the one proposed by the authors, but with significantly more variability / range of difficulty seems like it would be strictly more useful, and possibly last longer as a concrete target for the community.  As it stands, I worry that this benchmark will be \"solved\" within approximately 6 months.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iBA9WfXwCWX", "reviewer": "Reviewer_sTb2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper can be thought of as two (related) papers:\n\n1. A proposed benchmark for learning constraints from demonstrations in continuous (approximately deterministic) domains. Here constraint is taken to be an assertion that sum of unknown random variables (either in expectation or per episode) is less than a particular value. The goal of this family of problems is to identify the unknown constraints. This includes a series of domains with corresponding demonstrations, objectives, and constraints.\n\n2. An adaptation of maximum entropy based constraint inference that explicitly models distributions over the constraints. The adaptation is shown to perform well on the proposed benchmark making it a reasonable baseline for future comparisons.\n\n", "summary_of_the_review": "I think the area of constraint learning is indeed in need of a set of benchmarks. I think this paper is a good step in that direction, although I would argue has a fairly biased focus on continuous deterministic domains that is not emphasized enough as a limitation / bias.", "strength_and_weaknesses": "# Strength\n\n1. I agree with the paper that this (important) emerging sub-area is in need of common benchmarks.\n\n2. The domains and proposed constraints/demonstrations are certainly interest and cover both synthetic and \"real world\" scenarios.\n\n3. The proposed algorithm makes sense and builds on prior work in a pretty reasonable way.\n\n# Weakness\n\nAs I tried to highlight in the summary, this benchmark focuses on domains that are entirely deterministic (the synthetic ones) or approximately deterministic (the ones generated by human performance noise). Like with the development of maximum entropy inverse RL, it is worth distinguishing this from fundamentally stochastic domains, i.e., when causal entropy is not well approximated by \"non-causal\" entropy. Here I would argue that the gridworld domains that were directly disregarded offer a non-trivial test bed, e.g., for high-level planning in the presence of environment and multi-agent noise.\n\nIn particular, the proposed algorithm is really only suitable for (approximately) deterministic domains as is pointed out in Brian Ziebart's 2010 thesis. \n\n[Nitpick] Finally, I would argue that the proposed algorithm addresses a slightly different problem since it is privy to an implicit prior distribution over constraints?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9-ZRSlPk7Jy", "reviewer": "Reviewer_4BCv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper makes two contributions. First, it proposes a new benchmark for Inverse Constrained Reinforcement Learning (ICRL) for continuous control tasks that provides environments along with location constraints in those environments that an agent has to satisfy, as well as expert demonstrations obtained through PPO-Lag for each of these. The benchmark includes continuous control tasks in MuJoCo and separately a continuous driving control task. Second, the paper proposes a new method, Variational Inverse Constrained Reinforcement Learning (VICRL) which uses variational inference to infer a full posterior over possible constraints. The paper compares VICRL to previously proposed constraint inference methods and finds that VICRL slightly outperforms the alternatives.", "summary_of_the_review": "It is risky to write a paper that makes two contributions simultaneously. Doing so risks that neither component alone will be done sufficiently well to merit publication. Unfortunately I think that is the case with this paper. I personally think the introduction of the benchmark is likely to have a greater impact on the community than the VICRL technique, as the VICRL method does not statistically outperform either of the two state-of-the-art baselines for the presented tasks. However, as mentioned in the weaknesses section, the benchmark is not quite general enough to be used as a comprehensive assessment of agents’ abilities to learn constraints from demonstrations. I therefore do not think the paper can be accepted as is, and would encourage the authors to expand the benchmark part of their paper to consider a greater range in the types of constraints that are investigated.\n", "strength_and_weaknesses": "Strengths\n* Clarity of presentation. With the exception of some minor typos here and there, some of which I point out below, the paper is well written and clear. The benchmark is well motivated from the perspective of a key missing piece for comparing different approaches to ICRL, and the method description was straightforward to follow.\n\n* Potential for impact. The proposed benchmark seems like it would meaningfully fill a current gap to make reproducibility and comparisons across different methods for constraint inference easier. While I have some concerns that the benchmark as defined is too limited for meaningfully advancing the field as a whole, I believe in the general idea and think that if done well it could have a significant impact.\n\n* Baseline comparisons. The baselines compared to in this paper are up-to-date methods that highlight different approaches to learning and inferring constraints from demonstrations. This aids in the interpretation of the results for the proposed VICRL method.\n\n* Code looks reasonably straightforward to use with clear README which will be valuable for anyone interested in actually using the benchmark.\n\nWeaknesses\n* The VICRL method does not statistically outperform either of the presented state-of-the-art baselines across tasks (GACL or MECL). While it may perform comparably to prior techniques, there is no environment for which it statistically significantly outperforms baselines (based on Figures and Table B.2 of appendix). In order to then still show value of VICRL over the baselines, more information is needed as to where VICRL shines by having a posterior distribution over constraints. While what it learns appears to be interpretable based on the appendix, it is not obvious that the posterior distribution helps with this. So what exactly is the variational method providing in terms of interpretability, efficiency, or performance? If there is no clear answer to this, the paper might be better written as a benchmark only paper, with some extensions to the considered tasks.\n\n* Proposed benchmark is lacking constraint and environment representations that the community already thinks are critical.\n  * Grid-world representations not included, despite being mentioned by the authors in several places as a major focus of study for a subset of the RL community.\n  * Constraints specified mostly as location constraints do not cover the vast varieties of constraint types that matter for robotics and RL applications\n  * Many constraints that are interesting from a robotics standpoint include multiple variables, and sometimes multiple objects, and are often time-varying.\n  * For example, consider those that describe the dynamics inside common simulators like PyBullet or MuJoCo, we have constraints like fixed (two bodies move together as one), revolute (fixed axis of rotation around which an object can move), or sliding. In safety applications, constraints are very likely to be time-varying, such as avoiding coming into contact with any humans. \n  * None of these are represented by the current benchmark. This risks developing methods that are overly specific to static location-based constraints, which would not reflect what the community ultimately wants for downstream applications.\n\nNot a weakness per se, but I didn’t understand how it could be possible that the constraint inference methods could out-perform PPO-Lag for the blocked ant environment. Could the authors elaborate on this?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster