Toward Exploratory Inverse Constraint Inference with Generative Diffusion Verifiers
Authors: Runyi Zhao
Sheng Xu
Bo Yue
Guiliang Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Inverse Reinforcement Learning
Generative Diffusion Model
Primary area: reinforcement learning
TL;DR: We propose an exploratory inverse constraint learning algorithm for inferring a diverse set of feasible constraints from offline dataset.

Abstract: An important prerequisite for safe control is aligning the policy with the underlying constraints in the environment. In many real-world applications, due to the difficulty of manually specifying these constraints, existing works have proposed recovering constraints from expert demonstrations by solving the Inverse Constraint Learning (ICL) problem. However, ICL is inherently ill-posed, as multiple constraints can equivalently explain the experts' preferences, making the optimal solutions not uniquely identifiable. In this work, instead of focusing solely on a single constraint, we propose the novel approach of Exploratory ICL (ExICL). The goal of ExICL is to recover a diverse set of feasible constraints, thereby providing practitioners the flexibility to select the most appropriate constraint based on the practical needs of deployment. To achieve this goal, we design a generative diffusion verifier that guides the trajectory generation process using the probabilistic representation of an optimal constrained policy. By comparing these decisions with those made by expert agents, we can efficiently verify a candidate constraint. Driven by the verification feedback, ExICL implements an exploratory constraint update mechanism that strategically facilitates diversity within the collection of feasible constraints. Our empirical results demonstrate that ExICL can seamlessly and reliably generalize across different tasks and environments. The code is available at https://github.com/ZhaoRunyi/ExICL.

Published as a conference paper at ICLR 2025
TOWARD EXPLORATORY INVERSE CONSTRAINT IN- FERENCE WITH GENERATIVE DIFFUSION VERIFIERS
Runyi Zhao' Sheng Xu' Bo Yue', Guiliang Liu't 'School of Data Science. The Chinese University of Hong Kong, Shenzhen trunyizhad shengxul, boyue j link. cuhk edu. en liuguil Scugh .du.cn
ABSTRACT
An important prerequisite for safe control is aligning the policy with the under- lying constraints the environment. In many real-world applications, due to the difficulty of manually specifying these constraints, existing works have proposed recovering constraints from expert demonstrations by solving the Inverse Con- straint Learning (ICL) problem. However, ICL is inherently ill-posed, as multiple constraints can equivalently explain the experts' preferences, making the optimal solutions not uniquely identifiable. In this work, instead of focusing solely on a single constraint, we propose the novel approach of Exploratory ICL (ExICL). recover a diverse set of feasible constraints, thereby pro-
The goal of ExICL viding practitioners the flexibility to select the most appropriate constraint based on the practical needs of deployment. To achieve this goal, we design a gen- erative diffusion verifier that guides the trajectory generation process using the probabilistic representation of an optimal constrained policy. By comparing these decisions with those made by expert agents, we can efficiently verify a candidate constraint. Driven by the verification feedback, EXICL implements an exploratory constraint update mechanism that strategically facilitates diversity within the col- lection of feasible constraints. Our empirical results demonstrate that ExICL can seamlessly and reliably generalize across different tasks and environments. The code is available at hnttttt://// thu.. com/ whhaoRuni/Exxee
INTRODUCTION
In recent years, Reinforcement Learning (RL) agents have demonstrated remarkable performance in a variety of virtual games and environments by extensively exploring and exploiting the entire state-action space (Mnih al. 2015 Silver et al., 2018 Vinyals al., (019)). However, real- world applications often prioritize the safety and reliability of decisions, requiring RL policies to operate under restricted regions or spaces in realistic environments. To learn such policies, safe RL methods typically update the policy within the bounds of constraints (Liu et al. [022]). However, in practical applications, these constraints are often not readily available and can be challenging to specify manually, particularly in complex environments.
Recent advances in Inverse Constraint Learning (ICL) propose recovering the constraints followed by expert agents from their demonstration (Scobee & Sastry 2020). Previous methods (Malik al.) 2021 Kim et al. 2023) typically extend the classical Inverse Reinforcement Learning (IRL. frame- work to learn the constraint model under known rewards. However, IRL. is essentially an ill-posed problem (Ng Russell 2000). and the optimal solution is often non-identifiable. When it comes to ICL, we find multiple constraints difficult to identify the real constraints. In resolving the problem of unidentifiable solution, pre- vious ICL solvers typically rely additional assumptions, such the expert agent implementing a regularized policy (Malik et al. (02T), or the ground-truth constraint set having a minimum cov- erage of state-action pairs (Scobee & Sastry/ 2020). While these assumptions reduce the number of candidate constraints, there is no guarantee that the real constraint can be uniquely identified or accurately characterized by these assumptions.
equivalently explain expert demonstrations, which makes
"Equal contributions. 'Corresponding author: Guiliang Liu, fugailiang@euhh.cdii

---

Published as a conference paper at ICLR 2025
To address these issues, in this work, propose Exploratory Inverse Constraint Learning (Ex- ICL) algorithm, which learns the set of feasible constraints with which the agent can accurately recover the expert demonstration. To deploy these constraints in practical applications, practitioners can select from the feasible set based on domain knowledge or specific requirements. Although the concept of a feasible solution set has been theoretically analyzed by IRL solvers (Metelli al. 202T) and extended to ICL settings (Yue et al., 2024] (2025). the development of practical implementations remains largely unexplored. This is due two main reasons (see Figure D: 1) The difficulty in verification: To verify the feasibility of a candidate constraint, prior studies must tackle forward constrained RL problem. This involves multiple rounds of policy model updates under the con- straint, rendering the recovery of a feasible set of constraints computationally intractable, especially in complex environments. 2) The lack of exploratory mechanisms: Previous ICL, algorithms focused primarily on identifying a single constraint, lacking an exploratory mechanism to infer diverse set of feasible constraints.
In response to the difficulty in verification, in this work, our ExICL algorithm utilizes Generative Diffusion Verifier (GDV) to accelerate verification. Operating within an in-context learning frame- work, GDV can generate the optimal trajectory for given constraint model without necessitating updates to model parameters. This capability is enabled by a generative optimization framework, where the GDV guides the generative process of a diffusion model with the probabilistic represen- tation of an optimal policy under a specified constraint. During the trajectory generation process, the constraint model must accurately predict the cost of trajectories under varying levels of noise. Motivated by this requirement, we designed a noise-robust objective for updating constraints. The divergence between the generated trajectories and the expert trajectories efficiently determines the feasibility of constraint and guides the updates to our constraint model.
Upon identifying a feasible constraint, our ExICL algorithm initiates an exploratory update iden- tify additional constraints. To facilitate this, we develop a strategic exploration mechanism that enhances the diversity of feasible constraints learned through a contrastive learning objective. In particular, we characterize each constraint by the predicted feasibility (or cost) values assigned to each state-action pair. Building on this characterization, we refine the constraint update objective to encourage divergence between these values and those of other constraints within the feasible set. This exploratory update mechanism enables learning of diverse set of feasible constraints.
To empirically validate ExICL method, assess its performance across diverse set of tasks (including navigation, locomotion, and autonomous driving) and under various types of constraints (such spatial, dynamic, and kinematic). The results demonstrate that ExICI. outperform other baseline methods from multiple perspectives, including 1) inferring more accurate constraints, 2) enhancing the diversity of the inferred constraint set, and 3) boosting training efficiency. These empirical studies consolidate the validity of our EXICL.
Constrained Reinforcement Learning
Generative Diffusion Venfier
Constraint Set Go
Single Constraint
Sample Data
Yes If Match Expert
Random Noise
Explore
Iterate until Canvergence
Trejectory
Inverse Constraint Update
Constraint Guided Denoising
If Match Expert
Exploratory
Constraint
No
Update
Value Update
Policy Net.
No
Update
Constraint Model
Trajectory
Constraint
Figure 1: The flowchart of classic ICL. (left) and our EXICL (right). The key differences are: 1) To verify constraint, ICI. iteratively updates the policy and value functions using samples from the dataset, whereas ExICL employs guided generation approach, simplifying the verification process. 2) while the objective of classic ICL infer single constraint model, ExICL adopts an exploratory constraint learning strategy aimed at identifying a broader, feasible set of constraints.
2 PROBLEM FORMULATION
Constrained Reinforcement Learning (CRL). The CRL problem commonly studies Constrained Markov Decision Process (CMDP) setting M, := (S,A, T, r, c, e, Ag. 7), where: 1) S and A de- note the space of states and actions. 2) 7 (8') de|ss, a) defines the transition distribution. 3) r(s, a) and
2

---

Published as a conference paper at ICLR 2025
c(s, a) denote the reward and cost function (we assume 0). 4) defines the threshold of the constraint, where 6 = 0 refers to a hard constraint, enforcing absolute satisfaction, while e > denotes soft constraint, permitting a certain degree of constraint violation. 5) 1l denotes the initial state distribution. 6) 7 E [o, 1) is the discount factor. In our empirical study, we mainly study an episodic setting where the game ends some terminating state or time horizon. The goal of CRL policy т E AS is to maximize the expected discounted rewards under known constraints
T
T
max ET 3.PP
Σ ("r((1; an)] s.t. TT,, PT.P t=
Σ D=
E''o(se c(s8, ae)
S
(PI)
Inverse Constraint Learning. While traditional CRL problems often assume that cost signals c(-) are directly observable from the environment, in many real-world scenarios, we typically have ac- cess to expert demonstrations DE rather than observing the cost signals directly. To address this challenge, recent study (Scobee & Sastry 2020 introduced Inverse Constrained Reinforcement Learning (ICRL). The objective of ICRL is to infer the underlying constraint models from expert demonstrations, enabling any policy to reproduce these demonstrations by applying the recovered constraints. To achieve it, previous works (Malik et al. 2021) apply the Maximum likelihood Esti- mation (ML.) objective under the maximum entropy framework:
arg arg PM. (DE = arg max
II TEEDE
(rx)) g (T
(I)
7!Pal
Inspired by (Malik et al. 2021), we approximate 1 (Mc(T) = I dh (s,, az) such that (st, at) E
10, 1] indicates the permissibility of performing action at state The MLE gradient W.PM (Dr|C) can be transformed to:
VuPM, = ET+-D* [V log (T") - (TM( V log (F)]
(2)
where 1) the cost function can derived by c(s, a) = log Φ(s, a). 2) f denotes the estimated trajec- tory based on the CMDP Me with the learned cost c. Under this setting, ICRL algorithms typically assume that reward signals are observable and the goal is to recover only the constraints, in contrast to Inverse Reinforcement Learning (IRL) (Ziebart et al. 2008), which aims to learn rewards from an unconstrained MDP.
Identifiability Issue in ICRL. Like many other inverse optimization problems (Arora & Doshi 2021]. ICRL is essentially ill-posed since various combinations of rewards and constraints can plain the same expert demonstrations, which makes it difficult to identify the ground-trute constraint uniquely. Striving for the identifiability of solutions, pioneer work (Scobee & Sastry (0200) intro- duced the concept of a minimum constraint under a discrete CMDP. This constraint comprises the smallest number of state-action pairs necessary for an expert agent to reproduce expert demonstra- tions, ensuring that the optimal constraint is as concise as possible. A continuing work (Malik et al.) 2021) extends the concept of the minimum constraint to continuous state-action spaces by incor- porating a weighted regularizer into the constraint update objective. However, this extension faces several challenges: 1) Finding the minimum constraint in a continuous state space is intractable. 2) The scale of regularization is highly sensitive to the chosen weighting term, which can not guarantee the solution unique. 3) Even when we capture the minimum constraint, there is no guarantee that this constraint is the one respected by the actual expert agents.
Beyond focusing on the exact minimum constraints, inspired by the recent theoretical advancement in IRL (Metelli et al.) 2021), intriguing but less explored solution is to infer the set of feasible constraints. Under this setting, recent studics (Lindner et al.] 2022]] Metelli et al.) 2023) developed a theoretical framework for characterizing the feasible set of solutions for inverse optimization prob- lems. However, these studies focus on reward learning instead of the constraint inference problem. Besides, their results are only applicable under discrete state-action spaces, but the realistic ap- plication aligns better with continuous state-action space. In the meantime, while previous works commonly rely on online interaction with the environment, we focus on an offline setting. This is because in practice data collection is expensive (e.g., in robotics, educational agents, or healthcare) or dangerous (e.g.. in autonomous driving. or healthcare). Motivated by the above considerations, we propose the following offline ICRI. problem considering the feasible constraint set.
Definition 2.1. The problem of offline inference for feasible constraint set can be characterized by a pair q (M, Dº), where M is CMDP e (CMDP without the cost) and DO (DE D B) is the
3

---

Published as a conference paper at ICLR 2025
offline demonstration such that D {sh dr r Ne In denotes expert demonstrations and D F denotes the dataset generated by non-expert agents. cost model c x A X H + R is feasible for Q if 7h is an optimal policy for the CMDP M U ie., T² I['ee We denote by Cpp the set of feasible cost functions for Q namely feasible constraint set.
In solving the ICSI problem above, a critical prerequisite is to efficiently assess the feasibility of constraint by determining whether an expert policy can be learned under this constraint. However, traditional ICRL solvers (Scobee & Sastry] 2020 Malik et al. 2021| Liu et al. (024) are based on bi-level optimization framework, which requires solving both a forward CRL problem and an Inverse Constraint Learning problem. Updating both policy and constraint function is computationally intensive, significantly impacting the efficiency of learning the feasible constraint set.
EXPLORATORY INVERSE CONSTRAINT LEARNING
To learn diverse set of feasible constraint, we introduce the Exploratory Inverse Constraint Learn- ing (ExICL. algorithm (T). Striving for efficient assessment of candidate constraints, ExICL lever- ages a Generative Diffusion Verifier (GDV) to evaluate whether expert demonstrations produced under the examined constraint (Section 3.1). This optimization operates in context, thus it bypasses the need to update model parameters, resulting in a significant increase in computational efficiency. To ensure that the constraint model can effectively guide the denoising process, ExICL implements a noise-robust constraint learning (Section 3.2). Additionally, ExICL employs an iter- ative exploration process that strategically updates the constraint function, thereby facilitating the identification of broad range of feasible and diverse constraints (Section 3.3
be
3.1 DIFFUSION PLANNER FOR GENERATIVE VERIFICATION
Learning the Planner from Offline Demonstration. Inspired by the Diffuser (Janner et al. 2022 Ho et al. 2020), we follow the diffusion probabilistic models (Sohl-Dickstein et al. |015 Ho al, 2020) and formulate planning trajectory generation task through learned iterative denoising diftusion process pa Pa("" (1'). The data distribution induced by the denoising process is given by:
Po( pa (T()) II po ((( where (e(T"-117) = -N ,),,
where i E [0, I] denotes the diffusion step and each r¹ = (sh ap- at......... s/ af) where & E [0, denotes the planning step (i.e., the time step of action execution in the environment).
This learned denoising process is trained to reverse forward diffusion process ((r"( that slowly corrupts the structure of trajectories by adding noise. The corresponding distribution is:
((> = ((7)) II (((((1=11) 2=0
where ((+')>"1)=N(( 1-----T-11 ,31)
(4)
Under these denoising and diffusion processes, q(7²) denotes the data distribution, and g(r²) indi- cates the standard Gaussian prior. To learn the demoising model pe (r(=-1(r') from the offline dataset D we optimize its parameters by maximizing the negative likeliiio of observed trajectories via constructing variational lower bound over the individual steps of denoising:
(oogg)(oggdr p(7))) 2 By(rary log (r:1)
Σ log (o (7
(5)
17771170)
Note that the (T) = po ) approximate the distribution of trajectories within the offline demonstration Do, and there is no guarantee a trajectory t ~ (T generated under this model will be safe. By leveraging this property, GDV can efficiently verify the feasibility of constraint by determining whether it can guide the diffusion planning process to generate a safe trajectory.
Verifying Constraint via Guided Denoising. To verify the feasibility of constraint, GDV utilizes the guided sampling strategy (Janner et al.) (022) and perturbs the distributions in the iterative denoising process. At each denoising step, the perturb distribution (7( can be represented as:
po (T) p(T|O0.s 1) P.M. (Oo.r ()7)p)(T)
4

---

Published as a conference paper at ICLR 2025
where Ou:r is binary variable denoting whether the outcomes are desired. Without considering the constraint, the probabilistic inference framework for RL. (Levine 2018) defines
T
pOO..- = 1,-) pa(T)p(OOoT - 1/7) - pp(0)) II M(s++1100 az) (t) e r(sease) (7)
where 1) (s(()(s)) denotes the behavior policy that generates the offline dataset. 2) po(T) =
p(s0) [I p(5 sse, at) (T))ss)))) denotes the trajectory distribution in the offline dataset D
and 3) p(Ou:r 1|T) eLi (8).))) the optimality model.
By extending this advancement to CMDP Me. the optimal probabilistic representation for con- strained policy model P.M. (Oo.T = 1/7) defined by:
Pm. PM. (O: 1/7) e Ci (s).a))
E, d(s c(s ac) ( Se E, Dr c(t,, ax)l >e
(8)
Although the CRL objective essentially non convex,,, this problem general has zero duality gap: Theorem 3.1. (CRL has zero duality gap and the Slater's 'ss condition holds for PP.
Paternain et al. (2019). ) Suppose that and e are bounded then strong duality holds for Ply, i.e, Pl DI".
min max Egg P.PT As0 x
Σ
(r(ena))))) Ac(st, Ac
(DI)
Correspondingly, P.M. (Ourr 1/T) (Equation
8 can be represented as its dual format by trans-
forming the constraint into penalty such that:
(T) (T(M), (Oo.T 1/7) - pe
where À denotes the Lagrange multiplier. P.M. (T) represents the distribution of trajectories sub- jected to reward maximizing and constraint satisfying objectives, which can model the trajectories generated under given constraint. Since P.M, (r) is modeled by a diffusion model, the denoising process transitions can be approximated as Gaussian (Sohl-Dickstein et al.) 2013) such that:
P=(""=177' (0.r) (Ho(T')
Eg., E') where ge V,PM, (Our 1/7) |== (10)
Such denoising process does not involve parameter updating over the diffusion cost/reward- value models, and thus it is more efficient than the classic CRL solver. By comparing the generated trajectories P.M. (7) with the expert ones E GDV efficiently validates the accuracy of inferred constraints. During this process, to ensure that the GDV can accurately guide trajectory generation and facilitate the discovery of diverse of feasible constraints, we must ensure the following:
The constraint model can accurately guide the trajectory generation in the GDV. Specifically, the model must be able predict the cost of trajectories with added Gaussian noise accurately.
Our algorithm includes mechanism to generate large number of candidate constraints.
In the following sections, we introduce the approach to learning noise-robust constraints with (T) and the exploration strategy for learning a diverse set of constraints.
3.2 NOISE-ROBUR CONSTRAINT UPDATE
To ensure that the constraint model accurately estimates the cost of noisy trajectory, we collect noisy samples (7+ and f0:1) during the diffusion process by introducing noise into 5" and f. Subsequently, we update the classic constraint inference loss (2) to noise-robust version in the following:
( log de o :)) k=
E By(r-.a:1oo) (Σ log 1=0
ERADE
EANDP
Eygorrr oo)
(11)
where 1) DP denotes the nominal trajectory data generated by policy under CMDP M. (with the estimated cost before update) and 2) (=*, i) I (s),,, at, denotes the permissibility function for accurately determines whether the noise-augmented trajectory 7¹ is feasible. In this
manner, the corresponding cost estimation (s) at) = log d_,, (s af) is noise-robust, which can accurately guide the trajectory generation during the diffusion process.
5

---

Published as a conference paper at ICLR 2025
3.3 STRATEGIC EXPLORATION FOR CONSTRAINT UPDATE
Based on the above noise-robust objective, we enhance the algorithm's capability for actively dis- covering feasible constraints by implementing dynamic exploration algorithm. To achieve this goal, we extend the object (TT) by proposing an exploratory constraint update objective designed for strategic exploration, detailed follows:
Σ log (T 1)) DD 1 1=0
Σ log ow of' (+,. :1) ( F), (12)
DE By (:10)) (ro)
where ve refers regularization term, controlling the sparsity and the diversity of constraint func- tions ow To learning constraint representation. motivate by contrastive learning (He et al. 2020).
follow the InfoNCE loss (van den Oord et al. (0188) and implement (i(w,, 2) as:
Vo 2) Epp
log
(13)
Σowez (n (s..)) (er,a)))
where å denotes the regularization parameter controls the scale of sparsity, dist(:,) -) indicates the dis- tance metric, here we choose to be the / norm. Z denotes the set of feasible constraints that have already been discovered. Intuitively, this regularization term eLoo) (e,,,a) encourages the constraint function to assign higher feasibility values state-action pairs, improving the sparsity of the inferred constraints. Simultaneously, this objective fosters the diversity of these constraints, allowing them to be distinguishable from previously learned constraints. Since constraint model can be characterized by its predicted cost values at different state-action pairs, we require different constraint models assign different feasibility values &w the same state-action pairs
Implementation. Based on the above design, A Algorithm|
| Iillustrates the our implementation.
Algorithm Exploratory Inverse Constraint Learning (ExICL)
Require:
Offline dataset DO (DE, D D- }). contrastive exploration rounds M Randomly initialize feasibility function @w and σ²
Initialize the set of feasible constraint Z +- {0} and the Lagrange parameter À x= 0
Train diffusion based trajectory predictor (T( based on D with objective (5) for each exploration coefficient 8 do Initialize the subset of feasible feasibility constraint z + {0} for exploration round
0, .., M do Set the initial feasibility function o = 66 Initialize the set of predicted trajectories Dp {0) while The predicted trajectory f * 7 VT E D do
Generate a trajectory with c(-)
log 6th (-) as costs and (10 as denoising process.
Collect the predicted trajectory Dp = Dp (ff
Update by minimizing the loss A.Firbm (c(T)
Update the feasibility function Φ via objective 12) (based on Dy, 2 and 8) end while
Expand the subset of feasible constraints z = z U { end for Expand the feasible constraint set Z = Z U zó and reset the feasibility function Φi db end for
4 EMPIRICAL EVALUATION
We empirically evaluate the effectiveness of EXICL by its capability of 1) accurately inferring var- ious types of constraints (e.g., spatial, dynamic, and kinematic) under PointMaze and robot control environments (Section 4.1). D. 2) exploring various kinds of constraints with strategic exploration in constraint updates (Section 4.2 and 3) accelerating the process of constraint inference (Section 4.3). We empirically evaluate the effectiveness of ExICL by its capability of 1) accurately inferring var- ious types of constraints (e.g., spatial, dynamic, and kinematic) under PointMaze and robot control

---

Published as a conference paper at ICLR 2025
environments (Section 4.1)
2) exploring various kinds of constraints with strategic exploration in
constraint updates (Section
4.2 and 3) accelerating the process of constraint inference (Section 4.3
Experiment Setting. Our experiments inves- tigate the proposed ExICL in continuous en- vironments. Specifically, distinct PointMaze environments with differ- constraints in Figure 2 See Appendix A. for further details. To evaluate model per- formance in more challenging tasks, we ex- tend three robot control environments in Mu- JoCo (Todorov et al., 2012) by incorporat- ing distinct predefined constraints into differ- ent tasks. The examine tasks include 1) Obsta- cle HalfCheetah, where we introduce spatial constraint that prevents the robot from moving backward; 2) Limited- Speed Walker, where design dynamic constraint to control the robot's maximum forward speed: 3) Blocked Ant, where we add a kinematic constraint on the robot's 's leg an- gular velocity, limiting the size of each movement. Please refer to Appendix A.2 for further details. The autonomous experiment setting can be seen in B.3
construct three
Figure Constrained PointMaze UMAZE, L, and 2WALLS from left to right, where the blue, red, and dark regions indicate the starting, desti- nation, and constrained locations.
the following evaluation metrics: 1) Cumulative Reward, which adds up the total rewards obtained throughout the entire episode; 2) Cumulative Cost, which adds the total costs obtained throughout the entire episode and 3) Feasible Cumulative Reward, which quantifies the accumulated rewards before any constraint violations occur. Each experiment is repeated with five random seeds, and the results are reported as the mean + standard deviation (std). The detailed settings and random seeds are reported in Appendix
Evaluation Metrics. To quantify model performance, following (Malik al.) 2021
4.1 CONTROL PERFORMANCE: QUANTIFYING THE ACCURATENESS OF CONSTRAINTS
In this experiment, we investigate whether ExICL can effectively learn the accurate constraints by assessing whether the learned constraints can facilitate the reproduction of expert demonstrations.
Constrained PointMaze Environments. In this experiment, we design three different PointMaze environments, each featuring unique constraints, as shown in Figure 2 The agent's objective is to navigate from the starting location to the target location while successfully avoiding the imposed constraints. As for performance demonstration, largest 8. Since controls the level regularization on sparsity, our setting is to align with previous setting of ICRL solvers that favor the sparsity of constraints, thereby providing a fair com- parison with previous works.
selected the constraints discovered under the
Comparison Methods. We mainly compare the proposed ExICL with the following baselines: 1) Behavior Cloning (BC), which learns policy by directly imitating actions from expert demonstra- tions; 2) Least Square Inverse Q-Learning (LS-IQQ) (Al-Hafez et al., 2023), which infers reward value functions from offline data to replicate the expert policy. 3) Inverse Constrained Superior Dis- replicatt tribution Correction Estimation (ICSDICE) (Quan al. 2024), which utilizes reward information and solves regularized dual optimization problem for safe control by exploiting the dataset.
Table 1: PointMaze evaluation performance. Each value is reported the mean std over 100 runs and 5 seeds. We highlight the best results with the highest rewards or lowest violations in bold.
PhinMMueeU-MAZZ
PointMuee 2WALLS
Peintuue-LL
Methods
Reward Cost
Reward
Cost
Reward Cost
Reward
Cos
Reward wi Cost
BC
0.88 0.110
2,09 1330
0.75 0,10
050 0016
3.29 3.04
0.66
092 1
115 3.71
092 0.0
1S-1Q
0.54 +
10.38 7.68
0.54 0.20
082+-024
22.36 +999
0.53 -3--00
0.6 276-000.10
14.7+
0.48
BACL (ours)
1.00 :: 0.00
0,01 0.80
0.99 = 0.00
1010000
081 0.00
0.99 0.01
10:1.000
0.00 0.09
101 EUI0
Results Analysis. Table [ presents our evaluation results, revealing that ExICL consistently out- performs other baselines by achieving higher cumulative rewards and lower rates of constraint vi- olations. A key factor contributing to this performance that classic methods such as LS-IQ and BC do not intentionally model the constraint. Consequently, there is guarantee that the agent

---

Published as a conference paper at ICLR 2025
will maintain safe distance from the wall, often resulting in substantial costs and reduced feasible rewards.
Visualization. To better illustrate the learned constraint, Figure shows the validation of trajectory level constraints. We find that ExICL accurately captures the fcasibility trajectories.
S
Foosbbling
Peusialiny
Feastbling
sa
Figure 3:
Visualization of the constraints learned by ExICL under the PointMaze-UMAZE, 2WALLS, and environments. Each point indicates the predicted cost within a trajectory
Constrained MuJoCo Environments. Three different MuluCo environments that differ in agent and constraint are designed for this experiment. The agent's objective is to cover the maximum possible distance within a unit time while prevent violating the constraints.
Comparison Methods. To perform a more comprehensive evaluation, we add more baselines to pre- viously compared ones: 1) SMODICE (Ma al. 20022), which leverages dual and offline reward function to optimize the policy; 2) SeliDICE-Constrare
which replaces the DICE objective used in ICSDICE with OptiDICE (Lee et al. 2021); and 3) MRODICE-Constraiie which incorporates information by adding environment rewards to the SMODICE's learned discriminator reward.
Table 2: MuJoCo evaluation results. The baseline results adapted from (Quan et al. 2024). Each value reported as the mean + std over 10 runs and 5 seeds. Bold denotes safe methods with maximum rewards.
Obstacle- Hastael--alfChhetahh
Limited-Wian
Blocked-Ant
Methods
Reward w/o Cost
Cost
Reward w/o Cost
Cost
Reward w/o Cost t
Cost
BC
731 = 693
0,40
-7.
0,01 0.02
876 138
0.04 0.002
LS-IQ
2175 775
0.04 0,05
603 =: 203
0.00
-63 163::200 208
0.40
SMODICE
3565 1 345
0.13 0.111
2334 1: 238
0.52
1410
0.33 0.007
SMODICE
3829 661
0.30 0.24
1871 1 155
0.18 0.151
1763 180
0.53 1.003
OptIDCCE-
597
0.03 = 0.06
1538 + 283
0.01 0.001
3070
0,01 0.000
ICSDICE
2315 7400
0.04 = 0.04
1587 +: 308
0.01 0.002
3073 103
0,01 0.00
ExICL (ours)
5298 1: 480
0.00 1: 0.00
1862
0.00 0.00
3061 199
0.01 1.001
Results Analysis. Table shows shows the evaluation results in high-dimensional robot control tasks. We find that ExICL consistently achieves fewer constraint violations across all three environments, regardless of the constraint type, demonstrating the effectiveness of the inferred constants. In con- trast, offline IL methods, such as BC. IS-IQ. and SMODICE, generally fail ensure safety, even when reward information is incorporated. This limitation is expected, as these methods are not ex- plicitly designed to address safety concerns. Interestingly, OptiDICE-c, with its normalization and soft-chi divergence tricks, and ICSDICE, utilizing superior DICE approach tailored for constraint learning, exhibit satisfactory performance with low costs and relatively high rewards. However, neither method surpasses ExICL in terms of rewards, except in the Ant environment where perfor- mance is comparable. This especially evident in HalfCheetah environment, where ExICL's exploration ability enables the agent to effectively pursue rewards.
4.2 EXPLORATORY PERFORMANCE: EVALUATING THE DIVERSITY OF CONSTRAINTS
In this section, we study the exploratory performance our method by quantifying the diversity of constraints within the learned feasible constraint set.
Constrained PointMaze Environments. In this experiment, we study whether the ExICL can learn diverse set of constraints by analyzing the cost values assigned to the expert trajectories and
8

---

Published as a conference paper at ICLR 2025
expert ones. Ideally, the constraints in the feasible set should assign different costs to the same trajectory, in the meantime, the expert trajectories should consistently have lower costs.
Comparison Methods. In this study, we primarily compare several common exploration methods, including 1) Random-Noise, which involves directly adding noise to the parameters of learned con- straints to discover alternative constraints; and 2) Random Initialization, where we repeatedly learn different constraints starting from normal distributed initial model parameters other than zero con- stant initialization for other methods.
Sandom Nobe Ie-4 Partfook IMAZE bad, 779
Rancom Ine Peirthaze Pui-traze--M4222 INA22 bas mi]
Ex-CC FrerrMaze UKKZ 1MM
240
100
100
Codt VaLe
20-
Nean Car.
Near Cart
Cat
Aancom Noiin In-4 Polettape UN477 57p
Sandom Pobrriaaz UMZE expret
EetCI roblans UMAZE euzert 77
Get 2
Modell
Model Indea
5.00
Figure 4: The delta varying exploratory results for both constraint-violating trajectories (top) and expert trajectories (bottom) of CointMaze-UMAZZ environment. Each data point corresponds to the cumulative costs for trajectory. Three exploration strategies are presented: random noise (left), random initialization (middle) and EX-ICL (right).
Figure 4 iillustrates the exploratory results in the PointMaze- UMAZE environment, with exploratory models varying in regularization parameter 8. Additional ò- d-varying exploratory results for the other two environments, PointMaze-L and PointMaze-2Walls, are displayed in Figure 7 and Figure X in the Appendix, and the exploratory results depending on exploration rounds are displayed in Figure implies the enlarging sparsity of the constraints. Our results reveal that the diversity of reasible constraints discovered by random noise and initialization methods is less effective compared to that achieved by our ExICI. method. Another intriguing observation is the costs of constraint-violatie have a significantly higher variance than those of expert trajectories. This is because, to recover the constraint, ICRL algorithms must increase the cost values of bad trajectories above the threshold On the other hand, for the cost values of expert trajectories, ICRL algorithms must guarantee their values to be smaller than e. In this work, our is set to close to zero, so the scale of variances for bad trajectories is much larger than those of expert trajectories. By implementing strategic exploration, EX-ICL exploration strategy successfully identifies this diverse of feasible cost models, causing the variance of predicted cost values to be higher.
The decrease of cost value as regularization parameter 5 increase in Figure
trajectories
Robot Control Tasks. Figure 5 visualizes the robot trajectory segments during evaluation in three MuJoCo environments. From left to right are the results without constraint model (nominal), and with three different constraint models during exploration. The constrained value of the last frame is displayed, where yellow indicates constraint violations and green indicates safety. We present the average reward and cost per step for each figure. We observe that different constraint models lead to distinct behaviors, with red circles highlighting the key differences. For example, in the HalfCheetah environment, the robot moves varying distances due to the exploration process influenced by the constraint model. Similarly, the Walker robot is encouraged to lift its foot higher to increase speed, while the Ant robot explores in the northward direction. Appendix reports numerical results.
4.3 LEARNING EFFICIENCY: HOW FAST A CONSTRAINT CAN BE INFERRED
We conduct empirical study to compare the learning efficiency of our EXICL algorithm with other baselines within the PointMaze environment. The learning efficiency is measured by data points used for training as emphasized in offline reinforcement learning where the dataset collection

---

Published as a conference paper at ICLR 2025
Planning
Planning Costo
Planning Cost1
Planning Cost2
Spe
Rewyard-2.00 Caat-O.53
Reward- Cost-0.00
Roward-1 Cest-0.00
Roward Cest-0.06
spedd1131
Reward-2.98 2.92. Oost=0.90
Rewarden 144, C.0t
Rgward-1..4 Cost-0:00
Rewarda1.8 Cost-D.00
414-5-0 025
233 Cost=O.38
Reward-23 -3.00 Cost=0.95
Reward-3.01 Costeq.0c
Rewerd-3 00, Cost
Figure 5: Visualization the exploration results in MuJoCo environments. Each row represents an environment with identical frames for comparison.
more costly (Levine et al. 2020). The data efficiency curve illustrated in Figure 6 below depicts the trend of performance data point usage increases. The curie represents the progression till the discovery the first valid constraint, aiming to offer a fair comparison with previous work that studies only one constraint. The area circled with orange ellipse represents the training phase of the GDV model and reward value model. Among the compared methods, we find that our ExICL exhibits the highest sample efficiency. This efficiency stems from its ability learn feasible constraints that yield significant rewards while utilizing minimal number of data points. This is attributed our GDV model, which performs a generative denoising process to validate candidate constraints without relying on frequent policy updates, thereby significantly improving sample complexity.
1150
IC50
ICS
Fasiie Rowrros
Date Po
Data Paint
Data Prant
Figure 6: Training efficiency of three methods in three PointMaze environments.
CONCLUSION
In the paper, we introduced the ExICL algorithm, which designed learn diverse set of con- straints from an offline demonstration dataset. By proposing GDV model, we significantly accel- erated the verification of candidate constraints. Additionally, we developed strategic exploration mechanism that updates constraints and efficiently expands the constraint set, thereby facilitating the discovery of varied constraints. To validate our method, we conducted experiments across a diverse array of tasks, including navigation, locomotion, and autonomous driving, and under various types of constraints. Our results demonstrate that ExICL significantly outperforms other baseline methods accurate constraints, discovering diverse constraints, and enhancing learn-
in terms of learning ing efficiency. A promising direction for future work involves expanding ExICL to more practical environments, such quadrapedal and humanoid robot control.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work supported in part by Shenzhen Science and Technology Major Program un- der grant 17Z0202409003044000000 Shenzhen Fundamental Research Program (General Pro- gram) under grant JCYJ20230807 14202005, Guangdong-Shennzhnn Joint Research Fund under grant 2023A15151 10617, 17, Guangdong Basic and Applied Basic Research Foundation under grant 2024A1515012100 and Guangdong Provincial Key Laboratory of Mathematical Foundations for
Artificial Intelligence (2023B1212010001)
REFERENCES
Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, and Jan Peters. Ls-iq: Implicit reward regularization for inverse reinforcement learning. arXiv preprini arXiv:2303.0059) 2023.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, meth- ods and progress. Artif. Intell., 297:103500 2021.
Johann Brehmer, Joey Bose, Pim de Haan, and Taco S. Cohen. EDGI: equivariant diffusion for plan- ning with embodied agents, In Advances in Neural Information Processing Systems, (NeurIPS), 2023.
Glen Chou, Dmitry Berenson, and Necmiye Ozay. Learning constraints from demonstrations with grid and parametric representations. The International Journal of Robotics Research, 40(10-11); 1255-1283, 2021.
Zeyu Feng, Hao Luan, Pranav Goyal, and Harold Soh. Ltldog: Satisfying temporally-extende symbolic constraints for safe diffusion-basel planning. CoRR, abs/2405.042335 2024.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In Neural Information Processing Systems ((NeurIPS)), pp- 4028 4039, 2021.
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang. Bin Zhao, and Xue- long Li. Diffusion model an effective planner and data synthesizer for multi-task reinforcement learning. In Advances in Neural Information Processing Systems, (NeurIPS). 2023.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, (CVPR), pp. 9726-9735. Computer Vision Foundation / IEEE, 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS). 2020.
Jifeng Hu, Yanchao Sun, Sili Huang. Siyuan Guo, Hechang Chen, Li Shen, Lichao Sun, Yi Chang, and Dacheng Tao. Instructed diffuser with temporal condition guidance for offline reinforcement learning. CoRR, abs/2306.04875, 2023.
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, (ICML), volume 162 of Proceedings of Machine Learning Research, pp. 9902-9915, 2022.
Konwoo Kim, Gokul Swamy, Zuxin Liu, Ding Zhao, Sanjiban Choudhury, and Zhiwei Steven Wu. Learning shared safety constraints from multi-task demonstrations, In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2023.
Robert Krajewski, Julian Bock, Laurent Kloeker, and Lutz Bekstein. The highd dataset: A drone dataset of naturalistic vehicle trajectories german highways for validation of highly automated driving systems. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 21 11118222.... 2018. doi: 10.1 (0.1109/TTSC.200 2..565555....
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference Machine Learning (ICMI., pp. 6120-6130, 2021.

---

Published as a conference paper at ICLR 2025
Sergey Levine.
CoRR, Reinforcement learning and control probabilistic inference: Tutorial and review. 2018.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. CoRR, abs/2005.01644 2020.
Wenhao Li. Efficient planning with latent diffusion. In The Twelfth International Conference on Learning Representations, (ICLR), 2024.
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, (ICML), volume 202, pp. 20035- 20064, 2023.
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. In International Conference on Machine Learning, (ICML), volume 202, pp. 20725-207444 2023.
David Lindner, Andreas Krause, and Giorgia Ramponi. Active exploration for inverse reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 55:5843-5853 2022.
Guiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezaee, and Pascal Poupart. Benchmarking con- straint inference inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2023.
Guiliang Liu, Sheng Xu, Shicheng Liu, Ashish Gaurav, Sriram Ganapathi Subramanian, and Pascal Poupart. A comprehensive survey on inverse constrained reinforcement learning: Definitions, progress and challenges. arXiv preprint 2rXiv:2409.07569 2024.
Shicheng Liu and Minghui Zhu. Distributed inverse constrained reinforcement learning for multi- agent systems. In Neural Information Processing Systems (NeurIPS), 2022.
Shicheng Liu and Minghui Zhu. Meta inverse constrained reinforcement learning: Convergence
guarantee and generalization analysis. International Conference on Learning Representations (ICRL), 2024.
Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints in model-free reinforce- ment learning: A survey, In International Joint Conference on Artificial Intelligence (IJCAI), pp- 4508 400884515, 2021.
Yecheng Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Versatile offline imitation from observations and examples via regularized state-occupancy matching. In International Confer- ence on Machine Learning (ICML), pp. 14639-14660 2022.
Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International Conference on Machine Learning (ICML), pp. 7390-7399, 2021.
David L McPherson, Kaylene C Stocking, and S Shankar Sastry. Maximum likelihood constraint inference from stochastic demonstrations. In 2021 IEEE Conference on Control Technology and Applications (CCTA), pp.
IEEE, 2021.
Alberto Maria Metelli, Giorgia Ramponi, Alessandro Concetti, and Marcello Restelli. Provably effi- cient learning of transferable rewards. In International Conference on Machine Learning (ICML), pp. 7665-7676. PMLR, 2021.
Alberto Maria Metelli, Filippo Lazzati, and Marcello Restelli. Towards theoretical understanding of inverse reinforcement learning. In International Conference on Machine Learning (ICML), volume 202, pp. 24555-24599 PMLR, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):599-33. 2015.
Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In International Conference Machine Learning (ICML), pp. 663-670, 2000.
12

---

Published as a conference paper at ICLR 2025
Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadif- fuser: Diffusion model as conditional planner for offline meta-rl. In International Conference on Machine Learning. (ICML), volume 202. pp. 26087-26105, 2023.
Dachyung Park, Michael Noseworthy, Rohan Paul, Subhro Roy, and Nicholas Roy. Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning. In Conference on robot learning, pp. 1005-1014. PMLR, 2020.
Santiago Paternain, Luiz F. O. Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. In Advances in Neural Information Processing Sys- tems (NeurIPS), pp. 7553-7563, 2019.
Claudia Pérez-D' Arpino and Julie A Shah. C-learn: Learning geometric constraints from demon- strations for multi-step manipulation in shared autonomy. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 4058-4065. IEEE, 2017.
Guanren Qiao, Guiliang Liu, Pascal Poupart, and ZhiQiang Xu. Multi-modal inverse constrained reinforcement learning from a mixture of demonstrations. In Neural Information Processing Systems (NeurIPS), 2023.
Guorui Quan, Zhiqiang Xu, and Guiliang Liu. Learning constraints from offline demonstrations via superior distribution correction estimation. In International Conference on Machine Learning, ICML, 2024.
Dexter Scobee and S. Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2020.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play Science, 362(6419): 1140- 1144, 2018.
Jascha Wohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamies. In International Conference Machine Learning, (ICML), volume 37, pp. 2256-2265, 2015.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5026-5033, 2012.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1 003748, 2018.
Oriol Vinyals, Igor Babuschkin, Wojciech Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(77722:350-3355 2019.
Xiao Wang, Hanna Krasowski, and Matthias Althoff. Commonroad-rl: A configurable reinforce- ment learning environment for motion planning of autonomous vehicles. 2021 IEEE Interna- fional Intelligent Transportation Systems Conference (ITSC), pp. 466-472. IEEE, 2021.
Wei Xiao, Soun-Hsuan Wang, Chuang Gan, and Daniela Rus. Safediffuser: Safe planning with diffusion probabilistic models. CoRR, ahs/2306.00148, 2023.
Sheng Xu and Guiliang Liu. Uncertainty-awen aware constraint inference in inverse constrained rein- forcement learning. In The Twelfth International Conference on Learning Representations, 2023.
Sheng Xu and Guiliang Liu. Robust inverse constrained reinforcement learning under model mis- specification. In International Conference Machine Learning, ICML, 2024.
Bo Yue, Jian Li, and Guiliang Liu. Provably efficient exploration in inverse constrained reinforce- ment learning. arXiv preprint arXiv:2409. 2024.
13

---

Published as a conference paper at ICLR 2025
Bo Yue, Shufan Wang, Ashish Gaurav, Jian Li, Pascal Poupart, and Guiliang Liu. Understanding constraint inference in safety-critical inverse reinforcement learning. The Thirteenth Interna- tional Conference on Learning Representations, 2025.
Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang. Diffusion models for reinforcement learning: A survey. CoRR, 0bs/2311.01222 2023.
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, pp. 1433-1436 2008.
14

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION AND ENVIRONMENTAL DETAILS
A.1 POINTMAZE ENVIRONMENTS
We designed three PointMaze environments: one with grid of 5 x 5 cells and two others with 7 grids. Each cell spans an area of 0.5 m x 0.5 m. The center the grid is positioned at the origin (0,0). Constraints applied specific cells marked "1" the corresponding maze map. The outer walls, also marked as "1", , are impassable, while the inner regions allow the agent to move freely.
The agent is modeled as a 2-DoF ball actuated by forces in the Cartesian x- and y-directions. The agent's s objective is to navigate through the maze and reach designated target. The goal is consid- ered reached when the Euclidean distance between the ball and the target is less than 0.45 m. The reward system assigns a value of when the agent reaches the goal, while all other cells have re- ward of O. Similarly, entering a constrained location incurs cost of 1. The environment terminates after a maximum of 150, 500, or 250 time steps, depending on the specific maze.
The state space is continuous and comprises four dimensions: the agent's and coordinates, as well as the linear velocity in both directions. The action space also continuous, consisting of accelerations in the z- and y-axes.
A.2 MuJoCo ENVIRONMENTS
Our simulated environments are constructed using Mujoco. We derived state-action function c(s, a) from state-dependent function c(s') with e(s, a) (ys=P(e(ss)))((s))) [c(s)]] Below are more details regarding the environments used:
1. HalfCheetah (Obstacle): These environments are adapted from Liu et al. 2023). In this setup, the agent controls robot that moves faster backward than forward. The rewards are based on the distance covered between consecutive time steps, along with penalties tied to the magnitude of the actions. Additionally, a constraint restricts movement to areas where the X-coordinate is greater than -3, forcing the robot move forward only.
2. Ant (Blocked): these environments, the agent manages a robot that moves forward and gains rewards based the distance traveled. However, constraint limits the robot's leg angular velocity to prevent excessive force on the ground. The limit is set at 1.
3. Walker (LS)): In these environments, the agent controls robot that moves forward and earns rewards for traveling distances. However, there is a speed limit is set at 1, resulting in reduced rewards compared to an environment without such a constraint.
A.3 OFFLINE DATASETS
Offline Dataset for Navigation Tasks. This offline dataset collects a total number of 3255 trajec- tories consisting of around 1.5 10t state-action pairs in three environments. More precisely, we collect 1638 trajectories for PointMaze- MantMaze-UMMZE. 667 trajectories for PointMaze-2WV 2allss, and 950 trajectories for Po each individual environment, the trajectories can be categorized into three parts: 1) expert trajectories generated by the expert policy trained under the PPO- PPO-Lagrangian algorithm and incorporates stochasticity 0.05, allowing for random actions; 2) constraint-violatinn trajectories created by a policy that accelerates the agent's movement directly toward the terminating location, with stochasticity of 0.1; 3) random trajectories generated by the uniformly random policy. The proportion of the number of pairs in each kind of trajectory is around 5 1.
Offline Dataset for Robot Control Tasks, We use the public offline dataset provided by (Quan et al., 2024). Specifically, this offline dataset includes total number of 250 trajectories, which obtains 200 suhoptimal trajectories (each with 1000 steps) and 50 expert trajectories from a PPO- lag algorithm.
A.4 MODEL ARCHITECTURES
We construct our generative diffusson verifier and reward value model following the official im- plementation of (Janner et al. 2022), both with U-Net based architecture. We also design
15

---

Published as a conference paper at ICLR 2025
cost value model similar to the reward value model with U-Net based architecture but differs in outputting horizon H-length feature takes value in [0, representing the feasibility Φ (St, at) for each state- action pair in the H H-length trajectory. And the cost value is explicitly calculated by
Ve -'''sss ( Σ 7 log (s) a). Note that diffusion time i is explicit input 1=0
into the network and embedded by an MLP to perceive the denoising process. Thus the cost value model accurately predict the cost value of noisy trajectory and it guide the generation. Environment-dependen model hyperparameters are presented in beloo..
A.5 EVALUATION DETAILS
The evaluation MuloCo conducted over 10 runs using 5 random seeds, while the evaluation in the PointMaze environments is carried out over 100 runs with random seeds.
Table 3: List of the utilized hyperparameters in the navigation tasks in PointMaze and MuJoCo environments.
PrerMMoe- LMAZZ
PhenAbor JWALLS
Paeethoos-1
Liminn-WWlllrr
Blicked-Be Ani
Commonseadd Velncty-C40
Disamann Fucker
0.9
0.99
0.50
0.99
0.99
11:9
Enoccit Cellectd
Poiie
Intial Laguange Maliptic
10
160
Lagaue Maliplen Leansie Rate
DAL
001
000
81
0.1
01
011
Model Hartooes
13
13
33
Grd Medel Luariig Ros
led
164
164
164
161
164
Coe Model Upter Strp
Defecsn Time Featers Timonoirs
Dittass Thme Thiken Elmrenss
138
138
138
138
Hileen Kanal
N.N U.No
Concoltion Layen Disccnies
(xxxxxx30)
(xx26
(1x26xxx2)
16

---

Published as a conference paper at ICLR 2025
B ADDTIONAL RESULTS
B.1 ADDITIONAL EXPLORATORY RESULTS OF THE CONSTRAINTS IN POINTMAZE ENVIRONMENT
The following 2 figures illustrate the output cost value of the cost models trained under different exploratory regularization parameters 8 for PointMaze-L. and PointMaze-2 2-Walls environments re- spectively.
Random Noisa 3 PaintMaze 2WALLS bad traj
Random Init PointMaze 2WALL5 bod tra)
EXHICL Pointfaxs 2WALLS bad 2R
70
50
CocCVauu
10-
Modell
Index
PointMaae expert_trs
Ink export
PotrtMaze 2WALL5 expert
Phan
Coo.
Coal
60
index
Figure 7: The delta varying exploratory results for both bad trajectories (top) and expert trajectories (bottom) of PointMaze-2WALA. environment. Each data point corresponds to the cumulative costs for trajectory. Three exploration strategies are presented: random noise (left), random initialization (middle) and Ex-ICL (right).
Rendom Noise 3.001 PointMaaeei bad ins)
Random PrintMueed bad, (nn)
Ex-it niMaark bed Lng
60
Cost
Ivan Corl
Cost
Colf
calea
calce
2303
Randtom Nelse 0.501 More-t
Random In Poi-MMaree expert
Fx4CL PointWWeee
Vean Cost
Mean Cot
Moan
Coit
Modil Index:
Modwi Index
2200
Figure 8: The delta varying exploratory results for both bad trajectories (top) and expert trajectories (bottom) of PointMaze-L environment. Each data point corresponds to the cumulative costs for trajectory. Three exploration strategies are presented: random noise (left), random initialization (middle), and Ex-ICL (right)
17

---

Published as a conference paper at ICLR 2025
We also illustrate the cost value- exploeetxlooraiinnnoo round figures below, in which the cost model is trained under the largest to demonstrate the effectiveness of our exploration method:
Ex- PolndMaze- UMAZE M bed_tral
ExHCL 2WALL boo_1n)
Ex- XL PaindMaze--- ted_tral
140
120
100
Phant
Cosl
Exp O1 Round
Exp Exploratien on Round
Exploration Round
expert_trg traj
PaintMaze 2WALLS caport.tra
LM export,
ContVste
20-
20
20
Figure 9: The exploration round varying exploratory results for both bad trajectories (top) and expert trajectories (bottom) of PointMaze-UMAZI PointMaze-2WALL and PointMaze-L environments. Each data point corresponds the cumulative costs for trajectory. Three exploration strategies are presented: random noise (left), random initialization (middle) and Ex-ICL (right).
As we expected, the cost values of the same trajectory gradually vary from the original ones, as the InfoNCE term in encourages them to be distinct.
B.2 NUMERICAL ANALYSIS FOR THE DIVERSITY OF THE CONSTRAINT IN MuJoCo ENVIRONMENT
For numerical analysis, we use five different constraint models during exploration to esti- mate the cost values of identical trajectory seg- ments in each environment and report the mean and standard deviation results across them. The results, shown in Table 4 indicate that: 1) con- straint models effectively distinguish between safe and violated segments, assigning higher cost values to violated trajectories and lower values to safe ones; and 2) the std results across the five constraint models suggest that the exploration process successfully induces variability among the constraint models.
Table 4: Costs values across varying exploration epochs. The reported values are the mean + std over 5 different constraints.
Environment
HalfCheetah
Walker
Ant
Violated Segments Safe Segments
49.21::.7
23.942..
36.6:1.4 10.1+2.1
6.1+10.7
10.9++0.8
B.3 ADDITIONAL PERFORMANCE EXPERIMENTS IN COMMONROAD ENVIRONMENT WITH HIGHD DATASET
To demonstrate the effectiveness of our method in a more realistic environment, we conducted exper- iments on performance CommonRoad-P RL (Wang et al. 2021) Environment with a velocity <00 constraints. We chose the processed HighD Krajewski et al. 2018 data given by (Liu et al. 2023), aligning with baseline method (Quan et al. 2024). Our methods outperformed the baseline methods in terms of trajectory cumulative cost>0 rate while achieving comparable performance cumu- lative reward and AÚC metric. Such a result suggested that our method maintains effectiveness in complicated realistic environments and broadens its future applications.
18

---

Published as a conference paper at ICLR 2025
Table 5: Performance measured by reward, cost, and Area Under Curve (AUC) of different methods. The reported value of reward and cost are the mean + std over 500 runs under random seeds
Methods
BC
OptiDice-c
ICSDICE
EX-ICL(ours)
Expert
reward
-2.3+22.3
-1.6+3.77
10.7=11.5
10.5+2.2
14.0
cost 1.4%
12%
6.5%
3.2%
0.8%
AUC
NA
0.76
0.81100011
0.84100.06
NA
C PERFORMANCE OF DEMONSTRATIONS, BASELINE, AND OUR METHOD
To support our claim that our method achieved expert-level performance in all MuJoCo locomotion environments and thus the large or small performance gaps between our method and baseline (Quan al. 2024) are determined by the gap between expert demonstration and baseline performance, we reported the cumulative feasible reward and the cumulative cost of expert demonstration, suboptimal demonstration, our method, and baseline (Quan al. 2024) in the below Table 6
Table 6:
MuloCo demonstration and policy cumulative feasible reward and cumulative cost. Bold denotes expert demonstration comparable performances.
Obstaele-HaifChetthhh
Limited- Walker
Blocked-Aut
Methods
Reward w/o Cost
Cost
Reward w/o Cost
Cost
Reward w/o Cost 1
Cost
Expert
4915 1170
0.00 + 0.00
1870 12
1.64 + 1.67
3059 276
7.22 2.74
Suboptinal
3784 1: 1455
487 1: 482
1003 1. 757
309 243
732 595
233 1:313
ICSDICE
2315 1. 740
0.04 1 0,04
1587 1. 308
0.01 1 0.02
3073 103
0.01 0.00
ExICL (ours)
5298 480
0.00 0.00
1862 + 29
0.00 0.00
3061
0.01
D RELATED WORKS
In this section, we introduce the previous works that are most related to our approach.
Inverse Constraint Learning. Inverse Constraint Learning (ICL) aims recovering constraints encoded in demonstrations to autonomously define and reuse constraints. The ICL problem is inher- ently ill-posed since there can be multiple combinations of reward and constraint pairs that explain the optimality of expert behaviors. To address this ambiguity, conventional ICL methods assumed
the constraints are generated from certain constraint templates(Chou et al. 2021 Park al. 2020) Pérez-D' Arpino & Shah 2017). while recent approaches leveraged neural networks to represent constraints over discrete state-action spaces (Scobee Sastry 2020) and continuous state-action
spaces (Malik et al.] 2021 Liu et al. (023). Continuous works explore ICL under the environment with soft constraint (Garg et al. [021] multiple agents (Liu & Zhul 2022, 2024 Qiao et al. 2023)
stochastic transition dynamics (McPherson al.) 2021 Xu & Liu (023), multiple tasks (Kim et al.
2023) and robust optimization framework (Xu & L.u.. 2024). A recent study (Quan al. 2024)
has explored the method of ICL from offline dataset. However, these methods mainly focus on identifying a specific constraint without exploring the approach to infer a diverse set of constraints.
Diffusion Planner for RL. Due to the impressive generative capabilities, some recent studies have explored using diffusion models as planners to simulate environmental dynamics and solve RL prob- lems in a model-based based manner (Zhu et al. 2023). Among these studies, Diffuser (Janner et al.) 2022] guided the denoising process in diffusion models using probabilistic representation of an RL policy. To further enhance model performance, subsequent research has expanded this cept. AdaptDiffuser (Liang et al. (202)) incorporated evolutionary planning, while EDGI (Brehmer al. (202)) focused on planning by leveraging the geometric structure in the task. Additionally, studies such as MTDiff (He et al. (023) explored multi-task planning, TCD (Hu et al., (023) uti- lized temporal information for guiding the planning, LatentDiffuser (Li 2024) examined planning in latent action spaces, (Ni et al., (223) studied the diffusion planning under meta RL setting and HDMI (Li et al. 2023) developed hierarchical decision framework for diffusion policy. In the realm of sarety-critiaal applications, while recent studies such as SafeDiffuser (Xiao et al.) (023)) and ( et al. 2024) have proposed safe planning using diffusion models under given constraints, the extension of this method to the inverse RL problem remains unexplored.
19


Reviews:
[{"review_id": "MGRdkUH8et", "reviewer": "Reviewer_X5Jn", "summary": "This paper proposes ExICL to tackle Inverse Constraint Learning problem, which aims to recover a diverse set of feasible constraints through an exploratory constraint update mechanism. The designed generative diffusion verifier utilizes the guided sampling strategy to verify the feasibility of explored constraints. This paper also aims to guarantee the robustness of feasible constraints discovery by accurately estimating the cost of noisy trajectory.", "questions": "1. My biggest confusion is about how the reward and cost are defined, respectively. Usually reward is defined as the negative cost if cost is positive, but in this paper, it seems not. Can you explicitly show how they are defined and how different they are?\n2. In section 4.2, on line 286, how is $\\phi_\\omega(s_t^i, a_t^i, i)$ defined? \n3. In section 4.3, can you explicitly give the expressions for dist$[1, \\phi_\\omega(s_t, a_t)$ and dist$[\\tilde\\phi_\\omega(s_t, a_t), \\phi_\\omega(s_t, a_t)])$?\n4. In algorithm 1, ``Updating $\\lambda$ by minimizing the loss $\\mathcal{L} = \\lambda \\mathbb{E}_{\\hat\\tau\\sim \\tilde{p}_M}[c(\\tau) - \\epsilon]$, why is no reward term involved here to update $\\lambda$? Another question related to this in Table 2: there is a significant discrepancy between the magnitudes of the Reward and Cost. Could you provide some insight into this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Introduction clearly states the current issues in Inverse Constraint Learning and the related works section is complete.\n2. The experiments are comprehensive, demonstrating the effectiveness of the proposed approach.", "weaknesses": "1. The contributions claimed in this paper are not apparent to me. Contents in 4.1 is quite close to what has been proposed in [1], and the non-convex objective theorem is inherited from [2]; the ambiguity of how things are defined in section 4.2, 4.3 impairs the significance of contributions again. There are many math notations are not defined or briefly mention. I will list each of them below in the question section. I found it confusing and hard to see how the idea works.\n2. Again, theorem 4.1 seems related to some existing conclusion from Paternain's paper [2], and this theorem is critical as it supports the zero duality gap for non-convex objective. The theorem stated in this paper is not quite the same as what is shown in [2], as the constraints here are not constant but are functions, but constants in [2]. There is supposed to be a connection shown here to support the theorem or a direct proof. A typo follows the theorem in Equation (9): $\\lambda\\epsilon$ might be missing at the end in the exponential term.\n\n[1]  Janner, Michael, et al. \"Planning with Diffusion for Flexible Behavior Synthesis.\" International Conference on Machine Learning. PMLR, 2022.\n[2] Paternain, Santiago, et al. \"Constrained reinforcement learning has zero duality gap.\" Advances in Neural Information Processing Systems 32 (2019).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WJXfv755YI", "reviewer": "Reviewer_zyAA", "summary": "The paper tackles the safe reinforcement learning problem using a diffusion model and guidance to train a set of feasibility functions. Unlike traditional inverse constraint learning, which is difficult to verify whether a candidate constraint is feasible and returns a single constraint, the paper's algorithm rapidly recovers a diverse set of constraints once the diffusion model is trained on expert data. The paper's algorithm outperforms baselines on constrained mazes and Mujoco experiments regarding performance and sample efficiency.", "questions": "- How are you selecting the constraint out of the constraint pool discovered by Ex-ICL for the experiment section?\n- Why does Figure 4's Ex-ICL figure have so much larger variance for bad trajectory cost value than other methods?\n- How sensitive are the results to exploration coefficient \\delta and exploration round m? Also, would it be instructive to showcase model performance for Ex-ICL that only searches over a single \\delta?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea of amortizing the ICL loop cost by pre-training a diffusion model is interesting.\n- The paper provides convincing empirical results that show the superiority of their method compared to the baselines of their experiments, both for reward and cost. It also investigates how reliable feasibility functions are on expert non and non-expert data.\n- While they have not directly demonstrated the advantages of having multiple constraint candidates returned by the algorithm (aside from possibly making search more efficient), this seems like a practical feature to have for real world use cases.", "weaknesses": "- While the authors list computational concerns as one of the advantages Ex-ICL has over ICL, they do not conclusively show Ex-ICL's computational advantage. Figure 6 shows that Ex-ICL is more sample efficient in constraint inference, but a true test of computational efficiency should also take into account diffusion model training time.\n- The experiments on maze and Mujoco are comprehensive but are fairly simple. For example, the baseline paper [1] includes a more realistic experiment on traffic scenarios.\n- There's not enough detail in the main paper or appendix on methodology (how is \\phi parameterized?)\n\n[1] Guorui Quan, Zhiqiang Xu, & Guiliang Liu (2024). Learning Constraints from Offline Demonstrations via Superior Distribution Correction Estimation. In Forty-first International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KjdF1E8va6", "reviewer": "Reviewer_56R9", "summary": "The authors consider inverse constraint learning, and improve on previous work by constructing an algorithm that can generate a set of constraints, and verify those constraints by applying techniques developed in diffusion modelling for RL. In particular, the authors construct a guidance term that is the gradient of a set of feasiblity terms, which they can use for on the fly verification of the proposed feasilibity functions, thereby eliminating a costly second optimization loop. The authors test their proposed algorithm on a variety of RL benchmarks.", "questions": "See also questions under \"weaknesses\"\n- Do the authors have some intuition why their method seems to outperform baselines significantly for HalfCheetah, marginally for Limited-Walker and only ties for Blocked-Ant?\n- In the MuJoCo experiments, is the reward presented in Table 2 the feasible reward? I.e. are rewards truncated after a constraint has been violated? It seems that that would be the more inveresting metric to report, I would recommend the authors report that metric, and if they already do so make it clear it is that metric.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The technique makes clever use of the advanatages found in diffusion techniques: being able to modify the policy at run time by applying guidance terms\n- The paper strikes a good balance of building a new method out of existing elements.", "weaknesses": "The authors seem to omit some details of their mechanism, which I think are quite crucial to the paper. These are:\n- how is reward treated? Is a separate reward model that is (1) differentiable, and (2) conditioned on diffusion time (i in the author's notation) trained following Janner et al? These details are not present in Alg. 1, but are necessary to evaluate the gradient p_Mc in eqns (9) and (10). \n- It is also not made clear whether in (9) and (10) the feasibility functions and reward are made to condition on diffusion time i, as I would expect it should since only tau_i is available at i.\n- After algorithm 1 is complete, how is the final policy constructed for the experiments? Perhaps this is as simple as running eqn. (9) and (10) a final time, but this is not specified either. \n- After algorithm 1 completes, how are constraints chosen by the practitioner as the abstract says? How do the authors choose what constraints they apply when sampling their final evaluations? This is stated in the abstract but is not discussed in the paper at all. \n- How is constrained data collected? Is there an expert that already includes the constraint?\n\nMinor:\n- A few scattered grammar errors could be addressed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)