A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challenges
Authors: Guiliang Liu
Sheng Xu
Shicheng Liu
Ashish Gaurav
Sriram Ganapathi Subramanian
Pascal Poupart
Venue: Accepted by TMLR ()

Abstract: Inverse Constrained Reinforcement Learning (ICRL) is the task of inferring the implicit constraints that expert agents adhere to, based on their demonstration data. As an emerging research topic, ICRL has received considerable attention in recent years. This article presents a categorical survey of the latest advances in ICRL. It serves as a comprehensive reference for machine learning researchers and practitioners, as well as starters seeking to comprehend the definitions, advancements, and important challenges in ICRL. We begin by formally defining the problem and outlining the algorithmic framework that facilitates constraint inference across various scenarios. These include deterministic or stochastic environments, environments with limited demonstrations, and multiple agents. For each context, we illustrate the critical challenges and introduce a series of fundamental methods to tackle these issues. This survey encompasses discrete, virtual, and realistic environments for evaluating ICRL agents. We also delve into the most pertinent applications of ICRL, such as autonomous driving, robot control, and sports analytics. To stimulate continuing research, we conclude the survey with a discussion of key unresolved questions in ICRL that can effectively foster a bridge between theoretical understanding and practical industrial applications. The papers referenced in this survey can be found at https://github.com/Jasonxu1225/Awesome-Constraint-Inference-in-RL.

Published in Transactions on Machine Learning Research (01/2025)
A Comprehensive Survey on Inverse Constrained Reinforce- ment Learning: Definitions, Progress and Challenges
Guiliang Liu
Crrrudtiong@cuuu.ccc..c
Sheng Xu-
shengrul @link. wdw.
Shicheng Liu?
s115539@psu.cdl edu
Ashish Gauray2.4
sshiis..gourrv@uuuteeii..o
Sriram Ganapathi Subramanian
$2ganapa@xxaaertoo.cc
Pascal Poupart
ppoupart@wwaterloo.cm
School of Data Science, The Chinese University of Hong Kong, Shenzhen, Cheriton School of Computer Science, University of Waterloo, Pennsylvania State University,
Veelor Institute
Reviewed on OpenReview: https: openreview. net/ forun? id= WUQaBi.JqyP
Abstract
Inverse Constrained Reinforcement Learning (ICRL) is the task of inferring the implicit constraints that expert agents adhere to, based on their demonstration data. As an emerg- ing research topic, ICRL has received considerable attention in recent years. This article presents a categorical survey of the latest advances ICRL. It serves as a comprehensive reference for machine learning researchers and practitioners, as well as starters seeking to comprehend the definitions, advancements, and important challenges in ICRL. We begin by formally defining the problem and outlining the algorithmic framework that facilitates constraint inference across various scenarios. These include deterministic stochastic en- vironments, environments with limited demonstrations, and multiple agents. For each con- text, we illustrate the critical challenges and introduce a series of fundamental methods to tackle these issucs. This survey encompasses discrete, virtual, and rcalistic environ- ments for evaluating ICRL agents. We also delve into the most pertinent applications of ICRL,, such as autonomous driving, robot control, and sports analytics. To stimulate continuing research, we conclude the survey with discussion of key unresolved ques- tions in ICRL that effectively foster bridge between theoretical understanding and practical industrial applications. The papers referenced in this survey can be found at https:///ithuu:
Inference-in-Hi
1 Introduction
To ensure the reliability of Reinforcement Learning (RL) algorithm within safety-critical applications, it is crucial for the agent to have knowledge of the underlying constraints. However, in many real-world tasks, the constraints are often unknown and difficult to specify mathematically, particularly when these constraints are time-varying, context-dependent, and inherent to the expert's own experience. For example, Figure [ shows contemporary example of a highway merging task, where the ideal constraints depend on the traffic road conditions well the weather.

---

Published in Transactions on Machine Learning Research (01/2025)
Weather:
Speed: Low
Traffic: Heavy
Weather:
Speed: Fast
Traffic: Light
Car
Car
Distance 1m
Distance 70m
Car
8
Distance
Distance 90m
Ego
Ego Car
Speed 60 m/s
Speed 90 m/s
Figure 1: An example of the context-sensitive distance constraint between vehicles during merge the highway. Under proper weather conditions, when vehicle speed is relatively low and traffic congestion is high, the distance between cars can be reduced. However, in adverse weather conditions, when vehicles are moving fast and traffic sparse, it becomes necessary to increase the distance between cars to ensure safety.
An effective approach to resolve the above challenges is Inverse Constrained Reinforcement Learning (ICRL), which infers the implicit constraints obeyed by expert agents, utilizing experience collected from both the environment and the observed demonstration dataset. These constraints, learned through a data-driven approach, can effectively generalize across multiple environments, thereby providing more comprehensive explanation of the expert agents' behavior and facilitating safety control in downstream applications.
To infer the underlying constraints, ICRL alternates between updating an imitating policy with Constrained Reinforcement Learning (CRL) and learning constraint function via Inverse Constraint Inference (ICT), until the imitation policy can reproduce expert demonstrations. Figure shows an illustrative example of the learning process of ICRL in a grid-world environment where the agent seeks the shortest path from the start state to the goal state, In the absence of any constraint, the agent optimizes a policy that initially returns a direct path from the start state to the goal state (Figure Round 1). However, since this path does not imitate the expert path, the agent infers that the orange region must be infeasible, given that it is not visited by the expert demonstration. In Round the agent optimizes revised policy subject to this constraint that produces a new path. Once again, the agent infers that the orange region must be infcasible since it is not visited by the expert demonstration. The process keeps on alternating between constrained policy optimization and constraint inference until the resulting policy imitates expert. demonsttrtions.
Round
Round
Round
Round
Goal
Goal
Goal
Goal
Figure 2: A running example of ICRL, which alternates between policy updates and constraint inference each round. The expert policy and the imitation policy are represented by the black and blue curves, respec- tively. The newly inferred constrained region in each round highlighted in orange, while the constrained region inferred in previous rounds depicted in gray.
ICRL, evolving research area, has received significant attention in recent years. This article offers comprehensive introduction to ICRL including recent advancements.
1.1 The Significance of this Survey
The survey makes the following contributions.
2

---

Published in Transactions on Machine Learning Research (01/2025)
Table I: A structured summary of the key ICRL algorithms that are reviewed in this article.
Method&
State/Action
Environment
Dataset-
Agent-
Reference
Framework
Space
Dynamics
Coverage
Numbers
Scobee & Sastry
2020
Maximum Entropy
Discrete
Deterministic
Full
Single
Malik el al.
2021
Maximum Entropy
Continuous
Deterministic:
Full
Single
Mc Pherson
2021
Maximum Causal Entropy
Discrete
Stochastic
Full
Siugle
Baert al.
2023
Maximum Causal Entropy
Continuous
Stochastic
Full
Single
Gauray
2023
Deep Constraint Correction
Continuous
Stochastic
Full
Single
Xu & Liu 2024b
Robust Optimization
Continuous
Stochastic
Full
Single
Papadimitrion et al.
2023
Bayesian Framework
Discrete
Stochastic
Partial
Single
Liu et al. 2028
Varistional Inference
Continuous
Stochastic
Partial
Siugle
Subramanian al.
2024
Coufidence Estimation
Continuous
Stochastic
Partial
Single
Xu Lau 2024a
Generative Model
Continuous
Stochastic
Partial
Single
Quan al.
2024
Distribution Correction Estimation
Contimuous
Stochastic
Partial
Single
Papadimitrion &r Brown
2024
Preference-Basee Estimation
Continuous
Stochastic
Partial
Single
Park ct al.
2019
Bayesian Framework
Discrele
Stochastic
Full
Single
Jang al.]
2023
Reward Decomposition
Continuous
Stochastic
Full
Siugle
Lindner
2024
Linear Programming
Continuous
Deterministic
Full
Multiple
Kim ct al.
2023
Multi- Masti IRL
Continuous
Deterministie
Full Multiple
Qino ct al.
2023
Multi-Modality RL
Continuous
Deterministic
Full
Multiple
[Jhu & Zhu
2022
2023a
Maximum Likelihood
Contimous
Stochastic
Full
Multiple
In-Depth Guide to ICRL. More specifically, we introduce the forward and backward procedures of ICRL
within the framework of a Constrained Markov Decision Process (CMDP). To substantiate the rationale behind ICRL, we highlight fundamental diffcrences in comparison to Inverse Reinforcement Learning (IRL) and Inverse Optimal Control (IOC).
Overview of Recent Advancements. To illustrate the recent developments in a structured format, Ta- ble [ presents the existing ICRL methodologies under various learning contexts and environments. Specifi- cally: 1) In deterministic environments, we present the Maximum Entropy (MEnt) ICRL approach, which investigates both discrete and continuous domains. 2) In stochastic environments, our focus shifts to model the Maximum Causal Entropy (MCEnt) and soft constraints. 3) To manage demonstration datasets that can only encapsulate partial knowledge of the enviromment, we provide an overview of the distribution-basell data-augmented, and offline ICRL techmiques. 4) We introduce approaches to learn constraints and rewards simultancously. 5) This survey explores ICRL the context multi-agent scttings, dotailing how infer shared or individual constraints based on different lypes of agents involved.
Motivation for Future Research. To pave the way for future research, our survey encompasses evaluation benchmarks under discrete, virtual, and realistic environments. Additionally, we discuss the practical usage of ICRL by introducing its potential applicalions ions in diverse fields such as autonomous driving, robol control, and sports analytics. Finally, we outline some open problems and challenges of ICRL, hereby highlighting the areas where future research can contribute significantly and bridge existing gaps.
1.2 Organization of Contents
As ICRL is an emerging field, this article is intended to serve as a comprehensive guide for readers keen to learn about it. The structure this article is outlined as follows: 1) In Section 2 we provide a thorough introduction to the foundations and termènologies of ICRL. This includes the mathematical definitions of RL, Constrained RL, and ICRL, well as the methods employed to regularize the learned constraints. 2) Section
3 delves into the research topics related to ICRL, such as Inverse Reinforcement Learning (IRL) and Inverse Optimal Control (1OC). We highlight their fundamental differences from ICRL, thereby underscoring the unique value of ICRL. 3) Scction ! and Section 5 explore recent advancements in ICRL. This includes discussions about constrained inference algorit derived from determinislic and stochastic enviromments. Section and Section [ describe recent progress in ICRL from partial demonstration data and multiple agents in the environment. 4) Section 7 investigates approaches simultaneously infer constraints and rewards. 5) Section g presents the benchmarks and real-worle applications of ICRL. 6) Section 10 concludes this survey and outlines important challenges and open questions for future ICRL research.
3

---

Published in Transactions on Machine Learning Research (01/2025)
2 Background and Notation
In this section, establish notation and provide the essential background for a better understanding of the remainder of the survey. A synopsis of the primary notation and abbreviations is available in Table 2
Table 2: The main notation and abbreviations throughout the paper.
Symbols and abbreviations
Meaning
M.
Markov Decision Process (MDP) and Constrained MDP. State, action, and reward in an MDP.
8, 0, r dss.),, 7, Mo
Cost, and the corresponding threshold in constraint. Discount factor and the initial state distribution (i.e., po)- Expert demonstrations dataset.
DE
EE, R, IT
Expert policy, nominal (i.e., imitation) policy, and the set of policies. Expert trajectory, nominal trajectory, and the set of trajectories. Occupancy measure in accordance with policy π. The feasibility of performing actions under given state. Labels for the agent identity.
TE f, E p
(s,aa)
k
w, 0, &
Model parameters for the constraint, policy and density models. State-action and state-only value functions. Trajectory entropy and causal entropy.
(ss, a), ((s)
H()(T)), H(aorTl (0.T)
MEnt
Maximum Entropy.
MCEnt
Maximum Causal Entropy.
CRL
Constrained Reinforcement Learning.
ICI
Inverse Constraint Inference.
IRL
Inverse Reinforcement Learning.
ICRL
Inverse Constrained Reinforcement Learning.
Specifically, we use lower-case letters (e.g., s) to denote scalars, bold lower-case letters denote vectors (e.g., s) and bold upper-case letters (e.g., S) to denote matrices. In the rest of this section, introduce key definitions for the task of Reinforcement Learning (Section 2.1 Constrained Reinforcement Learning (Sec- tion 2.2 and Inverse Constrained Reinforcement Learning (Section 2.3 as well as constraint regularization (Section 2.4 methods.
2.1 Reinforcement Learning
Reinforcement, learning (RL) algorithms are generally based on episodic Markov Decision Process (MDP) M (utton & Barto 2018), which can be defined by tuple (S.A, PR.PT, T. T.)o) where: 1) S and A denote the space of states and actions. 2) pT (8 (8'18, a) and (r|s, a) define the transition and reward distributions. 3) y E (0,1) the discount factor. 4) T e [0, 2o) defines the planning horizon. 5) #4 = pps) denotes the initial state distribution. In principle, this MDP terminates
time step T, though this planning horizon T is not fixed. For example, a game may 1) terminate when the agent reaches a terminating or goal state, 2) assign a lerminating probability associated with each state.
In an episodic MDP M, the objective is to solve the sequential decision problem to maximize the (discounted) cumulative reward by learning policy [[als). Given that the literature on ICRL commonly adopts the Maximum Entropy (MEnt) framework, our survey aligns with the ICRL setting by focusing on the MEnt RL objective [Haarnoja et al. 2017). The MEnt RL objective can be described as follows:
1 In'r(ss an) + H(T))
Jr arg max E
ERr.PT to.o
(1)
where H(T)) represents the policy entropy weighted by a. Incorporating such an entropy regularizer of- fers several key advantages: 1) It appropriately model the bounded rationality in human behaviors
4

---

Published in Transactions on Machine Learning Research (01/2025)
(see Section 2.4 2) It leads to soft representation of the optimal policy, which can better model the sub-optimal behaviors and is more robust to stochasticity in the environment. 3) Depending on the problem settings, it can represent either trajectory-level entropy H(T(T)) = -Enrr>log [T)] (see Sec-
tions and 4.2 or causal entropy (i.c., discounted cumulative step- wise entropy (Bloem & Bambos 2014))
H (0:0: (0:T) Eep /Et= 78 log [a)](s))] (see Section 5.1 and 5.20, accommodating the dynamics in both deterministic and stochastic environments, In the decision process, the agent generates a trajectory T" [So, .y.,...011,, and [[ = p(5a) II= (s7 a,). In this paper, we use EM to denote
the set of trajectories in the MDP M.
2.2 Constrained Reinforcement Learning
Constrained Reinforcement Learning (CRL) typically considers a constrained optimization problem under a Constrained Markov Decision Processes (CMDPs) Me. = PR. PT, ((pc,;))Jvs 7, 7, which augments the original MDP M by adding constraints. pe, (cls, a) denotes stochastic constraint function D with an associated bound 62z where indicates the index of a constraint, and the costs are positive and bounded, i.e., c(s, a) [0, mmax). In this context, CRL agents consider constrained optimization problem.
Discounted Cumulative Constraints. We consider a CRL problem where the goal is to find policy x that maximizes expected discounted rewards under a set of cumulative soft constraints:
I
arg max Epp T. π
Σ
tr((), at)
+
1 H(T)
(2)
t=0
T
8.1. E Ep
Σ
s'cs (s), an)
Sei ViE
where r(5t, dr) = Epx (ralss, at) and a(s,, at) = Exc, (calsr, at). This formulation is most useful under an infinite decision horizon (T = oo), where the constraints consist of bounds on the expectation of discounted cumulative cost values. Assuming 6 > 0, formula (2) can effectively represent soft constraints since permits visiting some state-action pairs with high cost (, >> and the expectation of the discounted additive cost is smaller than the threshold (e:).
long the chance of getting there is small,
Assuming the Markov chain induced by transition pr (8++1 8t, at) and policies क E II to be irreducible and aperiodic (thus, the visitation frequency follows a unique stationary distribution), the constraint in Equation can be equivalently represented as:
E pi (s.a) [[(s,a)) Sa Vi E [1,7]
(3)
with the normalized occupancy measure:
T -1 Σ (s
(4)
Here P s,a, a) defines the probability of reaching at time step by implementing policy in the MDP M. This constraint representation shows the constraint is linear (and thus convex) with respect to PJ (instead of т). Based on the occupancy measure, the CRL problem in 2 can be represented as:
arg max E ((((())) Vi
(r(s,a)
log log ((ass)
(5)
s.t. E p (s.a) (s) [1,11
where we slightly modify the original objective 2 by transforming the entropy into causal entropy. Although this optimization problem is non-convex, its Lagrange dual has no duality gap (Paternain al., 2019
In the context of deterministic environment, PC, (c)s,a) be simplified as c(s.,) which uniquely determines the based state-action pair.

---

Published in Transactions on Machine Learning Research (01/2025)
Hence, we transform the CRL problem 5 into its dual form:
arg min (x.) [((s.0) 1 a
log [(als) Σ X Bosi(an)) (c(s,a)) -ca
-c]]
(6)
where A denotes the optimal Lagrange multiplier for the ith constraint. For simplicity, let's consider only one constraint, and we have:
arg arg min - E Eppa(e.)) ((())) Xe(ssa) 1
log log m(als)) X*e
a
Trajectory-based Constraints. An alternative approach is to define constraints on trajectories (i.e.. sequences of state-action pairs) denoted by T:
T En'r + H(T))
arg max Ep PT
(8)
1=0
s.t. Or>(pr,)).mg [c(T)] Sa E
Depending on how we define the trajectory cost c(T), the trajectory constraint can be more expressive than the cumulative constraint, for example, Transformers (Vaswani et al. 2017) can be applied to predict the trajectory cost in way that does not decompose into sum of step- wise costs. This trajectory constraint is useful in the episodic MDP with limited (but not necessarily fixed) planning horizon.
Hard versus Soft Constraints. When is set to the minimumn cost among all trajectories (i.e., Cmiin arg min, (T), where previous work commonly sets Gmin = 0 ), it imposes a hard constraint on the problem. This ensures that the agent incurs a minimal cost for all trajectorics. Conversely, when c chosen to be greater than Cmin, signifies soft constraint. This scenario permits the agent to ineur higher costs than for some trajectories, provided that other trajectories incur costs less than and the expected value of the costs does not exceed e.
2.3 Inverse Constrained Reinforcement Learning
In practice, instead of observing the constraint costs, of- ten have access to expert demonstrations DE = {TR} gener- ated by the expert (with policy Tg) that satisfy the under- lying constraints. The class of methods that aim to learn a policy from expert demonstrations is named apprenticcship learning (Abbeel Ng 2004). Among these methods, behav- ior cloning follows a supervised learning approach where the learner directly maps states to actions by observing the ex- pert's demonstrations. However, this approach is sensitive to prediction errors. Divergence from the expert trajectory could lead to a cascade of errors in a sequential decision-making prob- lem. Furthermore, it is particularly challenging to transfer the imitation policy to a new environment with different dynamics from the learning environment. Beyond learning from expert demonstrations, a growing body of rescarch emphasizes lcarning from human interventions (Li al. 2022 Basich et al., 2023 Spencer ee al. 2022). This approach addresses the limitations of demonstration- based learning, which restricted to scenarios where humans can directly operate the system to produce demonstrations.
Forward Step: Polly Upate (T:: 11+4))
Sampling Trajectory:
Constraint
Interations
Function: fe(d+t1)
Ditss
Inverse Step: Constraint Upate
(Φ: $1.1)
Expert Trajectory: De
Figure 3: The flowchart of ICRL.
Striving for robustness and generalizability, in this study, we focus on the preference modcling approach where the agent must first recover the rewards optimized and constraints respected by expert agents, and imitate experts by optimizing the CRL objective under these constraints. This is a challenging task since there might be various equivalent combinations of reward distributions and constraints that can explain the same export demonstrations (Ziebart al. 22008) Striving for identifiability, ICRL algorithms simplify the problem by assuming that. rewards observable, and the goal is to recover only the constraints that best explain the

---

Published in Transactions on Machine Learning Research (01/2025)
expert data (Scobee & Sastry 2020). To better elucidate the objectives of ICRL, discuss the technical differences ween ICRL and other relevant methods, including Inverse Reinforcement Learning (TRL.) Inverse Optimal Control (IOC), and algorithms for simultaneous recovery of both rewards and constraints in Section 3 The inference process of ICRL often involves alternating between updating an imitating policy and updating a constraaint function. Figure summarizes the main procedure ICRL. ICRL solvers involve learning the cost functions from an expert demonstration dataset Dg. This essentially solving tri-level optimization problem (Kim et al. 2023
max max minE (sg r(sg, ag) - xx(ss,, ag) )- log (ggt(alss)) E (( (s,a) a(s Ad(s, a) 1
a
log π ((als)) de
(9)
where the cost function e, the Lagrange parameter and the policy function are subject to optimization at each level.
2.4 Regularizing the Learned Constraints
ICRL is essentially an ill-posed problem since there exist different constraints that can equivalently explain the expert demonstration, and thus the true constraints are not uniquely identifiable, illustrated in Figure 4 This characteristic makes identifying the true underlying constraints difficult task. To mitigate these challenges and enhance the identification of the optimal constraints, an effective method is to add regularization into constraint learning. Popular constraint regularization methods include:
Case
Case 2
Case
goal
goal
goal
Top Row
Inferred
Inferred
Constrained States
Inferred Constrained
Constrained
Centrel
States
Bottom
start
start
start
Figure 4: Examples the ICRL solutions. In these illustrations, the initial location and the final destination are represented by red and blue circles, respectively. The expert demonstrations, signified by dark curves, are directly observable. The three distinct constraints recovered by ICRL algorithms, highlighted as gray regions, provide valid explanations for expert behaviors.
Minimal Constraints. An effective regularization approach involves limiting the complexity of the con- straint. In discrete domain, this can be understood as identifying the constraint set (i.c., the unsafe set) C, which includes only the minimum number of state-action pairs that are critical for explaining the expert behaviors (Scobee & Sastry| 2020). For example, in Figure 4 the central row constraint is most critical since it explains why the agent intentionally avoids certain regions by taking a greater number of steps, conse- quently diminishing the cumulative rewards. In contrast, the constraints depicted in the top and bottom rows are less significant as they do not exert substantial influence on the agents' behavior. In the contin- nous domain, constraining the complexity often implics enhancing the sparsity of the cost in the underlying CMDP Malik al.] 2021). Standard sparsity regularizers (e.g., the E norm) be directly incorporated into the objective of constraint inference.
Bounded Rationality. In decision-making theory, the concept of bounded rationality (Simon 1990) sug- gests that individuals tend to choose satisfactory decisions rather than optimal ones. Regarding the lask of ICRL, we assume that expert agents employ a mixed strategy (i.e., (g(a)) s) 2 0, 0,V(s, a) & C) where the probability of seleeting sub-optimal actions is always greater than zero unless the statc-action pair (s, is infeasible. useful representation this mixed strategy can be attained by optimizing the policy entropy, which be achieved by maximizing either the trajectory-level entropy in deterministic environments (see
7

---

Published in Transactions on Machine Learning Research (01/2025)
Section 4 the causal entropy in stochastic environments (see Section 5 Bounded rationality is referred to indirect regularization because the entropy primarily affects the policy representation, which in turn influences constraint estimation.
Interpretable Constraints. In the pursuit of Explainable AI (XAI), it is essential for the constraints to be interpretable, such as identifying which feature most significantly impacts the cost function. Besides, XAT can necessitate that the cost function possesses physical significance. For instance, Liu al. (2023) defined the cost function as c(s, a) 1 o(s, a) and Qiao et al. (2023) defined the cost function as c(s, a) log d(s, a), where the feasibility function 6(s, a) represents the probability that executing action a in state is safe, Besides, XAI tochniques can be incorporated into the training or design of the cost function to understand the reasons behind the safety of particular action or state.
3 Related Topics
In this section, we cover the research topics that are relevant yet distinet from ICRL. While studying these topics is not our primary focus, we provide an overview of their key definitions and algorithms, and most importantly, highlight the crucial differences with ICRL in order to aid readers gaining a more comprehensive understanding of our study.
3.1 Inverse Reinforcement Learning
Inverse Reinforcement Learning (IRL) is the method for deducing the reward function within an MDP based on observed expert demonstrations. Multiple algorithmic frameworks have been developed for addressing the IRL problem. These include methods like maximun marginal (Ng & Russell 2000), Maximum Entropy (MEnt) Ziebart et al. 2008), Bayesian inference (Ramachandran & Amir 2007 and adversarial learning techniques Fuu et al. 2017).
To provide a concrete example, MEnt IRL, which is among the most extensively studied IRL algorithms, formulates the objective function as a two-player max-mi min game (Garg et al. 2021
max min L(T, = Epe (r(s, a)] - Ep. (r(s, a)] - H((T) - ((r) FRR HEll
(10)
where pe and Pr refer to occupancy measures derived by the expert and imitation policies, H denotes the entropy and vp a convex reward regularizer. Given that this a concave-convex max-min objective, the Lagrange duality gap between the dual problem and the original one effectively closed. This structure allows the use of established optimization techniques (e.g., Karush-Kuhn- nuuh-rrckeer (KKT) conditions) for deriving the optimal policy representation:
explQxo(sst,0 az) f explQ sf (st,
x(aess))=
(11)
where the function Qooft satisfies the soft Bellman equation:
(s, = a() + ( log Σ exp e (8L41,
(12)
Under this formulation, the goal to find the reward function that can maximize the likelihood of generating expert demonstrations, i.e., the corresponding objective is
arg max ETPEPR log r
II {(AR,1SE,)) (E(O.T]
(13)
A notable line of research, constrained inverse reinforcement learning (CIRL), integrates predefined con- straints into the IRL. framework. CIRL enables learning behaviors from expert demonstrations while ensur- ing adherence these constraints (Schlaginhaufen & Kamgarpour 2023. Ding & Xue 2022 Renard 2023) However, this line of research does not address the challenge of inferring the constraints.
8

---

Published in Transactions on Machine Learning Research (01/2025)
Comparing IRL versus ICRL: Based on the tri-level optimization objective for ICRL (Eq. 9 recent study (Hugessen et al. 2024 explored whether we can 1) apply an IRI. algorithm to recover F(s,a) a() X*((s,) a) and 2) learn an imitation policy by directly maximizing the cumulative rewards E 7'7(s,, at)) without considering the constrained optimization objective. While this simplification stabilizes the learning process, the reward learning method poses challenges in generalizing the learned constraints.
Generalization Differences. Within the frameworks of IRL and ICRL, the inferred reward and constraint functions are expected to generalize across environments. In other words, once we have learned reward or constraint functions, the goal is to reuse them in other environments. We show with a simple example that the generalization achieved by reward functions and constraint functions is different in new environments.
Consider a simple shortest path problem (Figure 3 where the shortest path goes through state & and the second shortest path avoids state sf, Suppose that the reward function is known and the shortest path earns reward of 10 while the second shortest path carns reward of 9. Suppose that the expert demonstrations always follow the second shortest path. Hence, we can learn constraint that makes 8 infeasible we can learn an additional reward term that gives penalty of -1 - 7 (where 1) > 0). In both cases, the learned constraint and reward functions induce optimal policies that are consistent with the expert behavior. Let's consider a new environment with the same shortest and second shortest paths, but different reward function. The shortest path still earns reward of 10, but the second short- est path carns reward of 9 - 2n. If we apply the learned constraint, optimal agent will avoid of since it is infeasible and follows the second shortest path. In contrast, if we apply the penalty, an optimal agent will follow the shortest path since the sum of the reward and penalty 9 - 77, which is greater than 2n for the second shortest path. Hence, constraints and penalties lead to different inductive biases for generaliza- tion in new domains. In this particular example, the constraint is independent of the transition dynamics and the reward fune- tion, while the effect of the penalty depends on the transition dynamics and reward function. Hence, in new environments, the constraint will make 8 infeasible no matter what the tran- sition dynamics and rewards are while the penalty may or may not ensure that S avoided depending on the reward function.
Goal
Goal
r(t) 10
r(t)
r(t)
r(T)
21
Generalize
IRL: ('ss'(((s))
ICRL: (M²)
Start
An example showing that general- izing F and constraint 1 (M)) = 0 learned in the training environment (Left) to new environment (Right) induce different opti- mal policies (the red path for F and the blue for Mo).
Figure 5:
Suppose we prefer constraint for generalization purposes. It is possible to simply use an inverse RL technique to find F(s, a) and then convert it into constraint function a). Note that F(s, r(s,a) + X a). In order to recover a(s, one needs to find the optimal Lagrange multiplier X of the Lagrangian dual, which is equivalent to solving the original constrained optimization problem. Hence, it is not clear that this will be simpler than directly learning a constraint. Nevertheless, this is a worthwhile direction for future work.
3.2 Constraint Inference in Inverse Optimal Control
Inverse Optimal Control (IOC) is a subfield that bridges the gap between machine learning and control theory. The primary objective of TOC problems is to deduce cost functions constraint functions by closely observing expert behaviors. An IOC algorithm is comprised of forward optimal control problem and backward inference problem. More specifically, the forward problem can be characterized as follows:
min fu(T) s.t. f((t) S Eeg, h.(r) Ch,i. Vi [1,1]
(14)
where function fo denotes the estimated cost (c.g., expectation or risk-sensitive metric) of the trajectorios, while functions fa and h4 establish the feasibility conditions through inequality and equality constraints. To deduce these constraints from the demonstration dataset DE, the backward inference problem can be
9

---

Published in Transactions on Machine Learning Research (01/2025)
formulated as:
find fi and h; s.1. Si(TE) S Ee h((E)) = th,i VTF E De, Vi [1,7], filT-.)) 2 (1 + Eh, e [1,
(15)
(16)
where T-a denotes an unsafe trajectory. Intuitively, for any given expert trajectory TE E Dg. is necessary for the condition functions fi and h4 to satisfy the constraints. Conversely, for every unsafe trajectory T-x: these condition functions must lead to violation of the constraints.
In this context, Chou et al. 2018 addressed the forward problem (as described in Equation 14 with the hit-and-run sampling algorithm (Kiatsupaibul et al. 2011b. This algorithm generates lower-cost trajectories (T-.) that comply with the constraints learned from the system. The design of sampling algorithms depends on the linearity and convexity of the underlying system dynamics. To solve the inverse problem, the state space is partitioned into discrete regions. A feasibility function is then learned to distinguish between feasible and infeasible states within these regions. Chou al. (2019) and Chou et al. (2020) extended these algorithms to continuous state spaces. They achieved this by employing parametric constraint functions and devising uncer uncertainly-awank constraints. These constraints were driven by robust optimization techniques and Bayesian inference, thereby enhancing the algorithms" adaptability and precision. A recent study (Papadimitriou & Li 2023) proposed an incremental greedy constraint inference algorithm, which aims to minimize the KKT residual of the optimal control problem and infer constraints by progressively expanding the constraint set.
While the aforementioned studies have developed general constraint inference algorithms that can be ex- lended to various lasks, portion of IOC research specifically concentrates on applications that implement particular types of constraints. For instance,
research focuses geometric constraints (Armesto al, 2017 Pérez-D' A'Apino & Shah, 2017), isoperimetric constraints (Wei et al. 2024b), and trajectory-oriented
constraints (Li & Berenson 2016 Mehr et al. 2016
1OC versus ICRL: Although Constraint Inference IOC (CI-10C) and ICRL address similar problems, their approaches to solving these problems are significantly different. The solution to IOC problems typically depends on the physical mechanism at play within dynamical system, for example, the task of balancing pendulum is commonly formulated as linear quadratic problem. In this scenario, future states can be depicted a linear mapping of prior states, with the primary objective being to develop strategy that minimizes the quadratic cost. In this context, IOC aims to construct closed-form solution by leveraging the known system dynamics or empirically estimating the model dynamics (i.e., model-based methods). In comparison, ICRL offers greater flexibility as it can not only incorporate model-based approaches to utilize system dynamics when available, but also leverage recent advancements in RL to learn a model-free controller without knowing or estimating the model dynamies (i.e., model-free methods). These distinctions underlie the fundamental differences between TOC and ICRL. In this study, primarily focus on the formulation of ICRL problems and their potential solutions.
In the following sections, as illustrated in Figure D we introduce the main methodologies of ICRL in a structured manner.
4 Constraint Inference in Deterministic Environments
Under a deterministic environment, the environmental dynamics, such as transition and reward, follow a deterministic mapping. To conduct constraint inference under environments with deterministic dynamics, the Maximum Entropy (MEnt) RI. approach among the most extensively studied TCRL, methods. Tn the rest of this section, we will introduce ICRL in both discrete and continuous domains under the MEnt framework.
4.1 Maximum Entropy ICRL in the Discrete Domain
Under the environment with deterministic dynamics, we start by considering the discrete state-action space. In this discrete domain, the goal of ICRL is to determine the most plausible set of constraints, denoted
10

---

Published in Transactions on Machine Learning Research (01/202))
Sestry (000))
Discete
ICRL
T. Constorre-aaree
Parket
Section ICRL
Make at (oz2))
HEre ICRL Contirazua Derrais
Determnist minblic Environient
Seztion 7,ICLL for both Rewardds Constraints
22 Biienll
Lu the g002: 2003ad0)
Convider 1t netwelie Bynamics
Decorpoottion ICRL
2.3 Resaad
Jang et (022)
Mo (2021)
3-1 Mitett Obccrtte Domain
Deeri at el (22z))
52 MCERn ICRL Continucus Domah
Section S. ICRL Stochastic Environment
ICRL Methods
8.1 Shanel Constrairi Experts
Kin st (2223)
Pasaciritriou (2123)
Consider Linked Skee Demonne nition bara
Section 8. ICRL from Mullipple Expert Agents
Consider the diffrrncee between agents
User at (2033)
6.1 Posteror Destrtuttion Comntraints
6.2 HuBs Modal Constraim Experts
Qleo wt (D02)
Subeamonisn (2020)
Nu (ugA0
62 Datayhugmerred Constraine
sn B. IDRL from Limited Do nonstrattons
Conskler the interaction among agents
Quan 2004)
833 Mult-Agent Conctrain Inference
Lud (200222
Papodiritizud (204)
Figure 6: Illustrating the structure of ICRL methods. We describe the sections of our survey, including their relations and key literature. Specifically, we begin with the simplest ICRL setting in Section 4 where we
assume deterministic dynamics. Over time, we gradually generalize to more realistic scttings, beginning with
stochastic dynamics (Section 5 then addressing demonstration data of limited size (Section 6 unknown
reward functions (Section and handling scenarios involving multiple agents (Section S
C", that can be incorporated into the original MDP M to explain the expert demonstrations Dg. To obtain the optimal policy representation under the maximum entropy principle, Sccobe & Sastry (2020) assume the expert demonstrratinns follow the underlying constraints, which are defined by identifiers 1 (ME(TT)) = 1. The derived probability of observing trajectories within the dataset DE can be represented as:
(DelC) = PP
II
e Mc (TE)
(17)
Prl
TEEDg
where |DEl represents the size of the expert dataset, and Me denotes the modified MDP after incorporating the inferred constraints. By employing the maximum likelihood method, the optimal constraint can be learned as follows:
C mag. GEC
(18)
Assuming the expert demonstrations are valid (i.e., (TE) = 1) and the rewards are known, the numerator in the likelihood PM,(De: (De(C)) independent the learned policy. Maximizing Pm, (D) can be equivalently modeled as minimizing the denominator Ze Σreem ting 1 (T) to 0 for the trajectories with a large exponential reward (r)) (i.e., large generating probability) and ensuring these trajectories do not overlap with Dg:
Intuitively, this is accomplished by set-
C arg max log (M -|C| CEC
(19)
s.t. DnE =
where Ej, fr E EM, (1)) (M. = 0} represents the set of trajectories rendered infcasible by the added constraints, |C| represents the size of the constraint set, and a serves as a hyperparameter that balances the trade-off between maximizing the probability and minimizing the constraint size. Intuitively, in contrast incorporating all statc-action pairs from non-expert trajectorics into the constraint. set, this objective effectively mullifies those non-expert trajectories that possess a large exponential reward (r(r) In terms of practical implementation, according to the objective in Equation 19 Scobee & Sastry (2020) devised
11

---

Published in Transactions on Machine Learning Research (01/2025)
constraint set by implementing greedy iterative constraint inference algorithm, which iteratively adds state into the constraint set.
4.2 Maximum Entropy ICRL in the Continuous Domain
Despite the convenience of discrete domains, in practice, underlying states and actions are often represented by continuous features, such as image patterns or text embeddings. This necessitates the modeling of contin- wous domains. In continuous state and action spaces, constructing the constraint set C* is computationally intractable. Therefore, Malik et al. (2021) proposed learning a binary classifier ow a) as a feasibility func- tion approximator to determine the probability that performing action a under state feasible, such that (M) ())r (s, a) denotes the probability that the trajectory is safe, and the corresponding cost can be defined as (T) = - - log ow (s. Note that, compared to previous works Chou et al. (2019) that establish feasibility solely based on states, defining feasibility on both state and action pairs allows for modeling constraints that are inherently dependent on the state-action context.
In deterministic environments, previous ICRL works, inchuding (Scobee & Sastry 2020 Malik et al. 2021
Liu et al. 2023 often model hard constraints (e 0). By employing the Karush-Kuhn-Tunn Tucker (KKT) condition and the interior point method with log barrier parameterized by B, we can derive the optimal policy and represent the probability of generating a trajectory т as follows:
r(r)+9 log
I Tow. (x(a))
PMe (T) = 7ew
(20)
where 1) B balances the reward maximization and cost minimization in the objective and 2) the parti- tion function be denoted Ze, =f f 1og)
d(saa)) dr. dr. Using this policy representation, the feasibility function du can be adjusted to maximize the log- likelihood of generating expert data, i.e. max, (ogpM.(Dr) Taking the gradient of Equation (20) with respect to w gives us:
V. log V log
II TEEDs
P.M (TE)
(21)
Σ TEEDE
B log
II [Sp,0]]ETR
(((E..0))) E, Eyna(r)
V log
II @w(s (e,0)Er
Similarly to constraint inference in a discrete environment, constraining all the states not covered by expert demonstrations a trivial solution. To find the most effective constraint, Malik et al. (2021) incorporated the regularizer on the sparsity of cost into the objective function, and the final objective becomes:
~- arg max P.M (DE) + (T,
II (s,
(22)
(s.a)et
In order to construct a precise feasibility function within high-dinensional spaces, Malik et al. (2021) pa- rameterized (-) using neural networks and updated its parameters in accordance with the aforementioned objective.
Intuitively, the maximum likelihood loss classifies policy-visited states infeasible and demonstraation-viiieee states feasible, but overlap cause conflicting updates, slowing training and misclassifying safe states. To address this, Peng & Billard (2024) proposed two-step Positive-Unlaben
Constraint Learning (PUCL) method, inspired by Positive- Unlabeled learning (Bekker & Davis 2020), which first identifies reliable in- leasible data and then trains a binary feasibility classifier as constraint function using both positive and reliable infeasible data.
5 Constraint Inference in Stochastic Environments
Due to the complexity of real-world applications, doterministically predicting future states is challenging. In these applications, the environment dynamics, such transition and reward functions, are often repre- sented by stochastic models. However, the MEnt-based algorithms (Section 4 assume deterministic training
12

---

Published in Transactions on Machine Learning Research (01/2025)
enviromments, without considering the inftuence of underlying stochasticity in the environment. Specifically, stochastic transilion functions introduce aleatoric uncertainly. Striving for reliable constraint inference, the policy representation (e.g., the maximum entropy policy in 17 and the constraint model (e.g., ow) must be sensitive to its influence.
In this section, order develop ICRL algorithms that are robust to the underlying uncertainty in the environment, we provide an overview of 1) Maximum Causal Entropy ICRL in both discrete (Section 5.1) and continuous (Section 5.2) domains, and 2) soft constraints (Section 5.3) that are compatible with the noise induced by stochastic dynamics.
5.1 Maximum Causal Entropy ICRL in the Discrete Domain
Similarly, we start by discussing the ICRL algorithm under the discrete domain. In the MDP with stochastic dynamics, the trajectory-level policy can be factorized into:
(r) = (o(())
II t=
x: (as (S)
(23)
Consequently, modeling the trajectory-level policy (formula 17 requires accounting for the influence of transition dynamics Py, which often unavailable to the agents during training (typically the model-free setting). An effective approach that can generalize MEnt ICRL to stochastic environments modeling the causal entropy (McPherson al. 2021) framework. The causal entropy can be represented as:
H Haaor (O:T) E PT
y log (acls))
(24)
where the step-wise policy r(ass) depends only the available information at each step.
A critical challenge to constraint inference in stochastic environments lies in the definition of the feasibility function. In this discrete enviromment, McPherson et al. 2021 define:
Me (s, a) = 1 (p(S++1 = S|S; = 8, A, = a) S (3))
(25)
Intuitively, it assesses the likelihood that executing an action a under a given state will result in an unsafe state the constraint set C, and examine whether this likelihood is less than (3)).
By following (Ziebart et al. 2010 Haarnoja et al. 2017). the optimal policy representation for Maximum Causal Entropy (MCEnt) Reinforcement Learning under feasibility function 1. can be represented follows:
Del (ona))
TM. (aclse)
Mo (s),
(26)
V (se)
Qo0ft (0, (St, ar) = r(s,, az) + Es (1 (82+1),,
(27)
veol (8t) = log Σ l Mo at
(s 9ot (oolt(anas) (snas)
(28)
In practical applications, these values can be calculated using soft Value Iteration, which is similar to conventional value iteration but replaces the hard maximum over actions with a log-sum-exp operation. By employing the aforementioned policy representation, the joint distribution within a horizon [ : T] can be expressed as:
PM. (A ap =s)
(29)
v (
0,
otherwise.
13

---

Published in Transactions on Machine Learning Research (01/2025)
From the aforementioned formula, it becomes evident that altering the constraint set C' only modifies the normalizing constant e ev,t (os). (eo) In accordance with the value function reprosentation (Equation 260, ex- panding the constraint set results in strict decrease in (st), subsequently maximizing the likelihood of observed demonstrations. By utilizing the maximum likelihood estimation, the problem involves incremen- tally expanding the constraint set Co by iteratively adding 3 (i.e., C. = C,UB) and determining the threshold (B) using the chance level specification algorithm (Vauquez- Chanlatte et al. 2018 for constructing MMa
5.2
Maximum Causal Entropy ICRL in the Continuous Domain
In the continuous environments, constructing the constraint set is computationally intractable, so similar Section 4.2 Baert et al. (2023) utilizes (s, a) to determine the probability that performing action a under state is fcasible. In this way, the constraint in the MCEnt objective can be updated to
-B -B
(T,
log ew (s, (s, S
(30)
The policy satislying the KKT condition can be represented as follows:
(aels))))) 8 (sna))
(31)
Vai (a))
000 (st, 01) = r(3),a)) + Alog + Es (eeei ($(+1)
(32)
vea (8t) = log /. Qat (on.a)
(33)
In practice, these value functions can be effectively approximated using Soft Actor- Actor-Critic (Haarnoja et al. 2018) for continuous action spaces. To develop the maximum likelihood approach for updating the parameters of cost functions, Gleave & Toyer (2022) introduced the concept of the discounted likelihood of trajectory:
Definition 5.1 (Discounted likelihood, Desenition 3.1 in (Gleave & Toyer 2022 ). The discounted likelihood of a trajectory T = (50, dg, $1, at...., -T, ST) under policy π is:
T-1
(T) = (o()))
II t=
pT (8:41 (a (arls))
(34)
Intuitively, when 7 1, the discounted likelihood is equivalent to the likelihood of 7 under the policy For 7 S 1, the probabilities of actions later in the trajectory are regularized by the power of 74. This regularization is useful in cases with an infinite planning horizon (T' + oo), as it prevents the trajectory likelihood for any policy from approaching 0. Based on the aforementioned definition, the log-like liceliiol of demonstrator's trajectories, sampled from demonstration distribution De, is then:
logp?y. (T)
7" log T.M. (arls)) + (o(s)) +
log PT (5)
(35)
IMi
T
(2ed (st, ar) V vaaft(ss)) + log (s0)
log pr (s)
(36)
Based on the above formula, recent works (Gleave & Toyer 2022 Baert al. 2023 showed that the
parameters of the constraints, denoted as w, can be updated to maximize the log- likelihood term Pog (T) by utilizing the following gradient estimate:
T
T
V. logp), (T) = Ep
V (wr an) - E,
7' V Vn
(37)
1=
Baert et al. (202)) defines ((r) Et- 7 (d),,, but in the we denote x(T)
T-
ITT-1 T-1 a)s,)) sa that ((T) can represent
the feasibility probability of trajectory and the cost c(7) E, log (t)
14

---

Published in Transactions on Machine Learning Research (01/2025)
5.3 Inferring Soft Constraints with ICRL
Unlike the hard constraints that guarantee constraint salisfaction, soft constraints account for noise in sensor measurements, such as those caused by stochastic transition functions (PT) or cost functions (pc). This consideration helps address potential violations that may arise in expert demonstrations duc to the stochastic dynamics in the environment. Specifically, the soft constraint can be represented as Ept Ei de) S which akin to the cumulative constraint (Eq. 2 but the soft constraints require a threshold e > 0. To better align with the soft constraint inference, the Inverse Soft Constraint Learning (ISCL) algorithm taken by (Gaurav et al. 2023) employs the Deep Constraint Correction (DC*) framework (Donti et al.) 2021), which transforms a constrained problem into an unconstrained problem
by introducing a non-differentiable ReLU term. To extend this approach ICRL, Gaurav al. (2023) represented the CRL objective as:
T Σ relo -
arg max E Ep
Σ ((1()); ARELU
E mour
(38)
Upon obtaining the optimal policy during a specific run, ISCL incorporates it into the candidate policy set II such that II = NU.. Accordingly, the ICI objective in ICRL can be expressed as:
arg min ETy MM0
((c( + ARELU E Ep.-Rg
a(()) -c]
(39)
It is important to note that the first expectation is taken over the mixture policies mmiz, while the second expectation is derived from the expert dataset Dr. ISCL constructs rucls wmir as a weighted combination of the candidate policies in II.
6 Constraint Inference from Limited Demonstrations
ICRL algorithms typically infer constraints from expert demonstrations, which necessarily have limitod cov- erage of the underlying environment. Upon updating the constraint model, due to the limited amount of training data, epistemic uncertainty arises in game states that fall outside the data distribution. To be more specific, in an ICRL task, the training dataset Dtrain for constraint inference records the expert demon- strations and nominal trajectories generated by the imitation policy, i.e., D {TI,E. E, fx. Epistemic uncertainty arises when the constraint model is asked to predict the costs of state-action pairs (8, â) that are out of the training data distribution. This issue is closcly related to the false correlation problem in oflline RL (Jin et al.) 2021 Xie et al.) 2021), which is due to insufficient data coverage, lead- ing to overestimated action values and suboptimal policies. Approaches to mitigate this challenge include conservative value estimation (Kumar al., 2020 uncertainty modeling (Deng et al., 2002)), and policy constraints (Wu al. 2022). In the realm of ICRL, the algorithms discussed earlier primarily concen- trate on addressing aleatoric uncertainty induced by the stochastic transition dynamics in the environment. Addressing the influence of epistemic uncertainty remains a critical problem in ICRL literature.
In this section, we provide an overview of ICRL algorithms that account for epistemic uncertainty, including the approach to estimating the posterior distribution of constraints (Section 6.1 data-augmented constraint inference (Section 6.2). and offline constraint inference (Section 6.3
6.1 Modeling the Posterior Distribution of Constraints
The aforementioned ICRL methods typically learn a constraint function that best differentiates expert tra- jectories from generated ones. To better handle the limited training data, in this section, we introduce the ICRL method that models the posterior distribution of constraints.
Bayesian Posterior Estimation. In contrast these traditional methods that primarily rely on maxi- mum likelihood estimation, Papadimitriou et al. (2023) applied the Maximum--APPosteriott (MAP) approach
15

---

Published in Transactions on Machine Learning Research (01/2025)
to address the epistemic uncertainty during constraint inference. This led to the development of the Bayesian Inverse Constraint Reinforcement Learning (BICRI) model that infers posterior probability distribution over constraints based on demonstrated trajectories.
BICRL is primarily designed to infer the constraint se C E {0, 131s) in the discrete state space S. This is achieved by sampling candidate solutions, consisting of a candidate constraint set Ö and a Lagrange multiplier X, from their inferred distributions at the previous run. Within the CMDP Me constructed on the candidate constraint set Ć BICRI. computes the likelihood of expert demonstration TE under the MCEnt framework (see Section 5.1 80:
T-1 pr (s 0z) ग (asl())))s = constant
T
T-1 II = e
goot
(T (o(s)) II ==
II =
(40)
where the value functions can be defined by:
C.A. (1t, az) =((s; az) + A log I M... (s, ar) + Es CCH4 (86+1)
(41)
( = log / O2f (oria) da
(st)
(42)
The feasibility identifier 1 Me.. (st, az) defines whether performing the action a in state will lead to the transition to a constrained state $1+1 E E (also see Equation (25)) and these (action)-value functions can be computed by dynamic programming (Sutton & Barto 2018
Within the m runs, BICRL utilizes the learned distribution over constraint sets at previous (m - 1)th as prior (i.e., Peo (C, A) Pm-1(C,A(TE)) The resulting posterior distribution can be represented as:
Pm (C,EE) (Mc.A(TTE C, (C,
(43)
p(TE)
To initialize Papadimitrion et al. (0023 selected an uninformative prior. By utilizing the above posterior, the Maximum Posteriori (MAP) estimates for the constraint sets and the penalty reward can obtained as:
CMAP, arg max pCC, AlT) C,A
(44)
and the Expected Posteriori (EAP) estimates can be obtained as:
CeAP, YEAP [CC.
(45)
Since sampling the constraint set C from a continuous state space is computationally intractable, BICRL is mainly validated with discrete state spaces. How to extend this algorithu to continuous domains remains an open problem that requires further exploration.
Variational Inference. The aforementioned Bayesian updates depend on the samples derived from the Monte-Carlo Markov Chain (MCMC). However, this method tends to encounter intractability issues environments with continuous state and action spaces. To tackle these problems, [Lu al. (2023) proposed Variational Inverse Constrained Reinforcement Learning (VICRL), which infers the approximated posterior distributions of constraints to caplure uncertainty in the remonstration dataset.
Specifically, VICRL infers the distribution of a feasibility variable & such that (ols,, to which an action a should be allowed in a specific state s. The instance & can define a soft constraint given by: (s, a) = - o, where ~ p(-1s, a). As + continuous variable within the range [0, 1), Liu et al. 2023 parameterized p(øls, a) using Beta distribution:
a) measures the extent
((s,a) ( Beta(Q Bw) where (ow,Pu) = log[1 exp(ff(ss
(46)
3ICRR assumes discrete state space, and refers the cardinality of finite
16

---

Published in Transactions on Machine Learning Research (01/2025)
Here fo is implemented by multi-layer network with 2-dimensional outputs (for and 3). In practice, the true posterior p(o)Dr) is intractable for high-dimensional input spaces, so ICRL learns approximate posterior ((([p) by minimizing Dui ((dDe))l((dD)) This is equivalent to maximizing an Evidence Lower Bound (ELBo):
Bg pog. Dui a [((
(47)
(Pelo) (Pelo) can be implemented using the trajectory likelihood (Eq. 20 within the MEnt framework, and the discounted likelihood (Eq. 340 within the MCEnt framework.
In this case, the log- likelihhoodd term pog,
The primary challenge in VICRL is defining the KL divergence. Aiming for ease in computing mini-batch
gradients, (Liu al. 2023 approximated Dur [(d))))l with E(s.)eer Dur ( a)ll(()). As both the
posterior and the prior tollow Beta distributions, the KL divergence according to the Dirichlet VAE (Joo et al. 2020 can be represented as:
T'a+ + log ((a9))(8)) =log log [((00 +,30)/
DM ((o)) (l)(())
(48)
[(a)T(B))
+ (a (a--a") a²)
[(aaaa (b)()(())
(/6 + B)] + (8- 88) (03) 3)]
where 1) [a", and [a, are parameters from the prior and 2) the posterior functions and T and 1/ denote the gamma. and the digamma functions. Note that the goal of ICRL is to infer the smallest constraint for explaining expert behaviors. While previous mothods often use a regularizer E[1 - Φ(7) (Malik et al., 2021) for punishing the scale of constraints, this KI-divergence term extends it by further regularizing the variances of constraints.
Confidence-Aware Constraint Inference. The aforementioned methods typically sample cost functions or constraints from the estimated distribution. However, when dealing with epistemic uncertainty, a more ideal approach is to first assess the confidence level in the estimated constraints. By doing s0, one can ensure that only those constraints that meet a desired confidence threshold are utilized, thereby enhancing the reliability of the constraints used in the model. To achieve this goal, (Subramanian et al. (2024) introducod Confidence-Aware Inverse Constrained Reinforcement Learning (CA-ICRL), which incorporates a confidence level alongside set of expert demonstrations. This approach outputs a constraint that is at least as restrictive as the true underlying constraint based on desired confidence level. The parameters of the constraint models w are given by:
W arg max
F-1 Σ r(T) + log Bull)) -g) 10--ogg -log Zw
(49)
LEDS
where & denotes the desired confidence level, Zu denotes the partition function (similar to the Ze in Equa- tion 20) and (C) denotes the quantile function (i.e., inverse cumulative distrribution) of distribution P at threshold g. Similar to (Liu et al. 2023),, |Subramanian et al. (2024) utilized the Beta distribution for mod- cling the feasibility of trajectories, and the parameters (aw, Bw) are computed by aggregating the point-wise influence signals from each state-action pair within the trajectory. This is done with a deep set network that has the property of producing higher (aw,, parameters as the number of expert demonstrations increases, effectively reducing the epistemic uncertainty. The confidence-aware estimate of the feasibility of a trajectory is denoted by:
(T)
F-11 (8 {)
(50)
In settings where developers can gather more expert demonstrations on a need basis, this confidenco-aware framework can also be used to determine when we can stop gathering expert demonstrations ensure that desirable level of performance is achieved with desired confidence level.
17

---

Published in Transactions on Machine Learning Research (01/2025)
6.2 Data- Autta-negeeeedd Inverse Constraint Inference
Apart from modeling the posterior distribution of constraints, an alternative approach to handling epistemic uncertainty involves augmenting the dataset. Epistemic uncertainty arises due to the limited training data and the model's lack of knowledge about Out-of-Distribution (OoD) data. An effective measure of epistemic uncertainty is [[w: y|2; D) (Smith & Gal 2018 van Amersfoort et al. 2020), which quantifies the amount of information gained by the model w when observes the true label y for a given input x; i.e,, the greater the uncertainty of the model regarding the data, the more additional information it can obtain once the true label y is observed. Under the setting of ICRL, to reduce the epistemic uncertainty, Xu & Liu (2024a) added the regularizer [[w;; o|T, D) into the ICI objective (Equation 21
to propose the following objective:
T
E ETEEDE
(s of }/]]]
E R
log (S,, ax]] at (w;
(51)
where D denotes the training dataset consisting of expert demonstration Dg and imitation demonstrations D. Since the mutual information term / is compulationally intractable, Xu & Liu (2024)) showed it can be em- pirically approximated by [(w; o|T D) = H[p(olT, D)] M Σm H[p(o]T; wm]] where Wmn ~ <(w). Specifically, 1) H[p(oFF wm)] defines the entropy of a constrained model parameterized by wm. 2) Hfp[olF, D)] E [0, oo) measures the amount of information required to describe the feasibility & of an exploratory trajectory based on the given training dataset D. By substituting them in formula 51], we obtaín the following objective:
-1
T
T
Σ EDE 1
loglow (s. of af]]]
EED
1 ac]]] [=), - -aHlp(o1T + aH[p(o/):um))
(52)
M
Inspired by Smith & Gal 2018 Xu & Liu (024aa) used dropout layers (Sriaastaaa et al. 2014 for approx-
imating the distribution of model parameters q(w). Besides, to reduce the conditional entropy Hlp(0/ D) Xu & Liu 20244) proposed expanding the training dataset by adding generated trajectories {(TG, Q.)) To more specific, the augmented expert datasct constructed by DE DE U To. Véc = 1, and the aug-
mented nominal dataset is constructed by DG = =Bu TG, Véaa 0. By substituting them in objective 52
we arrive at the following objective:
T
T
DE
Loglowm.e
(8 af E
log Bowl (8,, ai]] + [
(53)
M
m
t=
t=0
In order to generate the trajectories TG, Xu & Liu (2024a) designed a Flow-based Trajectory Generation (FTG) algorithm that extends the Generative Flow Network (GFlowNet) (Bengio et al., 2021 to generate a diverse set of trajectories based on the dataset and desendde rewards. Similar to the generative data synthesizer (Zhu al., 2023), FTG explores various combinations of "points" (such as state-action pairs in RL) within sequential data. This design enables FTG to generate trajectories in which the sequence of states and actions deviates from those in the training data samples through multiple rounds of random sampling (Li et al. 2023). The dataset augmented with these trajectories can more accurately characterize the underlying distribution of feasible and infeasible trajectorics, thereby reducing the incortainty associated with the parameters of the constraint model.
6.3 Offline Inverse Constraint Inference
The aforementioned methods primarily focus the impact of limited expert demonstrations. However, the imitation policy can be learned through interaction with the environment. In contrast, a more stringent scenario is Offline ICRL, where the agent must infer constraints and learn imitation policies based solely on a fixed dataset, without access to the environment for additional interactions.
Specifically, in Offline ICRL, we are given an offline datasct Do ( af r No Within this datasot, expert trajectories are denoted as Dg {sf a r NR C Do. These expert trajectories are generated by optimal policy E. adhering to the unobserved constraint function c(s,a). More formally, n m
18

---

Published in Transactions on Machine Learning Research (01/2025)
arg max, E p== [r(s, (r(ss,)) a)], subject to [c=e)((s,a]] (p" denotes the occupancy measure by following the expert policy )²). An estimated cost function è is a feasible solution to ICRL and only the optimal
solution z can reproduce d To achieve offline ICRL, Quan et al. (2024) mainly study hard constraints with e 6 0. The CRL problem can be formulated as:
max MaxEp (r((s,))(
(54)
s.l. (s, SO, A8,a
By extending this objective to the offline ICRL problem, this objective can be updated to:
max (0)) Eplr(s,a]] - B,Dy(pllP))
(55)
s.t. EaEAp(s,a) = (1-7)po(s) + Ereswael d' (psssss ,a), as E
where o represents the visitation distribution in the offline dataset DO, Dy(plpo denotes the a f-divergence between two distributions, and 3, denotes the weighting parameter. Intuitively, instead of maximizing only the reward, we augment the cumulative reward with a divergence regularizer prevent it from deviating beyond the coverage of the offiine data. Note that this objective forms dual problem for CRL (Sikchi et al. 2024).
To align with the above Distributional Correction Estimation (DICE) (Lee et al. 2021 objective, Quan et al. (2024) translate the constraint inference problem into the problem of estimating the superior distribution set as follows.
Definition 6.1 The of superior distributions (the distribution. is on state-action pairs. e.g-, the nor- malized occupancy measure defined in Equation 4) denoted as D, is defined as those distributions that are generaled by cerlain policy and achieve higher cumuuaatie rewards than experts, denoled D = (p= Ep-(r(s, a)] >> (,)))(s,
Intuitively, for any superior distribution p² E D, there's at least one state-action pair (s, with p² ps(sa) > 0 that constitutes a violation of the constraints. A straightforward method to identify fcasible cost function e begins with estimating the set of superior distributions D. Subsequently, positive cost is assigned to at least one state-action pair within the support of each distribution p² included in this set. This assignment is done while ensuring that the chosen state-action pair is not covered by the expert demonstrations.
A recent study (Papadimitriou & Brown 2024 explored an alternative scenario in which preferences among demonstrations are available. Specifically, Papadimitriou & Brown (2024) developed the Preference- Based Bayesian Inverse Constraint Reinforcement Learning (PBICRL) algorithm. This method extends the Bradley-Tery Terry model to the context of constraint inference. In this approach, constraints are inferred by maximizing the likelihood that demonstrations with higher preferences (D+) are more effective than those with lower preferences )D_). This log-likelihood function represented as follows:
E.en. ((())
C(w,c)= Σ REDI
logp(T, > =
Σ
log
E. (r())__(())))))))))))())))) (()))= + 2 Exer, (((s)+ Te 2,er,
GED SEED
(56)
where 1) w denotes the weighting term for the cost 2) 3 denotes the inverse temperature parameter, 3) T, indicates the length of a trajectory Tr and 4) m the margin parameter between group D4 and D_..
7 Simultaneous Inference of Rewards and Constraints
Previous ICRL algorithms typically focus on learning a constraint function based on a known reward function. However, these methods struggle in environments where both constraint and reward signals are absent. Under this setting, an intriguing yet relatively unexplored extension of ICRL is simultancously infer rewards and constraints from expert demonstrations. However, since both IRL and ICRL are inherently ill-posed due
19

---

Published in Transactions on Machine Learning Research (01/2025)
to the ambiguity in identifying the reward function and constraint function, concurrently inferring rewards and constraints can substantially amplify the complexity and ambiguity of identifying the true underlying rewards and constraints.
In this section, we provide an overview of algorithms that learn both rewards and constraints, inchuding
constraint-based Bayesian IRI. (Park al.) 2019) (Section 7.1) that learns different local rewards and local constraints from different expert trajectory segments, Bexximum-lkkelihoo ICRL (Liu & Zhu 2022; 2023b a) (Section 7.2) that learns global rewards and constraints from complete trajectories, and transferable
constraint learning (Jang et al. 2023 that jointly infers task reward and residual task agnostic constraint pairs from demonstrations.
7.1
Constraint-Baser Bayesian Inverse Reinforcement learning
While the existing IRL and ICRL methods typically learn single reward function or single constraint function from the expert trajectories, Park el. al.
2019 proposed Constraint-based Bayesian Nonparametric
IRL (CBN-IRL) algorithm that learns multiple local rewards/goals and local constraints from different expert trajectory segments. Specifically, Park et al.
(0099) splits the expert trajectory TE into multiple partitions {(,))-1- Within each partition 6, CBN-IRL learns a goal function (s(s) that assigns a positive reward to the destination s and zero other states, i.e., (s()) ((s x5) where of is the goal state. CBN-IRT, also learns a constraint function (s) + {0,1} assigns a value of one infeasible or unsafe states and zero to feasible safe states. Following this setting, Park al. (2019) proposed the Maxximmm-A-Postern (MAP)
objective lo infer the constraint c.(s) and goal (x))) for each segment.
c²' arg max II PTESsly Cr (osl
(57)
where P(TE.J Meys Cyg) denotes the likelihood of generating the expert trajectory segment TEJ in the jth partition 'j, p({)(e-)) refers to the probability of transferring from other partitions t-j to partition ij, and p(go. co) denotes the prior. Partitioning the expert trajectory into local segments increases the sparsity of rewards and reduces the complexity of constraint inference for each partition.
7.2 Bi-Level Optimization Inverse Constrained Reinforcement Learning
While CBN-IRL learns multiple local reward/ /goa functions and local constraint functions from different expert trajectory segments, Liu & Zhu 2002 2023a b) proposed to learn a single reward function and constraint function for the complete expert trajectories. Specifically, bi-level optimization problem:
Liu
Zhu (2023a formulated a
max ETNDE
y' log Fe (r)or
t=
T - [t]l + H((Te.) ) + EynDE
T
s.t. c²(r):= = = min Epp Rojei 40
Σ t=
Σ t=
((((apls))
(58)
The upper level aims to learn a reward function r to maximize the log- likelihood of the expert trajectories DE where Trę is the constrainod soft Bellman policy Liu & Zhu (2022 2023a bb) under the reward function r and constraint function c. It is proven in Liu & Zhu (2022 2023a) that the constrained soft Bellman policy maximizes the entropy-regularizze cumulative reward-minus cost:
Tr: arg max E pp Ep 50
Σ (r((,,, c(st, H(m))
(59)
The lower-level objective function
be partitioned into two parts. The first part
Epr Sa ~ [r(*,, 7 ar) - c(s), + H(Ter) uses adversarial learning to encourage a constraint
20

---

Published in Transactions on Machine Learning Research (01/2025)
function c that makes the best policy Ty: perform the worst. The second part BynDE Eo re((asss)) penalizes constraint functions where the expert has high cumulative constraint.
However, this bi-level formulation 58 still suffers from unidentifiabilty, Lee. infinitely many combinations of reward and constraints can explain the expert trajectories. Thercfore, Liu & Zhu (2022) assume that the reward function and constraint function are linearly parameterized, so that they can prove and leverage the strictly convex property of the lower-level problem to find a unique constraint function and thereby guarantee the convergence of the reward function.
7.3 Reward Decomposition Inverse Constrained Reinforcement Learning
Jang et al. (023 proposed decomposing the reward function into task reward rp E RR and the constraint- related residual reward Te R, given the constrained demonstrations DE and the task reward space Rp S R. Here, Tp is to produce an expert-like but unconstrained behavior, defined based on predefined task- relevant features. In contrast, re represents negative constraint-cost function, expressed r. -c(s, a). By designing Transferable Constraint Learning (TCL) algorithun, the learned Ye is lask-agnostie and transferable across tasks to produce constrained behaviors.
Following the Q-dccomposition framework (Russell & Zimdars, 2003), Jang ct al. 2023) designed an opti- mization problem that simultaneously learns the overall reward from demonsttaatinn DE through inverse RL and decomposes into an optimal reward pair (rt r2) using a reward decomposition (RD) approach:
(r r.) arg
max rp U.ER,
min JIRL (r, n: DE) JuD(Tp: n; Rp))) [
(60)
s.t. r r= Tp + rc
where JIRL and JRD are objective functions and is output policy associated with the overall reward
Specifically, JIRL is the maximum causal entropy IRL Ziebart et al. 2010) objective function aiming to infer a total reward r and its associated policy æ that can produce behaviors similar to demonstrations Dz:
(R n; DE) E,,aND [r(s, (ass,a)]E,,gn E,,as [r(s, a)] H(TT)
(61)
JRD is the reward decomposition function designed to determine T. from the overall reward r, based on the assumption of an additive decomposition Tр + Te- Jang et al. (2023) utilized rр to define a tpsk-specific policy Tp, which governs all task-relevant but unconstrained actions. By identifying the largest Yp that closely approximates r, the residual reward re can be derived. This residual reward, being task-agnostic, enables its corresponding policy generalize across different tasks. To achieve this, the objective function JRD is formulated minimize the action divergence between Tp and m:
JRD(F m: R,) En-pp [DKL. (Tp(. ()([T( [)]]
(62)
where PM is the state-visitation frequencies for π,, Tp define the policy learned under the rp and DKL (-) represents a Kullback-Leible (KI.) divergence between two given policies. In Jang et al. (2023), task- relevant features are manually selected to serve as the basis vectors for the task space. When such features are not readily available, alternative options include using features designed for sparse rewards or binary indicators, which can effectively represent the task structure.
8 Constraint Inference from Multiple Expert Agents
In practice, demonstration datascts may be generated by multiple agents. For instance, on an open road, vehicle trajectories could be produced by variety of human drivers, each possessing different driving skills (e.g., risk-averse and risk-seeking) and operating different types of vehicles (i.e., trucks, vans, or cars). These environments require considering the behaviors of multiple heterogeneous and homogeneous agents.
Unlike Multi-Agent Reinforeement Learning (MARL) /Zhang et al. 2021). ) which solves only forward control problems, ICRL typically involves solving backward inverse constraint inference problem using provided
21

---

Published in Transactions on Machine Learning Research (01/2025)
expert demonstrations. To adapt ICRL to multi-agent settings, we define three separate levels of Multi-Agent ICRL:
1) Expert trajectories generated by multiple types of agents, all adhering to the same constraint but optimizing different reward functions. The key challenge at this level is: How can we infer the shared constraint respected by various types of agents?
2) Expert trajectories are generated by multiple types of agents, each respecting different constraints. The key challenge lies in: How can different constraints be inferred for various types of agents?
3) The demonstration dataset is produced by multiple agents acting simultaneously. The challenge here is to determinc: How can cooperative and competitive behaviors among the agents be modeled in order to infer appropriate constroints for explaining their behaviors?
To better differentiate these challenges, in the first two research topics, assume the expert demonsttations are generated by multiple experts, but the forward control policy is conditioned on a single agent type by treating the states of other agents as background. However, for the final challenge, interactions among multiple agents must be considered to model the joint behavior of different agents. In this section, we study several existing ICRL algorithms that aim to lackle these challenges.
8.1 Inferring a Shared Constraint from Multiple Expert Demonstrations
In order to infer a consistent constraint respected by multiple types of expert agents, Lindner et al. (2024)
studied a setting where the expert agents optimize different rewards under shared constraint. To represent the constraint, Lindner al. (2024) defined the safe set as the convex hull of the feature expectations of the expert demonstrations:
-C = =onov(DD) = E >c 2 0 and Σ A =1
(63)
where fTTT,K) EFR.EPP EE 2't f( at]] estimates the feature expectations derived by the h expert policy TEAA- The inferred raint essentially establishes 'worst-case (pessimistic) constraint', as opposed to indispensable one (i.e., the minimal constraint in Section 2.4p. Under this inferred constraint, policies whose feature expectation is not in the convex hull represented by the weighted combination of expert Intuitively, policy exploring the
policies' feature expectations are considered to be constraint-violatio state-action pairs uncovered by (i.e., located outside the support of) expert demonstration is automatically considered infeasible.
In Lindner al. (2024), instead of learning constraint function, the approach assumes that any unseen behavior is unsafe. It enforces constraints by requiring the learner to act convex combination of the demonstrated safe trajectories. The key advantage of this method is that it climinates the need to know the reward function that the expert was optimixing. However, by restricting the learner to merely replicate the expert's demonstrated behavior, this approach limits the ability to generalize effectively and may result in highly suboptimal performance on new tasks. To solve this problem, Kim et al. (2023) additionally leverages side information in the form of a reasonable set of constraints, enabling policy performance guarantees.
Specifically, Kim et al. (023) considers inverse constraint learning under the assumption that the reward function, expert demonstrations, and class of potential constraints F, are available. It is assumed that the ground-truth constraint c* lies within the convex and compact set F.. Building upon this, Kim et al. (2023) formulates the problem of inferring a shared constraint from multi-task demonstrations as follows:
K Σ
max min max cEf T EILAEA AEN
(J -xxc) - Xxc))
(64)
where J( Eyonn Et f(s ai) denotes the value of policy under reward cost function f. based the observed K samples of the form (rx, (7 1)
22

---

Published in Transactions on Machine Learning Research (01/2025)
8.2 Multi-Modal Constraint Inference from Mixture of Expert Demonstrations
Instead of assuming the agents respect the same constraint, Qiao et al. (2023) studied expert data De that record demonstrations from multiple experts who respect different kinds of constraints. To infer these constraints from mixture of expert demonstrations, (Qiao et al., 2023) proposed Multi-Modal Inverse Constrained Reinforcement Learning (MM-ICRT) algorithm that performs unsupervised agent identification and multi-modal policy optimization to learn agent-specific constraints.
Specifically, MM-ICRL trained Row-bbsed density estimator Pu(s, based on Masked Auto-regressive Flow (MAF) Paapamakarios et al. 2017). This density estimator is trained to maximize the log-likelihood of trajectories generated by specific agent k. The agent's trajectory-lovel identifier can be represented using
exp Hmm Pola,a)k)
the softmax representation such that (k(())) Σn exp (najer po(s,a)k)) After learning the density model [X]
(s alk), MM-ICRL divides DE into sub-datasets (Dk) A= by: 1) initializing the dataset D3 0 and 2) VT; E DE: adding 7' into DR if l = arg max, (k)r). We repeat the above steps for all k E K. Based on the identified expert dataset Dx. Qiao et al. (2023) performs agent-specific constraint inference to learn the conditional feasibility function ow (st, kx) and updated the parameters w by computing the gradient of the conditional likelihood function:
N
T
V logglg[((22 (p({xlo, =
(s af - -NE+ (-1b) (-16) V.
loglo ac1h1]]
(65)
t=
This inverse constraint objective relies on the nominal trajectories & sampled with the conditional imitation policy (=(kk)) MM-ICRL learns the imitation policy by following the objective:
policy optimization
min E(()) [r(T) + as logle (617)] (gg])(((17)]] + (az (1){1[(T(1)]
(66)
x
s.t. E Ex( k))
7' log (s, 2e
1=0
Intuitively, this objective expands the reward signals with a log-probability term (kl))) which encour- ages the policy to generate trajectorics from high-density regions conditioning on a specific agent typc. This approach ensures that the learned policies (-11k) K are differentiable.
The MM-ICRL method alternates between executing agent-specific constraint inference and optimizing multi- modal policies. This process is accompanied by updating density models using the acquired limitation policies until they successfully reproduce the expert demonstration.
8.3 Inverse Constraint Inference from Multi-Agent Environment
For expanding ICRL to model the cooperative behaviors among multiple agents, one effective approach is to adopt a Constrained Markov Game (CMG) framework where the action space is = II denoting that multiple agents perform simultaneously. In a CMG environment, Liu & Zhu (2022) considered collaborative multi-agent setting, where multiple agents collaboratively recover the expert constraints. Assuming additively decomposed cost function e(s,a) Σk ca(8, a), the corresponding constraint be represented by:
k A [
E PT
Σ
2 Σ t=
(8) Se
(67)
To find a statistical distribution model for the policy æ based on the above constraint, Liu & Zhu (2022) formulated the problem based the MCFnt framework (see Section 5.1 and theoretically showed that the
23

---

Published in Transactions on Machine Learning Research (01/2025)
optimal solution follows the constrained soft Bellman policy given by:
K
T
TT M.M. arg max Ex Ex PT
Σ
T
A A. 40
Cs(s, ax)
Σ 1=0
Br. Br.T Ho (f llg (ogr(a......xlss))) axlss))))
(68)
where represents some fixed feature mapping for expert k, An and de are the optimal dual variables. Given the representation of the optimal multi-agent policy TM,. the constraint is updated by maximizing the log-likelihood of generating the expert demonstrations by utilizing the optimal policy:
T
e = arg max
Σ TEEDg t=
Σ
r (ag,1s.a
(69)
9
Benchmarks and Applications
In this section, we introduce the existing benchmarks for evaluating ICRL algorithms and explore the po- tential applications of ICRL in addressing critical real-world challenges.
9.1 Benchmarks
Discrete Environments. Grid- World environments are among the most well-studied discrete environments for evaluating the performance of ICRL algorithms. Specifically, previous works (Scobee & Sastry 2020;
McPherson et al. 2021 Papadimitrion et al., 2023 Glazier et al. 2021 Gaurav et al. 2023) added some
obstacles to a grid map and examined whether their algorithms can locate those obstacles by observing expert demonstrations.
Setting
Setting 2
Setting
Setting
6
5
4
3
2th
1
3th
401 Tral
0
4th
1
2
3
4
5
6
0
1
2
6
6
5
Figure 7: The Wordd Environments: We randomly select five expert trajectories (denoted as 0th Traj to 4th Traj) for visualization purposes. The starting point and destination are denoted by red and blue circles, respectively. The primary objective of the ICRL algorithms is to deduce the constrained region, which is unobservable and illustrated by the gray color.
Figure 7 presents four distinct Grid-World environments. These environments have been chosen for their ease of visualization and result analysis, and is relatively convenient. to expand these simple gird-worlds into more complicatod scenarios, for example, Bacrt et al. (2023): Xu & Liu (2024b) developed Grid-World environments with stochastic transition dynamics, while Qiao et al. (0023) added multiple types of con- straints to Grid-Worlds. While Grid-Worrd environments, with their dimeniinni and discrete state spaces (represented by x-y coordinates), offer several key benefits, they present challenge in generalizing model performance to the environment with high-dimensional and continuous states. By explicitly consid- ering the stochasticity, MCFnt-ICRL algorithms (Section outperform those based on MEnt framework (Section
Virtual Environments. Evaluating RL algorithms directly in real applications can be inefficient, costly, and potentially lead to critical safety issues. As alternative, simulated virtual environments offer
24

---

Published in Transactions on Machine Learning Research (01/2025)
effective platform for testing RL algorithm performance. These environments enable episodic replay and efficient exploration, mitigaling many of the challenges associated with real-world applications.
Among various game simulators, MuJoCo (Todorov et al. 2012) has been extensively employed to evaluate the performance of ICRL in terms of recovering location constraints in robot control tasks (Malik et all 2021 Liu et al. 2023 Qiao al.) (023)). For instance, if an agent observes that certain locations are never visited by expert agents, it can reasonably infer that these locations are likely to be unsafe. To construct proper virtual environments for validating ICRL algorithms, Liu et al. (2023 modified some popular Mu- JoCo environments (including Half-cheetah, Ant, Inverted Pendulum, Walker, Hopper and Swimmer, see Figure 8 by incorporating some predefined constraints into each environment. Table summarizes the environment settings. The constraints are added to the X-coordinate, moving velocity, and angular velocity of the robot body under different controlling tasks. During the evaluation, instead of directly observing these added constraints, the agent has access to the expert demonstrations and the goal to recover these constraints. Within these environments, MEnt-ICRL (Malik al. 2021) was the first propose inferring constraints represented by a neural network to accommodate continuous features. Subsequent studies, in- cluding VICRL (Liu et al., 2023), UA-ICRL (Xu & Liu 2024a), and AR-ICRL (Xu & Liu 2024b), have explored stochastic dynamics by introducing noise into the transition functions of the environments. Specif- ically, in environments such as Blocked Half-Cheetah, Blocked Ant, and Crippled Walker, Xu & Liu (024b) examined various types of noise, including fully random noise, partially random noise, and adversarial noise. Utilizing a robust optimization framework, AR-ICRL has demonstrated superior performance under these diverse noise conditions.
Figure 8: Mujoco environments. From left to right, the environments Half-choetah, Ant, Inverted
Pendulum, Walker, and Swimmer.
Table 3: The virtual environments with different types of constraints.
Name
Constraint Types
Obs. Dim.
Act. Dim.
Constraints
Position Blocked chef-cheetah
Spatial Constraint
Coordinate 2
Position- Blocked Ant
Peeition-Blocked
Spatial Constraint
113
8
X- C.orrdinaee 2 -3
Leg-Blocked Ant
Kinematic Constraint
113
Leg Angular Velocity
Limited-Speed Ant
Dynamic Constraint
113
Moving Speed 0.5
Position- Biased Pendulum
Spatial Constraint
X X-Coordinate 2 -0.015
Blockeed Walker
Spatial Constraint
18
6
X-Coordinate 2 -3
Crippled Walker
Kinematic Constraint
18
6
ThTghhh Angle]| s 0.6
Limited-Speed Walker
Dynamic Constraint
18
Moving Speed
Position-Blocked Swimmer
Spatial Constraint
10
2
X- X-Coordinate x 0.5
Leg-Blan Blg-Blocked Hopper
Kinematic Constraint
Leg Angular Velocity S 0.3
Realistic Environment. Realistic environments denote RL environments whose dynamics are grounded by real-world datasets. Under this setting, Liu et al. (2023) formulated a. Highway Driving (HighD) environment (Figure g The key objective of the HighD environment was to investigate whether an agent capable of inferring the constraints adhered to by human drivers and safely navigating self-driving car, referred to the 'ego car', to its destination.
Specifically, Liu et al. (2023) utilized a HighD dataset (Krajewski et al. 2018) that records naturalistic vehicle trajectories from German highways. Within each scenario, HighD contains information about the static background (e.g., the shape and the length of highways), the vehicles, and their trajectories. The HighD environment control in this scenario. The game context, which constructed by following the background and the trajectories of other vehicles, reflects the driving environment in real life. The observed features are collected
constructed by randomly selecting a scenario from the dataset and an ego car for
25

---

Published in Transactions on Machine Learning Research (01/2025)
Observation Radius
Destination
600
Figure 9: (Figure in (Liu et al. 2023 The Highway Driving (HighD) environment. The ego car is blue, other cars are red. The ego car can only observe the things within the region (marked by blue). The goal is to drive the ego car to the destination (in yellow) without going off-road, colliding with other cars, violating time limits and other constraints (e.g., speed and distance to other vehicles).
by the Road-RR, toolkit. (Wang et al. 2021 Table A summarized the studied constraints, including a car speed constraint and a car distance constraint which ensures the ego car can drive a safe speed and keep a proper distance from other vehicles.
Table 4: The raints for realistic environments (Table 3 in (Liu et al. 2023
Type
Name
Constraint Types
Obs. Dim.
Act. Dim.
Constraints
HighD Velocity Constraint
Dynamic Constraint
76
2
Car Velocity I 40 m/s
Realistic
HighD Distance Constraint
Dynamic Constraint
76
2
Car Distance 2 20 m
9.2 Applications
In this section, we introduce the potential applications of TCRL in solving practical problems.
Autonomous Driving. Designing autonomous agents to control vehicles open roads is challenging task, particularly when considering the long-tail safety-critical events (Yan et al. 2023). Learning policy that can develop a secure driving strategy across diverse driving scenarios is difficult without explicitly modeling safety constraints. However, the optimal constraints for autonomous driving should be context-senne senssiivve and attuned to driving behaviors, aspects often unknown in real-world applications. On the other hand, there has been a significant release of high-quality, open- open-source datasets that document naturalistic vehicle trajectories in various key driving scenarios. These include highways (Krajewski et al. 2018), intersections (Bock et al. 2020 Zhan et al.] 2019), roundabouts (Krajewski et al.) 2020), and entrances and exits of highways (Moers et al. 2022). These recorded trajectories serve as a testament to the expertise of human driving behaviors. As such, the application of ICRL to infer constraints from human demonstrations becomes an essential aspect of this field.
Embodied AI. Embodied AT refers to artificial intelligence systems that interact with the physical world through sensors and actuators, enabling them to perform tasks in real-world environments. As crucial component of embodied AI, generalizable policy learning represents significant area of interest within the robotics and artificial intelligence communities. Many real-world robotic applications require safety guarantees during the design of control policies, which not only optimize task performance but also adhere to some implicit constraints imposed by safety, ethics, or operational requirements, such as safe navigation, human-robot interaction, and robot manipulation (Dulac-Arnold et al.) 2021 Brunke et al.] 2022 Zhao
et al. 2023). While tasks in robotics typically have specific objectives, such as reaching a destination or handling an item, their underlying constraints often remain ambiguous. For instance, a robot may need keep certain distances from people and objects to avoid areas that are not explicitly defined. In these cases, algorithms must infer these unknown constraints by obscrving expert demonstrations and exploring the environment with a known reward function. An immediate approach to accomplishing this task is ICRL, which infers these constraints and learns a policy that adheres to them, thereby enabling robots to align their actions with desired behaviors, ensuring safer and more efficient interactions across different environments. For example, Palafox et al. (2023) proposed a name-theorete approach to multi-robot collision avoidance, utilizing rotating hyperplane constraints that are learned from expert demonstrations.
to maintain specific poses (i.e., natural motions (Hansen et al., 2024() during operation
26

---

Published in Transactions on Machine Learning Research (01/2025)
Decision Making in Healthcare. Recent advancements in AI healthcare have explored the application of RI. to develop AT assistants for diagnosis and disease treatment, (Yu et al. 2021). However, the policies derived from these applications can sometimes lead to unsafe behaviors, such as administering excessive drug dosages, making inappropriate adjustments to medical parameters, or implementing abrupt changes in medication dosages. To learn safe decisions, a common method for learning sale policies is CRL, but the success of CRL significantly relies on the accurate and trustworthy representation of constraints. However, in healthcare, designing the constraints based solely on prior knowledge is challenging, The effectiveness of many healthcare applications depends on integrating the underlying experience of medical experts. In this context, ICRL provides a promising method for extracting constraints from expert demonstration data thereby inducing more reliable decision-making systems in Healthcare.
Sports Analytics. In professional team sports, winning the game often involves complex interactions between multiple players who execute a series of movements within a confined play court and limited playtime. This scenario shares numerous similarities with the propertics of MDPs, making RL a compatible method for modeling the behavior of professional athletes. Several previous studies (Decroos et al. 2019 Liu & Schulte, 2018 Liu et al. 2020) have leveraged policy evaluation algorithms to assess the individual contributions of cach player towards maximizing specific rewards, such as the probability of winning the game. However, in real-world situations, players often prioritize their short-term preferences over the specified long-terny tangtermm goals. Accordingly, a significant challenge in sports analytics is to gain an understanding of and provide explanations for the behavior of these professional athletes, including the motivations behind their movements (Albert) et al. 2017). A previous study (Luo et al. 2020) expanded on MEnt IRL to infer the rewards function. With this foundation, ICRL algorithms to. For precise inference of constraints, it's crucial for the algorithm to account for both the cooperative behaviors of teammates and the competitive actions of opponents. As such, sports analytics can serve as a significant application for assessing the efficacy of multi-agent ICRL algorithms.
be applied to infer the constraints that professional players adhere
10 Conclusion
As an important research lopic for enabling safe control, ICRL has received considerable atlention in recent years. This paper systematically defines the problem of ICRL while distinguishing it from related research topics and summarizes the recent progress in conducting ICRL under discrete and continuous environments, from a limited demonstration dataset and multiple expert agents. We introduce the benchmark and potential applications of TCRL. To facilitate future research, we outline some open questions regarding ICRL:
10.1 Open Questions in Short- Term
While these open questions remain unexplored, we can still refer to a variety of established techniques, methods, and methodological frameworks for guidance. These encompass the following subjects:
Gamo-Thcoretic Multi-Agent Constraint Inference. In the process of inferring constraints from multi-agent environment, prior methodologies (Lu & Zhu 2022) typically assume cooperative scenario where each agent adheres to unique individual constraints. Furthermore, the behavior satisfying one agent's 's constraints is presumed not to conflict with the actions of others. However, in real-world applications, such as autonomous driving in open traffic, drivers often engage in competitive interactions, particularly under conditions of heavy traffic (Ding et al.] 2023). The movement of one vehicle can significantly affect the likelihood of other vehicles reaching their distance constraints. In such circumstances, the task of constructing a tameeeheo model to capture the competitive behavior of agents and determining the potential equilibrium among the policies (i.e., strategies) of different agents is an important direction of future work. A potential solution involves extending existing RL solvers from general-sum games (Bai et al. 2020 Song et al. 2022 to ICRL scenarios. However, these methods primarily focus on the forward control problem. Incorporating them into inverse constraint. inference remains a substantial challenge.
Inverse Constrained Learning from Offline Dataset. Traditional ICRL algorithms typically adopt an online learning paradigm, wherein the agent iteratively amasses experience through interaction with the environment and leverages that experience to infer constraints and update its policy. However, in numerous
27

---

Published in Transactions on Machine Learning Research (01/2025)
scenarios, online interaction may be impractical, primarily due to the high cost of data collection (for instance, in robotics, educational agents, or healthcare), or potential risks (such as in autonomous driving or healthcare) (Levine et al. 2020). Conversely, many real-world datasets are already replete with rich experionces from both expert and sub-optimal agents. Consequently, a more pragmatic approach in ICRL involves learning constraints from offline datasels, independent of environmental interactions. In this context, Quan et al.] (2024) introduced an offline ICRL algorithm for learning hard constraints while the problem of inferring soft constraints remains unresolved. Besides, empirical results indicate that performance is unstable and highly sensitive to the choice of hyperparameters and the underlying distance metric (e.g.. the proportion of expert, but sub-optimal controlling trajectories significantly impacts model performance. Developing a robust. method to reliably infer constraints continues to pose a significant challenge in offline ICRL. Potential solutions include leveraging more stable and robust methods from offline inverse RL (uu ct al. 2017 Zeng ct al. 2023) for learning constraints.
Dj in Equation 55p. Additionally, the composition of offline constraint-violating and constraint-satisfyii
Inferring Generalizable Constraints. In real-world scenarios, the dynamics of deploying environments can be disrupted by unforeseen noise umpredictable uncertainties inherent to the system. Without consid- cring these disturbances, the ICRL algorithm's 's reliability could be significantly compromised, particularly in safety-critical applications such as autonomous driving or robotic surgery. These disturbances can greatly vary across different systems. For instance, in the task of highway merging (refer to Figure [ systems from different locations may be influenced by diverse disturbances stemming from weather conditions, tempera- ture variations, and local driving styles. To handle these challenges, Xu & Liu (2024b) devised an ICRL algorithm under the robust optimization framework (Ben-Tal et al.] 2009). However, this model primarily focuses the mismatched transition functions between the training and deployment environments. It does not account for other critical factors, such as mismatched rewards and the resulting changes in occupancy measures, which are also crucial for comprehensive robustness assessment. To solve these questions, it is significant to generalize the existing robust RL (Wang & Zou 2021 Panaganti et al.] 2022 Wang et al. 2024a) and inverse RL (Viano et al. 2021 Wei et al. 2023) frameworks to the ICRL setting. More impor- fantly, the generalization of these learned constraints to real-world control scenarios has not been extensively studied. The test beds are mostly based on simulated environments rather than real-world settings. A cru- cial direction for future rescarch is to demonstrate the effectiveness of these methods in real-world control applications, such as facilitating collision-free robot manipulation policies.
Inferring Dynamic Constraints. Previous ICRL studies typically focus on learning static constraints from offline expert demonstrations. However, in the real world, such static constraints may be ineffective because of the dynamic and ever-changing nature of environments. Constraints practical scenarios often depend on timo-sensitive factors such as moving obstacles, interactions with other agents, or shifts in envi- ronmental conditions (e.g., weather or lighting changes). Static constraints fail to capture this variability, which can lead to suboptimal or unsafe behavior when deploying learned policies in dynamic settings. For instance, in autonomous driving, the safe distance between vehicles depends on their relative velocities, traf- fic density, and weather conditions. A static constraint inferred from the offline datasct might indicate a fixed minimum distance, which could be either overly conservative or dangerously lax under specific condi- tions. Similarly, in robotic manipulation, obstacles may move unpredictably, and the robot must adjust its constraints dynamically to avoid collisions while completing task efficiently. Dynamic constraints require more complicated modeling approaches that account for temporal changes and adapt in real-time. Unlike static constraints, they must reflect. the evolving feasibility of actions based on the current state, past. tra- jectory, and expected future conditions. Future work could leverage sequence modeling approaches, such as recurrent neural networks (RNNs) (Medsker et al. 2001] and Transformers (Vaswani et al. 2017 to capture the temporal evolution of constraints. Furthermore, employing online inverse RI. (Self et al. 2020 Lian
et al. 2021 Wang& & Zou 2021 techniques to design online ICRL algorithms capable of iteratively refining constraints based on new observations also serves as a promising research direction.
Constraint Inference for LLMs. The concept of constraint inference, traditionally rooted in RL, has significant potential in aligning large language models (LLMs) with specific ethical, safety, operational requirements. LLMs, such as GPT-style models (Floridi & Chiriatti 2020 Achiam et al.) 2023), generate outputs based on learned patterns in vast corpora of text. However, this capability comes with challenges
28

---

Published in Transactions on Machine Learning Research (01/2025)
in ensuring that their responses adhere to desired constraints, especially in safety- crittcali applications like healthcare and content moderation (Wei et al. 2024a Kumar et al. 2023 Liu et al. 2024). Enforcing proper constraints is a crucial strategy for preventing LLMs from generating harmful content. Recent studies, such as Dai et al. (2024), have explored the use of Lagrange methods to align LLMs with safety requirements. However, instead of relying on a cost model learned from preferences and subsequently used for performing forward-constrained optimization, an intriguing alternative to infer these constraints directly from observed data without preferences. This approach establishes a meaningful connection between reinforcement learning from human feedback (RLHF) and ICRL. By leveraging constraint inference techniques, LLMs can be more effectively aligned with safety and ethical guidelines, ensuring their outputs remain within well-defined and adaptable boundaries.
10.2 Open Questions in Long-Term
Although the methods for realizing these long-term objectives are not fully explored, these issues are critical for the future development of ICRL.
Thcorctical Grounding for ICRL. Many recent works have developed theorctical understandings of IRL, Notably, to handle the identifiability issue, Kim el al.] (2021) studied how identifiability relates to properties of the MDP model and proved necessary and sufficient conditions for identifiability in deterministic MDP with the MEnt-RL objective. Alternatively, Lindner et al. (2022) defined the feasible reward set featuring the region of rewards that can equivalently explain the expert behaviors. A contimuing work by Metelli et al. 2023) formally introduced the problem of estimating the feasible reward set, the corresponding PAC requirement, and the minimax lower bound on the sample complexity.
In the ICRL problems, it is crucial to understand the identifiability of learned constraints and the convergence of inference. Yet, existing research primarily focuses on inferring rewards rather than constraints (Metelli ct al. 2021 Lindner et al. 2022. Metelli et al.] 2023). Contrary to the unconstrained IRL problems, the optimality of ICRL is interconnected with the nature of constraints (including hard, soft, or probabilistic constraints) and the chosen constrained optimization method, such as the Lagrange method or Interior Point Methods. Defining the procise identifiability conditions and the fcasible constraint set can be quite challenging. Recently, an ICRL study {Yue et al. 2024) defined the feasible constraint set for ICRL problems. However, their focus was primarily on simplified discrete environments. Therefore, how to develop novel strategies tailored to the unique intricacies of ICRL still remains a major challenge.
Constraints. Recent advancements in generative diffusion models, such as OpenAT's Sora Google's s Ved have shown remarkable performance in video generation. However, criti- cal challenges remain in applying these token-by-token sequential generators to simulate real-world scenarios. These data-driven simulators often produce videos that violate fundamental physical laws and common rules observed in the real world. Recent advancements in embodied AT simulators have highlighted a critical issuc: without ensuring physical realism, skills learned in simulated environments may not be transferable to realistic settings. This gap underscores the importance of developing simulations that accurately mimic real-world physics to effectively guide robots in practical applications (Hua et al. 2024 Nasiriany et al. 2024 Wang et al. 2024b). To address these challenges, a key direction for future work at TCRL. involves extracting these laws and rules from real-world vidoos by treating them as expert demonstrations and representing the physical rules through constraints. Consequently, developing suitable representation for these constraints and a reliable approach to inferring them from real-world videos presents significant challenges.
Learning
Acknowledgments
This work is supported in part by Shenzhen Fundamental Rescarch Program (General Program) under grant 07202300077114000003
Guangdongg-Seenheer Joint Research Fund under grant 2023A15151 10617, Guangdong Basic and Applied Basic Research Foundation under grant 2024A1515011003 and Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence 012331212210001
httt:://opeeaa..cm////eeee/////
29

---

Published in Transactions on Machine Learning Research (01/2025)
References
Picter Abbeel and Andrew Y. Ng. Apprenticeship learning via. inverse reinforcement learning. In Carla E. Brodley (ed.), International Conference on Machine Learning, (ICML), volume 69, 2004.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Nge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint AXiu:2303.08772 2023.
Jim Albert, Mark Glickman, Tim Swartz, and Rund H Koning, Handbook of Statistical Methods and Analyses in Sports. CRC Press, 2017.
Leopoldo Armesto, Jorren Bosga, Vladimir Tvan, and Sethu Vijayakumar. Efficient learning of constraints and generic null space policies. In IEEE International Conference on Robotics and Automation (ICRA), pp. 1520 1526, 2017.
Mattijs Baert, Pietro Mazzaglia, Sam Leroux, and Pieter Simoens. Maximum causal entropy inverse con- strained reinforcement learning. arXiv preprint 0rXiv:2305.08857 2023.
Yu Bai, Chi Jin, and Tiancheng Yu. Near-optimal reinforcement learning with self-play. In Advances in Newral Information Processing Systems, (NeurIPS), 2020.
Connor Basich, Saaduddin Mahmud, and Shlomo Zilberstein. Learning constraints on autonomous behavior from proactive feedback. In 2023 IEEE/RS] International Conference on Intelligent Robots and Systems (IROS), pp. 3680 3687, 2023.
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning, 109 (4):719-766) 760, 2020.
Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust oplimizalion, volume 28. Princeton university press, 2009.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. In Advances in Newral Information Processing Systems (NewrIPS), 2021.
Michael Bloem and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. In IEEE Conference on Decision and Control, CDC, pp. 4911 4916. IEEE, 2014.
Julian Bock, Robert Krajewski, Tobias Moors, Steffen Runde, Lennart Vater, and Lutz Eckstoin. The ind dataset: drone dataset of naturalistic road user trajectories at german intersections. In 2020 IEEE Intelligent Vehicles Symposium (TV), pp. 1929-1934, 2020. doi: 10100//V7440..222000000
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning- beasengg-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:411-444, 2022.
Glen Chou, Dmitry Berenson, and Necmiye Ozay. Learning constraints from demonstrations. In Workshop on the Algorithmic Foundations of Robotics, WAFR 2018, volume 14, pp. 228-245, 2018.
Glen Chou, Neemiye Ozay, and Dmitry Berenson. Learning parametric constraints in high dimensions from demonstrations. In Conference on Robol Learning (CoRL), pp. 1211-12 1230, 2019.
Glen Chou, Dmitry Berenson, and Necmiye Ozay. Uncertainty-aware constraint learning for adaptive safe motion planning from demonstrations. In Conference on Robot Learning (CoRL), pp. 1612 1639, 2020.
Josef Dai, Xuchai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: safe reinforcement learning from human feedback. In International Conference on Learning Representations, 2024.
30

---

Published in Transactions on Machine Learning Research (01/2025)
Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. Actions speak louder than goals: Valuing player actions in soccer. In The 25th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 1851-1861, 2019.
Zhihong Deng, Zuyue Fu, Lingxiao Wang, Zhuoran Yang, Chenjia Bai, Tianyi Zhou, Zhaoran Wang, and Jing Jiang. Falsc correlation reduction for offiine reinforcement learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Fan Ding and Yexiang Xue. X-men: guarantoed xor-maximum entropy constrained inverse reinforcement learning. In Uncerlainly in Artificial Inlelligence, pp- 589 589-598, 2022.
Wenhao Ding, Chejian Xu, Mansur Arief, Haohong Lin, Bo Li, and Ding Zhao A survey on safety-crilical driving scenario generation A methodological perspective. IEEE Transactions on Intelligent Transporta- tion Systems(TIPS). 2023.
Priya L. Donti, David Rolnick, and J. Zico Kolter. DC3: A learning method for optimization with hard constraints. In International Conference on Learning Represenlations, ICLR, 2021.
Gabriel Dulac-Aroodd Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110(9))2419-2444 2021.
Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-694 -64, 2020.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. CoRR, abs/1710.11248, 2017.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q lcarning for imitation. In Neural Information Processing Systems (Neurips), pp. 4028-4039, 2021.
Ashish Gaurav, Kasra Rezaee, Guiliang Liu, and Pascal Poupart. Learning soft constraints from constrained expert demonstrations. In International Conference on Learning Representations (ICLR), 2023.
Aric Glazier, Andrea Loreggia, Nicholas Mattei, Taher Rahgooy, Francesca Rossi, and Kristen Brent Ven- able. Making human-like trade-ofTs constrained environments by learning from demonstrations. CoRR, abs/2109.11018, 2021.
Adam Gleave and Sam Toyer. A primer on maximum causal entropy inverse reinforcement learning. CoRR, htt//2033.14009) 2022.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning (ICML), pp. 1352-1361, 2017.
Tuomas Haarnoja, Aurick Zhou, Picter Abbecl, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning ICML, volume 80, pp. 1856-1865, 2018.
Nicklas Hansen, Jyothir SV, Vlad Sobal, Yann LeCun, Xiaolong Wang, and Hao Su. Hierarchical world models as visual whole-body humanoid controllers. arXiv preprint orXiw:2405. 18418, 2024.
Pu Hua, Minghuan Liu, Annabella Macaluso, Yunfeng Lin, Weinan Zhang, Huazhe Xu, and Lirui Wang. Gensim2: Scaling robot data generation with multi-modal and reasoning 1mss. arXiv preprint 10t:22600.06455 2024.
Adriana Hugessen, Harley Wiltzer, and Glen Berseth. Simplifying constraint inference with inverse rein- forcement learning. In Advances in Neural Information Processing Systems, 2024.
Jachwi Jang, Minjae Song, and Dachyung Park. Inverse constraint learning and generalization by transferable reward decomposition. IEFF Robolács and Automation Letters, 9(1):279 2266, 2023.
31

---

Published in Transactions on Machine Learning Research (01/2025)
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rI? In International Conference on Machine Learning, pp. 5084-5090 2021.
Weonyoung Joo, Wonsung Lee, Sungrae Park, and Il-Chul Moon. Dirichlet variational autoencoder. Pattern Recognit., 107:107514, 2020.
Seksan Kiatsupaibul, Robert L. Smith, and Zelda B. Zabinsky. An analysis of a variation of hit-and-run for uniform sampling from general regions. ACM Trans. Model. Comput. Simul., 21(3):16:1116::) 2011.
Konwoo Kim, Gokul Swamy, Zuxin Liu, Ding Zhao, Sanjiban Choudhury, and Steven Z Wu. Learning shared safcty constraints from multi-task demonstrations. Advances in Neural Information Processing Systems, 2023.
Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in inverse reinforcement learning. In International Conference on Machine Learning, 2021.
Robert Krajewski, Julian Bock, Laurent Kloeker, and Lutz Eckstein. The highd dataset: A drone datasel of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems. In International Conference on Intelligent Transportation Systems (ITSC), pp. 2118-2125, 2018.
Robert Krajewski, Tobias Moers, Julian Bock, Lennart Vater, and Lutz Eckstein. The round dataset: A drone dataset of road user trajectories at roundabouts in germany. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1-6, 2020.
Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. Certifying 1lm safety against adversarial prompting. arXiv preprint arXiv:2309.0270 027005 2023.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Newral Information Processing Systems, 191179-1191 2020.
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Fung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning (ICML), pp. 6120-6130, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXie preprinl 2rXiv:2005.01443 2020.
Changshuo Li and Dmitry Berenson. Learning object orientation constraints and guiding constraints for narrow passages from one demonstration. In International Symposium on Experimental Robotics (ISER), volume 1, pp. 197-210, 2016.
Quanyi Li, Zhenghao Peng, and Bolei Zhou. Efficient learning of safe driving policy via human-ai copilot optimization. In International Conference on Learning Representations (ICLR), 2022.
Yinchuan Li, Shuang Luo, Haozhi Wang, and Jianye Hao. Cflownets: Continuous control with generative flow networks. In International Conference on Learning Represenlations (ICLR), 2023.
Bosen Lian, Wenqian Xuc, Frank Lewis, and Tianyou Chai. Online inverse reinforcement learning for nonlinear systems with adversarial attacks. International Journal of Robust and Nonlinear Control, 31 (14)66646-6677, 2021.
David Lindner, Andreas Krause, and Giorgia Ramponi. Active exploration for inverse reinforcement learning. In NewrIPS, 2022.
David Lindner, Xin Chen, Sebastian Tschiatschek, Katja Hofmann, and Andreas Krause. Learning safety constraints from demonstrations with unknown rewards. In International Conference on Arlificial Intelli- gence and Statistics, pp. 2386-238 2394, 2024.
Guiliang Liu and Oliver Schulte. Deep reinforcement learning ice hockey for context-aware player evalu- ation. In International Joint Conference on Artificial Intellggecce (IJCAT), pp. 3442 3448, 2018.
32

---

Published in Transactions on Machine Learning Research (01/2025)
Guiliang Liu, Yudong Luo, Oliver Schulte, and Tarak Kharrat. Deep soccer analytics: learning action- value function for evaluating soccer players. Dala Mining Knowledge Discovery, 34(5):1531-155, 2020.
Guiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezaee, and Pascal Poupart. Benchmarking constraint inference in inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2023.
Shicheng Liu and Minghui Zhu. Distributed inverse constrained reinforcement learning for multi-agent systems. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Shicheng Liu and Minghui Zhu. Learning multi-agent behaviors from distributed and streaming demonstra- tions. In Advances in Neural Information Processing Systems (NeurIPS), 2023a.
Shicheng Liu and Minghui Zhu. Mela inverse constrained reinforcement learning: Convergence guaranlee and generalization analysis. In International Conference on Learning Representations (ICLR), 2023b.
Zixman Liu, Xiaolin Sun, and Zizhan Zheng. Enhancing 11m safety via. constrained direct preference opti- mization. arXiv preprint ar arXiv:2408. Fft:2408.02478 2024.
Yudong Luo, Oliver Schulte, and Pascal Poupart. Inverse reinforcement learning for team sports: Valuing actions and players. In International Joint Conference on Artificial Intelligence (IJCAI), pp. 3356 0356-3663, 2020.
Shchryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International Conference on Machine Learning (ICML), pp. 7390 7399, 2021.
David Livingston McPherson, Kaylene C. Stocking, and S. Shankar Sastry. Maximum likelihood constraint inference from stochastic demonstrations. IEEE Conference on Control Technology and Applications (CCTA), pp. 1208-1213, 2021.
Larry R Medsker, Lakhmii Jain, et al. Recurrent neural networks. Design and Applications, 5664667) 2001.
Negar Mehr, Roberto Horowitz, and Anca D. Dragan. Inferring and assisting with constraints in shared autonomy. In IEEE Conference on Decision and Control (CDC), pp. 6689 6696. IEEE, 2016.
Alberto Maria Metelli, Giorgia Ramponi, Alessandro Concetti, and Marcello Restelli. Provably efficient learning of transferable rewards. In International Conference on Machine Learning, 1CML, volume 139, pp. 7665-7676, 2021.
Alberto Maria Metelli, Filippo Lazzati, and Marcello Restelli. Towards theoretical understanding of inverse reinforcement learning. arXiv preprint arXiv:2304.1200 2023.
Tobias Moers, Lennart Vater, Robert Krajewski, Julian Bock, Adrian Zlocki, and Lutz Eckstein. The exid datasct: A real-world trajectory dataset of highly interactive highway scenarios in germany. In 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 958 08-964, 2022.
Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. arXiv preprint Xrx:::2406.0 02523, 2024.
Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning, In International Conference on Machine Learning, TCML, pp. 663-670,44 2000.
Fernando Palafox, Yue Yu, and David -elll Learning hyperplanes for multi-agent collision avoid- ance in space. arXiv preprint 2=Xix:2311.09450 2023.
Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learn- ing using offline data. Advances in neural information processing systems, 35:32211 1022113310 -22244, 2022.
33

---

Published in Transactions on Machine Learning Research (01/2025)
Dimitris Papadimitrion and Daniel S. Brown. Bayesian constraint inference from user demonstrations based margin-respecling preference models. In IEEE International Conference on Robolics and Aulomalion, ICRA, pp. 15039 15046. IEEE, 2024.
Dimitris Papadimitrion and Jingqi Li. Constraint inference in control tasks from expert demonstrations via inverse optimization. CoRR, hbs//2344033377 2023.
Dimitris Papadimitriou, Usman Anwar, and Daniel S Brown. Bayesian methods for constraint inference in reinforcement learning. Transactions Machine Learning Research, 2023.
George Papamakarios, Tain Murray, and Theo Pavlakou. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems (NewrIPS), pp. 2338 2347, 2017.
Daehyung Park, Michael D. Noseworthy, Rohan Paul, Subhro Roy, and Nicholas Roy. Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning. In Conference on Robot Learning, CoRL, volume 100, pp. 1005-1014. 2019.
Santiago Paternain, Luiz F. O. Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained re- inforcement learning has zero duality gap. In Advances in Neural Information Processing Systems, pp- 7553 7563, 2019.
Baiyu Peng and Aude Billard. Positive-umlabeled constraint learning (pucl) for inferring nonlinear continuous constraints functions from expert demonstrations. arXiv preprint arXiv:2408.01622 2024.
Claudia Pérez-D'Arpae D'Arino and Julie A Shah. C-learn: Learning geometric constraints from demonstrations for multi-step manipulation in shared autonomy. In IEEE International Conference on Robotics and Aulomation (ICRA), pp. 4058 4065, 2017.
Guanren Qiao, Guiliang Liu, Pascal Poupart, and Zhiqiang xu. Multi-modal inverse constrained reinforce- ment learning from a mixture of demonstrations. In Advances in Neurul Information Processing Systems (NeurIPS), 2023.
Guorui Quan, Guiliang Liu, et al. Learning constraints from offline demonstrations via superior distribution correction estimation. International Conference on Machine Learning ( (ICML), 2024.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In International Joint Conference on Artificial Intelligence, IJCAI, pp. 2586-2591, 2007.
Titouan Renard. Provable convergence guarantees for constrained inverse reinforcement learning, 2023.
Stuart Russell and Andrew Zimdars. Q-decomposition for reinforcement learning agents. In International Conference on Machine Learning, pp. 656 663, 2003.
Andreas Schlaginhaufen and Maryam Kamgarpour. Identifiability and generalizability in constrained inverse reinforcement learning. In International Conference on Machine Learning, ICML, volume 202, pp. 30224 30251, 2023.
Dexter R. R. Scobee and S. Shankar Sastry. Maximum likelihood constraint inference for inverse reinforce- ment learning. In International Conference on Learning Representations (ICLR), 2020.
Ryan Self, Moad Abudia, and Rushikesh Kamalapurkar. Online inverse reinforcement learning for systems with disturbances. In American control conference (ACC), pp. 1118 1123, 2020.
Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual RL: Unification and methods for reinforcement and imitation learning. In International Conference on Learning Representations, ICLR, 2024.
Herbert A Simon. Bounded rationality. Utility and probebility, pp. 15 15-18, 1990.
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. The Conference on Uncertainly in Artificial Intelligence. (UAT), pp. 560-569, 2018.
34

---

Published in Transactions on Machine Learning Research (01/2025)
Ziang Song, Song Mei, and Yu Bai. When can learn general-sum markov games with large number of players sample-efficinttly? In International Conference on Learning Representations, (ICI.R), 2022.
Jonathan Spencer, Sanjiban Choudhury, Matthew Barnes, Matthew Schmittle, Mung Chiang, Peter Ra- madge, and Sidd Srinivasa. Expert intervention learning: An online framework for robot learning from explicit and implicit humaan leedback. Aulonomous Robols, pp. 15, 2022.
Nitish Srivastava, GeolTrey E. Hinton, Alex Krizhevsky, Πya Sulskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 15(1): 1929-1958, 2014.
Sriram Ganapathi Subramanian, Guiliang Liu, Mohammed Elmahgiubi, Kasra Rezaee, and Pascal Poupart. Confidence aware inverse constrained reinforcement learning. In International Conference on Machine Learning (ICML), 2024.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems IROS, pp. 5026-5033. IEEE, 2012.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In International Conference on Machine Learning (ICML), pp. 9690- 9700, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnell (eds.), Advunces in Neural Information Processing Systems, pp. 5998 6008, 2017.
Marcell Vazquez-Chanlatee Susmit Jha, Ashish Tiwari, Mark K. Ho, and Sanjit A. Seshia. Learning task specifications from demonstrations. In Advances in Neural Information Processing Systems (NeurIPS), pp. 3372-5382, 2018.
Luca Viano, Yu-Tin Huang, Parameswaran Kamalaruban, Adrian Weller, and Volkan Cevher. Robust inverse reinforcement learning under transition dynamics mismatch. In Neural Information Processing Systems (Neurips), 2021.
Xiao Wang, Hanna Krasowski, and Matthias Althoff. Commonroad-rl: A configurable reinforcement learning environment for motion planning of autonomous vehicles. Tn 24th IEEE International Intelligent Trans- portation Systems Conference, 1TSC 2021, Indianapolis, IN, USA, September 19-22, 2021, pp. 466 472. IEEE, 2021.
Yue Wang and Shaofeng Zou. Online robust reinforcement learning with model uncertainty. Advances in Newral Information Processing Systems, 34:7193-722 12299,,-,6,--- 2021.
Yue Wang, Alvaro Velasquez, George Atia, Ashley Prater-Bennette, and Shaofeng Zou. Robust. average- reward reinforcement learning. Journal of Artificial Intelligence Research, 80:719 803, 2024a.
Yufei Wang, Zhou Xian, Feng Chen, Tsun-HHsua Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. In International Conference Machine Learning, ICML, 2024b.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does 1lm safety training fail?
Advances in Newral Information Processing Systems, 36, 2024a.
Qinglai Wei, Tao Li, Jie Zhang, Hongyang Li, Xin Wang, and Jun Xiao. Isoperimetric constraint inference for discreto-time nonlinear systems based on inverse optimal control. IEEE Transactions on Cybernetics, 2024b.
35

---

Published in Transactions on Machine Learning Research (01/2025)
Ran Wei, Siliang Zeng, Chenliang Li, Alfredo Garcia, Anthony D McDonald, and Mingyi Hong. A bayesian approach to robust inverse reinforcement learning. In Conference Robot. Learning, pp- 2304-2322, 2023.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for
offline reinforcement learning. Advances in Neural Information Processing Systems, 35:31278 35:31278-31291, 2022.
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman--onsistent pessimism for offline reinforcement learning. Advances in Neurol Information Processing Systems, 34:6683 6694, 2021.
Sheng Xu and Guiliang Liu. Uncertainty-aware constraint inference in inverse constrained reinforcement learning. In International Conference on Learning Representations (ICLR), 2024a.
Sheng Xu and Guiliang Liu. Robust inverse constrained reinforcement learning under model misspecification. In International Conference on Machine Learning, ICML, 2024b.
Xintao Yan, Zhengxia Zou, Shuo Feng, Haojie Zhu, Haowei Sun, and Henry X Liu. Learning naturalistic driving environment with statistical realism. Nature Communications, 14(1):2037, 2023.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey.
ACM Computing Surveys (CSUR), 55(1):1 1-13,, 2021.
Bo Yue, Jian Li, and Guiliang Liu. Provably efficient exploration in inverse constrained reinforcement learning. arviv preprint arXiv:2409.15963 2024.
Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. When demonstrations meet generative world models: A maximum likelihood framework for offline inverse reinforcement learning. Advonces in Newral Information Processing Systems, 36:65531-65565, 2023.
Wei Zhan, Liting Sun, Di Wang, Haojic Shi, Aubrey Clausse, Maximilian Naumann, Julius Kümmerle, Hendrik Königshof, Christoph Stiller, Arnaud de La Fortelle, and Masayoshi Tomizuka. INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Sce- narios with Semantic Maps. arXiv: 1900. 110000000000088 fes, cess], September 2019.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pp. 321 384, 2021.
Weiye Zhao, Rui Chen, Yifan Sun, Ruixuan Liu, Tianhao Wei, and Changliu Liu. Guard: A safe reinforce- ment learning benchmark. arXiv preprint arXiv:2305.1362 2023.
Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang, Diffusion models for reinforcement learning: A survey. CoRR, abs/ Abs/2311.01223, 2023.
Brian D. Ziebart, Andrew Maas, J. Andrew Baguell, and Anind Dey. Maximum entropy inverse reinforcement learning. In AAAT Conference on Artificial Intelligence (AAAT), pp. 1433-1438, 2008
Brian D. Ziebart, J. Andrew Bagnell, and Anind K. Dey. Modeling interaction via the principle of maximum causal entropy. In International Conference on Machine Learning (ICML), pp. 1255-1262. 2010.
36


Reviews:
[{"review_id": "2lM4uAKp6Z", "reviewer": "Reviewer_nG5E", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "My expertise does not include inverse constrained reinforcement learning (ICRL), so I approached this manuscript with the intent to learn more about the topic. However, I found the paper challenging to follow for a reader without a background in ICRL. Firstly, it seems that this paper is based on a finite-horizon MDP, which the authors should make clear. There are also other settings, such as infinite-horizon MDPs and POMDPs, which should ideally be discussed in a review. This led me to wonder: is ICRL only applicable to finite-horizon MDPs?\n\nFurthermore, I noticed that the optimization problem in equation (1) includes a policy entropy term, which is not that common in general reinforcement learning problems, and no reference was provided by the authors. Has this problem formulation become a standard paradigm in ICRL? Although the authors discuss the benefits later, they do not directly address this question. For these two fundamental points, I was unable to find answers in the manuscript. This indicates that the survey paper does not extend the discussion but simply summarizes the current work.\n\nThe survey paper's structure also makes it difficult for readers to follow. The transitions between sections are weak, and the connections between different parts are not well-established, making the overall flow feel disjointed. More improvment to writing quality and structural coherence is needed. For instance, while Figure 3 summarizes the main procedure of ICRL, I could not find an introduction to the mathematical model of ICRL. What is the core optimization problem that ICRL aims to solve?\n\nStarting from Section 4, titled \"Constraint Inference in the Deterministic Environment,\" the writing becomes increasingly challenging for readers, with extensive explanations lacking sufficient elaboration. The taxonomy of different categories is also not clearly explained or compared. I hope the authors can restructure the paper to improve clarity and cohesion.\n\nThe paper does have its strengths. It introduces an emerging topic that is gradually gaining attention in the academic community, which makes it a timely and relevant contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper surveys the latest advances in ICRL, providing definitions, key challenges, and applications in areas like autonomous driving, robot control, and sports analytics, aiming to bridge theoretical understanding with practical applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I request the authors can restructure the paper to improve clarity and cohesion, add a mathmatical problem fomulation of ICRL.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d4iYKXhnh8", "reviewer": "Reviewer_PDYr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper covers a wide range of ICRL methodologies, making it a valuable resource for researchers new to the field.\n- The clear categorization of approaches based on environments and constraints helps build a structured understanding of ICRL applications.\n- I appreciated the discussion on open questions, which sheds light on both short-term and long-term research directions for the community.\n\n**Weaknesses**\n\n- While the survey categorizes the methodologies effectively, it could benefit from a more analytical comparison of these approaches, highlighting their relative strengths and weaknesses.\n- Some of the challenges and limitations mentioned are well-known in the broader reinforcement learning literature and aren't unique to ICRL. Including relevant references to existing discussions or explaining why current solutions aren't applicable would strengthen the paper.\n- A minor point: some terminology is used inconsistently across sections—for example, \"permissibility functions\"—which might confuse readers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper offers a comprehensive survey of recent developments in Inverse Constrained Reinforcement Learning (ICRL), focusing on how implicit constraints can be inferred from expert demonstrations. The authors begin by establishing foundational definitions and draw comparisons between ICRL and related fields like Inverse Reinforcement Learning (IRL) and Inverse Optimal Control (IOC). They systematically categorize ICRL methods across various settings, including deterministic and stochastic environments, multi-agent scenarios, and cases with limited or partial demonstrations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. At the end of Section 2.4, could you clarify whether \"Interpretable Constraints\" refers to a regularization method? Is φ(s, a) a learned model or a fixed function? How does learning $\\phi(s, a)$ differ from learning $c(s, a)$, and how does this relate to interpretability?\n2. In Section 3.2, you mention that \"ICRL can incorporate recent advancements in RL to learn model-free controllers without knowing or estimating the model dynamics (i.e., model-free methods).\" Does this imply that ICRL focuses only on model-free methods, or does it also consider model-based methods?\n3. Please clarify the terminology in Section 4.2. Is $\\phi_w(s, a)$ a permissibility function? Earlier, a permissibility function is defined as \"a function learned to distinguish between feasible and infeasible states within these regions.\" Does the concept of a permissibility function apply only to states, or can it also apply to state-action pairs?\n4. In Section 6, the issue of epistemic uncertainty seems closely related to the false correlation problem in offline RL. It would be beneficial to add a brief discussion and reference to the false correlation problem.\n5. In Section 8, you mention three levels of Multi-Agent ICRL. Could you elaborate on the key differences between the third level and the first two? Do the first two levels consider agents operating sequentially rather than simultaneously?\n6. In Section 10, you discuss open questions in ICRL. It would be helpful to cite some papers that have attempted to address these questions, even if they aren't specific to ICRL, since these challenges are common in other RL domains.", "broader_impact_concerns": "No concerns about broader impacts.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LxYSzSMdiX", "reviewer": "Reviewer_4Ecj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\nThe paper is well structured. The authors try to present some key details about the setup and implementation of each paper in an organized way. To the best of my knowledge most of the relevant research work has been included. I have included a few additional papers later in my review that the authors might want to consider. The authors also try to shed some light on the possible future directions that might be of interest.\n\n**Weaknesses**:\n* Surveys that also benchmark different methods tend to be even more valuable in my view, although I understand that this is a significantly more involved task. I think that regarding the future direction the authors should try and elaborate a bit more on the \"challenges part\" and possible future research avenues. For example, the case of inferring dynamic constraints (like moving obstacles) could be discussed. Also for example, constraint inference could be potentially utilized in aligning LLMs so that they respect constraints. \n\n* The authors also do not mention anything about the complexity of the methods. In many cases such arguments were not done in the original papers and so it is hard to tabulate complexities for different methods, but I think that some comments can be added regarding that. \n\nSome additional references that you might want to consider are listed at the end.\n\nIn the section \"Inverse Constrained Learning from Offline Dataset\" I believe that except for Qual et al. constraint inference from offline data has been studied in [1].\n\nHere are some typos/rewordings/minor fixes for your convenience.\n\nIn (32) and (41) the integration is over $a$ but a does not appear in the integral.\n\nIn (50) three different datasets appear. What is the relationship of $\\mathcal{D}$ with the others?\n\n\"By utilizing the posterior distribution over constraint sets at previous...\" I think this sentence can be rewritten to be a bit more clear\n\nIn (52) the subscripts in the expectations appear to be in bold for some reason\n\nAlso it would be nice to start a section with text and not with Figures. This happens in section1 and 2.4.\n\nIn (5) maybe include $\\forall i \\in [1,N]$ as in (6)\n\nIn (11) it is not clear which is the original problem of this dual. I assume it is (5)\n\n\"task of inferring the implicit constraints followed by expert agents\" maybe a different word than followed is more appropriate\n\nSometimes when referring to a single person \"and inherent to experts’ own experience\" it should be expert's. This happens a few times in the paper.\n\n\"Discounted factor\" should be Discount factor\n\nThe term nominal is used a few times in the paper but it is not clearly defined somewhere.\n\nwhere $\\omega_m\\sim q(\\omega)$ Specifically, there is a . missing at the end of the sentence.\n\n\"constraints function $c(s,a). i.e. \\pi^{\\epsilon}\\mathbb{E} =$ ...\" avoid starting a sentence with i.e.\n\n\"Within each partition ι, CBN-IRL learns\" in multiple sections in the paper the letter i appears without the dot on top. I think it would be clearer if you used just i. If it is done because you want to use i only for the index of a constraint then it is fine.\n\n\"proposed to learn single reward function\" --> to learn a single\n\n\"here f is implemented by a multi-layer\" $f$ or $f_{\\omega}$?\n\nI would avoid having equations between text, especially when they affect the spacing between lines. In those case just use write them as an equation. See for example below (56). Also when writing an optimization problem use a different line for the constraint (the s.t. part)\n\n\"., locating outside the support of\" -->located\n\nIn (59) there is both $\\pi$ and $\\pi(\\theta)$ appearing. Is this a typo?\n\nIn (61) and a few other places there is no gap between the $argmax$ and the expectation\n\n\"for extracting expert constraints from their demonstration data\" maybe rephrase to constraint from expert demonstration data?\n\n\" independent of environmental interaction.\" --> interactions\n\nsometimes a colon is used to introduce an equation bit sometimes it is missing. (for example (53) and (54)). Also there is no need for a colon to introduce definition 6.1\n\nIn (18) maybe just use $|C|$ for the size of the set\n\n \"Striving for reliable constraint inference, the policy represent\" maybe you meant representation?\n\n \"domains, 2) soft constraints\" instead of , maybe use and\n\n \"In the continuous environment, constructing\"--> environments\n\nIn definition 5.1 is should be $s_T$\n\n\"but the soft constraints requires a threshold\"--> require\n\n\"by expert demonstration is a trivial \"--> demonstrations\n\n\"Lagrange method or Interior Points Method.\"--> Interior Point Methods\n\n \"The parameters of the constraint models $\\omega$ is given by:\"--> are\n\n in page 4 when the cost $c$ is discussed I think it would be clearer if it was written as $c(s,a)$ as it is a function. That appleid for example in table 2 and above eq. (2)\n\nIn (19) the partition should be $Z_c$ or $Z_{c_{\\omega}}$?\n\n\"in RL to learn model-free controller without\"--> a model-free...\n\n\" While studying these topics are not our primary focus, we\"--> is not\n\n\n\n**References**:\n\n[1] D. Papadimitriou et al. \"Bayesian Constraint Inference from User Demonstrations Based on Margin-Respecting Preference Models\", ICRA 2024\n\n[2] C. Basich et al. \"Learning Constraints on Autonomous Behavior from Proactive Feedback\", IROS 2023\n\n[3]. J. Jang et al. \"Inverse Constraint Learning and Generalization by Transferable Reward Decomposition\", RA-L\n\n[4] K. Kim et al. \"Learning Shared Safety Constraints from Multi-task Demonstrations\" NIPS 2023\n\n[5] F. Palafox et al. \"Learning Hyperplanes for Multi-Agent Collision Avoidance in Space\"\n\n[6] Q. Wei et al. \"Isoperimetric Constraint Inference for Discrete-Time Nonlinear Systems Based on Inverse Optimal Control\", TCYB\n\n[7]  F. Ding et al, \"X-MEN: Guaranteed XOR-Maximum Entropy Constrained Inverse Reinforcement Learning\", UAI 2022\n\n[8] B. Peng et al. \"Positive-Unlabeled Constraint Learning (PUCL) for Inferring Nonlinear Continuous Constraints Functions from Expert Demonstrations\", arxiv\n\n[9]  B. Yue et al. \"Provably Efficient Exploration in Inverse Constrained Reinforcement Learning\", arxiv", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents to the best of my knowledge the first survey in the area of constraint inference. The authors discuss recent developments in constraint inference in reinforcement leaning but also in control tasks.The authors provide motivation for the research area and they follow to discuss most of the recent advances. They also discuss some open problems as a possible direction for future research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be great if the authors could address the weaknesses listed above.  Regarding the third bullet point, there is no need for trying to derive computational complexities, but some discussion or table with some comments would be a good addition to the paper.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewer nG5E still has some concerns (around the problem formulation), I believe that the benefits outweigh the downside of publishing the paper. 

Reviewer 4Ecj: "As far as I know this is the first survey paper in the area of constraint inference in RL. I think that the authors presented most of the up to date work in this area. Although I personally prefer surveys that also review the approaches quantitatively, I understand that in this case this might be trickier given that there is high variability in the setups for which each method was developed."

Reviewer PDYr: "I recommend accepting this submission. The authors have made valuable improvements in clarifying key concepts and addressing the reviewer's requests, which improves the paper’s readability and applicability. These updates make the work more accessible to a broader audience, thereby increasing its potential impact as a survey paper."

Reviewer nG5E: "I read the revised paper again. However, I still have some concerns regarding the problem formulation in the ICRL framework, particularly the inclusion of entropy in the objective function. While the manuscript emphasizes the benefits of incorporating entropy, it lacks a fundamental and substantive comparison with approaches that do not include entropy. This comparison is crucial to justify the rationale for using entropy at a deeper, foundational level."
Decision: Accept (poster)