Multi-Modal Inverse Constrained Reinforcement Learning from a Mixture of Demonstrations
Authors: Guanren Qiao
Guiliang Liu
Pascal Poupart
zhiqiang xu
Venue: NeurIPS 2023 poster (2023)
Keywords: Inverse Constrained Reinforcement Learning
Learning from Demonstrations
Muti-Modal Learning
TL;DR: We introduce a Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) algorithm to infer multiple constraints corresponding to various types of agents from a mixture of expert demonstrations.

Abstract: Inverse Constraint Reinforcement Learning (ICRL) aims to recover the underlying constraints respected by expert agents in a data-driven manner. Existing ICRL algorithms typically assume that the demonstration data is generated by a single type of expert. However, in practice, demonstrations often comprise a mixture of trajectories collected from various expert agents respecting different constraints, making it challenging to explain expert behaviors with a unified constraint function. To tackle this issue, we propose a Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) algorithm for simultaneously estimating multiple constraints corresponding to different types of experts. MMICRL constructs a flow-based density estimator that enables unsupervised expert identification from demonstrations, so as to infer the agent-specific constraints. Following these constraints, MMICRL imitates expert policies with a novel multi-modal constrained policy optimization objective that minimizes the agent-conditioned policy entropy and maximizes the unconditioned one. To enhance robustness, we incorporate this objective into the contrastive learning framework. This approach enables imitation policies to capture the diversity of behaviors among expert agents. Extensive experiments in both discrete and continuous environments show that MMICRL outperforms other baselines in terms of constraint recovery and control performance.

Multi-Modal Inverse Constrained Reinforcement Learning from a Mixture of Demonstrations
Guanren Qiao', Guiliang Liu" Pascal Poupart? Zhiqiang Xu
Schooo of Data Science, The Chinese University of Hong Kong, Shenzhen, University of Waterloo, 3 Vector Institute, 'Mohamed bin Zayed University of Artificial Intelligence 223040241llink cuhk edu. en, liugui snnguiliang@euhee edu. cn, ppoupart@uwaterlo ca, zhiqiang. ataigiing.xu@mbzaa xu@mbzuai. ac. ae
Abstract
Inverse Constraint Reinforcement Learning (ICRL) aims recover the underlying constraints respected by expert agents in a data-driven manner. Existing ICRL algorithms typically assume that the demonstration data is generated by single type of expert. However, in practice, demonstrations often comprise mixture of trajectories collected from various expert agents respecting different constraints, making it challenging to explain expert behaviors with a unified constraint function. To tackle this issue, we propose Multi-Modie Inverse Constrained Reinforcement Learning (MMICRL) algorithm for simultaneously estimating multiple constraints corresponding to different types of experts. MMICRI. constructs flow-based den- sity estimator that enables unsupervised expert identification from demonstrations, so as to infer the agent-spece specific constraints. Following these constraints, MMI- CRL imitates expert policies with novel multi-modal constrained policy optimiza- tion objective that minimizes the the unconditioned one. To enhance robustness, incorporate this objective into the contrastive learning framework. This approach enables imitation policies to cap- ture the diversity of behaviors among expert agents. Extensive experiments in both discrete and continuous environments show that MMICRL outperforms other base- lines in terms of constraint recovery and control performance. Our implementation is available at: https://gthhub.com http:///ttbb.oom///iiggaaaemm/nn Mawprree//MMMiiMtoal-lll--eeeeee Moddll-Invers--onsttree Reinforcement-Lcarmingg
policy entropy and maximizes
Introduction
A fundamental prerequisite for achieving safe Reinforcement Learning (RL) is that the agents' policies must adhere to the underlying con- straints in the environment 21 However, in many real-world applications (e.g., robot control and autonomous driving), the ideal constraints tume-varying. contex--dependent, and inher- ent to experts' own experience. These con- straints are hard specify mathematically and may not be readily available to RL agents in policy updates.
Inverse Step:
Forward Stop: Constrained
Agent Specific
Constraint function
Policy Optimization
Constraint inforence
x 21 11, V2 E
#L (2) 45 ( (z)
Identified
trjectories By...
Eapert Data
Unsupevised Agent Identifiration
Density Model Update:
Mudel
Yz
De (2) D2
maximize, (t,1k)
Mixture of Demonstrations Dr
promising approach for learning latent con- straints is Inverse Constraint Reinforcement Learning (ICRL) 3 5. As a data-driven technique, ICRL recovers the underlying constraints
Figure 1: The flowchart of MMICRL.
*Corresponding author: Guiliang Liu, email: Tuguiliangccuhk.cu..co cuh...du.khnnn
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

respected by expert agents from their demonstrations and utilizes these constraints to support down- stream applications. Existing ICRL methods B 4 g commonly assumed that all expert demonstrations follow the same constraints, and they approximated these constraints with a unified constraint model, whereas, in practice, the demonstration data may be collected from various agents, and these agents might follow different or even conflicting constraints. It is problematic to leverage a single constraint model to explain the behaviors of diverse agents. For example, in the context of autonomous driving [2], the vehicle-distance constraints followed by trucks and cars should differ, and misapplying these constraints could lead to serious traffic accidents.
To differentiate expert demonstrations, previous RL approaches [10]
inferred the la- tent structure of expert demonstrations and identified expert agents by examining their behavioral preferences. However, these methods were specifically designed for imitation learning rather than constraint inference. Moreover, as their identification process primarily relies on an agent classifier that evaluates state-action pairs, there is no theoretical guarantee that the optimal model is identifiable.
In this work, we propose the Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) algorithm for estimating agent-specific constraints from a mixture of expert demonstrations (Figure I MMICRL extends the traditional maximum entropy framework IEI with agent-conditioned entropy minimization and unconditioned entropy maximization subject to a permissibility constraint. The resulting policy representation facilitates inferring agent-specific constraints by alternating between the following steps: 1) Unsupervised Agent Identification. MMICRL conducts trajectory- agent identification using flow-based density estimator. The optimal results are identifiable since each agent's ss policy must correspond to unique occupancy measure 115 2) Agent-Spp -peccffc Constraint Inference. Leveraging the identified demonstrations, MMICRL estimates a permissibility function for distinguishing expert demonstrations from sub-optimal ones, based on which we construct constraints for each type of agent. 3) Multi-Modal Policy Optimization. MMICRL measures the accuracy of inferred constraint models by comparing the similarity between expert trajectories and those generated by imitation policies under the inferred constraints. To capture the diversity of behaviors exhibited by multiple agents, we incorporate policy optimization within the contrastive learning framework. We treat the generated trajectories as noisy embeddings of agents, which serve as compact representations of their behaviors. Utilizing the contrastive estimation methods [16), we can enhance the similarity between embeddings for agents of the same type, while simultancously maintaining the distinctiveness of diverse agent types.
We empirically demonstrate the performance of our method by conducting experiments in both dis- crete (e.g.. Gridworld) [7] and continuous environments (e.g.. MuJoCo) [17]. MMICRL significantly surpasses other baselines terms of distinguishing different agents, adhering to the true constraints, and optimizing control performance. To examine the robustness of MMICRL we investigate its ability to recover from incorrect agent identification and subsequently infer the correct constraints.
2 Related Works
Inverse Constrained Reinforcement Learning. Prior ICRL methods typically learned constraints from demonstrations under the maximum entropy framework [1). Some research [88] 19) employed constraints to differentiate between feasible and infeasible state-action pairs in Constrained MDP, but these studies were restricted to inferring discrete constraints in environments with known dynamics. subsequent work 4 extended this approach continuous state-action spaces with unknown transition models by utilizing neural networks to approximate constraints. Inspired by Bayesian g inferred probability distributions over constraints. To better model demonstrations, 6 extended ICRL to infer soft constraints rather than hard ones, and 23 explored ICRL under the multi-agent setting. Striving for efficient comparisons, [7] established an ICRL benchmark across various RL domains, such as Gridworld, robot control, and autonomous driving. However, these algorithms primarily target inferring constraints for a single agent type, without considering the distinct constraints associated with multiple agent types.
Inverse Reinforcement Learning 22 21 22),
Learning from a Mixture of Expert Demonstrations. Multi-task Inverse Reinforcement Learning (IRL) 24
25 aims to leam from mixture of expert demonstrations. Some previous studies
10
26
12 27 utilized the Generative Adversarial Imitation Learning algorithm 28 to model the 11
13 29 learned interpretable representations of behavioral policies and inferred the latent representation of expert demonstrations in an unsupervised way. Specifically, [30] learned a Variational Auto-Encoe Encoder (VAE) [I] where the encoder infers the latent
behaviors of multiple agents. [10

---

factors of variation from mixed demonstrations and the decoder models different types of expert behaviors, To strengthen the consistency between the learned policy and the types of agents, [[2] 32] included "burn-in demonstrations" for updating the imitation policies. These methods, however, were proposed for imitation learning or rewards recovery (i.e., for IRI.) X instead of constraint inference.
3 Problem Definition
Constrained Mixture-Agent Markov Decision Process. To support constraint inference for different agents, formulate the environment
Constrained Mixture- Markov Decision Process (CMA-MDP) (34) M which can be defined by tuple (S, ž, R. PT, {(Pc. < 7; where: 1) S. A and Z denote the space of states, actions, and latent code (for specifying expert agents). 2) pT (s' (,, a) and R(s, a) define the transition and reward functions. 3) Pc, (cls, a, 2) refers to an
agent-specific cost model with an associated bound e, (z), where j indicates the index of a constraint. 7 y is the discount factor and (s(() defines the initial state distribution. CMA-MDP assumes the agents are differentiable by examining their policies, implying that different agent types cannot share identical policies. We aim to elucidate these differences with the constraints associated with each agent type.
In contrast to Multi-Agent Reinforcement Learning (MARL) 35, where multiple agents can act concurrently, CMA-MDP allows only one agent to operate at given time. Nevertheless, since the agents in CMA-MDP might adhere to distinct constraints, they can develop different optimal control policies or strategies. However, standard MARL frameworks do not explicitly differentiate between agent types, nor do they distinguish their respective policies and constraints.
Policy Update under Conditional Constraints. We introduce Constrained Reinforcement Learning (CRL) based on CMA-MDP. For an agent identified by z, the goal of CRL is to find a policy x (a|s,, that maximizes expected discounted rewards under the conditional constraints:
I [<]2)) max PTT.PT.T Σ >'r0 + (T((T) s.t. Po-(0o-77 [c((1 s {e(=) E [0. (1)
where H(x) denotes the policy entropy weighted by 3, and we follow 4 to define the trajectory cost
<(T(2) = (s Φ(s, a/2) where the permissibility function Φ(s,, a|2) indicates the probability that performing action under a state s is safe for agent z. CRL commonly assumes that the constraints are known, whereas in practice, instead of directly observing the constraint signals, we often have access to expert demonstrations that follow the underlying constraints, and thus the agent must recover the constraints from the demonstration dataset.
Constraint Inference from a Mixture of Expert Dataset. Based on the CMA-MDP, the goal of ICRL is to discover the underlying constraints respected by different types of expert agents from their demonstrations, To achieve it, we consider the Maximum Entropy framework 44] and represent the likelihood function as follows:
N
II L.ee, exp [r(o(()] M (1(0))) 1=1
p(DFld) = (
(2)
where 1) N denotes the number trajectories in the demonstration dataset, 2) ZMro s is a normalizing term, 3) the permissibility indicator 1 &M (7()) can be defined by ((>()=)) = IT= (e(s),a))) aflz).
and 4) the agent identifier 1,,ee, determines whether the trajectory (() generated by the agent z.
By following constraint inference can be formulated as inferring de by maximizing this likelihood function. common approach is parameterizing de with neural network models and updating the model parameters with the agent-specific expert data [4] 6 7]. However, the major challenge of this task lies in our lack of knowledge about the agent's 's identity that generates the expert trajectories (i.e., Leep, is unknown). For example, a vehicle trajectory dataset does not label the specific type of vehicles, or these labels are coarse- coarse-grained, incomplete, and noisy. On the other hand, to explain the optimal policy with the learned constraints, the types of agents and the constraints must be consistent. If we evaluate the policy of an agent = under the constraint for another type of agent z' (e.g., evaluate the driving strategy of a truck under the constraint for car), the performance will be substantially compromised. As a result, for a specific agent z, the expert trajectories (r") are optimal while the trajectories generated by other experts {Tzr, T } are supposed to be sub-optimal
3

---

or even infeasible. To capture the correct constraints, the ICLR algorithm must identify the agent corresponding to each trajectory in an unsupervised manner.
4 Inverse Constrained Reinforcement Learning for Mixture of Experts
In order to infer constraints for various expert agent types, we introduce the Multi-Modal Inverse Con- strained Reinforcement Learning (MMICRL) algorithm (Figure MMICRL employs conditional imitation policy to capture the diverse agent behaviors, facilitating unsupervised agent identification. Technically, require the agents' policies 1) exhibit high entropy when the agent type (specified by z) is unknown, and 2) collapse to specific behavioral mode when the type is determined. The objective for MMICRL can be expressed as:
Minimize - 01 a1H((T(T)) a2H[[T(F1=]] f(T), [(T|=)dr= = =, and / [(1])) (())(()) 2 e
(3)
Subject to / ((T)))))(T))T = N S(),
TED,
(Proof is provided in Appendix B.) where f(.) represents latent feature extractor, [[T[T[2)] denotes the agent-specifie policy entropy, and (T)(r)] signifies the entropy without knowledge of the agent type. The weighting parameters o1 and 02 (both > 0) determine the balance between conditional entropy minimization and general entropy maximization. This objective differs from the traditional Maximum Entropy Inverse Reinforcement Learning (MEnuRL) [14 in two ways: 1) the objective also minimizes an entropy conditioned on agent types, and 2) it incorporates an additional constraint related to the policy's 's permissibility (the last constraint). Given this objective, the optimal representation for the trajectory likelihood (i.e., trajectory policy) is:
Proposition 4.1. Let p(z(T) denote the trajectory-lever agent identifier, let r(T) = denote the trajectory rewards, let Z MMO denote a normalizing term. The optimal policy of the above optimization problem can be represented as:
An
f()
(T|=
1
exp
On Eg (x) [og(p(2(7))
r(r) ((T))))) -1-12
(4)
Z
MM'
02 01
Building upon this policy representation 4 we present the key steps of MMICRL which involves
iteratively performing: unsupervised agent identification for calculating p(=l7) (Section 4.1). 2) conditional inverse constraint inference (Section 4.2h for deducing 6(T[=), and 3) multi-modal policy
update (Section 4.3) for approximating (T]2). MMICRL alternates between these steps until the imitation policies reproduce expert trajectories, sigmifying that the inferred constraints align with the ground-truth constraints.
4.1 Unsupervised Agent Identification
MMICRL identifies expert trajectories (i.e., learning p(z[7)) in an unsupervised manner. Previous works commonly determined the agents" identities by examining the state-action features [10] [2 with the classifier p(2) 82, az). Nevertheless, different agents may exhibit similar behaviors under certain contexts or at specific time steps within trajectory. which makes this point-wise identification
problematic (c.g., see our experimental results of InfoGAIL [TO] in Section 5).
To derive a reliable agent identifier, MMICRL performs trajectory-Ievel identification by estimating an agent-specific density. Specifically, we define state-action density (i.e., normalized occupancy measure) Pa (s, a) (1 - - 500 7 = s/rr) where p(s; = x s/T) is the probability density of state at time step following policy æ. Based on this density, we consider the theorem:
r'p(,
Proposition 4.2 (Theorem 2 of [15). Suppose p is the occupancy measure that satisfies the Bellman flow constraints: E. p(s, a) = (o((s) + + p(s', a)PT(s')s a) and p(s, a) Z O. Let the policy
defined by: Ta(als) = )da""")da" then Tp is the only policy whose occupancy measure is p.
p(s.a)
Density Estimation. The aforementioned theorem provides crucial insight for agent identification: "one can identify an expert agent by examining the occupancy measures in the expert trajectories". I Leveraging this insight, we design a state-action density estimator to compute density using Conditional Flow-based Density Estimator (CFDE). CFDE estimates the density of input variables in the training data distribution under an auto-regressive constraint. Moreover, to enhance our

---

density estimator's 's sensitivity to the behavior of different agents, the estimator also conditions on the agent type. so p(= 22 I1, p 11 where == (s. a) defines an event. We implement p(z,/22 (1.-111 2) (2 ju; (x)p(a;))²) where 14; V/, (0 1: 1, 2) and ar tlo, (11:1-1; The neural function vi is implemented by stacking multiple MADE layers [3]] The corresponding agent
identifier can be represented by Bayesian rule, p=((2 _ m() po We assume uniform
II Iea. Po(s,a)z)
prior p(),, thereby deriving the following form: (=( Σ. Ilomeer po(s,al='))
Agent Identification. After learning the density model Pvl ( a|2) with CFDE, we divide DE into sub-datasets {D&)a=1 1-2 by: 1) initializing the dataset D= = and 2) VT, E Dg, adding T' into D2 arg max; ((.))Er (s, We repeat the above steps for all E Z.
4.2 Agent-Specific Constraint Inference
Based on the identified expert dataset, we have Pa = 1, VT, E D2g and thus log (w( = so the likelihood function can be simplified to:
N
p(D=ló,2) = II
exp [r(t:)]] )( -1
22
(5)
IM ==1
where ((Tz (N)
II (s.))er! (8, a|z) and the normalizer 2 = exxxl((T))) - dr. By
defining
A2 a1-a2"
we can then define the log- log-likel logīp(D, 2)] as:
[(((r Σ
+ n log II ds a 11) N Nlog explr(t)] II ob dr1z) df (=0
(6)
t=
We parameterize the instantaneous permissibility function with w, i.e., construct (st, 0x|2) and update the parameters by computing the gradient of the above likelihood function (the derivation resembles that of [4),, so V log [D(Dz 2)] can be defined as:
N
T
n lo
de (s)
1 |=)] oz
NE Me (11)
n log (Be ae [d|=)]]
(7)
This inverse constraint objective relies the nominal trajectories sampled with the conditional policy T (T|=) (also see Figure D.. For simplicity, we denote as π (T]22). In the following, we will introduce our approach to learning (T](z).
4.3 Multi-Modal Policy Optimization
By definition, the policy 7 (T[22) is trained maximize cumulative rewards subject to constraint Ex-x(-1e) [log d(T[=)] 2 e. To be consistent with our MMICRL objective in formula (3), we design
the multi-modal policy optimization objective in the following:
min (-1=))
Ex'r(se,a)) a1 [[] + [)]()())) s.t. Ex(-1=) Σ 7 log dw (s, 1=
t=
This objective extends maximum entropy policy optimization by minimizing an additional agent- conditioned entropy H|x(r|z)], which limits the variance of policy distribution for specific type of agent. The balance between these entropy terms is controlled by aj and ag. Since H[T(T)] = H[<(r]==) EEzppt)) *) (=17)] + H(2), and by removing the uniform prior p(=) (since it is independent of the policy update), the objective 8 can be simplified to:
(9)
h
min -E+(1=) (r(T) + a1 loglp) + (a2 a1 [[([((((2) s.t. (x(+))
C t=
rd log Dw as)
2
Intuitively, this objective expands the reward signals with a log-probability term (2 which encourages the policy to generate trajectories from high-density regions for specific agent type. This approach ensures that the learned policies T+(1=)== are differentiable.
Learning Diverse Policies via Contrastive Estimation. In practice, directly augmenting the reward with a probabiityy term (as in objective 9) may lead to sub-optimal policy (7). This issue
5

---

arises because the log- proobbbbyyyy term assigns a large penalty to trajectories with low (2( In such cases, the controlling policy becomes more sensitive to density estimation (since Pio (2|7)
softmax (P=((T]=)) see Section 4.1) rather than the reward signals. Balancing the trade-off between reward and density maximization by identifying an optimal weight on is challenging, especially when predictions from P& (T|22 are less accurate at the beginning of training (i.e., during the cold start).
To resolve the above issues, we consider replacing the identification probability with contrastive estimation method by constructing the following objective for policy optimization:
(10)
min (- (((T) + on Tooe (T, V1 12.....11) + (o2 on (+(((())) s.t. Ex(((=)) (E'
r" log dh (8, (s,
where V defines the probing sets (constructed with the density estimator in Algorithm I. Given specific agent type 2, these probing vectors are among the most representative data points since they are located in a high-density region conditioning agent types (& + 2). Inspired by the InfoNCE loss [16 37]
and low-density region conditioning other Lce can be calculated as:
E(g,dr)ev, fal(se, at), (32, ax)]
exp
LoelT, Vi....121) =
log
(II)
ΣEEz exp
E(aayev, Sal(St, at), (5,
where f. denotes the score function for measuring the similarity between the features from different state-action pairs (we use cosine similarity in the experiment). To interpret (Te (T, Vi.....Zi). ). we can
treat (s, a) E {T, }=} as positive embeddings for (-112) since they are generated by this policy. On the other hand, (5, a) { are negative embeddings for x-112) since they generated by controlling with other policies T-) (where 2 + 2). Considering the generation is influenced by the stochasticity in environment dynamics (e.g., transition functions), we can equivalently view the generation process as injecting the environmental noise into the policy, and thus Noise Contrastive Estimation (NCE) (38] becomes a compatible tool for learning differentiable policies. Specifically, since embeddings in V3 belong to high conditional density region in p(-[z) (see Algorithm I the knowledge from the density estimator has been integrated into policy updates.
In essence, the integration of contrastive learning into policy optimization helps the algorithm to better understand the relationships between agents, their behaviors, and the corresponding expert trajectories, resulting in improved performance for tasks involving diverse agent types. Algorithm] 2 introduces the detailed implementation of MMICRL.
Algorithm 1: Probing_5 Sets
Input: Agent type trajectory dataset D, and conditional density model p(-12) Initialize probing sets V₂- = (0]:
for T= E D= do
Find (8, à) = arg max( log poi (s, a/2) 1 Es 1844 log pv: (8, alž); Store the probing points V.- = V. U a));
end
Output: V/
Experiments
Running Setting. For comprehensive comparison, we employ consistent evaluation metrics across all environments. These metrics include 1) Constraint Violation Rate, which assesses the likelihood of a policy violating a constraint in given trajectory, and 2) Feasible Cumulative Rewards, which calculates the total rewards accumulated by the agent before violating any constraints. The demonstration data are assumed to have a zero violation rate. We run experiments with three different seeds and present the mean + std results for each algorithm. To ensure a fair comparison, we maintain uniform settings for all comparison baselines. Appendix A.2 reports the detailed settings.
Comparison Methods. We employ an ablation strategy to progressively remove components from MMICRL (Algorithm [ and consider the following variants: 1) MMICRL-LD excludes our contrastive estimation approach, using the objective (9) directly for policy updates. 2) Inspired by 10] InfoGAIL-ICRI, replaces the trajectory density model with discriminator to identify agents based on state-action pairs, extending GAII. (28] to distinguish between distinct agents'
trajectories. 3) MEICRL [ eliminates the agent identifier and expands traditional Maximum

---

Algorithm 2: Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL)
Input:
Expert demonstration Dr, upper level iterations N, lower level iterations B Initialize the policy To, (-1,; 2), constraint function (+12) and density model Pvo (+12);
for i ..,, do
Initialize an ensemble of imitation datasets {Dx se and each dataset D. {0): for 1, 1......... B do
Randomly sample a 2j, and generate imitation trajectories Tz, with To. (o. (.1.,2))
Update the density model by maximizing: log Pa. (Ts, 12y)
Store the generated trajectories and the code D= = D2 U {(Tz) 2/)}:
end
Divide DE into sub-datasets (DF, 121 by utilizing the density model (see Section 4.1 for k = 1....... B do
Sample an agent type from the prior ~ p(z);
Sample an expert trajectory TE,s from the 2th expert dataset: TE,s ~ DE.s;
Sample nominal trajectory from the t generated dataset: f D3;
Update the constraint function dw, with the objective end
E E do for
Construct probing set Vz ProbingSetss(,:D:; Ps (11)))lrri update the policy with the objective (10)
end end
Entropy (ME) IRL methods to infer Markovian constraints in a model-free environment. 4) Binary Classifier Constraint Learning (B2CL) constructs deterministic binary classifier directly for constraint inference, bypassing the need for the maximum entropy framework.
5.1 Empirical Evaluations Discrete Environments
The discrete environments enables us to visualize the recovered constraints and trajectories generated by various agents. We create 7x7 Gridworld map and design four distinct constraint map settings. In Figure the leftmost column illustrates expert trajectories and constraints for each configuration. Notably, the first three settings (rows 1-3 in Figure 2) incorporate
based Gridworld, widely studied RL environment that
different types of constraints, while the final setting (the last row in Figure 2 includes three constraint types. The primary goal for each agent is to navigate from the starting point to the endpoint while minimizing the number steps and avoiding their respective constraints. To facilitate constraint inference, a demonstration dataset containing expert trajectories provided for each environment 7
Figure dissplass the ground-truth trajectory map alongside the learned trajectory maps. Appendix C. summarizes the detailed performance in terms of feasible cumulative rewards, constraint violation rate, and destination-reachiin
rate. Without implementing agent identification, both B2CL and MEICRL can only restore single constraint type. Although InfoGAIL-ICRL incorporates agent identification, its performance is unsatisfactory, resulting in incomplete destination-reachingg In contrast, MMICRL-LD and MMICRL exhibit significant improvements; they can identify various agent types corresponding to different constraints and generate accurate trajectories that successfully reach the endpoint. Notably, the enhancements provided by MMICRL are more prominent, and the trajectories produced by the MMICRL algorithm closely resemble expert demonstrations.
5.2 Empirical Evaluations in Continuous Environments
Table 1: Continuous environment setting,
In continuous environments, we evaluate ICLR algorithms by whether they can infer location constraints for various types of robots (robots can only operate within designated areas). The task involves directing the robot navigate out of the danger zone, represented by the constraint.
Setting
Dim.
1 constraint
2nd constraint
Half-chectah
X
x2
Antwall
121
X
x2
Swimmer
12
x
0.01
0.01
Walker
xg
0.1 -0.1
x2
7

---

Via parto
vis rarto
BiLL Loleegee
Figure 2: We utilize blue, red, and orange colors to represent the constraints trajectories corre- sponding to each agent type. The red circle and blue triangle indicate the starting and ending points of each environment, respectively. Since MECL and B2CL recover only single, unified constraint, they produce just one type of trajectory (Marked by the red color).
We base our continuous environments on MuJoCo 17 For each environment, we configure two types of robots-ooe moving forward and the other moving backward each with opposing constraints. Table provides an overview of the settings. Our goal is to distinguish their respective constraints and enable each robot type to maximize rewards without violating constraints.
Figure 4 and Table 2 illustrate the constraint violation rate and feasible cumulative rewards for each agent across various environments. To evaluate the accuracy of the discovered con- straints, Figure Billstrratss the distribution of x- coordinate values for all states visited by agents of different types. Compared to other baselines, MMICRL consistently exhibits lower constraint violation rates and higher cumulative rewards, averaged over all agents. This superiority can be attributed to MMICRL's ability to capture the di- versity of constraints by inferring agent-specific constraints and learning heterogeneous policies through contrastive estimation. While certain baselines may achieve higher cumulative rewards for a specific agent, none of these algorithms consistently outperform the others. (refer Appendix C.2 for other complete results)
Figure 3: From left to right, the environments are Blocked Antwall, Blocked Hall-Cheetah, Blocked Swimmer, and Blocked Walker. The first row shows the results of InfoGAIL-ICRL, followed by MMICRL. The Blue and orange areas represent the trajectories of two types of agents.
5.3 Robust Test: Can MMICRL Recover from the Early Mistakes?
To test the robustness of MMICRI, we enforce MMICRL to make incorrect agent identifications and assess whether MMICRL recover from these errors. Specifically, in the first round of training, rather than using our density estimator, we randomly assign incorrect labels to 20% of expert trajectories. These incorrect identifications can impact ongoing constraint inference and policy updates. To recover from these mistakes, the MMICRL algorithm must accurately model diverse agent preferences and rectify the incorrect trajectories to match the appropriate agent type.

---

Table 2: MuJoCo testing performance. We report the average feasible rewards and constraint violation rate in 100 runs. The best average performance is highlighted bold.
Method
Blacked Cheetah
Blocked
Blocked Swimmer
Blocked Walker
Feusible Cumulative Rewards
T.CCL.
5.031:17 23
1.3
1,28:477.40 E.1
7.8ME++
MEICRL
4,85E+1 33445
1800+4 3.31E+3
2.015-2 1.30-1/8.00E-1
2.61E+1/
7.3335+1
InfoGALL-ICRL
2.22E+2/ 1.33E+2
3.73E+2 1.11++2
2.38E-1 / 7.900-1
2,15E+1
1.58E+3
MMICRL-LD
4.32F+3/2.5011
1.82F 2.21E+4
2.66F-2/6.100 10
5.12222
MMICRI.
6.12F+3/3.00E43
2.13E+4/
/..8:442.117:4 2,17E+4
4.07:22/6.488-22
8 1.53E+3
Constraint Violation Rate
RICI.
1004-0% 1076-/0//777--7 24%
33% 4244 677424
62%
100%-+00%
100% 7 100%=0%%
MEICRL
125%
00% /67.124%
79% 11%%
100%
100%
InfoGAIL-ICRA
25%:116% /
933
349-188% / 199
13%
100%
100% 0RR10%
MMICRL-LD
23% /
34%6=24%
00 00%=0%
11%=
34%-+t233%
52%=25%
+0%%+
MMICRI.
0%-0%
00% 14-+0%
0% +0% /0%-+0%
35%++-33% /
31 % +-22% / 25% 22%
hessbbe cumulattee Howerds
Blocked Antwell 2.0
FreNtii Cimdaatoe tlewerds Freediie Clmmdaetse Blweeds Freeille Tummlettvv Spondds Fewetii Cumulatte flwwerk
Diocked Antwall 1
Constreet Vastattoo Batt
Blocked Antwell D.
Constrent Vistation Batt
Blocked Antwall e.1
20000
20000
0.8t
66
6.
10000
10000
044
C4
W0T
10000
15000
10000
15000
5000
10000
15000
5000
30000
15000
Eptsode
Episade
sode
sode
ressble Lamakribe Nemards
6200
Blocked Half-Cheetah
2_0
Blocked Hall-Cheetah 2_1
10 WAT staeoon ware
3.00
Blocked Half-Cheetah
L0 mer stowemo ware
1.00
Blocked Holf-Cheetal
0.75
a000
2000
2000
0.25
0.25
0.00
0.00
2000
3000
4000
2000
4000
2000
4000
2000
4000
ressbbe Cumulaatiee Homardh reseibe Cummattti Cumuattee Howards
1,50
Blocked Welk z.0
Blocked Welker
COnTAATT Vesaeo wate
Blocked Walkee With 0
CONTTAaT Vesaeoo kate
Blocked WilkerWithPos z
1000
1500
0,50
500
0.25
10000 20000
30000 40000
10000 20000
30000 40000
0.00
0
10000 20000
30000 40000 Epicode
0
10000 20000
30000 40000
Episode
Episode
Eplcode
100
Blocked SwimmerWWthPers
0
Blocked Swimmer\Withoos 2
Constrent Vanletton Bate
1..0
Blocked Swimmert th Pos
CONTENEE Corearro Connee sate
2.00
Blocked Swimmert th Pos
400
600
0.8
400
200
200
C4
0.00
2500
500C
7500
10000
7500 Epeodd
5000
10000
2500
7500
10000
2500
7500 10000
EsGode
Edsode
Ediodd
Figure 4: The feasible cumulative rewards (left two columns) and constraint violation rate (right two columns). We denote the results for different agents with z_0 and =_1. From top to bottom, the environments are Blocked Antwall, Blocked Calf-Cheetahh Blocked Walker, and Blocked Swimmer.
Figure 5 displays the perfor- mance of MMICRL in both continuous and discrete set- tings. We observe that it can successfully recover multi- ple constraints correspond- ing to different types of agents. The errors made at the beginning of training do not have a significant im- pact on the overall perfor- mance, which demonstrates the robustness of MMICRL.
Blacked 2
Becos Arevell
Bldded Armaal
Bicore Arrvel
23000
2000
Ep1s0s
Figure 5: The constraint violation rate (left two columns), and feasible cumulative rewards (right two columns) in Blocked Ant and Gridworld- Setting Appendix C.3 shows complete results).
5.4 Empirical Evaluations in Realistic Environments
9

---

To demonstrate the generalization capability of the model, we conducted some preliminary studies in realistic autonomous driving environment. This environment is constructed by utilizing the HighD> dataset (For more details, check 7 and ( For extracting features of cars and roads, we use the features collector from Commonroad RL [39]. The constraint that we are interested in are 1) Car distance 2 20m (agent 0) and 2) Car distance 2 40m (agent 1). Figure Tigure shows the distribution of car distance in expert demonstrations for agent and agent I. We aim to investigate whether the MMICRL algorithm can differentiate between two different types of cars based on distance constraints.
cbsttacle distence ecpert demonatration
120-
201
20-
40
60
30
Figure 6: Car distance distribution.
Agerz-0 owstacie diitarce, siual
50
150
90
-0
100
30
10
0
120
1471,44 Jeed 2q wt_.eetaccaaa.. strr
w_cbitaceedieeeee Pastale_ddrteeeeettu strual
350
220
230
150
220
100
20
50
20
100
20
50
100
100
Figure 7: MMICRL performance in realistic environment. We test four different settings (each column refers to one setting.). We visualize the car distance in the states generated by the imitation policies learned under the inferred constraints for agent O (upper row) and 1(lower row).
In the four settings shown in Figure 7 we find that in the first row, the observed distances of the first type of car are all above 20 meters, while in the second row, the distances of the other type of car mostly above 40 meters. This suggests that the MMICRL algorithm shows promising preliminary results in this context. In the future, we will further optimize the algorithm to achieve lower constraint violation rate and higher effective rewards in a wider range of real-world environments.
Limitations
Omitting Agent Interactions: MMICRL does not consider the interactions between agents or how they can potentially collaborate compete to satisfy joint constraint. Future research could extend the game theory to ICRL for implementing multi-agent constraint inference.
Experiment in Virtual Environment. We evaluate our algorithm in virtual games rather than real-world applications (e.g., autonomous driving). This is due to the lack of ICRL benchmark for a mixture of experts. Future work can explore the application of MMICRL in realistic scenarios.
7 Conclusion
In this work, we introduce the MMICRL algorithm differentiate multiple constraints corresponding to various types of agents. MMICRL incorporates unsupervised constraint inference, agent-specific constraint inference, and multi-modal policy optimization. To demonstrate the advantages of our method over other baselines, we investigate whether MMICRL can accurately perform multiple constraint inference in both discrete and continuous environments.
Acknowledgments and Disclosure of Funding
The work is supported in part by the National Key R&D Program of China under grant No2022ZD01 16004, by Shenzhen Science and Technology Program DDSYS2021102111144 14455555 and by the Start-up Fund UDF01002911 1 of the Chinese University of Hong Kong, Shenzhen. We acknowledge the funding from the Canada CIFAR AI Chairs program, and the support of the Natural Sciences and Engineering Research Council of Canada (NSERC).
10

---

References
[1] Javier Garefa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research (JMLR), 16: 16:1437-1480, 2015.
[2] Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints in model-free reinforcement learning: A survey. In International Joint Conference on Artificial Intelligence (IJCAI), pages 4508 4008, 2021.
[3] Dexter R. R. Scobee and S. Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. In International Conference on Learning Representations, (ICLR), 2020.
[4] Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforce- ment learning. In International Conference on Machine Learning (ICML), pages 7390-7399, 2021.
[5] Mattijs Baert, Pietro Mazzaglia, Sam Leroux, and Pieter Simoens. Maximum causal entropy inverse constrained reinforcement learning. arXiv preprint arXiv:2305.02887, 2023.
[6] Ashish Gaurav, Kasra Rezaee, Guiliang Liu, and Pascal Poupart. Learning soft constraints from constrained expert demonstrations, In International Conference Learning Representations, (ICIR), 2023.
[7] Guiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezaee, and Pascal Poupart. Benchmarking constraint inference in inverse reinforcement learning. In International Conference on Learning Representations (ICIR), 2023.
[8] Dimitris Papadimitriou, Usman Anwar, and Daniel S Brown. Bayesian methods for constraint inference reinforcement learning. Transactions on Machine Learning Research, 2023.
[9] Robert Krajewski, Julian Bock, Laurent Kloeker, and Lutz Eckstein. The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems. In International Conference on Intelligent Transportation Systems (ITSC), pages 2118-2125. 2018
[10] Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems (NeurIPS), pages 3812-3822, 2017.
[11] Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[12] Alex Kuefler and Mykel J. Kochenderfer. Burn-in demonstrations for multi modal imitation learning. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 1071-1078, 2018.
[13] Ziyu Wang, Josh Merel, Scott E. Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems (NeurlPS), pages 5320-5329, 2017.
[14] Brian D. Ziebart, Andrew L. Maas, Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI), pages 1438, 2008.
[15] Umar Syed, Michael H. Bowling, and Robert E. Schapire. Apprenticeship learning using linear programming. International Conference on Machine Learning (ICML), pages 1032-1039, 2008.
[16] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, CoRR, abs/ 003748, 2018.

---

[17] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model--ased control. In International Conference on Intelligent Robots and Systems IROS, pages 5026-5033. IEEF, 2012.
[18] Dexter R.R. Scobee and S. Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2020.
[19] David Livingston McPherson, Kaylene C. Stocking, and Shankar Sastry, Maximum likelihood constraint inference from stochastic demonstrations. In IEEE Conference on Control Technology and Applications (CCTA), pages 1208-121,, 2021.
[20] Daehyung Park, Michael D. Noseworthy, Rohan Paul, Subhro Roy, and Nicholas Roy. Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning. In Annual Conference on Robot Learning (CoRL), pages 1005-1014, 2019.
[21] Alex James Chan and Mihaela van der Schaar. Scalable bayesian inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
[22] Daniel S. Brown, Scott Nickum, and Marek Petrik. Bayesian robust optimization for imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[23] Shicheng Liu and Minghui Zhu. Distributed inverse constrained reinforcement learning for multi-agent systems. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[24] Monica Babes, Vukosi Marivate, Kaushik Subramanian, and Michael L. Littman. Apprenticeship learning about multiple intentions. In International Conference on Machine Learning (ICML), pages 897-904, 2011.
[25] Christos Dimitrakakis and Constantin A. Rothkopf. Bayesian multitask inverse reinforcement learning. In Recent Advances in Reinforcement Learning- European Workshop (EWRL), pages 273-284, 2011.
[26] Igor Vozniak, Matthias Kluseh, André Antakli, and Christian Müller. Infosalgail: Visual attention-empooered imitation learning of pedestrian behavior in critical traffic scenarios. In International Joint Conference on Computational Intelligence (IJCCI) pages 325-337, 2020.
[27] Jiahao Lin and Zongzhang Zhang. ACGAIL: imitation learning about multiple intentions with auxiliary classifier gans. In Trends in Artificial Intelligence Pacific Rim International Conference on Artificial Intelligence (PRICAI), pages 321-334, 2018
[28] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 4565-4573, 2016.
[29] Yunzhu Li, Jiaming Song, and Stefano Ermon. Inferring the latent structure of human decision- making from raw visual inputs. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[30] Fang-I Hsiao, Jui-Hsuan Kuo, and Min Sun. Learning a multi-modal policy via imitating demonstrations with mixed behaviors. CoRR, abss/193..1004, 2019.
[31] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning (ICML), pages 1 2014.
[32] Raunak P. Bhattacharyya, Blake Wulfe, Derek J. Phillips, Alex Kuefler, Jeremy Morton, Ransalu Senanayake, and Mykel Kochenderfer. Modeling human driving behavior through genera- tive adversarial imitation learning. IEEE Transactions Intelligent Transportation Systems, 24:2874-2883 2023.
[33] Chelsea Finn, Paul F. Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. CoRR, abs/161 L.0355,, 2016.
[34] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning (ICML), pages 22-31, 2017.
12

---

[35] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agen agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pages 321-384, 2021.
[36] Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. MADE: masked autoen- coder for distribution estimation. In International Conference on Machine Learning (ICML), volume 37, pages 881-889, 2015.
[37] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In International Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9735, 2020.
[38] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: new estimation princi- ple for unnormalized statistical models. In International Conference on Artificial Intelligence and Statisties, (AISTATS), volume 9, pages 297-304, 2010.
[39] Xiao Wang. Hanna Krasowski, and Matthias Althoff. Commonroad-rl: configurable rein- forcement learning environment for motion planning of autonomous vehicles. In International Intelligent Transportation Systems Conference (ITSC), 2021.
13


Reviews:
[{"review_id": "AMs0o8Ippo", "reviewer": "Reviewer_KNkP", "summary": "# Problem Statement\nThe paper addresses a significant problem in Inverse Constraint Reinforcement Learning (ICRL), which is the assumption that all expert demonstrations follow the same constraints. This assumption is problematic because in real-world scenarios, demonstration data may come from various agents who follow different or even conflicting constraints. Therefore, using a single constraint model to explain the behaviors of diverse agents can lead to inaccuracies.\n\n# Main Contribution\nTo tackle this issue, the paper introduces the Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) algorithm. This approach allows imitation policies to capture the diversity of behaviors among expert agents. The paper demonstrates that MMICRL outperforms other baselines in terms of constraint recovery and control performance in both discrete and continuous environments.\n\n# Methedology\nThe proposed MMICRL estimates multiple constraints corresponding to different types of experts, allowing for a more accurate representation of diverse agent behaviors. It uses a flow-based density estimator for unsupervised expert identification from demonstrations, infers agent-specific constraints, and optimizes a multi-modal policy that minimizes the agent-conditioned policy entropy and maximizes the unconditioned one.\n\nThe key steps of MMICRL include unsupervised agent identification, conditional inverse constraint inference, and multi-modal policy update.\n\n1. Unsupervised Agent Identification: MMICRL identifies expert trajectories in an unsupervised manner. It performs trajectory-level identification by estimating an agent-specific density. The algorithm uses a Conditional Flow-based Density Estimator (CFDE) to compute a density using a state-action density estimator. The agent identifier is represented by the softmax representation.\n\n2. Agent-Specific Constraint Inference: Based on the identified expert dataset, the likelihood function can be simplified. The algorithm parameterizes the instantaneous permissibility function with $ω$ and updates the parameters by computing the gradient of the above likelihood function.\n\n3. Multi-Modal Policy Optimization: The policy is trained to maximize cumulative rewards subject to constraint. The objective expands the reward signals with a log-probability term, which encourages the policy to generate trajectories from high-density regions for a specific agent type. The algorithm integrates contrastive learning into policy optimization, helping it to better understand the relationships between agents, their behaviors, and the corresponding expert trajectories.\n\nThe algorithm alternates between these steps until the imitation policies reproduce expert trajectories, signifying that the inferred constraints align with the ground-truth constraints.\n\n# Experiments\n\nThe paper conducts experiments on the MMICRL algorithm in both discrete and continuous environments, using Constraint Violation Rate and Feasible Cumulative Rewards as evaluation metrics. In discrete environments based on Gridworld, MMICRL successfully identifies various agent types and generates accurate trajectories. In continuous environments based on MuJoCo, MMICRL consistently exhibits lower constraint violation rates and higher cumulative rewards, outperforming other baseline methods. A robustness test shows that MMICRL can recover from initial incorrect agent identifications, demonstrating its ability to accurately model diverse agent preferences and its robustness in handling errors.", "questions": "- Should the summation in line 178 be a production instead?\n- Why does the equation in line 181,\n$p_{\\psi}(z|\\tau) = \\frac{\\exp \\left(\\prod_{(s,a) \\in \\tau} p_{\\psi}(s,a|z)\\right)}{\\sum_{z'} \\exp \\left(\\prod_{(s,a) \\in \\tau} p_{\\psi}(s,a|z')\\right)}$ hold? It would be great if the authors could elaborate more on the derivation.", "limitations": "The authors acknowledge two valid limitations in their work. Firstly, their MMICRL method doesn't consider potential collaboration or competition between agents to meet joint constraints. Secondly, their evaluations are conducted in virtual environments, not real-world applications, due to the absence of a suitable ICRL benchmark.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "# Originality and significance\nThis work poses a unique problem setup of multi-modal ICRL, which nevertheless has substantial real-world impact for applications such as autonomous driving or human-robot interaction. The proposed method is sophisticated and innovative, requiring no supervision other than expert demonstrations.\n\n# Quality\nA comprehensive analysis, along with an ablation study, is conducted in comparison to various baselines, effectively validating the efficacy of the proposed method.", "weaknesses": "- The number of agent types needs to be preset, and such number tested in the paper has been limited (mostly 2 or 3). The results would be more significant if the effectiveness of the method could be demonstrated with more complex data of larger scale.\n- Please enlarge the font size in figure 3, and also adjust the opacity of the bars so that the overlapping distributions are both visible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we sincerely value your time and effort in evaluating our work. Your insights have been valuable to our work. We have prepared comprehensive responses and clarifications to address each point you raised. We hope these responses can resolve your concerns.\n\n1. *\"The number of agent types needs to be preset, and such number tested in the paper has been limited (mostly 2 or 3). The results would be more significant if the effectiveness of the method could be demonstrated with more complex data of larger scale.\"*\n\n**Our Response**. We have supplemented our findings with further experimental results, enclosed in the attached PDF. This includes an increase in the number of ground-truth constraints and a study examining the model's performance when the number of these constraints and the preset agent types are mismatched. These additional results can provide clearer insight into our ICRL's performance.\n\n2. *\"Please enlarge the font size in Figure 3, and also adjust the opacity of the bars so that the overlapping distributions are both visible.\"*\n\n**Our response.** We appreciate your kind suggestions and we have revised the manuscript accordingly.\n\n3. *\"Should the summation in line 178 be a production instead?\"*\n\n**Our response.** Thank you for your careful reading of our manuscript. It should be a production in line 178. We apologize for any confusion caused and appreciate the valuable suggestions.\n\n4. *\"Why does the equation in line 181, hold? It would be great if the authors could elaborate more on the derivation.\"*\n\n**Our response.** We're thankful for your suggestions. Indeed, this equation utilizes the softmax representation. We've developed an agent-conditioned density model, $p(\\tau|z)=\\prod_{(s,a)\\in\\tau} p(s,a|z)$, to calculate the density of state-action pairs. However, the agent identifier in MMICRL uses expert trajectories as input to predict the most likely agent generating these trajectories. We need to convert the density estimate to the agent identifier $p_\\phi(z|\\tau)$, a transformation commonly achieved via the softmax function."}, {"review_id": "rGqdk27qS5", "reviewer": "Reviewer_ZL9H", "summary": "This paper considers the problem of estimating constraints from a dataset consisting of a mixture of expert trajectories, and proposes an algorithm (MMICRL) for solving that problem.\nMMICRL proceeds iteratively, first estimating which trajectories belong to which agent class using a density estimation approach (over state-action occupancy), and second infers constraints on a per-agent class basis (using a MLE constraint inference procedure).\nThe policy is learned concurrently using an agent-unconditional max entropy, agent-conditional min entropy objective.\nExperiments in gridworld and mujoco environments show that MMICRL outperforms baselines as measured by permissible expected return and constraint violation rate.\n", "questions": "1. How is evaluation performed given that the agent-class is identified in an unsupervised manner? It seems like the relevant z value must be inferred either heuristically or manually in order to evaluate e.g., constraint violations. If so, this seems like a limitation of the approach, the reason being that identifying which z value to use in a practical setting would be challenging (given the use of CIRL in the first place implies complex / high-dimensional constraints).\n2. Is it correct that the method is limited to a setting with a fixed number of agent classes? This appears to be the case based on the algorithm. If so, (1) how robust is the algorithm to the number of classes chosen being incorrect? And (2) this also seems like a major limitation in that often real world data doesn’t cluster into a discrete set (but rather exists along a continuous spectrum). \n3. Section 3 has this line: “CMA-MDP assumes the agents are differentiable by examining their policies, implying that different agent types cannot share identical policies”. Would you elaborate on both the differentiability claim and the implication?\n", "limitations": "See weaknesses and questions sections\n", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "### Originality\n- The paper proses what seems to be a novel setting of inverse constrained RL with a mixture of experts\n- The combination of methods (unsupervised agent clustering, constraint inference, policy learning) used in MMICRL appears novel, the application of certain methods (state-action occupancy density estimation, contrastive objective) to ICRL seems novel, and some aspects seem entirely novel (in-class min entropy, out-of-class max entropy).\n### Quality\nEvaluation\n- The evaluation of the proposed algorithm is quite strong. Appropriate baselines are considered (given the novel setting they double as ablations), and experiments are well-designed to illustrate the advantages of the approach - the grid world demonstrates the behavior in an interpretable manner, and the mujoco experiments demonstrate the usefulness of function approximation in the setting.\n- Experiments are run multiple times over different random seeds, with aggregate performance reported.\n- Experiments investigating susceptibility of the algorithm to initial poor performance do a good job of conveying how robust the algorithm is in practice\n### Clarity\n- The algorithm and experiments are presented fairly clearly, with generally useful / clear figures, a specification of the algorithm, and results clearly summarized / visualized.\n", "weaknesses": "### Significance\n- Does constraint inference in the context of a mixture of expert demonstrations makes sense as a problem formulation? If different experts disagree on what is or is not a constraint, then does it make sense to formulate the problem as constraint inference (or would rewards be more appropriate)? The motivation for ICRL is typically that the behavior of experts is often best explained by a simple reward function + a set of constraints [1], but that does not seem to be the case here. The actionable critiques are (1) the paper does not apply the method in a realistic setting and (2) the examples given in the paper of real applications (driving datasets with unlabeled car types) are unrealistic (and relies on partial observability (of the object type) more than a mixture of experts).\n### Originality\n- Constraint inference in the context of a mixture of expert behavior does seem novel, but the analogous extension for reward learning is not (e.g., [2,3,4])\n- MMICRL largely combines existing methods (density estimation, constraint inference, policy optimization) adapting for the multi-modal case (with some novel contributions noted in the strengths)\n### Quality\nAlgorithm\n- What can be shown theoretically about the algorithm? For example, does it converge under some assumptions? Discuss of such topics seems missing from the paper\nEvaluation\n- Related to the point about significance: the evaluation settings are unrealistic. An example with real world data would dramatically improve the quality of the evaluation. \nClarity\n- It’s not clear until later in section 4 that the algorithm iteratively performs agent identification / clustering, which makes section 4.1 confusing when first reading through\n- Figure 4 could be improved wrt clarity. Perhaps aggregating over environments would help.  \n\n[1] Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning \\\n[2] LiMIIRL: Lightweight Multiple-Intent Inverse Reinforcement Learning \\\n[3] Dealing with multiple experts and non‑stationarity in inverse reinforcement learning: an application to real‑life problems \\\n[4] Nonparametric Bayesian Inverse Reinforcement Learning for Multiple Reward Functions\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we sincerely value the time and effort you have devoted to evaluating our work. To address each point you raised, we have prepared comprehensive responses. We hope these responses can resolve your concerns.\n\n1. *\"How is evaluation performed given that the agent class is identified in an unsupervised manner? ... identifying which z value to use in a practical setting would be challenging\"*\n\n**Our Response.** During the evaluation process, **we must assign the identified agents to a specific type of expert agent.** This is achieved by a majority voting system. To illustrate, let's consider a classified expert dataset, denoted as $D_{z}$ (line 182 of our paper), where 90\\% of the trajectories are generated by agent type 1 (this information is available during the evaluation.), so we would label this class as type 1. Subsequently, the corresponding ground-truth constraint related to type 1 would be used to compute the constraint violation rate.\n\nIn a **practical application**, such as autonomous driving, we can run our MMICRL algorithm to derive constraints for different agents. To apply these constraints, we must associate identified agents with actual vehicles, which is made by matching the distribution of the identified expert dataset $\\mathcal{D}_{z}$ to the distribution observed in the target vehicle. It's crucial to note that this matching wouldn't be feasible without the expert dataset identified by our MMICRL.\n\n2. *\"Is it correct that the method is limited to a setting with a fixed number of agent classes?\"*\n\n**Our Response.** In our MMICRL algorithm, the count of agent types of interest, denoted as $|\\mathcal{Z}|$, is a hyperparameter. However, it doesn't necessarily match the precise number of expert types but rather serves as an upper limit. Therefore, if the actual expert types are fewer than $|\\mathcal{Z}|$, MMICRL will 1) either create additional subclasses for an actual expert type or 2), assigns no expert trajectory to one class, rendering $\\mathcal{D}_{z^{''}}=\\{\\emptyset\\}$, which can then be discarded in the subsequent iteration. This behavior doesn't compromise the algorithm's effectiveness. However, if the actual expert types exceed $|\\mathcal{Z}|$, implying $|\\mathcal{Z}|$ no longer serves as an upper limit, it could lead to sub-optimal performance.\n\n3. *\"How robust is the algorithm to the number of classes chosen being incorrect?\"*\n\n**Our Response.** The robustness in MMICRL can be justified by experiment. Please check our additional results in the attached PDF file.\n\n4. *\"This also seems like a major limitation in that often real-world data doesn’t cluster into a discrete set ...\"*\n\n**Our Response.** In real life,  we believe there are many examples where agents can be classified into discrete sets. For instance, vehicles of different types, and sports players in different positions. For cases where clear distinctions are difficult to make, we can **discretize the continuous sets** and then use MMICRL to infer the constraints.\n\n5.*\"Section 3 has this line: \"CMA-MDP assumes the agents are differentiable by examining their policies ...\". Would you elaborate on both the differentiability claim and the implication?\"*\n\n**Our Response.** The assumption is consistent with the design of our MMICRL algorithm. If two agent policies are identical, our unsupervised agent identifier would not differentiate them. Consequently, there's no requirement or benefit to regard them as distinct agent types or account for their behaviors using separate constraints.\n\n6. *\"What can be shown theoretically about the algorithm? For example, does it converge under some assumptions? ...\".*\n\n**Our response.**  While we acknowledge the **significance of such understanding for ICRL**, theoretical results for ICRL are, to our knowledge, currently **lacking**. A potential route is extending theoretical findings from IRL to ICRL, but enabling this extension remains a complex issue. because:\n\n- The advancements in IRL theory depend on defining feasible sets of rewards. However, defining a feasible set of **constraints** is challenging due to the variability of constraints (e.g., hard, soft, and probabilistic constraints) and the constrained optimization method (e.g., Lagrange method, Interior points). These issues are notably **absent in IRL**, where the forward problem is unconstrained.\n\n- IRL theories usually deal with a simplified setting involving bounded rewards and a tabular environment. In contrast, our work explores environments with a continuous state space and multiple expert policies and demonstrations. The gap between rigorous theoretical understanding and our ICRL setting is substantial.\n\n7. \"If different experts disagree on what is or is not a constraint, then does it make sense to formulate the problem as constraint inference (or would rewards be more appropriate)?\"\n\n**Our response.** It is challenging to explain the conflicting behaviors among expert trajectories through reward inference with IRL. For instance, in a situation where 50\\% of expert agents move left and the other half move right, IRL would assign near-zero rewards to both actions, considering that each action is deemed sub-optimal 50\\% of the time.\n\nOur approach indeed aligns with the motivation for ICRL, as we assume all expert agents adhere to the same known reward function. We account for their disagreements by learning a series of distinct constraints.\n\n8. *\"Concerns about the unrealistic evaluation.\"*\n\n**Our response.** Section 5.4 Limitations have acknowledged the absence of a realistic benchmark.  A recent study ([Ref.7]) describes an environment tailored for autonomous driving tasks, albeit **limited to a single agent type**. A possible route is extending this environment to include multiple agent types. Our methodologies are elaborated on in the attached PDF."}, {"review_id": "ShImMFmU4O", "reviewer": "Reviewer_ieEm", "summary": "The paper introduces a new algorithm called Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) to address the challenge of recovering multiple underlying constraints from a mixture of trajectories demonstrated by different types of expert agents. The algorithm utilizes a flow-based density estimator to identify experts from the demonstration data and infer agent-specific constraints. MMICRL then employs a novel multi-modal constrained policy optimization objective to imitate expert policies, considering both conditioned and unconditioned policy entropy. The approach is further enhanced using contrastive learning to improve robustness. Experimental results in discrete and continuous environments demonstrate that MMICRL outperforms other baseline algorithms in terms of constraint recovery and control performance.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a new algorithm, MMICRL, which addresses the challenge of recovering multiple constraints from a mixture of trajectories demonstrated by different types of expert agents. This is a novel and important problem in the field of Inverse Reinforcement Learning (IRL). MMICRL utilizes a flow-based density estimator to identify expert agents from demonstration data. This approach allows for unsupervised expert identification and contributes to the ability to infer agent-specific constraints accurately. The incorporation of the policy optimization objective into the contrastive learning framework enhances the robustness of the algorithm. This addition strengthens the learning process and improves the overall performance of MMICRL.\n\nThe paper presents extensive experiments conducted in both discrete and continuous environments to evaluate the performance of MMICRL. The experimental results demonstrate that MMICRL outperforms other baseline algorithms in terms of constraint recovery and control performance, indicating the effectiveness and superiority of the proposed approach.", "weaknesses": "I am not an expert in IRL, but I know some RL algorithms [1] make use of mixture models and their modelling methods can even be applied in other fields, like computer vision[2]. I am wondering if is it possible to make these modelling methods in IRL and what is the advantage of MMICRL compared with these methods.\n\n[1] Ren, Jie, et al. \"Probabilistic mixture-of-experts for efficient deep reinforcement learning.\" arXiv preprint arXiv:2104.09122 (2021).\n\n[2] Xia, Xiaobo, et al. \"Pluralistic image completion with gaussian mixture models.\" Advances in Neural Information Processing Systems 35 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we sincerely value the time and effort you have devoted to evaluating our work. To address each point you raised, we have prepared comprehensive responses and clarifications. We hope these responses can resolve your concerns.\n\n1. *\"I am not an expert in IRL, but I know some RL algorithms [1] make use of mixture models and their modeling methods can even be applied in other fields, like computer vision[2]. I am wondering if is it possible to make these modeling methods in IRL.\"*\n\n**Our response.**  We thank the reviewer for providing us with the methods for reference and comparison. Although these works could serve as important relevant works (**We will make sure our related works section can reflect this relevance**), they are not directly applicable to solving ICRL problems. In the following, we elaborate on their difference to our MMICRL and potential extension to ICRL.\n\na) In the study, \"Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning\", the authors examined the **unconstrained** forward control problem that learns ** control policy** from **observed** environmental dynamics, such as state and rewards. Their approach employs a Gaussian Mixture Model (GMM) to model a **multi-modal** policy, which they term a mixture of experts.\n\nSeveral key differences distinguish their approach from ours:\n\n- Our method, MMICRL, addresses an **inverse** optimal control problem, learning **missing** dynamics signals, notably constraint signals, from provided **expert demonstration**.\n- MMICRL operates within a **constrained** Markov Decision Process (MDP), in contrast to their unconstrained model.\n- Instead of modeling multi-modal expert behavior with a single unified policy, MMICRL **differentiates** expert trajectories through agent identification and learns imitation policies **based on the expert type**.\n\nSubstituting our PPO-Lag policy representation with a GMM does not account for the contradictory behavior seen in expert demonstrations, particularly in safety-critical situations.\n\nb) In the research, \"Pluralistic Image Completion with Gaussian Mixture Models\", the primary focus is on constrained pluralistic image completion. The relationship between the sub-procedure and pluralistic results is modeled using a Gaussian Mixture Model (GMM).\n\nTheir methodology differs from ours in two main ways:\n\n- MMICRL models the control problem as sequential decisions within a Markov Decision Process (MDP). It's unclear if the task of pluralistic image completion can be mapped similarly to a control problem within an MDP.\n\n- MMICRL learns constraints from expert demonstrations. In contrast, the constraints in pluralistic image completion are either manually provided or derived from the dataset.\n\n2. *\"what is the advantage of MMICRL compared with these methods.\"*\n\n**Our response.** Since the methods primarily target different problems and settings than MMICRL, direct comparisons may be challenging. However, we argue the necessity of MMICRL based on the following points:\n\n- MMICRL can be concurrently applied to both discrete and continuous environments and various inverse control tasks.\n- MMICRL can estimate agent-specific constraints from a mixture of expert demonstrations, providing an explanation for their contradictory behavior in game contexts.\n- In comparison to other baselines, MMICRL stands out in maintaining a low constraint violation rate while achieving high rewards, demonstrating its effectiveness.\n- Using a GMM policy representation is not a direct substitute for our method. Our approach distinctly excels in differentiating between various types of agents."}, {"review_id": "BNFH2rt1GE", "reviewer": "Reviewer_Gwnr", "summary": "The paper proposes the algorithm Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) for imitation learning mixture of expert demonstrations with various constraints. The algorithm includes agent identification, agent-specific constraint inference and multi-modal policy optimization. The problem is challenging, the proposed method seems to be novel, and the simulation experiments show the effectiveness of the proposed method.\n", "questions": "From Eq. (9) to (10), why using the contrastive estimation method helps with handling the trade-off of reward maximization and density maximization? The contrastive estimation loss also induces an additional term from the reward.\n\nIn Eq. (10), how is the reward of the trajectory estimated conditioned on the current policy? Is there any value function estimation in the policy optimization process?\n\n", "limitations": "The limitations of the work are well-discussed.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I think the problem of multi-modal learning from a mixture of experts with constraints is indeed a challenging one, and the proposed algorithm has necessary components for handling several difficulties. \n\nThe paper has good writing, and the method is described clearly with necessary derivation details. Although lots of mathematical notations are used, it’s good to see intuitive explanations for most terms.\n\nThe experiments are evaluated on both discrete and continuous environments. The improvement of the proposed algorithm over baseline methods is significant, although not consistent across tasks.\n", "weaknesses": "Overall the paper has a clear description of the reasons for different design choices, but there are still several places confusing for me, detailed in the Questions section.\n\nFor proposition 4.2, please provide the complete conditions for the statement. For the case with a limited number of samples and estimated occupancy measure, what is the change for this statement?\n\nIn Table 2, it could be better to provide the constraint violation rates in the expert demonstration data as a reference. Does the demonstration data have zero violation rate? It may be mentioned somewhere in the paper that I missed.\n\nFor Fig. 4, is it possible to have zero or lower constraint violation while remaining high rewards for tasks Blocked WalkerWithPos and Blocked SwimmerWithPos. \n\nIt could be better to see the method tested on some more realistic cases like vehicle driving dataset, etc.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we sincerely value your time and effort in evaluating our work. Your insights have been valuable to our work. We have prepared comprehensive responses and clarifications to address each point you raised. We hope these responses can resolve your concerns.\n\n1. *\"For proposition 4.2, please provide the complete conditions for the statement. For the case with a limited number of samples and estimated occupancy measure, what is the change for this statement?\"*\n\n**Our response.**  Thanks for your suggestions. We agree that a thorough explanation of proposition 4.2 is necessary. This proposition is in fact citing the Theorem 2 of [15]. The occupancy measures $\\rho$ should satisfy the Bellman flow constraints. That's to say:\n\\begin{align}\n\\sum_a{\\rho(s,a)}=P_0(s)+\\gamma\\sum_{s^{\\prime},a}\\rho(s^{\\prime},a)P_{\\mathcal{T}}(s^{\\prime}|s,a) \\text{ and }{\\rho(s,a)}\\geq0\n\\end{align}\nThis constraint is often applied in the dual problem for solving MDP. Since our expert trajectories are assumed to be optimal in solving the primal control problem in MDP, their occupancy measures $\\rho^\\pi(s,a)$ should generally satisfy the constraints.  \n\n[15] Umar Syed, Michael H. Bowling, and Robert E. Schapire. Apprenticeship learning using linear programming. In International Conference on Machine Learning (ICML), pages 1032–1039, 2008.\n\n2. *\" In Table 2, it could be better to provide the constraint violation rates in the expert demonstration data as a reference. Does the demonstration data have zero violation rate? It may be mentioned somewhere in the paper that I missed.\"*\n\n**Our response.** That's a very good point. We agree that more detailed clarification is needed. In fact, ICRL algorithms typically focus on 1) soft constraints, which demand constraint satisfaction in expectation (i.e., violation rate > 0), and 2) hard constraints requiring absolute constraint satisfaction (i.e., violation rate = 0). While soft constraints are particularly useful in stochastic environments, they're challenging to estimate. For simplicity, we've chosen to focus on hard constraints, which exhibit a zero violation rate in the expert demonstration, in this study.\n\n3. *\"For Fig. 4, is it possible to have zero or lower constraint violation while remaining high rewards for tasks Blocked WalkerWithPos and Blocked SwimmerWithPos.\"*\n\n**Our response.**  In our experiments, we managed to attain a low constraint violation in **certain** random seeds, but not in **all** of them. The reported results represent **averages** across all tested seeds. Consistently achieving a low constraint violation rate is challenging in both environments. Similar findings are illustrated in Figure 2 of [7], where the MECL (ICRL) algorithm fails to uphold a zero constraint violation rate in the Walker and Swimmer environments. Notably, their experiments focus on a single type of agent, and our task is even more challenging. Future research will be required to make further progress in this regard.\n\n[7] Guiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezaee, and Pascal Poupart. Benchmarking constraint inference in inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2023.\n\n4.*\" It could be better to see the method tested on some more realistic cases like vehicle driving dataset, etc.\"*\n\n**Our response.** In Section 5.4 Limitations, we acknowledged the absence of a realistic benchmark to test our MMICRL algorithm. We notice a recent study ([Ref.7]) provides an environment tailored for autonomous driving tasks, albeit **limited to a single agent type**. A possible route is extending this environment to include multiple agent types and deriving corresponding constraints. Our methodology and findings are elaborated in the attached PDF.\n\n5.*\"From Eq. (9) to (10), why using the contrastive estimation method helps with handling the trade-off of reward maximization and density maximization? The contrastive estimation loss also induces an additional term from the reward.\"*\n\n**Our response.** This is an excellent question. Empirically, we conducted an **ablation study** in our experiment (see MMICRL-LD) by omitting our contrastive estimation approach and using objective (9) directly for policy updates. We noticed sub-optimal control performance in our early experiments, which **motivated** us to explore alternative methods. The contrastive estimation technique is a straightforward, effective method for learning feature representations. Compared to directly applying log density, the information-theoretic InfoNCE loss, which transforms **multi-label-classification** into **positive-negative samples differentiation**, serves as a more applicable loss for learning differentiable policies. This method has already shown stable performance in more complex applications.\n\n6.*\"In Eq. (10), how is the reward of the trajectory estimated conditioned on the current policy? Is there any value function estimation in the policy optimization process?\"*\n\n**Our response.** ICRL typically assumes a known reward function[4]. In our MMICRL, we assume all experts share the same reward, and explain their distinct behaviors by different constraints. To estimate the cumulative rewards in a trajectory, we follow [4] and utilize of PPO-Lag (Proximal Policy Optimization Lagrange) for policy optimization. \n\nIn PPO-Lag, we have an estimation of the value function. The algorithm learns both a reward-based value function and a cost-based value function for calculating expected rewards and costs for a policy. \n\n[4] Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International Conference on Machine Learning (ICML), pages 7390–7399, 2021."}]

Author rebuttal:
Dear Reviewers, Area Chairs, and Program Chairs,

We sincerely appreciate the valuable comments and suggestions provided, as they have been instrumental in enhancing our work. In light of your feedback, we can improve our work with detailed clarifications, comprehensive explanations, and supplementary experimental results. To facilitate understanding and mitigate any potential ambiguity, we provide a summary of the major updates below:

1. **Validating the Robustness with Additional Experiment.** To address the concerns raised by reviewer ZL9H, we evaluate our model's performance across different numbers of agent types. The algorithm maintains its efficiency even when the parameter surpasses the actual number of agent types. However, a parameter smaller than the actual count may influence the performance of constraint inference. In response to reviewer KNkP's concerns, we test in a Gridworld environment with a larger number of agent types, where the algorithm consistently exhibits satisfying performance. The results can be found on the attached one-page PDF.

2. **Clarifying the Advantage of MMICRL.** As it is requested by the reviewer ieEm, we clarify the distinctions between MMICRL and other methods, given that they are proposed for different tasks. Additionally, we provide a concise summary of the unique advantages offered by the MMICRL algorithm.

3. **Justifying Various Design Decisions**. We furnish the comprehensive conditions for Proposition 4.2. Additionally, we illuminate our motivation for adopting the contrastive estimate method, which effectively handles the balance between reward maximization and density maximization.

4. **Responding to Queries About Evaluation Process and Problem Formulation**. We detail our approach of allocating the identified agents to distinct expert agent types as part of the evaluation process. Besides, We introduce why explaining conflicting behaviors among expert trajectories through reward inference with IRL is a challenging task, so we think MMICRL proves to be an efficacious instrument for modeling the behaviors of multiple agents.

5. **Adding Results in Realistic Environments**. As suggested by reviewers ZL9H and Gwnr, we have added the experiment results of our MMICRL method in the realistic environment of autonomous driving on highways. We mainly study the car distance constraints (check results in the attached one-page PDF).

As far as we know, we are the first to differentiate multiple constraints corresponding to various types of agents. In order to demonstrate the generality of our method compared to other baselines, we examine the capability of MMICRL to accurately perform multi-modal constraint inference in both discrete and continuous environments. We believe that our benchmark can facilitate the development of more mature multi-model ICRL algorithms in other meaningful settings.

Meta-review: The paper proposes a new approach that can deal with having differing constraints in the expert demonstrations used for Inverse Constraint Reinforcement Learning (ICRL). The approach is extensively evaluated experimentally.

The reviewer found the method interesting and sound. Initial concerns included having to specify the number of modes and only being tested on low numbers of modes, the need for more realistic experiments, some unclear details, and missing theoretical guarantees.

The authors provided additional experimental results, details, and in-depth discussion in the rebuttal. Those addressed the (major) concerns of the reviewers.

3 reviewers are now in favor of accepting the paper, the "borderline reject" reviewer did not react after the rebuttal but his/her concerns have been well addressed during the discussion phase. While the paper certainly has limitations (as also pointed out by the authors), I believe it is an important step forward and should be presented to the community, such that future research can build on top of it.
Decision: Accept (poster)