Title: Multi-Modal Inverse Constrained Reinforcement Learning from a Mixture of Demonstrations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Inverse Constrained Reinforcement Learning

Learning from Demonstrations

Muti-Modal Learning
Abstract: Inverse Constraint Reinforcement Learning (ICRL) aims to recover the underlying constraints respected by expert agents in a data-driven manner. Existing ICRL algorithms typically assume that the demonstration data is generated by a single type of expert. However, in practice, demonstrations often comprise a mixture of trajectories collected from various expert agents respecting different constraints, making it challenging to explain expert behaviors with a unified constraint function. To tackle this issue, we propose a Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) algorithm for simultaneously estimating multiple constraints corresponding to different types of experts. MMICRL constructs a flow-based density estimator that enables unsupervised expert identification from demonstrations, so as to infer the agent-specific constraints. Following these constraints, MMICRL imitates expert policies with a novel multi-modal constrained policy optimization objective that minimizes the agent-conditioned policy entropy and maximizes the unconditioned one. To enhance robustness, we incorporate this objective into the contrastive learning framework. This approach enables imitation policies to capture the diversity of behaviors among expert agents. Extensive experiments in both discrete and continuous environments show that MMICRL outperforms other baselines in terms of constraint recovery and control performance.

Multi-Modal Inverse Constrained Reinforcement Learning from a Mixture of Demonstrations

Abstract

Inverse Constraint Reinforcement Learning (ICRL) aims to recover the underlying constraints respected by expert agents in a data-driven manner. Existing ICRL algorithms typically assume that the demonstration data is generated by a single type of expert. However, in practice, demonstrations often comprise a mixture of trajectories collected from various expert agents respecting different constraints, making it challenging to explain expert behaviors with a unified constraint function. To tackle this issue, we propose a Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) algorithm for simultaneously estimating multiple constraints corresponding to different types of experts. MMICRL constructs a flow-based density estimator that enables unsupervised expert identification from demonstrations, so as to infer the agent-specific constraints. Following these constraints, MMICRL imitates expert policies with a novel multi-modal constrained policy optimization objective that minimizes the agent-conditioned policy entropy and maximizes the unconditioned one. To enhance robustness, we incorporate this objective into the contrastive learning framework. This approach enables imitation policies to capture the diversity of behaviors among expert agents. Extensive experiments in both discrete and continuous environments show that MMICRL outperforms other baselines in terms of constraint recovery and control performance. Our implementation is available at: https://github.com/qiaoguanren/Multi-Modal-Inverse-ConstrainedReinforcement-Learning.

1 Introduction

A fundamental prerequisite for achieving safe Reinforcement Learning (RL) is that the agents' policies must adhere to the underlying constraints in the environment [1, 2]. However, in many real-world applications (e.g., robot control and autonomous driving), the ideal constraints are time-varying, context-dependent, and inherent to experts' own experience. These constraints are hard to specify mathematically and may not be readily available to RL agents in policy updates.

A promising approach for learning latent constraints is Inverse Constraint Reinforcement

InverseStep:

ForwardStep:

Agent-Specific

Constraint

Multi-Modal Constrained

function

PolicyOptimization

ConstraintInference

‘ZZA

b(- 12z) → Φi+1( 1z)

Identified

Imitation trajectories D1,...Dz

Unsupevised

Density

DensityModelUpdate:

AgentIdentification

Model

‘3ZA

DE →{D,..,D}

maximizep Pp(tz/z)

A Mixture of Demonstrations DE

Learning (ICRL) [3, 4, 5]. As a data-driven technique, ICRL recovers the underlying constraints

respected by expert agents from their demonstrations and utilizes these constraints to support downstream applications. Existing ICRL methods [3, 4, 6, 7, 8] commonly assumed that all expert demonstrations follow the same constraints, and they approximated these constraints with a unified constraint model, whereas, in practice, the demonstration data may be collected from various agents, and these agents might follow different or even conflicting constraints. It is problematic to leverage a single constraint model to explain the behaviors of diverse agents. For example, in the context of autonomous driving [9], the vehicle-distance constraints followed by trucks and cars should differ, and misapplying these constraints could lead to serious traffic accidents.

To differentiate expert demonstrations, previous RL approaches [10, 11, 12, 13] inferred the latent structure of expert demonstrations and identified expert agents by examining their behavioral preferences. However, these methods were specifically designed for imitation learning rather than constraint inference. Moreover, as their identification process primarily relies on an agent classifier that evaluates state-action pairs, there is no theoretical guarantee that the optimal model is identifiable.

In this work, we propose the Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) algorithm for estimating agent-specific constraints from a mixture of expert demonstrations (Figure 1). MMICRL extends the traditional maximum entropy framework [14] with agent-conditioned entropy minimization and unconditioned entropy maximization subject to a permissibility constraint. The resulting policy representation facilitates inferring agent-specific constraints by alternating between the following steps: 1) Unsupervised Agent Identification . MMICRL conducts trajectory-based agent identification using a flow-based density estimator. The optimal results are identifiable since each agent's policy must correspond to a unique occupancy measure [15]. 2) Agent-Specific Constraint Inference . Leveraging the identified demonstrations, MMICRL estimates a permissibility function for distinguishing expert demonstrations from sub-optimal ones, based on which we construct constraints for each type of agent. 3) Multi-Modal Policy Optimization . MMICRL measures the accuracy of inferred constraint models by comparing the similarity between expert trajectories and those generated by imitation policies under the inferred constraints. To capture the diversity of behaviors exhibited by multiple agents, we incorporate policy optimization within the contrastive learning framework. We treat the generated trajectories as noisy embeddings of agents, which serve as compact representations of their behaviors. Utilizing the contrastive estimation methods [16], we can enhance the similarity between embeddings for agents of the same type, while simultaneously maintaining the distinctiveness of diverse agent types.

We empirically demonstrate the performance of our method by conducting experiments in both discrete (e.g., Gridworld) [7] and continuous environments (e.g., MuJoCo) [17]. MMICRL significantly surpasses other baselines in terms of distinguishing different agents, adhering to the true constraints, and optimizing control performance. To examine the robustness of MMICRL , we investigate its ability to recover from incorrect agent identification and subsequently infer the correct constraints.

2 Related Works

Inverse Constrained Reinforcement Learning. Prior ICRL methods typically learned constraints from demonstrations under the maximum entropy framework [14]. Some research [18, 19] employed constraints to differentiate between feasible and infeasible state-action pairs in Constrained MDP, but these studies were restricted to inferring discrete constraints in environments with known dynamics. A subsequent work [4] extended this approach to continuous state-action spaces with unknown transition models by utilizing neural networks to approximate constraints. Inspired by Bayesian Inverse Reinforcement Learning [20, 21, 22], [8] inferred probability distributions over constraints. To better model demonstrations, [6] extended ICRL to infer soft constraints rather than hard ones, and [23] explored ICRL under the multi-agent setting. Striving for efficient comparisons, [7] established an ICRL benchmark across various RL domains, such as Gridworld, robot control, and autonomous driving. However, these algorithms primarily target inferring constraints for a single agent type, without considering the distinct constraints associated with multiple agent types.

Learning from a Mixture of Expert Demonstrations. Multi-task Inverse Reinforcement Learning (IRL) [24, 25] aims to learn from a mixture of expert demonstrations. Some previous studies [10, 26, 12, 27] utilized the Generative Adversarial Imitation Learning algorithm [28] to model the behaviors of multiple agents. [10, 11, 13, 29] learned interpretable representations of behavioral policies and inferred the latent representation of expert demonstrations in an unsupervised way. Specifically, [30] learned a Variational Auto-Encoder (V AE) [31] where the encoder infers the latent

factors of variation from mixed demonstrations and the decoder models different types of expert behaviors. To strengthen the consistency between the learned policy and the types of agents, [12, 32] included "burn-in demonstrations" for updating the imitation policies. These methods, however, were proposed for imitation learning or rewards recovery (i.e., for IRL) [33] instead of constraint inference.

3 Problem Definition

Constrained Mixture-Agent Markov Decision Process. To support constraint inference for different agents, we formulate the environment as a Constrained Mixture-Agent Markov Decision Process (CMA-MDP) [34] M ϕ , which can be defined by a tuple ( S , A , Z , R , p T , { ( p C i , ϵ i ) } ∀ i , γ, µ 0 ) where: 1) S , A and Z denote the space of states, actions, and latent code (for specifying expert agents). 2) p T ( s ′ | s, a ) and R( s, a ) define the transition and reward functions. 3) p C j ( c | s, a, z ) refers to an agent-specific cost model with an associated bound ϵ j ( z ) , where j indicates the index of a constraint. 4) γ is the discount factor and µ 0 ( s ) defines the initial state distribution. CMA-MDP assumes the agents are differentiable by examining their policies, implying that different agent types cannot share identical policies. We aim to elucidate these differences with the constraints associated with each agent type.

In contrast to Multi-Agent Reinforcement Learning (MARL) [35], where multiple agents can act concurrently, CMA-MDP allows only one agent to operate at a given time. Nevertheless, since the agents in CMA-MDP might adhere to distinct constraints, they can develop different optimal control policies or strategies. However, standard MARL frameworks do not explicitly differentiate between agent types, nor do they distinguish their respective policies and constraints.

Policy Update under Conditional Constraints. We introduce Constrained Reinforcement Learning (CRL) based on CMA-MDP. For an agent identified by z , the goal of CRL is to find a policy π ∗ ( a | s, z ) that maximizes expected discounted rewards under the conditional constraints:

where H ( π ) denotes the policy entropy weighted by β , and we follow [4] to define the trajectory cost c ( τ | z ) = 1 -∏ ( s,a ) ∈ τ ϕ ( s, a | z ) where the permissibility function ϕ ( s, a | z ) indicates the probability that performing action a under a state s is safe for agent z . CRL commonly assumes that the constraints are known, whereas in practice, instead of directly observing the constraint signals, we often have access to expert demonstrations that follow the underlying constraints, and thus the agent must recover the constraints from the demonstration dataset.

Constraint Inference from a Mixture of Expert Dataset. Based on the CMA-MDP, the goal of ICRL is to discover the underlying constraints respected by different types of expert agents from their demonstrations. To achieve it, we consider the Maximum Entropy framework [4, 14] and represent the likelihood function as follows:

where 1) N denotes the number of trajectories in the demonstration dataset, 2) Z M ˆ c ϕ is a normalizing term, 3) the permissibility indicator 1 M ˆ c ϕ ( τ ( i ) ) can be defined by ϕ ( τ ( i ) | z ) = ∏ T t =1 ϕ t ( s i t , a i t | z ) , and 4) the agent identifier 1 τ ( i ) ∈D z determines whether the trajectory τ ( i ) is generated by the agent z .

By following [4], constraint inference can be formulated as inferring ϕ t by maximizing this likelihood function. A common approach is parameterizing ϕ t with neural network models and updating the model parameters with the agent-specific expert data [4, 6, 7]. However, the major challenge of this task lies in our lack of knowledge about the agent's identity that generates the expert trajectories (i.e., 1 τ ( i ) ∈D z is unknown). For example, a vehicle trajectory dataset does not label the specific type of vehicles, or these labels are coarse-grained , incomplete, and noisy. On the other hand, to explain the optimal policy with the learned constraints, the types of agents and the constraints must be consistent . If we evaluate the policy of an agent z under the constraint for another type of agent z ′ (e.g., evaluate the driving strategy of a truck under the constraint for a car), the performance will be substantially compromised. As a result, for a specific agent z , the expert trajectories { τ ∗ z } are optimal while the trajectories generated by other experts { τ ∗ z ′ , τ ∗ z ′′ τ ∗ z ′′′ ,... } are supposed to be sub-optimal

or even infeasible. To capture the correct constraints, the ICLR algorithm must identify the agent corresponding to each trajectory in an unsupervised manner.

4 Inverse Constrained Reinforcement Learning for a Mixture of Experts

In order to infer constraints for various expert agent types, we introduce the Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) algorithm (Figure 1). MMICRL employs a conditional imitation policy to capture the diverse agent behaviors, facilitating unsupervised agent identification. Technically, we require the agents' policies to 1) exhibit high entropy when the agent type (specified by z ) is unknown, and 2) collapse to a specific behavioral mode when the type is determined. The objective for MMICRL can be expressed as :

(Proof is provided in Appendix B.) where f ( · ) represents a latent feature extractor, H [ π ( τ | z )] denotes the agent-specific policy entropy, and H [ π ( τ )] signifies the entropy without knowledge of the agent type. The weighting parameters α 1 and α 2 (both ≥ 0 ) determine the balance between conditional entropy minimization and general entropy maximization. This objective differs from the traditional Maximum Entropy Inverse Reinforcement Learning (MEntIRL) [14] in two ways: 1) the objective also minimizes an entropy conditioned on agent types, and 2) it incorporates an additional constraint related to the policy's permissibility (the last constraint). Given this objective, the optimal representation for the trajectory likelihood (i.e., trajectory policy) is:

Proposition 4.1. Let p ( z | τ ) denote the trajectory-level agent identifier, let r ( τ ) = λ 0 α 2 -α 1 f ( τ ) denote the trajectory rewards, let Z M ˆ c ϕ denote a normalizing term. The optimal policy of the above optimization problem can be represented as:

Building upon this policy representation (4), we present the key steps of MMICRL , which involves iteratively performing: 1) unsupervised agent identification for calculating p ( z | τ ) (Section 4.1), 2) conditional inverse constraint inference (Section 4.2) for deducing ϕ ( τ | z ) , and 3) multi-modal policy update (Section 4.3) for approximating π ( τ | z ) . MMICRL alternates between these steps until the imitation policies reproduce expert trajectories, signifying that the inferred constraints align with the ground-truth constraints.

4.1 Unsupervised Agent Identification

MMICRL identifies expert trajectories (i.e., learning p ( z | τ ) ) in an unsupervised manner. Previous works commonly determined the agents' identities by examining the state-action features [10, 11, 12] with the classifier p ( z | s z , a z ) . Nevertheless, different agents may exhibit similar behaviors under certain contexts or at specific time steps within a trajectory, which makes this point-wise identification problematic (e.g., see our experimental results of InfoGAIL [10] in Section 5).

To derive a reliable agent identifier, MMICRL performs trajectory-level identification by estimating an agent-specific density. Specifically, we define a state-action density (i.e., normalized occupancy measure) ρ π ( s, a ) = (1 -γ ) π ( a | s ) ∑ ∞ t =0 γ t p ( s t = s | π ) where p ( s t = s | π ) is the probability density of state s at time step t following policy π . Based on this density, we consider the theorem:

Proposition 4.2 (Theorem 2 of [15]) . Suppose ρ is the occupancy measure that satisfies the Bellman flow constraints: ∑ a ρ ( s, a ) = µ 0 ( s ) + γ ∑ s ′ ,a ρ ( s ′ , a ) Pτ ( s ′ | s, a ) and ρ ( s, a ) ≥ 0 . Let the policy defined by: π ρ ( a | s ) = ρ ( s,a ) ∫ ρ ( s,a ′ )d a ′ , then π ρ is the only policy whose occupancy measure is ρ .

Density Estimation. The aforementioned theorem provides a crucial insight for agent identification: " one can identify an expert agent by examining the occupancy measures in the expert trajectories ". Leveraging this insight, we design a state-action density estimator to compute a density using a Conditional Flow-based Density Estimator (CFDE) . CFDE estimates the density of input variables in the training data distribution under an auto-regressive constraint. Moreover, to enhance our

density estimator's sensitivity to the behavior of different agents, the estimator also conditions on the agent type. so p ( x | z ) = ∏ i p ( x i | x 1: i -1 , z ) where x := ( s, a ) defines an event. We implement p ( x i | x 1: i -1 , z ) = N ( x i | µ i , ( exp ( α i )) 2 ) where µ i = ψ µ i ( x 1: i -1 , z ) and α i = ψ α i ( x 1: i -1 , z ) . The neural function ψ is implemented by stacking multiple MADE layers [36]. The corresponding agent identifier can be represented by Bayesian rule, so p ψ ( z | τ ) = p ( z ) · p ψ ( τ | z ) ∑ z p ( z ) · p ψ ( τ | z ) . We assume a uniform prior p ( z ) , thereby deriving the following form: p ψ ( z | τ ) = ∏ ( s,a ) ∈ τ p ψ ( s,a | z ) ∑ z ′ ∏ ( s,a ) ∈ τ p ψ ( s,a | z ′ ) .

Agent Identification. After learning the density model p ψ ( s, a | z ) with CFDE, we divide D E into sub-datasets {D z } |Z| z =1 by: 1) initializing the dataset D z = ∅ and 2) ∀ τ i ∈ D E , adding τ i into D z if z = arg max z ∑ ( s,a ) ∈ τ i log[ p ψ ( s, a | z )] . We repeat the above steps for all z ∈ Z .

4.2 Agent-Specific Constraint Inference

Based on the identified expert dataset, we have p ω ( z | τ z ) = 1 , ∀ τ z ∈ D z , and thus log p ω ( z | τ z ) = 0 , so the likelihood function (4) can be simplified to:

where ϕ ( τ ( i ) z ) = ∏ ( s,a ) ∈ τ ( i ) z ϕ t ( s, a | z ) and the normalizer Z M ˆ c ϕ = ∫ exp[ r ( τ )] ϕ ( τ ) λ 2 α 1 -α 2 d τ . By defining η = λ 2 α 1 -α 2 , we can then define the log-likelihood log[ p ( D z | ϕ, z )] as:

We parameterize the instantaneous permissibility function with ω , i.e., construct ϕ ω ( s t , a t | z ) and update the parameters by computing the gradient of the above likelihood function (the derivation resembles that of [4]), so ∇ ω log [ p ( D z | ϕ, z )] can be defined as:

This inverse constraint objective relies on the nominal trajectories ˆ τ sampled with the conditional policy π M ˆ ϕ ( τ | z ) (also see Figure 1). For simplicity, we denote it as π ( τ | z ) . In the following, we will introduce our approach to learning π ( τ | z ) .

4.3 Multi-Modal Policy Optimization

By definition, the policy ˆ π ( τ | z ) is trained to maximize cumulative rewards subject to constraint E τ ∼ π ( ·| z ) [log ϕ ( τ | z )] ≥ ϵ . To be consistent with our MMICRL objective in formula (3), we design the multi-modal policy optimization objective in the following:

This objective extends maximum entropy policy optimization by minimizing an additional agentconditioned entropy H [ π ( τ | z )] , which limits the variance of policy distribution for a specific type of agent. The balance between these entropy terms is controlled by α 1 and α 2 . Since H [ π ( τ )] = H [ π ( τ | z )] + E z ∼ p ( z ) ,τ ∼ π ( τ | z ) [ log[ p ψ ( z | τ )] ] + H ( z ) , and by removing the uniform prior p ( z ) (since it is independent of the policy update), the objective (8) can be simplified to:

Intuitively, this objective expands the reward signals with a log-probability term log[ p ψ ( z | τ )] , which encourages the policy to generate trajectories from high-density regions for a specific agent type. This approach ensures that the learned policies π ( ·| z ) Z z =1 are differentiable.

Learning Diverse Policies via Contrastive Estimation. In practice, directly augmenting the reward with a log-probability term (as in objective 9) may lead to a sub-optimal policy [7]. This issue

arises because the log-probability term assigns a large penalty to trajectories with low p ψ ( z | τ ) . In such cases, the controlling policy becomes more sensitive to density estimation (since p ψ ( z | τ ) = softmax [ p ψ ( τ | z )] , see Section 4.1) rather than the reward signals. Balancing the trade-off between reward and density maximization by identifying an optimal weight α 1 is challenging, especially when predictions from p ψ ( τ | z ) are less accurate at the beginning of training (i.e., during the cold start).

To resolve the above issues, we consider replacing the identification probability with a contrastive estimation method by constructing the following objective for policy optimization:

̸

where V defines the probing sets (constructed with the density estimator in Algorithm 1). Given a specific agent type z , these probing vectors are among the most representative data points since they are located in a high-density region conditioning on z and a low-density region conditioning on other agent types ( ˜ z = z ). Inspired by the InfoNCE loss [16, 37] , L ce can be calculated as:

̸

where f s denotes the score function for measuring the similarity between the features from different state-action pairs (we use cosine similarity in the experiment). To interpret L ce ( τ, V 1 ,..., | Z | ) , we can treat ( s, a ) ∈ { τ, V z } as positive embeddings for π ( ·| z ) since they are generated by this policy. On the other hand, (˜ s, ˜ a ) ∈ {V ˜ z } ˜ z = z are negative embeddings for π ( ·| z ) since they are generated by controlling with other policies π ( ·| ˜ z ) (where ˜ z = z ). Considering the generation is influenced by the stochasticity in environment dynamics (e.g., transition functions), we can equivalently view the generation process as injecting the environmental noise into the policy , and thus Noise Contrastive Estimation (NCE) [38] becomes a compatible tool for learning differentiable policies. Specifically, since embeddings in V z belong to a high conditional density region in p ( ·| z ) (see Algorithm 1), the knowledge from the density estimator has been integrated into policy updates.

̸

In essence, the integration of contrastive learning into policy optimization helps the algorithm to better understand the relationships between agents, their behaviors, and the corresponding expert trajectories, resulting in improved performance for tasks involving diverse agent types. Algorithm 2 introduces the detailed implementation of MMICRL .

Algorithm 1: Probing_Sets

Input: Agent type z , trajectory dataset D z and conditional density model p ( ·| z ) Initialize a probing sets V = {∅} ;

̸

5 Experiments

Running Setting. For a comprehensive comparison, we employ consistent evaluation metrics across all environments. These metrics include 1) Constraint Violation Rate , which assesses the likelihood of a policy violating a constraint in a given trajectory, and 2) Feasible Cumulative Rewards , which calculates the total rewards accumulated by the agent before violating any constraints. The demonstration data are assumed to have a zero violation rate. We run experiments with three different seeds and present the mean ± std results for each algorithm. To ensure a fair comparison, we maintain uniform settings for all comparison baselines. Appendix A.2 reports the detailed settings.

Comparison Methods. We employ an ablation strategy to progressively remove components from MMICRL (Algorithm 2) and consider the following variants: 1) MMICRL-LD excludes our contrastive estimation approach, using the objective (9) directly for policy updates. 2) Inspired by [10], InfoGAIL-ICRL replaces the trajectory density model with a discriminator to identify agents based on state-action pairs, extending GAIL [28] to distinguish between distinct agents' trajectories. 3) MEICRL [4] eliminates the agent identifier and expands traditional Maximum

Entropy (ME) IRL methods to infer Markovian constraints in a model-free environment. 4) Binary Classifier Constraint Learning (B2CL) constructs a deterministic binary classifier directly for constraint inference, bypassing the need for the maximum entropy framework.

5.1 Empirical Evaluations in Discrete Environments

The discrete environments we use are based on Gridworld, a widely studied RL environment that enables us to visualize the recovered constraints and trajectories generated by various agents. We create a 7x7 Gridworld map and design four distinct constraint map settings. In Figure 2, the leftmost column illustrates expert trajectories and constraints for each configuration. Notably, the first three settings (rows 1-3 in Figure 2) incorporate two different types of constraints, while the final setting (the last row in Figure 2) includes three constraint types. The primary goal for each agent is to navigate from the starting point to the endpoint while minimizing the number of steps and avoiding their respective constraints. To facilitate constraint inference, a demonstration dataset containing expert trajectories is provided for each environment [7].

Figure 2 displays the ground-truth trajectory map alongside the learned trajectory maps. Appendix C.1 summarizes the detailed performance in terms of feasible cumulative rewards, constraint violation rate, and destination-reaching rate. Without implementing agent identification, both B2CL and MEICRL can only restore a single constraint type. Although InfoGAIL-ICRL incorporates agent identification, its performance is unsatisfactory, resulting in incomplete destination-reaching. In contrast, MMICRL-LD and MMICRL exhibit significant improvements; they can identify various agent types corresponding to different constraints and generate accurate trajectories that successfully reach the endpoint. Notably, the enhancements provided by MMICRL are more prominent, and the trajectories produced by the MMICRL algorithm closely resemble expert demonstrations.

5.2 Empirical Evaluations in Continuous Environments

In continuous environments, we evaluate ICLR algorithms by whether they can infer location constraints for various types of robots (robots can only operate within designated areas). The task involves directing the robot to navigate out of the danger zone, represented by the constraint.

Ground-Truth_Setting1

trajectory_B2CL

trajectory_MEICRL

trajectory_InfoGAIL-ICRL

trajectory_MMICRL-LD

trajectory_MMICRL

1th Traj, agent 1

1th Traj, agent 0

1th Traj, agent 0

1th Traj, agent 0

1th Traj, agent 0

1th Traj, agent 1

2th Traj, agent 0

3th Tral, agent 1

2th Traj, agent 0

3th Traj, agent 0

2th Traj, agent 0

3th Tral, agent 0

2th Traj, agent 1

3th Traj, agent 0

2th Traj, agent 1

3th Traj, agent 0

2th Traj, agent 0

3th Tra], agent 1

00

4th Traj], agent 1

Ground-Truth_Setting2

trajectory_B2CL

trajectory_MEICRL

trajectory_InfoGAIL-ICRL

trajectory_MMICRL-LD

trajectory_MMICRL

6

6

6

6

6

5

h Traj, agent 0

Traj, agent 1

Traj, agent 0

0 uafe

h Traj, agent 0

Traj, agent 0

Traj, agent 1

Traj, agent 0

Traj, agent 1

Traj., agent 0

h Traj, agent 1

Traj, agent 0

Traj, agent 1

Traj, agent 0

Traj, agent 0

Traj, agent 0

Traj,agent 0

th Traj, agent 0

qth Tral agent o

0

th Traj, agent 1

4

+

4

4

b

Q

4

Ground-Truth_Setting3

trajectory_B2CL

trajectory_MEICRL

trajectory_InfoGAIL-ICRL

trajectory_MMICRL-LD

trajectory_MMICRL

6

5

5

5

3

Traj, agent 0

h Traj, agent 1

th Traj, agent 1

Traj, agent 0

h Traj, agent 0

Traj, agent 1

h Traj, agent 0

ent1

Ground-Truth_Setting4

trajectory_B2CL

trajectory_MEICRL

trajectory_InfoGAIL-ICRL

trajectory_MMICRL-LD

trajectory_MMICRL

64

2

1th Traj, agent 2

1th Traj, agent 0

1th Traj, agent 0

1th Traj, agent 2

1th Traj, agent 0

1th Traj, agent 0

2th Traj, agent 1

2th Traj, agent 0

2th Traj, agent 0

2th Traj, agent 0

2th Traj, agent 1

2th Traj, agent 1

3th Traj, agent 0

3th Traj, agent 0

3th Traj, agent 0

3th Traj, agent 1

3th Traj, agent 2

3th Traj, agent 2

Webase our continuous environments on MuJoCo [17]. For each environment, we configure two types of robots-one moving forward and the other moving backward-each with opposing constraints. Table 1 provides an overview of the settings. Our goal is to distinguish their respective constraints and enable each robot type to maximize rewards without violating constraints.

Figure 4 and Table 2 illustrate the constraint violation rate and feasible cumulative rewards for each agent across various environments. To evaluate the accuracy of the discovered constraints, Figure 3 illustrates the distribution of xcoordinate values for all states visited by agents of different types. Compared to other baselines, MMICRL consistently exhibits lower constraint violation rates and higher cumulative rewards, averaged over all agents. This superiority can be attributed to MMICRL's ability to capture the diversity of constraints by inferring agent-specific constraints and learning heterogeneous policies through contrastive estimation. While certain

1500

1500

2000

1000

400

1000

1500

500

200

500

1000

500

-10

10

0

S'0=

0.0

0.5

-10

600

1500

1000

200

400

1000

005

100

200

500

0

50

50

0

200

Q

200

0

0.5

0'0

1.0

0

20

10

10

baselines may achieve higher cumulative rewards for a specific agent, none of these algorithms consistently outperform the others. (refer to Appendix C.2 for other complete results)

5.3 Robust Test: Can MMICRL Recover from the Early Mistakes?

To test the robustness of MMICRL , we enforce MMICRL to make incorrect agent identifications and assess whether MMICRL can recover from these errors. Specifically, in the first round of training, rather than using our density estimator, we randomly assign incorrect labels to 20% of expert trajectories. These incorrect identifications can impact ongoing constraint inference and policy updates. To recover from these mistakes, the MMICRL algorithm must accurately model diverse agent preferences and rectify the incorrect trajectories to match the appropriate agent type.

Method

Blocked Half-Cheetah

Blocked Walker

Feasible Cumulative Rewards

B2CL

5.03E+1 / 2.12E+3

1.33E+4 / 3.11E+3

3.28E+2 / 7.40 E-1

2.19E+1 / 7.88E+1

MEICRL

4.85E+1 / 3.45E+3

1.80E+4 / 3.31E+3

2.01E+2 / 8.30E-1

2.61E+1 / 7.33E+1

InfoGAIL-ICRL

2.22E+2 / 1.33E+2

3.73E+2 / 1.11E+2

2.38E+1 / 7.90E-1

2.15E+1 / 1.58E+3

MMICRL-LD

4.32E+3 / 2.56E+3

1.82E+4/ 2.21E+4

2.66E+2 / 6.10E+2

9.02E+2 / 5.12E+2

MMICRL

6.12E+3 / 3.06E+3

2.13E+4

/

2.17E+4

4.07E+2 / 6.48E+2

8.98E+2

/

1.53E+3

Constraint Violation Rate

B2CL

100%

±

0% / 67%

±

24%

33%

±

24% / 67

±

24%

62%

±

12% / 100%

±

0%

100%

±

0% / 100%

±

0%

MEICRL

100%

±

0% / 50%

±

25%

0%

±

0% / 67

±

24%

79%

±

14% / 100%

±

0%

100%

±

0% / 100%

±

0%

InfoGAIL-ICRL

25%

±

16% / 93%

±

5%

34%

±

18% / 37%

±

19%

73%

±

12% / 100%

±

0%

100%

±

0% / 0%

±

0%

MMICRL-LD

33%

±

24% / 34%

±

24%

0%

±

0% / 0%

±

0%

71%

±

16% / 34%

±

23%

52%

±

25% / 50%

±

25%

MMICRL

0%

±

0% / 0%

±

0%

0%

±

0% / 0%

±

0%

55%

±

23% / 28%

±

19%

31%

±

22% / 25%

±

22%

Rewards

BlockedAntwallz_0

BlockedAntwallz_1

Constraint Violation Rate

BlockedAntwallz_0

Constraint Violation Rate

BlockedAntwallz_1

20000

0.8

0.8

Feasible CumulativeP

Feasible CumulativeP

0.6

0.6

10000

B2CL

10000

B2CL

0.4

B2CL

0.4

MM

InfoGAIL-ICRL

MEICRL

InfoGAIL-ICRL

MEICRL

0.2

InfoGAIL-ICRL

MEICRL

0.2

InfoGAIL-ICRL

MEICRL

MMICRL-LD

MMICRL-LD

MMICRL-LD

MMICRL-LD

0

MMICRL

0

MMICRL

0.0

MMICRL

0.0

MMICRL

0

5000

10000

15000

0

5000

10000

15000

0

5000

10000

15000

0

5000

10000

15000

Episode

Episode

Episode

Episode

Feasible Cumulative Rewards

BlockedHalf-Cheetahz_0

Feasible Cumulative Rewards

BlockedHalf-Cheetahz_1

BlockedHalf-Cheetahz_0

Constraint Violation Rate

1.00

BlockedHalf-Cheetahz_1

6000

4000

0.75

0.75

4000

0.50

0.50

B2CL

2000

B2CL

B2CL

2000

MEICRL

InfoGAIL-ICRL

MEICRL

InfoGAIL-ICRL

MEICRL

InfoGAIL-ICRL

0.25

InfoGAIL-ICRL

0

MMICRL-LD

MMICRL

0

MMICRL-LD

MMICRL

MMICRL

MMICRL-LD

0.00

MMICRL

MMICRL-LD

0

1000

2000

3000

4000

0

1000

200030004000

0

1000

2000

3000

4000

0

1000

2000

3000

4000

Episode

Episode

Episode

Episode

eRewards

1500

BlockedWalkerWithPosz_0

Rewards

BlockedWalkerWithPos z_1

ViolationKate

1.00

BlockedWalkerWithPosz_0

ViolationRate

1.00

Blocked WalkerWithPos z_1

2000

CumulativeP

1000

Feasible CumulativeF

1500

0.75

0.75

Feasible(

500

InfoGAIL-ICRL

MMICRL-LD

B2CL

MEICRL

1000

500

InfoGAIL-ICRL

B2CL

MEICRL

MMICRL-LD

Constraint

0.50

0.25

InfoGAIL-ICRL

B2CL

MEICRL

MMICRL-LD

Constraint

0.50

0.25

InfoGAIL-ICRL

MEICRL

B2CL

MMICRL-LD

0

0

MMICRL

10000 20000 30000 40000

0

0

MMICRL

10000 20000 30000 40000

0.00

0

MMICRL

100002000030000 40000

0.00

0

MMICRL

10000 20000 30000 40000

Episode

Episode

Episode

Episode

Feasible Cumulative Rewards

BlockedSwimmerWithPosz_0

Rewards

BlockedSwimmerWithPosz_1

BlockedSwimmerWithPosz_0

BlockedSwimmerWithPosz_1

600

800

1.00

0.8

400

200

InfoGAILICRL

B2CL

MEICRL

MMICRL-LD

Feasible

200

400

InfoGAILICRL

B2CL

MEICRL

MMICRL-LD

0.6

0.4

InfoGAIL-ICRL

B2CL

MMICRL-LD

MEICRL

MAAL

InfoGAILICRL

MEICRL

MMICRL-LD

B2CL

0

MMICRL

0

MMICRL

MMICRL

MMICRL

0

250050007500 10000

0

250050007

7500 10000

0

25005000750010000

0

2500

5000750010000

Episode

Episode

Episode

Episode

Figure 5 displays the performance of MMICRL in both continuous and discrete settings. We observe that it can successfully recover multiple constraints corresponding to different types of agents. The errors made at the beginning of training do not have a significant impact on the overall performance, which demonstrates the robustness of MMICRL.

Constraint Violation Rate

Blocked Antwall z_0

Constraint Violation Rate

Blocked Antwall z_1

Feasible Cumulative Rewards

Blocked Antwall z_0

Rewards

Blocked Antwall z_1

20000

20000

0.4

0.4

15000

Feasible Cumulative R

15000

0.2

0.2

10000

10000

5000

5000

0.0

0

MMICRL

5000

10000

15000

0.0

0

MMICRL

5000

10000

15000

0

0

MMCRL

5000

10000

15000

0

0

MMICRL

5000

10000

15000

Episode

Episode

Episode

Episode

Constraint Violation Rate

0.6

Gridworld-Settingl-z_0

Constraint Violation Rate

0.3

Gridworld-Settingl-z_1

Feasible Cumulative Rewards

0.3

Gridworld-Setting1-z_0

Feasible Cumulative Rewards

0.7

Gridworld-Setting1-z_1

0.2

0.4

0.2

0.1

0.6

0.2

0.1

0.1

0.0

0.5

0.4

0.0

MMICRL

0.0

MMCRL

0.2

MMICRL

0.3

MMICRL

0

100

200

O0E

0

100

200

O0E

0

100

200

OOE

100

200

00E

Episode

Episode

Episode

Episode

5.4 Empirical Evaluations in Realistic Environments

To demonstrate the generalization capability of the model, we conducted some preliminary studies in a realistic autonomous driving environment. This environment is constructed by utilizing the HighD dataset (For more details, check [7] and [9]). For extracting features of cars and roads, we use the features collector from Commonroad RL [39]. The constraint that we are interested in are 1) Car distance ≥ 20 m (agent 0) and 2) Car distance ≥ 40 m (agent 1) . Figure 6 shows the distribution of car distance in expert demonstrations for agent 0 and agent 1. We aim to investigate whether the MMICRL algorithm can differentiate between two different types of cars based on distance constraints.

obstacle_distance_expert_demonstration

120

agento

agent 1

100

frequency

80

60

40

20

0

0

20

40

60

80

same lane leadinq obstacle distance

agent_o_obstacle_distance_visual

agent-o_obstacle_distance_visual

agent-o_obstacle_distance_visual

agent-o_obstacle_distance_visual

350

300

35

80

300

250

30

70

60

frequency

250

frequency

25

frequency

50

2

200

4

150

15

30

100

100

10

20

50

50

10

0

0

same_lane_leading_obstacle_distance

20

40

60

80

100

0

same_lane_leading_obstacle_distance

20

40

60

80

100

0

0

same_lane_leading_obstacle_distance

20

40

60

80

100

0

0

same_lane_leading_obstacle_distance

20

40

60

80

100

agent_1_obstacle_distance_visual

agent-1_obstacle_distance_visual

agent-1_obstacle_distance_visual

agent-1_obstacle_distance_visual

350

200

250

200

175

300

150

200

175

frequency

250

200

frequency

125

150

100

150

75

75

100

50

50

50

50

25

25

0

same_lane_leading_obstacle_distance

20

40

60

08

100

0

same_lane_leading_obstacle_distance

20

40

60

80

100

0

0

same_lane_leading_obstacle_distance

20

40

60

80

100

0

same_lane_leading_obstacle_distance

20

40

60

08

100

In the four settings shown in Figure 7, we find that in the first row, the observed distances of the first type of car are all above 20 meters, while in the second row, the distances of the other type of car are mostly above 40 meters. This suggests that the MMICRL algorithm shows promising preliminary results in this context. In the future, we will further optimize the algorithm to achieve a lower constraint violation rate and higher effective rewards in a wider range of real-world environments.

6 Limitations

Omitting Agent Interactions : MMICRL does not consider the interactions between agents or how they can potentially collaborate or compete to satisfy a joint constraint. Future research could extend the game theory to ICRL for implementing multi-agent constraint inference.

Experiment in Virtual Environment. We evaluate our algorithm in virtual games rather than real-world applications (e.g., autonomous driving). This is due to the lack of an ICRL benchmark for a mixture of experts. Future work can explore the application of MMICRL in realistic scenarios.

7 Conclusion

In this work, we introduce the MMICRL algorithm to differentiate multiple constraints corresponding to various types of agents. MMICRL incorporates unsupervised constraint inference, agent-specific constraint inference, and multi-modal policy optimization. To demonstrate the advantages of our method over other baselines, we investigate whether MMICRL can accurately perform multiple constraint inference in both discrete and continuous environments.

Acknowledgments and Disclosure of Funding

The work is supported in part by the National Key R&D Program of China under grant No2022ZD0116004, by Shenzhen Science and Technology Program ZDSYS20211021111415025, and by the Start-up Fund UDF01002911 of the Chinese University of Hong Kong, Shenzhen. We acknowledge the funding from the Canada CIFAR AI Chairs program, and the support of the Natural Sciences and Engineering Research Council of Canada (NSERC).

References

Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research (JMLR) , 16:1437-1480, 2015.

Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints in model-free reinforcement learning: A survey. In International Joint Conference on Artificial Intelligence (IJCAI) , pages 4508-4515, 2021.

Dexter R. R. Scobee and S. Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. In International Conference on Learning Representations, (ICLR) , 2020.

Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International Conference on Machine Learning (ICML) , pages 7390-7399, 2021.

Mattijs Baert, Pietro Mazzaglia, Sam Leroux, and Pieter Simoens. Maximum causal entropy inverse constrained reinforcement learning. arXiv preprint arXiv:2305.02857 , 2023.

Ashish Gaurav, Kasra Rezaee, Guiliang Liu, and Pascal Poupart. Learning soft constraints from constrained expert demonstrations. In International Conference on Learning Representations, (ICLR) , 2023.

Guiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezaee, and Pascal Poupart. Benchmarking constraint inference in inverse reinforcement learning. In International Conference on Learning Representations (ICLR) , 2023.

Dimitris Papadimitriou, Usman Anwar, and Daniel S Brown. Bayesian methods for constraint inference in reinforcement learning. Transactions on Machine Learning Research , 2023.

Robert Krajewski, Julian Bock, Laurent Kloeker, and Lutz Eckstein. The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems. In International Conference on Intelligent Transportation Systems (ITSC) , pages 2118-2125, 2018.

Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems (NeurIPS) , pages 3812-3822, 2017.

Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav S. Sukhatme, and Joseph J. Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS) , 2017.

Alex Kuefler and Mykel J. Kochenderfer. Burn-in demonstrations for multi-modal imitation learning. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) , pages 1071-1078, 2018.

Ziyu Wang, Josh Merel, Scott E. Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems (NeurIPS) , pages 5320-5329, 2017.

Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI) , pages 1433-1438, 2008.

Umar Syed, Michael H. Bowling, and Robert E. Schapire. Apprenticeship learning using linear programming. In International Conference on Machine Learning (ICML) , pages 1032-1039, 2008.

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR , abs/1807.03748, 2018.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems IROS , pages 5026-5033. IEEE, 2012.

Dexter R.R. Scobee and S. Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. In International Conference on Learning Representations (ICLR) , 2020.

David Livingston McPherson, Kaylene C. Stocking, and S. Shankar Sastry. Maximum likelihood constraint inference from stochastic demonstrations. In IEEE Conference on Control Technology and Applications (CCTA) , pages 1208-1213, 2021.

Daehyung Park, Michael D. Noseworthy, Rohan Paul, Subhro Roy, and Nicholas Roy. Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning. In Annual Conference on Robot Learning (CoRL) , pages 1005-1014, 2019.

Alex James Chan and Mihaela van der Schaar. Scalable bayesian inverse reinforcement learning. In International Conference on Learning Representations (ICLR) , 2021.

Daniel S. Brown, Scott Niekum, and Marek Petrik. Bayesian robust optimization for imitation learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Shicheng Liu and Minghui Zhu. Distributed inverse constrained reinforcement learning for multi-agent systems. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Monica Babes, Vukosi Marivate, Kaushik Subramanian, and Michael L. Littman. Apprenticeship learning about multiple intentions. In International Conference on Machine Learning (ICML) , pages 897-904, 2011.

Christos Dimitrakakis and Constantin A. Rothkopf. Bayesian multitask inverse reinforcement learning. In Recent Advances in Reinforcement Learning - European Workshop (EWRL) , pages 273-284, 2011.

Igor Vozniak, Matthias Klusch, André Antakli, and Christian Müller. Infosalgail: Visual attention-empowered imitation learning of pedestrian behavior in critical traffic scenarios. In International Joint Conference on Computational Intelligence (IJCCI) , pages 325-337, 2020.

Jiahao Lin and Zongzhang Zhang. ACGAIL: imitation learning about multiple intentions with auxiliary classifier gans. In Trends in Artificial Intelligence - Pacific Rim International Conference on Artificial Intelligence (PRICAI) , pages 321-334, 2018.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS) , pages 4565-4573, 2016.

Yunzhu Li, Jiaming Song, and Stefano Ermon. Inferring the latent structure of human decisionmaking from raw visual inputs. In Advances in Neural Information Processing Systems (NeurIPS) , 2017.

Fang-I Hsiao, Jui-Hsuan Kuo, and Min Sun. Learning a multi-modal policy via imitating demonstrations with mixed behaviors. CoRR , abs/1903.10304, 2019.

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning (ICML) , pages 1278-1286, 2014.

Raunak P. Bhattacharyya, Blake Wulfe, Derek J. Phillips, Alex Kuefler, Jeremy Morton, Ransalu Senanayake, and Mykel J. Kochenderfer. Modeling human driving behavior through generative adversarial imitation learning. IEEE Transactions on Intelligent Transportation Systems , 24:2874-2887, 2023.

Chelsea Finn, Paul F. Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. CoRR , abs/1611.03852, 2016.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning (ICML) , pages 22-31, 2017.

Kaiqing Zhang, Zhuoran Yang, and Tamer Ba¸ sar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control , pages 321-384, 2021.

Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. MADE: masked autoencoder for distribution estimation. In International Conference on Machine Learning (ICML) , volume 37, pages 881-889, 2015.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In International Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9726-9735, 2020.

Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In International Conference on Artificial Intelligence and Statistics, (AISTATS) , volume 9, pages 297-304, 2010.

Xiao Wang, Hanna Krasowski, and Matthias Althoff. Commonroad-rl: A configurable reinforcement learning environment for motion planning of autonomous vehicles. In International Intelligent Transportation Systems Conference (ITSC) , 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the algorithm Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) for imitation learning mixture of expert demonstrations with various constraints. The algorithm includes agent identification, agent-specific constraint inference and multi-modal policy optimization. The problem is challenging, the proposed method seems to be novel, and the simulation experiments show the effectiveness of the proposed method.\n\nSTRENGTHS:\nI think the problem of multi-modal learning from a mixture of experts with constraints is indeed a challenging one, and the proposed algorithm has necessary components for handling several difficulties. \n\nThe paper has good writing, and the method is described clearly with necessary derivation details. Although lots of mathematical notations are used, it’s good to see intuitive explanations for most terms.\n\nThe experiments are evaluated on both discrete and continuous environments. The improvement of the proposed algorithm over baseline methods is significant, although not consistent across tasks.\n\nWEAKNESSES:\nOverall the paper has a clear description of the reasons for different design choices, but there are still several places confusing for me, detailed in the Questions section.\n\nFor proposition 4.2, please provide the complete conditions for the statement. For the case with a limited number of samples and estimated occupancy measure, what is the change for this statement?\n\nIn Table 2, it could be better to provide the constraint violation rates in the expert demonstration data as a reference. Does the demonstration data have zero violation rate? It may be mentioned somewhere in the paper that I missed.\n\nFor Fig. 4, is it possible to have zero or lower constraint violation while remaining high rewards for tasks Blocked WalkerWithPos and Blocked SwimmerWithPos. \n\nIt could be better to see the method tested on some more realistic cases like vehicle driving dataset, etc.\n\nQUESTIONS:\nFrom Eq. (9) to (10), why using the contrastive estimation method helps with handling the trade-off of reward maximization and density maximization? The contrastive estimation loss also induces an additional term from the reward.\n\nIn Eq. (10), how is the reward of the trajectory estimated conditioned on the current policy? Is there any value function estimation in the policy optimization process?\n\nLIMITATIONS:\nThe limitations of the work are well-discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new algorithm called Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) to address the challenge of recovering multiple underlying constraints from a mixture of trajectories demonstrated by different types of expert agents. The algorithm utilizes a flow-based density estimator to identify experts from the demonstration data and infer agent-specific constraints. MMICRL then employs a novel multi-modal constrained policy optimization objective to imitate expert policies, considering both conditioned and unconditioned policy entropy. The approach is further enhanced using contrastive learning to improve robustness. Experimental results in discrete and continuous environments demonstrate that MMICRL outperforms other baseline algorithms in terms of constraint recovery and control performance.\n\nSTRENGTHS:\nThe paper introduces a new algorithm, MMICRL, which addresses the challenge of recovering multiple constraints from a mixture of trajectories demonstrated by different types of expert agents. This is a novel and important problem in the field of Inverse Reinforcement Learning (IRL). MMICRL utilizes a flow-based density estimator to identify expert agents from demonstration data. This approach allows for unsupervised expert identification and contributes to the ability to infer agent-specific constraints accurately. The incorporation of the policy optimization objective into the contrastive learning framework enhances the robustness of the algorithm. This addition strengthens the learning process and improves the overall performance of MMICRL.\n\nThe paper presents extensive experiments conducted in both discrete and continuous environments to evaluate the performance of MMICRL. The experimental results demonstrate that MMICRL outperforms other baseline algorithms in terms of constraint recovery and control performance, indicating the effectiveness and superiority of the proposed approach.\n\nWEAKNESSES:\nI am not an expert in IRL, but I know some RL algorithms [1] make use of mixture models and their modelling methods can even be applied in other fields, like computer vision[2]. I am wondering if is it possible to make these modelling methods in IRL and what is the advantage of MMICRL compared with these methods.\n\n[1] Ren, Jie, et al. \"Probabilistic mixture-of-experts for efficient deep reinforcement learning.\" arXiv preprint arXiv:2104.09122 (2021).\n\n[2] Xia, Xiaobo, et al. \"Pluralistic image completion with gaussian mixture models.\" Advances in Neural Information Processing Systems 35 (2022).\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of estimating constraints from a dataset consisting of a mixture of expert trajectories, and proposes an algorithm (MMICRL) for solving that problem.\nMMICRL proceeds iteratively, first estimating which trajectories belong to which agent class using a density estimation approach (over state-action occupancy), and second infers constraints on a per-agent class basis (using a MLE constraint inference procedure).\nThe policy is learned concurrently using an agent-unconditional max entropy, agent-conditional min entropy objective.\nExperiments in gridworld and mujoco environments show that MMICRL outperforms baselines as measured by permissible expected return and constraint violation rate.\n\nSTRENGTHS:\n### Originality\n- The paper proses what seems to be a novel setting of inverse constrained RL with a mixture of experts\n- The combination of methods (unsupervised agent clustering, constraint inference, policy learning) used in MMICRL appears novel, the application of certain methods (state-action occupancy density estimation, contrastive objective) to ICRL seems novel, and some aspects seem entirely novel (in-class min entropy, out-of-class max entropy).\n### Quality\nEvaluation\n- The evaluation of the proposed algorithm is quite strong. Appropriate baselines are considered (given the novel setting they double as ablations), and experiments are well-designed to illustrate the advantages of the approach - the grid world demonstrates the behavior in an interpretable manner, and the mujoco experiments demonstrate the usefulness of function approximation in the setting.\n- Experiments are run multiple times over different random seeds, with aggregate performance reported.\n- Experiments investigating susceptibility of the algorithm to initial poor performance do a good job of conveying how robust the algorithm is in practice\n### Clarity\n- The algorithm and experiments are presented fairly clearly, with generally useful / clear figures, a specification of the algorithm, and results clearly summarized / visualized.\n\nWEAKNESSES:\n### Significance\n- Does constraint inference in the context of a mixture of expert demonstrations makes sense as a problem formulation? If different experts disagree on what is or is not a constraint, then does it make sense to formulate the problem as constraint inference (or would rewards be more appropriate)? The motivation for ICRL is typically that the behavior of experts is often best explained by a simple reward function + a set of constraints [1], but that does not seem to be the case here. The actionable critiques are (1) the paper does not apply the method in a realistic setting and (2) the examples given in the paper of real applications (driving datasets with unlabeled car types) are unrealistic (and relies on partial observability (of the object type) more than a mixture of experts).\n### Originality\n- Constraint inference in the context of a mixture of expert behavior does seem novel, but the analogous extension for reward learning is not (e.g., [2,3,4])\n- MMICRL largely combines existing methods (density estimation, constraint inference, policy optimization) adapting for the multi-modal case (with some novel contributions noted in the strengths)\n### Quality\nAlgorithm\n- What can be shown theoretically about the algorithm? For example, does it converge under some assumptions? Discuss of such topics seems missing from the paper\nEvaluation\n- Related to the point about significance: the evaluation settings are unrealistic. An example with real world data would dramatically improve the quality of the evaluation. \nClarity\n- It’s not clear until later in section 4 that the algorithm iteratively performs agent identification / clustering, which makes section 4.1 confusing when first reading through\n- Figure 4 could be improved wrt clarity. Perhaps aggregating over environments would help.  \n\n[1] Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning \\\n[2] LiMIIRL: Lightweight Multiple-Intent Inverse Reinforcement Learning \\\n[3] Dealing with multiple experts and non‑stationarity in inverse reinforcement learning: an application to real‑life problems \\\n[4] Nonparametric Bayesian Inverse Reinforcement Learning for Multiple Reward Functions\n\nQUESTIONS:\n1. How is evaluation performed given that the agent-class is identified in an unsupervised manner? It seems like the relevant z value must be inferred either heuristically or manually in order to evaluate e.g., constraint violations. If so, this seems like a limitation of the approach, the reason being that identifying which z value to use in a practical setting would be challenging (given the use of CIRL in the first place implies complex / high-dimensional constraints).\n2. Is it correct that the method is limited to a setting with a fixed number of agent classes? This appears to be the case based on the algorithm. If so, (1) how robust is the algorithm to the number of classes chosen being incorrect? And (2) this also seems like a major limitation in that often real world data doesn’t cluster into a discrete set (but rather exists along a continuous spectrum). \n3. Section 3 has this line: “CMA-MDP assumes the agents are differentiable by examining their policies, implying that different agent types cannot share identical policies”. Would you elaborate on both the differentiability claim and the implication?\n\nLIMITATIONS:\nSee weaknesses and questions sections", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n# Problem Statement\nThe paper addresses a significant problem in Inverse Constraint Reinforcement Learning (ICRL), which is the assumption that all expert demonstrations follow the same constraints. This assumption is problematic because in real-world scenarios, demonstration data may come from various agents who follow different or even conflicting constraints. Therefore, using a single constraint model to explain the behaviors of diverse agents can lead to inaccuracies.\n\n# Main Contribution\nTo tackle this issue, the paper introduces the Multi-Modal Inverse Constrained Reinforcement Learning (MMICRL) algorithm. This approach allows imitation policies to capture the diversity of behaviors among expert agents. The paper demonstrates that MMICRL outperforms other baselines in terms of constraint recovery and control performance in both discrete and continuous environments.\n\n# Methedology\nThe proposed MMICRL estimates multiple constraints corresponding to different types of experts, allowing for a more accurate representation of diverse agent behaviors. It uses a flow-based density estimator for unsupervised expert identification from demonstrations, infers agent-specific constraints, and optimizes a multi-modal policy that minimizes the agent-conditioned policy entropy and maximizes the unconditioned one.\n\nThe key steps of MMICRL include unsupervised agent identification, conditional inverse constraint inference, and multi-modal policy update.\n\n1. Unsupervised Agent Identification: MMICRL identifies expert trajectories in an unsupervised manner. It performs trajectory-level identification by estimating an agent-specific density. The algorithm uses a Conditional Flow-based Density Estimator (CFDE) to compute a density using a state-action density estimator. The agent identifier is represented by the softmax representation.\n\n2. Agent-Specific Constraint Inference: Based on the identified expert dataset, the likelihood function can be simplified. The algorithm parameterizes the instantaneous permissibility function with $ω$ and updates the parameters by computing the gradient of the above likelihood function.\n\n3. Multi-Modal Policy Optimization: The policy is trained to maximize cumulative rewards subject to constraint. The objective expands the reward signals with a log-probability term, which encourages the policy to generate trajectories from high-density regions for a specific agent type. The algorithm integrates contrastive learning into policy optimization, helping it to better understand the relationships between agents, their behaviors, and the corresponding expert trajectories.\n\nThe algorithm alternates between these steps until the imitation policies reproduce expert trajectories, signifying that the inferred constraints align with the ground-truth constraints.\n\n# Experiments\n\nThe paper conducts experiments on the MMICRL algorithm in both discrete and continuous environments, using Constraint Violation Rate and Feasible Cumulative Rewards as evaluation metrics. In discrete environments based on Gridworld, MMICRL successfully identifies various agent types and generates accurate trajectories. In continuous environments based on MuJoCo, MMICRL consistently exhibits lower constraint violation rates and higher cumulative rewards, outperforming other baseline methods. A robustness test shows that MMICRL can recover from initial incorrect agent identifications, demonstrating its ability to accurately model diverse agent preferences and its robustness in handling errors.\n\nSTRENGTHS:\n# Originality and significance\nThis work poses a unique problem setup of multi-modal ICRL, which nevertheless has substantial real-world impact for applications such as autonomous driving or human-robot interaction. The proposed method is sophisticated and innovative, requiring no supervision other than expert demonstrations.\n\n# Quality\nA comprehensive analysis, along with an ablation study, is conducted in comparison to various baselines, effectively validating the efficacy of the proposed method.\n\nWEAKNESSES:\n- The number of agent types needs to be preset, and such number tested in the paper has been limited (mostly 2 or 3). The results would be more significant if the effectiveness of the method could be demonstrated with more complex data of larger scale.\n- Please enlarge the font size in figure 3, and also adjust the opacity of the bars so that the overlapping distributions are both visible.\n\nQUESTIONS:\n- Should the summation in line 178 be a production instead?\n- Why does the equation in line 181,\n$p_{\\psi}(z|\\tau) = \\frac{\\exp \\left(\\prod_{(s,a) \\in \\tau} p_{\\psi}(s,a|z)\\right)}{\\sum_{z'} \\exp \\left(\\prod_{(s,a) \\in \\tau} p_{\\psi}(s,a|z')\\right)}$ hold? It would be great if the authors could elaborate more on the derivation.\n\nLIMITATIONS:\nThe authors acknowledge two valid limitations in their work. Firstly, their MMICRL method doesn't consider potential collaboration or competition between agents to meet joint constraints. Secondly, their evaluations are conducted in virtual environments, not real-world applications, due to the absence of a suitable ICRL benchmark.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}