Title: Mixed-Initiative Multiagent Apprenticeship Learning for Human Training of Robot Teams
Venue: NeurIPS 2023
Accepted: yes
Keywords: Learning from Demonstration

Multi-Robot Systems

Teaching Robot Teams
Abstract: Extending recent advances in Learning from Demonstration (LfD) frameworks to multi-robot settings poses critical challenges such as environment non-stationarity due to partial observability which is detrimental to the applicability of existing methods. Although prior work has shown that enabling communication among agents of a robot team can alleviate such issues, creating inter-agent communication under existing Multi-Agent LfD (MA-LfD) frameworks requires the human expert to provide demonstrations for both environment actions and communication actions, which necessitates an efficient communication strategy on a known message spaces. To address this problem, we propose Mixed-Initiative Multi-Agent Apprenticeship Learning (MixTURE). MixTURE enables robot teams to learn from a human expert-generated data a preferred policy to accomplish a collaborative task, while simultaneously learning emergent inter-agent communication to enhance team coordination. The key ingredient to MixTURE's success is automatically learning a communication policy, enhanced by a mutual-information maximizing reverse model that rationalizes the underlying expert demonstrations without the need for human generated data or an auxiliary reward function. MixTURE outperforms a variety of relevant baselines on diverse data generated by human experts in complex heterogeneous domains. MixTURE is the first MA-LfD framework to enable learning multi-robot collaborative policies directly from real human data, resulting in ~44% less human workload, and ~46% higher usability score.

Mixed-Initiative Multiagent Apprenticeship Learning for Human Training of Robot Teams

Abstract

Extending recent advances in Learning from Demonstration (LfD) frameworks to multi-robot settings poses critical challenges such as environment non-stationarity due to partial observability which is detrimental to the applicability of existing methods. Although prior work has shown that enabling communication among agents of a robot team can alleviate such issues, creating inter-agent communication under existing Multi-Agent LfD (MA-LfD) frameworks requires the human expert to provide demonstrations for both environment actions and communication actions, which necessitates an efficient communication strategy on a known message space. To address this problem, we propose Mixed-Initiative Multi-Agent Apprenticeship Learning (MixTURE). MixTURE enables robot teams to learn from a human expert-generated data a preferred policy to accomplish a collaborative task, while simultaneously learning emergent inter-agent communication to enhance team coordination. The key ingredient to MixTURE's success is automatically learning a communication policy, enhanced by a mutual-information maximizing reverse model that rationalizes the underlying expert demonstrations without the need for human generated data or an auxiliary reward function. MixTURE outperforms a variety of relevant baselines on diverse data generated by human experts in complex heterogeneous domains. MixTURE is the first MA-LfD framework to enable learning multi-robot collaborative policies directly from real human data, resulting in 44% less human workload, and 46% higher usability score.

1 Introduction

In recent years, Multi-Agent Reinforcement Learning (MARL) has been predominantly used by researchers to optimize a reward signal and for learning multi-robot tasks. Nevertheless, RL generally suffers from key limitations such as difficulty in designing an expressive and suitable reward function for complex tasks [60, 2] which can lead to undesirable robot behavior [14, 60, 64], high sample complexity [29], and safety concerns due to direct robot-environment interactions for optimizing the policy [64]. These problems are further exacerbated in multi-robot scenarios where inter-robot interactions and environment dynamics can be more complex and task descriptions and objectives more ambiguous [60, 53, 52]. As such, accurate models of human strategies and behaviors achieved via imitation methods are increasingly important for safely and effectively deploying autonomous systems and aligning values motivating robot behaviors with human values [47, 22, 12].

Learning from Demonstration (LfD) attempts to learn the correct behavior (policy) from a set of expert-generated demonstrations rather than a reward function, which can result in lower sample

complexity and a learned policy that more closely reflects the human's preferred strategy [48, 47]. Unfortunately, extending existing single-agent LfD paradigms such as Behavioral Cloning (BC) [66] or Inverse Reinforcement Learning (IRL) [1] to multi-robot settings poses several challenges such as environment non-stationarity and existence of multiple equilibrium solutions (an agent's optimal policy depends on other agents' policies) [37, 27, 60]. One can adopt such frameworks directly in a centralized system. However, centralized systems are not scalable, are prone to single-node failure, and pose significant computation overhead, and therefore, decentralized approaches (e.g., limitedrange communication and local computations) have been more desired in multi-robot systems [20].

Prior work has shown that enabling communication among agents of a robot team creates a shared mental model of joint action-spaces and therefore, allowing coordinated action decisions [38, 65, 55, 19, 51] to handle challenges such as partial observability and environment dynamicity [25, 54, 50]. Although LfD can resolve the high sample complexity and reward shaping problems in RL, the task of MA-LfD can be onerous for the humans as they have to control multiple robots and imagine and simulate an appropriate Theory-of-Mind to create a communication strategy for the robot team.

Introducing the inter-agent communication [32] under current state-of-the-art (SOTA) Multi-Agent LfD (MA-LfD) frameworks such as Multi-Agent Generative Adversarial Imitation Learning (MAGAIL) [60] or Multi-Agent Adversarial IRL (MA-AIRL) [71] requires the human expert to provide demonstrations for both environment actions and communication actions . Such approaches assume that the human expert has access to an efficient communication strategy on a known finite message space in addition to a known strategy for taking environment actions [42]. This assumption, however, is invalid, since dynamic environments with multiple interacting agents will be too complex and unpredictable for humans to be able to develop cohesive and comprehensive inter-agent communication protocols. Even if such an efficient communication strategy exists, demonstrating both the task strategy as well as the communication strategy could potentially pose significant workload on the human expert, which in turn can affect the performance and quality of demonstrations. These issues become even more severe when the robot team is heterogeneous or of composite nature (i.e., agents with different observation- and action-spaces as well as different objectives) where agents must rely on communication to operate and fulfill their tasks correctly [54, 7, 41].

To address these challenges, we develop a distributed MA-LfD framework to efficiently incorporate a human expert's domain-knowledge of teaming strategies for collaborative robot teams and directly learn team coordination policies from expert human teachers. To this end, we propose MixedInitiative Multi-Agent Apprenticeship Learning (MixTURE) which enables robot teams to learn an expert's preferred strategy to act in an environment. MixTURE simultaneously learns end-toend emergent communication for the robot team to enhance team coordination, without the need for human generated data or an auxiliary reward function. To improve the quality of the learned inter-agent communication protocol, we reduce the entropy of a generated message given joint states and actions by maximizing the Mutual Information (MI) between messages and joint states. We demonstrate through empirical evaluation and a human subject experiment that our LfD-based MixTURE outperforms RL based methods due to reward function independence and low sample complexity. Furthermore, MixTURE significantly alleviates the human demonstrators' workload and time required to provide demonstrations, increases system usability, and improves overall collaboration performance of the robot team. Our key contributions are as follows :

We propose the MixTURE framework for learning robot teaming strategies from human expert demonstrations while simultaneously learning inter-agent communication through online interactions during training, without the need for expert data or an auxiliary reward.

We develop an MI maximization-based emergent communication learning model which reduces the entropy of a generated message for an agent given joint state-observations.

We evaluate MixTURE on real, diverse human-generated data, collected in a human-subject user study, and show that, in a complex multi-agent domain with heterogeneous tasks, we are able to achieve ∼ 42% - ∼ 77% higher performance and a significantly lower sample complexity. We also show that using MixTURE significantly improves workload and system usability relative to a benchmark MA-LfD framework. To best of our knowledge, this is the first work to train a MA-LfD framework on real human data.

We investigate the effects of demonstrating both environment actions and communication actions on a human expert's workload, demonstration quality, and system usability score. Our results show that a high-workload demonstration process in classic MA-LfD approaches

significantly ( p < . 001 ) reduces an expert's demonstration quality (measured by performance) and the system's usability score. MixTURE significantly improves these results; increasing a human's performance and experience engaging in MA-LfD.

2 Related Work

The literature for Multi-Agent LfD (MA-LfD) primarily aims to address the complexity of simultaneously training multiple agents under coordinated [43, 58, 46, 57, 68, 28] and uncoordinated tasks [6, 69, 40, 70]. In [61, 8] the MA-LfD problem is reduced into a single-agent problem by making the assumption that all agents share the same dynamics, observation spaces, and model parameters. In [34], a coordinated multi-agent IL approach is proposed which learns a latent coordination model along with the individual agent policies. In [60] the single-agent GAIL framework is extended for multi-agent scenarios along with a practical actor-critic method for multi-agent imitation. Similarly, in [71] the AIRL method was extended to the multi-agent settings. In [62] a scalable multi-agent LfD approach is proposed where a model-based heuristic method for automated swarm reorganization is leveraged to improve multi-agent task allocation problem. In [4] an expert feedback-based system is developed to address multi-agent path-finding problem. In [64] authors create an advising system to incorporate sub-optimal model-based heuristic policies to help improve MARL performance. Other prior work focused on learning human profile/behavior models for improved MA-LfD [45, 5]. More recently, Hoque et al. [26] proposed Fleet-DAgger , formalizing interactive fleet learning setting, in which multiple robots interactively query and learn from multiple human supervisors.

Nevertheless, applicability of these prior works in the collaborative multi-agent problems is considerably limited since none of these works explicitly address the inter-agent communication in dynamic and partially observable domains where agents not only need to take task-related actions, but also need to communicate and share information for coordination. Additionally, none of these prior work leverage real human-generated data for training to evaluate the approach against heterogeneity and diversity in human data. Our work addresses these limitations by eliminating the requirement for an expert to demonstrate a communication strategy. Instead, the human expert only needs to teach the robot team how to complete a task through demonstrations, and the robots will automatically learn a communication strategy that aligns with the expert's demonstrations. We also collect real human data to evaluate our method's ability to cope with variations in demonstration styles and strategies.

3 Problem Formulation: General MA-LfD with Heterogeneous Agents

We ground our problem formulation in a Markov Game (MG) [36] generalized to include partial observability and heterogeneous agents. We define a set of heterogeneous agents in a composite robot team (i.e., composed of different classes of robots) as agents that can have arbitrarily different state-, observation-, and action-spaces. The agents can also have different task objectives which, when enacted in coordination, enable the team to achieve a shared overarching mission. Accordingly, we define our generic MG as a 9-tuple: ⟨C , N , {S ( c ) } c ∈C , {A ( c ) } c ∈C , { Ω ( c ) } c ∈C , {O ( c ) } c ∈C , r, T , γ ⟩ . C is set of all available agent classes in the composite robot team and the index c ∈ C denotes the agent class. N = ∑ ⟨ c ∈C⟩ N ( c ) is the total number of collaborating agents where N ( c ) represents the number of agents in class c . {S ( c ) } c ∈C and {A ( c ) } c ∈C are discrete joint sets of state- and action-spaces, respectively. { Ω ( c ) } c ∈C is the joint set of observation-spaces, including class-specific observations. Agents of the same class have identical S , A , and Ω . γ ∈ [0 , 1) is the temporal discount factor for each unit of time and T is the state transition probability density function. At each timestep, t , each agent, j , of the c -th class can receive a partial observation o c j t ∈ Ω ( c ) according to some class-specific observation function {O ( c ) } c ∈C : o c j t ∼ O ( c ) ( ·| ¯ s ) . Next, each agent, j , of class c , takes an action, a c j t , forming a joint action vector ¯ a = ( a 1 1 t , a 1 2 t , · · · , a c 1 t , · · · , a c j t ) . When agents take the joint action ¯ a , in the joint state ¯ s and depending on the next joint-state, they receive an immediate reward, r (¯ s, ¯ a ) ∈ R , shared by all agents regardless of classes. Each agent, j , of a class, c , achieves its own objective by sampling actions from a stochastic policy π ( c ) j . The objective of each agent is then to maximize the team return (expected sum of discounted rewards), i.e., E π [ ∑ T t =0 γ t r t ] .

To directly learn the human's preferred strategy and resolve the reward specification problems [2] posed by RL, we leverage a demonstration dataset, D , provided by an expert, rather than the ground truth reward signal r employed in MARL. Unlike [71, 60] or [67], we do not assume multiple human experts in our MG to avoid the need for further coordination amongst the experts, which can be time consuming and expensive. D is a set of trajectories { τ c j } N ( c ) j =1 , where τ c j = { ( o c j t , a c j t ) } T t =1 is an expert trajectory collected by sampling a c j t ∼ π E ( a c j t | ¯ o t ) in which π E is the expert policy and ¯ o t is the joint observation that the expert has access to at time t . We further assume that D contains the entire supervision to the learning algorithm (i.e., no online interactions during training). We build the MixTURE architecture on the generative adversarial training [21]. Our distributed GAIL objective underlying MixTURE is shown in Eq. 1 where D ( c j ) θ is a local discriminator that classifies expert and policy trajectories for agent j of a class c , and π ( c j ) ϕ is the parameterized policy of agent j of a class c .

According to [21], under the GAIL objective in Eq. 1 and at optimality, the distribution of generated state-action pairs by ¯ π ϕ should match the distribution of demonstrated state-action pairs.

4 Mixed-Initiative Multi-Agent Apprenticeship Learning

Motivation and Problem Overview Consider a generic composite team of robots including agents with heterogeneous characteristics and task objectives. Without loss of generality, consider a robot team composed of perception-only and action-only robots. Under our problem formulation in Section 3, perception robots and action robots create two separate classes of agents that need to collaborate on an overarching mission. For instance, in an application of wildfire fighting, robots of the perception class (e.g., quadcopters) need to search an environment for firespots, while the action robots (e.g., fire-extinguishing ground robots) who cannot sense the environment are required to extinguish the firespots found by the perception robots [54, 55, 52, 3]. Note that neither of the robot classes are capable of accomplishing the task without the other class.

To teach a collaborative policy to such a robot team, one can leverage demonstrations from a team of humans where each member is an expert. Using a team of human experts, however, poses further challenges: (1) simultaneous access to several human experts is expensive and can be time consuming, and (2) a communication strategy (e.g., natural language) among the human demonstrators is required for coordination, which can be challenging to translate to robot domain due to ambiguity, colloquialisms, and context-dependent use [30, 23]. Additionally, humans' communications might not make sense to the robot agents as humans could be unaware of all agents' full local state spaces.

Alternatively, we can leverage the demonstrations from a single human with domain-knowledge regarding the entire mission objective. For example, a trainer/coach can play a simulated game of firefighting using the aforementioned perception and action robots and provide expert demonstrations for how to efficiently distribute agents and prioritize tasks for searching the environment and putting out the fire. The challenge of using a single human expert, however, is that in this case the human would also need to demonstrate communication-actions (i.e., what information should an agent broadcast at each state) on top of the environment-actions (i.e., moving around or dousing fire). To this end, humans would have to create and maintain a Theory-of-Mind (ToM) of each agent under this increased action-space dimensionality, which significantly increases workload [15, 17].

To resolve this problem, we propose taking separate initiatives for teaching the robots in the team how to operate (environment actions, a t , per observation, o t ) and how to communicate (communication message, z t , per state, s t ) such that a human expert would only be required to provide environment-action demonstrations and the robot team would automatically infer on their own a suitable communication policy for the underlying expert demonstrations. We call our approach Mixed-Initiative Multi-Agent Apprenticeship Learning (MixTURE).

MixTURE Architecture The proposed MixTURE architecture for human training of robot teams is shown in Fig. 1. At each timestep, each agent generates an embedding, representing agent's belief space, from its local observation. To handle agents' partial observability, the local observation embeddings are then passed into local recurrent policies for each agent. Each GRU policy receives the preprocessed features from the local observations as well as its own hidden state from previous timesteps. Therefore, the policy output depends only on the history of local observations and actions.

Forward

ai

a

Model

Mutual Information

tional

Comm.Processing

Comm.Processing

(a,Zji)

!0

→

Zij

Maximization

Comm.

FC²

FC2

MSE Loss

For i, maximize MI via:

ittent

!

αiZji

Oi

RecurrentPolicy

RecurrentPolicy

Preprocessing

Policy Section

GRU:T

GRU:T

Local

Optimization

个

↑

Discriminators

Module

embedding

embedding

(Ouai)

↑

↑

Di(ot,at)

ai

ioi

Optimization

Preprocessing

Preprocessing

BCELosS

FC}

FC1

(Oj,aj)

(PPO)

a

at

Agent i

Oi

Agent j

不

Expert Sample: De → (ot,at)

To learn from the human expert demonstrations stored in the dataset D E , as shown in Fig. 1, we build a distributed multi-agent GAIL architecture where each agent is equipped with a parametrized discriminator, D θ . The discriminators are trained via a Binary Cross Entropy (BCE) loss to distinguish between state-action pair samples from the expert dataset and those generated by the generator (i.e., local policies). The output of the discriminators are treated as local rewards, which are then combined to encourage collaboration and teaming behaviour among agents [31]. To learn the agent policies through this shared reward signal, we leverage the Proximal Policy Optimization (PPO) [49].

We enable inter-agent communication by adding an attentional communication module in which each agent is equipped with a fully-connected network that processes action-embeddings generated by the recurrent policies of an agent and those generated by its local teammates (i.e., messages, z ) to output an action-decision. We enable this message-passing by creating differentiable communication channels among agents. To maintain locality, these communication channels can be leveraged locally (i.e., a communication graph where edges only exist when robots are spatially within close proximity). For an agent, i , the action-embedding messages received from a teammate, j , are weighted by some learned attention coefficients, α ji , to assign message importance. Therefore, the input message for agent i at time, t , can be computed as m j → i t = ∑ N t ( i ) j =1 α ji z t ji where N t ( i ) represents the neighbors, j , of agent i at time t . Here, α ji are the learned attention coefficients that are computed via α ji = softmax j ( σ ( ¯ W att [ ω ¯ h i ∥ ¯ m j → i ])) . In this equation, ¯ W att are the learnable weights of the attention network, ∥ represents concatenation, σ is an activation function nonlinearity, and ¯ h represents the hidden states. The Softmax function is used to normalize the coefficients across all neighbors j . Such attentional communication can enhance the action-decision quality, particularly with increased number of agents or when the states may significantly vary in different parts of the environment [16]. Messages in our communication module are entirely learned via backpropagation.

Mutual Information Maximization-Based Differentiable Communication A challenge with the communication model learned via the described end-to-end differentiable channels is that the distribution of the messages broadcasted by an agent, i , given the state-observations, ρ ( z ij | ¯ o ) , can have a high variance. The desired behavior, instead, is that the agents employ a cohesive communication strategy in which an agent sends a consistent message when it observes relatively similar states.

To resolve this issue, we propose maximizing the Mutual Information (MI) between an agent's outgoing message and the joint state-observations. MI is a measure of the reduction in entropy of a probability distribution, X , given another probability distribution, Y , such that I ( X ; Y ) = H ( X ) -H ( X | Y ) , where H ( X ) denotes the entropy of X and H ( X | Y ) is the conditional entropy

of X given Y [33]. In our work, by maximizing the MI between the distribution of an agent's message, ρ ( z ij | ¯ o ) , and the joint observations, we reduce the entropy over messages and encourage the communication model to be more consistent. Maximizing the MI in this way encourages z ij to correlate with features within the observation distribution (i.e., mode discovery) [13, 44].

Unfortunately, a direct MI Maximization (MIM) between the message distributions and joint observations as formulated above, I ( z ij ; ¯ o ) = H ( z ij ) -H ( z ij | ¯ o ) , is intractable as it requires access to the true posterior, ρ ( z ij | ¯ o ) . Therefore, in keeping with prior work, we rely on the Evidence Lower Bound (ELBO) of the MI instead. As shown in prior work [13], by minimizing an MSE loss between a sample from the current message embedding and the approximate posterior, modeled as a normal distribution with constant variance, is equivalent to maximizing the likelihood of the posterior.

In practice, we build a distributed reverse model in the MixTURE architecture (shown in Fig. 1) to accommodate the mentioned MSE loss. The distributed reverse model for each agent has access to the local received messages as well as the global joint-actions taken by all agents such that the optimization results in a communication entropy-reduction mechanism at the team level. Since actions and generated messages are functions of the state-observations, each reverse model, Θ , can take in the joint actions, ¯ a , and all received messages, z ji , to estimate the outgoing message for agent, i , as ˆ z t ij = Θ i (¯ a, z t ji ) . The policy and reverse models are trained together in an end-to-end fashion to N

Wenote that while the addition of the MI maximization loss would make the overall loss function more complex, we believe there are further benefits that could be achieved via training the MIM reverse model, besides enhancing agent certainty in choosing actions. Particularly, we believe our MIM reverse model, when tuned well, has the potential to roughly cluster message embeddings based on observation-action pairs which can provide further useful information on the learned communication protocol for a task and further insight into interpreting the learned communication protocols.

Training and Execution We build the MixTURE framework in a Centralized Training for Decentralized Execution (CTDE) paradigm [19] to accommodate for the global joint-action inputs to the MI maximizing reverse models during training. We note that, the MIM reverse model is only used during training and is cut during the execution, and therefore, the learned policies can be executed fully decentralized. To optimize the policies based on the reward signal generated by the discriminators, we leverage the PPO algorithm [49]. Moreover, to enhance and stabilize the training we propose combining an offline BC loss, L BC c i = -1 N ∑ N i =1 π i ( a c i t | o c i t , z t ji ) , with the online GAIL loss. As shown by prior work [18], through this combination, the offline BC helps preserving ground knowledge that should be respected during training, while the online part helps with learning of new information encountered during execution. As such, putting together our distributed GAIL loss in Eq. 1, the standard clipped PPO loss [49], the offline BC loss, and the MIM loss introduced above, we present the full loss expression to train the MixTURE architecture as in Eq. 2, where ζ π and ζ D E are minibatches of trajectory segments belonging to current policy, π , and demonstration dataset, D E , respectively, N is the total number of agents and λ is a tunable scaling parameter.

5 Evaluation

We break the problem of evaluating our proposed architecture for teaching multi-agent coordination policies to (heterogeneous) robot teams into three research questions (RQ):

RQ1 Can MixTURE learn useful multi-agent coordination strategies from synthetic data (e.g, models of human experts / Oz-of-Wizard [63])?

RQ2 Can MixTURE learn from diverse data generated by real human experts?

RQ3 How challenging is it for a human expert to provide multi-agent demonstrations and does MixTURE alleviate these challenges (comparing workload and system usability)?

Environments In keeping with prior work in MARL and MA-LfD [60, 54], we selected three multiagent domains that are partially observable and require collaboration among heterogeneous agents (see Section 3). Please refer to the supplementary material for more details about the environments.

Predator-Prey10x10

Predator-Capture-Prey10x10

FireCommander10x10

episode length

80

75

75

50

50

60

25

25

40

0.0

2.5

5.0

7.5

0.0

2.5

5.0

7.5

0.0

2.5

5.0

7.5

timesteps

1e5

timesteps

1e5

timesteps

1e5

MARL

NCMA-GAIL

MA-GAIL

MixTURE

BC+DC

Predator-Prey (PP) [59]: the goal in this homogeneous (i.e., same class agents) domain is for N predator agents with limited vision to find a stationary prey and move to its location.

Predator-Capture-Prey (PCP) [54]: a new class of capture -agents, are introduced to the PP. In this heterogeneous domain, the goal of predator agents is the same, while capture agents must move to the prey location and capture it, without having any observation inputs.

FireComander (FC) [54, 56]: two classes of robots, perception and action robots, are required to collaborate to extinguish propagating firespots. Similar to the PCP domain, perception robots search the domain to find hidden firespots, and action robots with no observation must rely on communication to know where to put out the firespots using an extra action when on a fire. Unlike the PCP, in this complex domain firespots randomly spread over time and thus, the team must continue until all firespots are found and extinguished.

Baselines To investigate our RQ1 , we employ a variety of baselines, described below, all of which utilize the combined offline BC and online loss training scheme. To collect the synthetic dataset, the expert heuristic (Appendix 4) for environment-actions is a near-optimal search algorithm and for the communication, it is an anticipatory observation sharing mechanism inspired by prior work [11, 10].

MARL [49]: MA-PPO optimizing both environment-action and communication policies.

BC+DC [66]: MixTURE ablation trained only via offline BC loss with diff. communication.

NC MA-GAIL [60]: Non-communicative ablation of the MA-GAIL [60] framework.

MA-GAIL [60]: Full MA-GAIL trained on full dataset w/ demonstrated communication.

5.1 Human-Subject Experiment: Conditions and Procedure

To investigate our RQ2 and RQ3 , we conducted an IRB-approved human-subject user study in the FireCommander domain. For more experiment details please refer to the supplementary material.

Domain, Setup, and Procedure Our experiments were conducted in the context of a simulated multi-robot task, leveraging the FireCommander (FC) domain under different difficulty levels and modes. Depending on the level of difficulty, there can be multiple initial firespots, hidden from the human, that propagate randomly based on a fixed wall-clock rate. The human expert was responsible to strategically move the simulated robots to find and extinguish all firespots as fast as possible. The human subject was shown a performance score at the end of each round, computed based on existing, found, and extinguished fires. After briefing, each subject began by filling in a pre-questionnaire form (demographic and prior videogame experience) followed by reading through a series of detailed game instructions. To minimize the learning effect, subjects were allowed to practice all conditions until they felt comfortable. Next, each subject played six different rounds of the game (i.e., two conditions and three difficulty levels) in a randomly selected order. The demonstration data was fully stored to be later used for training. Finally, each subject was asked to fill some post-measurement forms.

Participants Werecruited 55 participants (mean age = 25 . 0 ± 2 . 67 ; 34 . 5% female). All participants were recruited through on-university-campus advertisement and were trained equally for the task.

Independent Variables and Conditions In this study, we seek to determine if MixTURE can learn multi-agent collaborative policies from diverse human generated data without demonstrated communication and we compare its performance against the MA-GAIL [60] with expert demonstrated

Easy

Medium

Hard

80

80

78

72

72

72

64

64

length

56

66

56

48

60

episode

48

40

54

40

32

48

24

32

42

16

24

36

8

16

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

timesteps

1e6

timesteps

1e6

timesteps

1e6

MARL

MA-GAIL

MixTURE

BC+DC

communication. We examine the performance across three levels of difficulty, i.e., easy, medium, and hard, which represent game complexity. As such, we utilize a 2 × 3 within-subjects experiment design varying across two abstractions: (1) only demonstrating environment actions for each robot at each time step (i.e., noComm condition), 2) demonstrating both environment actions and communication actions for each agent at each time step ((i.e., withComm condition)).

Metrics To evaluate our RQ2 and RQ3 , the demonstration data collected from the human subjects for both noComm and withComm conditions are used to train MixTURE and the benchmark MAGAIL [60], respectively. The algorithms are compared in terms of the learned policy performance (i.e., number of steps taken to win the game, where lower is better). Additionally, to address our RQ3 , we leverage the NASA-TLX Workload Survey [24] and the System Usability Scale (SUS) [9].

5.2 Results and Discussion

RQ1: Evaluation on Synthetic Dataset Weperform our evaluations across three different difficulty levels: (1) easy ( 5 × 5 domain, 3 robots), (2) medium ( 10 × 10 domain, 6 robots), and (3) hard ( 20 × 20 domain, 10 robots). Table 1 presents the full evaluation results for collaborative policies learned by MixTURE and the baselines in terms of number of steps taken to win the game (lower is better). The test results show the mean performance values calculated over ten trials of running the best training models. As shown, MixTURE achieves significant improvement over all baselines and in all domains. Additionally, the learning curves in Fig. 2 (medium case), show significant improvement in sample complexity. We believe our model provides a strong step towards learning collaborative policies in multi-robot systems by setting a new SOTA in complex heterogeneous tasks.

We also performed several ablation studies, such as investigating scalability and effects of the MIM reverse model for message reconstruction, discriminator architecture, and the combined offline BC loss. Due to space constraints, all ablation results are available in the supplementary material.

RQ2: Training and Evaluation Results on Human-Subject Dataset We train MixTURE and MA-GAIL [60] (and other baselines) on data collected in our human-subject user experiment to investigate our RQ2 . The results are presented in Fig. 3. Please see the Appendix for environment

details. As shown, MixTURE can learn high-quality multi-robot collaboration policies from diverse human-generated demonstrations. This is while purely relying on human demonstrations [60], does not succeed in this task. We hypothesize that a key point underlying MixTURE's success despite diversity and heterogeneity in human data is that the MIM-based differentiable communication channels provide the model with ability to reason about the underlying human demonstrations and cope with trajectory distribution through automatically finding a suitable communication protocol. This result also shows that a demonstrated communication policy by a human expert is not enough to efficiently coordinate the robot team in complex domains as the human is unaware of all agents' local states, which leads to the failure of existing MA-LfD frameworks such as the MA-GAIL [60]. We hypothesize that MA-GAIL trained on demonstrated communications fails because: (1) humans are not fully aware of all agents' full local state spaces and therefore, the demonstrated communication could be severely inefficient and sub-optimal. MixTURE instead learns an efficient communication policy automatically through gradient updates; (2) The laborious task of providing both environment and communication actions for the human demonstrators significantly deteriorates the quality of demonstrated policies, which can be confirmed by our subject-study results in next section.

We believe that this strong result shows great potential for the MixTURE model to efficiently teach multi-agent coordination and collaboration policies to robot teams through human demonstrations.

RQ3: Statistical Analysis We investigate our RQ3 by quantifying the workload and SUS measures reported by the human subjects. We hypothesize that:

H1 Demonstrating both an environment-action and a communication-action strategy for the robot team increases the human expert's workload and decreases the system's usability score.

H2 Demonstrating both an environment-action and a communication-action strategy for the robot team negatively affects human performance and the demonstration quality.

H1 : We test for normality and homoscedasticity and do not reject the null hypothesis in either case, using Shapiro-Wilk ( p > 0 . 32 and p > 0 . 96 ) and Levene's ( p > 0 . 39 and p > 0 . 09 ) tests for workload and SUS, respectively. For workload, we perform a paired t-test and find that using the MixTURE model w/o communication demonstration was rated statistically significantly lower than using MA-GAIL w/ demonstrated communication by the expert ( p < 0 . 001 ) on NASA-TLX workload scale. For system usability, using a similar a paired t-test we find that using the MixTURE model w/o communication demonstration led to a statistically significantly higher SUS than using MA-GAIL w/ demonstrated communication by the expert ( p < 0 . 001 ). As shown in Fig. 4, relaxing the need for demonstrating a communication strategy reduced the humans' workload by 44 . 3% and increased the systems' usability scale by 46 . 1% in our experiment.

H2 : We examine: (1) human's performance score in the game, (2) total tasks completed (i.e., fires extinguished), and (3) average demonstration time per step. We apply a paired samples Wilcoxon test and confirm that (see Fig. 5, from left to right), relaxing the need for demonstrating a communication strategy through MixTURE leads to achieving significantly higher performance score ( p < 0 . 001 ) by the human, better ability to scale to more complex scenarios with more tasks ( p < 0 . 001 ), and significantly lower demonstration time per step ( p < 0 . 001 ). Note that, the middle plot in Fig. 5 shows more tasks completed (i.e., more firespots killed) for withComm condition under easy and moderate

120

***

***

100

80

Scores

60

40

20

noComm

withComm

0

Workload

SUS

scenarios, which indicates a worse human performance in this conditions since existence of more firespots means inefficiency in extinguishing the fire before it spreads too large. On the other hand, under the noComm condition, humans can easily scale to larger domain sizes and more initial firespots. Note that, lower tasks completed for withComm condition under the hard case is attributed to failing the task and losing the game (i.e., hard game cut-off when score drops below 50).

Further Discussions and Limitations Demonstrating multi-agent strategies can be considered a highly involved and high-workload task, which in turn can affect a human's situational awareness and

noCommcondition

withCommcondition

110

★★★

★★*

firespotskilled]

40

***

★★*

10

★★★

★★★

★★*

score]

100

35

30

[game

06

25

Performance

80

Total Tasks[num.t

20

70

15

60

10

50

40

Easy

Moderate

Hard

Easy

Moderate

Hard

Easy

Moderate

Hard

optimality of demonstrations. We posit that a human's decision-making for one agent is influenced by knowledge of other agents' observations. However, we note that, classic MA-LfD methods, such as MA-GAIL, necessitate awareness all robots' states and observations and they require providing both environment and communication actions at the same time. In MixTURE, we relax this assumption by tasking the human to only provide environment actions for robots one at a time. Our real human subject study demonstrates this relaxation's feasibility, enabling successful demonstrations provided by humans for large and heterogeneous teams of simulated robots. In practice, one can use existing multi-agent datasets (e.g., collaborative assembly in ROBOTURK [39, 67] or Meta's STARDATA for StarCraft II [35]) or readily create a demonstration dataset using existing tools (e.g., ROS, Gazebo). MixTURE's contribution comes in at this stage, where we can learn collaborative multi-agent policies from such datasets.

The addition of the MIM loss to Eq. 2 can complicate the tuning process. While we never observed any performance decay as a result of the MIM loss, tuning the MIM loss to achieve consistent and significant performance improvement seems to be challenging. Finally, MixTURE currently does not account for demonstration sub-optimality. An interesting future direction is then to modify the MixTURE architecture to address suboptimal human demonstrations.

Conclusion We proposed the MixTURE model to learn multi-agent collaborative policies for a robot team, directly from human expert demonstrations. Using our method, a human expert can teach the robot team how to accomplish a task collaboratively via demonstrations and the team will automatically reason over and learn a communication strategy suitable for the underlying demonstrations. The learned communication helps the robot team to deal with the partial observability, reasoning about action-decisions to best respond to teammates' policies, and alleviate the effects of environment non-stationarity. We provided several empirical and experimental results, confirming MixTURE's strong ability to learn from expert heuristics and real diverse human generated data.

Broader Impact

Our experiments show that multi-agent learning from demonstration can be enabled for teams of cooperating heterogeneous robots via direct human teaching. This greatly assists robot collaboration through human domain knowledge and provides a strong next-step towards human-robot teaming. We note that our research could be applied for good or otherwise, particularly in an adversarial setting. Nonetheless, we believe democratizing this knowledge is for the benefit of society.

Acknowledgments

This work was supported by the Naval Research Laboratory (NRL) under grant number N0017321-1-G009 and grant number N00014-22-1-2834. The authors also would like to thank Manisha Natarajan for being a sounding board and guiding our statistical analysis.

References

Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), page 1, 2004.

David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.

Ian F Akyildiz and Ismail H Kasimoglu. Wireless sensor and actor networks: research challenges. Ad hoc networks, 2(4):351-367, 2004.

Ahmed Alagha, Rabeb Mizouni, Jamal Bentahar, Hadi Otrok, and Shakti Singh. Multi-agent deep reinforcement learning with demonstration cloning for target localization. IEEE Internet of Things Journal, 2023.

Feryal Behbahani, Kyriacos Shiarlis, Xi Chen, Vitaly Kurin, Sudhanshu Kasewa, Ciprian Stirbu, Joao Gomes, Supratik Paul, Frans A Oliehoek, Joao Messias, et al. Learning from demonstration in the wild. In Proceedings of the International Conference on Robotics and Automation (ICRA), pages 775-781. IEEE, 2019.

Sage Bergerson. Multi-agent inverse reinforcement learning: Suboptimal demonstrations and alternative solution concepts. arXiv preprint arXiv:2109.01178, 2021.

Matteo Bettini, Ajay Shankar, and Amanda Prorok. Heterogeneous multi-robot reinforcement learning. arXiv preprint arXiv:2301.07137, 2023.

Raunak P Bhattacharyya, Derek J Phillips, Blake Wulfe, Jeremy Morton, Alex Kuefler, and Mykel J Kochenderfer. Multi-agent imitation learning for driving simulation. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1534-1539. IEEE, 2018.

John Brooke et al. Sus-a quick and dirty usability scale. Usability evaluation in industry, 189(194):4-7, 1996.

Abhizna Butchibabu. Anticipatory communication strategies for human robot team coordination. PhD thesis, Massachusetts Institute of Technology, 2016.

Abhizna Butchibabu, Christopher Sparano-Huiban, Liz Sonenberg, and Julie Shah. Implicit coordination strategies for effective team communication. Human Factors, 58(4):595-610, 2016.

Letian Chen, Rohan Paleja, Muyleng Ghuy, and Matthew Gombolay. Joint goal and strategy inference across heterogeneous demonstrators via reward network distillation. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 659-668, 2020.

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 29, 2016.

Jack Clark and Dario Amodei. Faulty reward functions in the wild. Internet: https://blog. openai. com/faulty-reward-functions, 2016.

Mary L Cummings and Carl E Nehme. Modeling the impact of workload in network centric supervisory control settings. In Neurocognitive and Physiological Factors During High-tempo Operations, pages 23-41. CRC Press, 2018.

Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In Proceedings of the International Conference on Machine Learning (ICML), pages 1538-1546. PMLR, 2019.

Birsen Donmez, Carl Nehme, and Mary L Cummings. Modeling workload impact in multiple unmanned vehicle supervisory control. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 40(6):1180-1190, 2010.

Lydia Fischer, Barbara Hammer, and Heiko Wersing. Combining offline and online classifiers for life-long learning. In Proceedings of the International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2015.

Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 29, 2016.

Avinash Gautam and Sudeept Mohan. A review of research in multi-robot systems. In Proceedings of the IEEE 7th International Conference on Industrial and Information Systems (ICIIS), pages 1-5. IEEE, 2012.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.

Nate Gruver, Jiaming Song, Mykel J Kochenderfer, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning with latent variables. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 1855-1857, 2020.

Ibrahim A Hameed. Using natural language processing (nlp) for designing socially intelligent robots. In Proceedings of the Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), pages 268-269. IEEE, 2016.

Sandra G Hart and Lowell E Staveland. Development of nasa-tlx (task load index): Results of empirical and theoretical research. In Advances in Psychology, volume 52, pages 139-183. Elsevier, 1988.

Guy Hoffman and Cynthia Breazeal. Collaboration in human-robot teams. In Proceedings of the AIAA 1st Intelligent Systems Technical Conference (IntelliSys), page 6434, 2004.

Ryan Hoque, Lawrence Yunliang Chen, Satvik Sharma, Karthik Dharmarajan, Brijen Thananjeyan, Pieter Abbeel, and Ken Goldberg. Fleet-dagger: Interactive robot fleet learning with scalable human supervision. In Proceedings of the Conference on Robot Learning (CoRL), pages 368-380. PMLR, 2023.

Junling Hu, Michael P Wellman, et al. Multiagent reinforcement learning: theoretical framework and an algorithm. In Proceedings of the International Conference on Machine Learning (ICML), volume 98, pages 242-250, 1998.

Wonseok Jeon, Paul Barde, Derek Nowrouzezahrai, and Joelle Pineau. Scalable multi-agent inverse reinforcement learning via actor-attention-critic. arXiv preprint arXiv:2002.10525, 2020.

Sham Machandranath Kakade. On the sample complexity of reinforcement learning. University of London, University College London (United Kingdom), 2003.

Diksha Khurana, Aditya Koli, Kiran Khatter, and Sukhdev Singh. Natural language processing: State of the art, current trends and challenges. Multimedia Tools and Applications, pages 1-32, 2022.

Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.

Sachin G Konan, Esmaeil Seraj, and Matthew Gombolay. Iterated reasoning with mutual information in cooperative and byzantine decentralized teaming. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.

Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E, 69(6):066138, 2004.

Hoang M Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 1995-2003. PMLR, 2017.

Zeming Lin, Jonas Gehring, Vasil Khalidov, and Gabriel Synnaeve. Stardata: A starcraft ai research dataset. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AAAI), volume 13, pages 50-56, 2017.

Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine Learning Proceedings, pages 157-163. Elsevier, 1994.

Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.

Jean MacMillan, Elliot E Entin, and Daniel Serfaty. Communication overhead: The hidden cost of team cognition. 2004.

Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Proceedings of the Conference on Robot Learning (CoRL), pages 879-893. PMLR, 2018.

Negar Mehr, Mingyu Wang, Maulik Bhatt, and Mac Schwager. Maximum-entropy multi-agent dynamic games: Forward and inverse solutions. IEEE Transactions on Robotics, 2023.

Douglas De Rizzo Meneghetti and Reinaldo Augusto da Costa Bianchi. Towards heterogeneous multi-agent reinforcement learning with graph neural networks. arXiv preprint arXiv:2009.13161, 2020.

Manisha Natarajan, Esmaeil Seraj, Batuhan Altundas, Rohan Paleja, Sean Ye, Letian Chen, Reed Jensen, Kimberlee Chestnut Chang, and Matthew Gombolay. Human-robot teaming: Grand challenges. Current Robotics Reports, pages 1-20, 2023.

Sriraam Natarajan, Gautam Kunapuli, Kshitij Judah, Prasad Tadepalli, Kristian Kersting, and Jude Shavlik. Multi-agent inverse reinforcement learning. In Proceedings of the Ninth International Conference on Machine Learning and Applications (ICMLA), pages 395-400. IEEE, 2010.

Rohan Paleja, Andrew Silva, Letian Chen, and Matthew Gombolay. Interpretable and personalized apprenticeship scheduling: Learning interpretable scheduling policies from heterogeneous user demonstrations. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 33:6417-6428, 2020.

Georgios Th Papadopoulos, Asterios Leonidis, Margherita Antona, and Constantine Stephanidis. User profile-driven large-scale multi-agent learning from demonstration in federated human-robot collaborative environments. In Proceedings of the Human-Computer Interaction. Technological Innovation: Thematic Area (HCI), pages 548-563. Springer, 2022.

Peixi Peng, Junliang Xing, and Lili Cao. Hybrid learning for multi-agent cooperation with sub-optimal demonstrations. In In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3037-3043, 2021.

Lindsay Sanneman and Julie Shah. Transparent value alignment. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 557-560, 2023.

Stefan Schaal. Learning from demonstration. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 9, 1996.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

Esmaeil Seraj. Embodied team intelligence in multi-robot systems. AAMAS 2022 Doctoral Consortium, 2022.

Esmaeil Seraj. Embodied, intelligent communication for multi-agent cooperation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 37, pages 16135-16136, 2023.

Esmaeil Seraj, Letian Chen, and Matthew C Gombolay. A hierarchical coordination framework for joint perception-action tasks in composite robot teams. IEEE Transactions on Robotics, 2021.

Esmaeil Seraj, Andrew Silva, and Matthew Gombolay. Multi-uav planning for cooperative wildfire coverage and tracking with quality-of-service guarantees. Journal of Autonomous Agents and Multi-Agent Systems, 36(2):39, 2022.

Esmaeil Seraj, Zheyuan Wang, Rohan Paleja, Daniel Martin, Matthew Sklar, Anirudh Patel, and Matthew Gombolay. Learning efficient diverse communication for cooperative heterogeneous teaming. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 1173-1182, 2022.

Esmaeil Seraj, Zheyuan Wang, Rohan Paleja, Matthew Sklar, Anirudh Patel, and Matthew Gombolay. Heterogeneous graph attention networks for learning diverse communication. arXiv preprint arXiv:2108.09568, 2021.

Esmaeil Seraj, Xiyang Wu, and Matthew Gombolay. Firecommander: An interactive, probabilistic multi-agent environment for heterogeneous robot teams. arXiv preprint arXiv:2011.00165, 2020.

Andy Shih, Stefano Ermon, and Dorsa Sadigh. Conditional imitation learning for multi-agent games. In Proceedings of the 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 166-175. IEEE, 2022.

Marco AC Simões, Robson Marinho da Silva, and Tatiane Nogueira. A dataset schema for cooperative learning from demonstration in multi-robot systems. Journal of Intelligent & Robotic Systems, 99:589-608, 2020.

Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. arXiv preprint arXiv:1812.09755, 2018.

Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.

Adrian Šošic, Wasiur R KhudaBukhsh, Abdelhak M Zoubir, and Heinz Koeppl. Inverse reinforcement learning in swarm systems. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 1413-1421, 2017.

William Squires and Sean Luke. Scalable heterogeneous multiagent learning from demonstration. In Proceedings of the International Conference on Practical Applications of Agents and Multi-Agent Systems, pages 264-277. Springer, 2020.

Aaron Steinfeld, Odest Chadwicke Jenkins, and Brian Scassellati. The oz of wizard: simulating the human for interaction research. In Proceedings of the ACM/IEEE International Conference on Human Robot Interaction (HRI), pages 101-108, 2009.

Sriram Ganapathi Subramanian, Matthew E Taylor, Kate Larson, and Mark Crowley. Multiagent advisor q-learning. Journal of Artificial Intelligence Research, 74:1-74, 2022.

Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 29, 2016.

Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018.

Albert Tung, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Learning multi-arm manipulation through collaborative teleoperation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 9212-9219. IEEE, 2021.

Hongwei Wang, Lantao Yu, Zhangjie Cao, and Stefano Ermon. Multi-agent imitation learning with copulas. In Machine Learning and Knowledge Discovery in Databases, pages 139-156. Springer, 2021.

Xingyu Wang and Diego Klabjan. Competitive multi-agent inverse reinforcement learning with sub-optimal demonstrations. In Proceedings of the International Conference on Machine Learning (ICML), pages 5143-5151. PMLR, 2018.

Fan Yang, Alina Vereshchaka, Changyou Chen, and Wen Dong. Bayesian multi-type mean field multi-agent imitation learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 33:2469-2478, 2020.

Lantao Yu, Jiaming Song, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 7194-7201. PMLR, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel learning approach designed for robot teams to acquire a preferred policy for collaborative task completion using human expert-generated data. Additionally, the method enables the robots to gain an understanding of the theory of mind of each individual agent within the team.\n\nSTRENGTHS:\nThe paper presents a learning framework that enables robots to acquire teaming strategies through human expert demonstrations, while simultaneously learning the theory of mind between agents. The effectiveness of the proposed approach is demonstrated on synthetic and real data collected from humans. The method outperforms other baselines in accomplishing the defined task.\n\nWEAKNESSES:\n1.I agree that incorporating learning communication policies is crucial for both multi-agent learning from demonstrations (MA-LfD) and multi-agent reinforcement learning (MARL). However, the tasks presented in the paper may be considered relatively simple, which might not fully showcase the author's capabilities. It would be beneficial if the author could demonstrate the proposed methods in more complex 3D environments such as ITHOR or Habitat, or even leverage visual-based theory of mind datasets like \"Learning triadic belief dynamics in nonverbal communication from videos\" or \"BOSS: A Benchmark for Human Belief Prediction in Object-context Scenarios.\"\n2.I'm curious if the author has conducted a performance comparison between a single human expert and multiple experts (N) for each agent. Benchmarking the differences in performance could provide valuable insights into the impact of expert diversity on the proposed approach.\n3.Could the author elaborate on the rationale behind selecting GRU (Gated Recurrent Unit) over a transformer for the proposed MixTURE architecture? Understanding the specific reasons behind this choice would enhance the clarity and comprehensiveness of the paper.\n\nQUESTIONS:\nMy questions are listed in the weakness, i hope the authors can address them.\n\nLIMITATIONS:\nYes, the authors have addressed all the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a model, called MixTURE, for learning multi-agent collaborative policies from human demonstrations, addressing the challenges of coordinating heterogeneous agents in complex tasks. MixTURE leverages a mutual information maximization-based differentiable communication module to reason about and adapt to diverse human demonstrations. The authors evaluate MixTURE through synthetic and human-subject experiments in the FireCommander (FC) domain. In the synthetic evaluations, MixTURE outperforms baselines, achieving significant improvements across difficulty levels and setting a new state-of-the-art in learning collaborative policies for complex tasks.\nIn the human-subject experiments, the proposed approach demonstrates its ability to learn high-quality collaboration policies from diverse human-generated demonstrations, surpassing frameworks that solely rely on demonstrated communication. The model's ability to reason about human demonstrations and adapt to trajectory distributions is highlighted as a key factor in its success. The statistical analysis supports the claims made by the authors. It shows that relaxing the need for demonstrating communication reduces the workload for human experts and improves system usability. Furthermore, the results indicate that MixTURE achieves higher performance scores, better scalability to complex scenarios, and lower demonstration time per step compared to frameworks with demonstrated communication.\nOverall, the results support the effectiveness of MixTURE in learning collaborative policies, its ability to leverage human demonstrations, and its potential to enhance human-robot collaboration in complex multi-agent systems.\n\nSTRENGTHS:\n- Originality:\nThe paper introduces a novel approach by explicitly addressing the challenge of inter-agent communication in dynamic and partially observable domains.\nThe use of a differentiable communication module based on Mutual Information (MI) is a creative and original contribution.\nThe paper's focus on learning communication strategies from human demonstrations and addressing heterogeneity and diversity in human data sets it apart from previous methods.\nThe elimination of the need for an expert to demonstrate a communication strategy and allowing robots to automatically learn communication protocols is a useful idea.\n\n- Quality:\nThe technical details and algorithms are thoroughly explained, ensuring reproducibility and soundness of the proposed approach.\nThe paper demonstrates a fair experimental methodology, conducting evaluations in both synthetic and human-subject environments.\nThe inclusion of ablation studies and statistical analysis enhances the evaluation and supports the provided claims.\n\n- Clarity:\nThe paper is well-structured and organized, with clear sections that guide the reader.\nThe authors provide comprehensive explanations of the methodology, algorithms, and experimental setup.\nThe use of figures, tables, and visualizations aids in understanding the concepts and results.\nThe paper uses appropriate terminology and notation, making it accessible to readers familiar with the field.\n\n- Significance:\nThe paper addresses a significant research gap by explicitly considering the challenge of inter-agent communication in collaborative multi-agent problems.\nThe proposed approach has practical implications for teaching multi-agent coordination, as it eliminates the need for explicit communication demonstrations from experts.\nThe use of real human-generated data demonstrates the model's ability to cope with variations in demonstration styles and strategies, increasing its applicability in real-world scenarios.\nThe experimental results support the claims and show the effectiveness of the proposed MixTURE model, contributing to the advancement of Multi-Agent Learning from Demonstrations (MA-LfD).\n\nWEAKNESSES:\nA discussion on how MixTURE could be used for continuous (state/action) problems would be helpful.\n\nIt’s also unclear whether the proposed approach would be generalizable to other domains.\n\nQUESTIONS:\nCan you provide more insights into the generalization capabilities of the MixTURE model? How would it perform in unseen environments or more complex tasks? Or when it’s trained on a lower-difficulty level but tested on a harder one?\n\nCould you discuss how / whether sub-optimality of human demonstrations might be detected?\n\nLIMITATIONS:\nAre there specific scenarios or conditions where the MixTURE model might struggle or fail to learn effective communication strategies?\n\nAs stated previously, a discussion on whether using MixTURE for continuous (state/action) problems is not trivial would be useful.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Mixed-Initiative Multi-Agent Apprenticeship Learning (MixTURE) framework to teach a team of agents using demonstrations provided by individual human experts. It learns both a cooperative policy for each agent and the inter-agent communication policy for each agent. The proposed MixTURE, including the framework and the communication learning model, is the main contribution of the paper. Human subject studies on the proposed framework can also be viewed as a contribution.\n\nSTRENGTHS:\n+ Evaluating the framework with statistical analysis/comparison and human subject studies is a strength.\n+ Enabling multi-robot learning from demonstrations by an individual expert has the potential of improving multi-robot learning efficiency and human knowledge transfer to robot teams.\n\nWEAKNESSES:\n-  The problem of learning communications is not well justified or formulated. For example, what are assumptions on the protocol and constraints for the multi-robot communication (broadcasting or message passing)? As an example, connected autonomous vehicles use a broadcasting mechanism and follow a standard data format for communication. Why to use a differential communication channel? Are there examples of multi-robot systems using a differential channel for communication?\n\n- If communication is learned based on a mutual information metric in the proposed framework, how does learning communication benefit from human demonstrations? Learning communication seems to be a separate problem from LfD in this work.\n\n- The communication learning module needs the global joint action-observation to compute each agent’s outcome message. How is the joint action-observation information obtained during decentralized execution?\n\n- How the approach can be generalized to realistic multirobot settings and environments? In the experiments, agents have discrete states and actions, and the environment is a grid world. These  experiment settings do not well support the argument of human training of “robot” teams.  \n\n- The proposed framework follows a centralized training decentralized execution paradigm, but does not review related work on this paradigm or discuss its pros and cons. Especially, how important is multi-robot communication in centralized training decentralized execution methods?\n\n- Existing works on learning communications (e.g., when, what, who and how to communicate between multiple agents) are not reviewed. \n\n- As an opinion, advantages of using an individual human expert to provide demonstrations for a team of agents are not convincing, especially for in decentralized multi-robot systems. The general goal of LfD is to transfer human knowledge to robots/agents. Humans behave differently when working alone or with other teammates. In addition, when a single human expert controls a team of agents to perform a task, the expert still provides a sequence of decisions for different agents. However, decentralized teaming involves concurrent decisions, may have conflicts, and has tightly-coupled teaming activities (e.g., two robots carrying an object). All the above properties of teaming cannot be demonstrated by a single expert.\n\nQUESTIONS:\n- Both terms robots and agents are used in the paper. What are their differences?\n- Just curious, where does the name MixTURE (especially the TRUE) come from?\n\nLIMITATIONS:\nNo negative societal impact of the work is perceived.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a learning framework for multi-agent coordination using communication from human expert demonstrations: Mixed-Initiative Multi-Agent Apprenticeship Learning (MixTURE). A human expert teaches a team of robots to collaborate on a task by demonstrating actions for every robot, while the robot team learns a emergent communication protocol that encourages sending information about the joint observation. An attentional communication module and a mutual information maximization reverse model  are used. The framework is evaluated on three domains with different levels of difficulty and complexity. They also conduct a human-subject user study to collect real human data and assess the usability of their framework.\n\nSTRENGTHS:\n- The paper introduces a interesting framework that leverages both human demonstrations and emergent communication to learn effective coordination policies among several robots. \n- A comprehensive evaluation of their method on synthetic and real human data is conducted.\n\nWEAKNESSES:\n- The proposed framework assumes that the human expert has access to the aggregated / joint observation of all the robots at each time step and the human expert can command every robot's action simultaneously at each time step. However, this assumption is rather unrealistic for real-world robot teams or robot teams in simulation. \n- The synthetic environments that the framework is evaluated on is not large-scale enough compared to common testing environments for multi-agent reinforcement learning like SMACv2 (_SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning. Ellis, Benjamin and Moalla, Skander and Samvelyan, Mikayel and Sun, Mingfei and Mahajan, Anuj and Foerster, Jakob N. and Whiteson, Shimon._)\n- In the experimental results section, no qualitative analysis or visualization of the learned policies and communication protocols are provided. It would useful to observe how robots coordinate and communicate in different settings. \n- An expert heuristic is used to collect the expert demonstration dataset. However, heuristics can struggle to produce optimal demonstrations for more difficult environments. Then algorithms of multi-agent reinforcement learning still need to be used at the first place to collect the expert demonstration dataset.\n\nQUESTIONS:\n- How does the MixTURE framework scale to larger environments, more complex tasks or more agents? \n- What is the computational cost and communication cost of the framework?\n\nLIMITATIONS:\nNot adequately.\n- need to mention the strong assumptions on the expert's joint observation and action space", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a Multi-Agent Learning from Demonstration approach for learning multi-agent policies for collaborative tasks. The approach learns from human demonstrations of the joint policy, but does not require demonstrations of inter-agent communication, which is learned during training via online interaction. Results are presented on learning policies from both synthetic and real human data. A human study is performed demonstrating the increase in performance and decrease in workload for the proposed method, compared to methods that require demonstration of inter-agent communication.\n\nSTRENGTHS:\n- This paper relaxes a limiting assumption made by some prior works that a human expert provides demonstrations for both environment actions and communication actions, which can be taxing for a human to actually provide. In doing so, this work strikes a balance between prior RL and IL approaches for learning the action and communication policies.\n- The paper trains policies on both synthetic and human data. MixTURE is seemingly quite effective on learning from real human data.\n- The paper empirically verifies the motivating claim about the effect of removing communication actions from a human expert's workload, via a human study on 55 users.\n- A new \"mixed discriminator\" is proposed, where the discriminator is given global observations but only local actions. Per the discussion in Appendix 5.2, this choice exhibits improvements on the decentralized and centralized discriminators from MA-GAIL.\n- The work is well-written and clear. Source code is provided for reproducibility.\n\nWEAKNESSES:\n- The training objective has multiple moving parts (GAIL, PPO, BC, and MIM) which could potentially make training difficult; some additional discussion on this and how sensitive the method is to different weights on the losses would be helpful.\n- Contribution 2 is the mutual information maximization-based communication learning model. On the 5x5 PP and PCP environements, the MIM does not have a significant effect. In the FC environment, it moderately increases sample efficiency. Have ablations been run on the larger/more complex environments? This may provide more robust support for the claim that MI maximization is \"helpful for domains with increased task complexity,\" and in general may more strongly justify the inclusion of the MI objective, which otherwise complicates the training procedure.\n- Related to the above, in Section 5.1 the authors conjecture that the MIM is a key point in dealing with diversity in the human data, i.e., that it \"provides the model with the ability to reason about the underlying human demonstrations and cope with the trajectory distribution through automatically finding a suitable communication protocol.\" It would be valuable to discuss what communication protocol is learned in these settings (if it is interpretable) and why it so strongly outperforms MA-GAIL trained on the dataset with demonstrated communication actions.\n\nQUESTIONS:\nIn addition to the points described in the Weaknesses section, I would appreciate clarification on the following:\n- Could the authors clarify / provide more details on how the communication graph is generated? Do edges in the graph only exist when robots are spatially within close proximity (and if so, why is this a useful property) or do edges exist between every pair of agents?\n- How many trials are shown in Figures 2 and 3?\n- What is the process for hyperparameter tuning for the proposed method and the baselines?\n- What is the intuition for maximizing the MI between the agent's message and the joint observation (including observations that the agent does not have access to) as opposed to the agent's individual observation history?\n\nLIMITATIONS:\n- The authors note that the method does not currently account for suboptimal demonstrations. If the method is particularly sensitive to how the various losses are weighed, this may be an additional limitation in terms of the amount of tuning required.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}