Mixed-Initiative Multiagent Apprenticeship Learning for Human Training of Robot Teams
Authors: Esmaeil Seraj
Jerry Yuyang Xiong
Mariah L Schrum
Matthew Gombolay
Venue: NeurIPS 2023 poster (2023)
Keywords: Learning from Demonstration
Multi-Robot Systems
Teaching Robot Teams
TL;DR: In this work we present MixTURE, a multi-agent learning from demonstration approach that enables directly teaching teams of heterogeneous robots under a Dec-POMDP paradigm using real human generated data.

Abstract: Extending recent advances in Learning from Demonstration (LfD) frameworks to multi-robot settings poses critical challenges such as environment non-stationarity due to partial observability which is detrimental to the applicability of existing methods. Although prior work has shown that enabling communication among agents of a robot team can alleviate such issues, creating inter-agent communication under existing Multi-Agent LfD (MA-LfD) frameworks requires the human expert to provide demonstrations for both environment actions and communication actions, which necessitates an efficient communication strategy on a known message spaces. To address this problem, we propose Mixed-Initiative Multi-Agent Apprenticeship Learning (MixTURE). MixTURE enables robot teams to learn from a human expert-generated data a preferred policy to accomplish a collaborative task, while simultaneously learning emergent inter-agent communication to enhance team coordination. The key ingredient to MixTURE's success is automatically learning a communication policy, enhanced by a mutual-information maximizing reverse model that rationalizes the underlying expert demonstrations without the need for human generated data or an auxiliary reward function. MixTURE outperforms a variety of relevant baselines on diverse data generated by human experts in complex heterogeneous domains. MixTURE is the first MA-LfD framework to enable learning multi-robot collaborative policies directly from real human data, resulting in ~44% less human workload, and ~46% higher usability score.

Mixed-Initiative Multiagent Apprenticeship Learning for Human Training of Robot Teams
Esmaeil Seraj
Jerry Xiong
Georgia Institute of Technology eseraf30gatech. edu
Georgia Institute of Technology j 3xiong60@gatech edu
Mariah Schrum
Matthew Gombolay
University of California Berkeley mariahschrumberreeleyy edu
Georgia Institute of Technology matthew cottheewgombllay@ee. gattoollllycc....ccc...coo gatech. edu
Abstract
Extending recent advances in Learning from Demonstration (LfD) frameworks multi-robot settings poses critical challenges such as environment con-stationarity due to partial observability which is detrimental to the applicability of existing methods. Although prior work has shown that enabling communication among agents of a robot team can alleviate such issues, creating inter-agent communication under existing Multi-Agent LfD (MA-LfD) frameworks requires the human expert to provide demonstrations for both environment actions and communication actions, which necessitates efficient communication strategy on a known message space. To address this problem, we propose Mixed-Initiative Multi-Agen Apprenticeship Learning (MixTURE). MiXTURE enables robot teams to learn from a human expert-generated data a preferred policy to accomplish a collaborative task, while simultancously learning emergent inter-agent communication to enhance team coordination. The key ingredient to MixTURE's success is automatically learning communication policy, enhanced by model that rationalizes the underlying expert demonstrations without the need for human generated data a variety of relevant baselines on diverse data generated by human experts complex heterogeneous domains. MiXTURE is the first MA-LID framework enable learning multi-robot collaborative policies directly from real human data, resulting in 44% less human workload, and 46% higher usability score.
maximizing reverse
auxiliary reward function. MiXTURE outperforms
Introduction
In recent years, Multi-Agent Reinforcement Learning (MARL) has been predominantly used by researchers to optimize a reward signal and for learning multi-robot tasks. Nevertheless, RL generally suffers from key limitations such as difficulty in designing expressive and suitable reward function
for complex tasks [60] 2 which can lead to undesirable robot behavior [14 60 64], high sample
complexity [29]. and safety concerns due to direct robot-environment interactions for optimizing
the policy (64). These problems are further exacerbated in multi-robot scenarios where inter-robot interactions and environment dynamics can be more complex and task descriptions and objectives more ambiguous (60) (53 (2). As such, accurate models of human strategies and behaviors achieved via imitation methods are increasingly important for safely and effectively deploying autonomous systems and aligning values motivating robot behaviors with human values 47
22 [2]
Learning from Demonstration (LfD) attempts to learn the correct behavior (policy) from set of expert-gent ggeerrtedd demonstrations rather than a reward function, which can result lower sample
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

complexity and a learned policy that more closely reflects the human's 's preferred strategy Unfortunately, extending existing single-agent LfD paradigms such as Behavioral Cloning (BC) 66
48
47
or Inverse Reinforcement I Learning (IRI.) [1 to multi-robot settings poses several challenges such as environment statiionaon and existence of multiple equílibrium solutions (an agent's s optimal policy depends on other agents' policies) [37 27 60). One can adopt such frameworks directly in centralized system. However, centralized systems are not scalable, are prone to single-node failure, and pose significant computation overhead, and therefore, decentralized approaches (e.g., limited- range communication and local computations) have been more desired in multi-robot systems [20]
Prior work has shown that enabling communication among agents of robot team creates shared mental model of joint -sciinn-paces and therefore, allowing coordinated action decisions [38 65 [55]
19 ST to handle challenges such as partial observability and environment dynamicity [25] 54 [5].
Although LfD can resolve the high sample complexity and reward shaping problems in RI., the task of MA-LffD can be onerous for the humans as they have to control multiple robots and imagine and simulate an appropriate Theory-of- -f-- to create a communication strategy for the robot team.
Introducing the inter-agent communication 32 under current state-of-the-art (SOTA) Multi-Agent LID (MA-L(D) frameworks such as Multi-Agent Generative Adversarial Imitation Learning (MA- GAIL) 60 or Multi-Agent Adversarial IRL (MA-AIRL) [7]] requires the human expert to provide demonstrations for both environment actions and communication actions. Such approaches assume that the human expert has access to an efficient communication strategy on a known finite message space addition to a known strategy for taking environment actions 42]. This assumption, however, is invalid, since dynamic environments with multiple interacting agents will be too complex and unpredictable for humans to be able to develop cohesive and comprehensive inter-agent communica- tion protocols. Even if such an efficient communication strategy exists, demonstrating both the task strategy as well as the communication strategy could potentially pose significant workload on the human expert, which in turn can affect the performance and quality of demonstrations. These issues become even more severe when the robot team heterogeneous or of composite nature (i.e., agents with different observation- and action-spaces as well as different objectives) where agents must rely on communication to operate and fulfill their tasks correctly [54]] 711.
To address these challenges, we develop a distributed MA-LiD framework to efficiently incorporate a human expert's s domain-knowledge of teaming strategies for collaborative robot teams and directly learn team coordination policies from expert human teachers. To this end, we propose Mixed- Initiative Multi-Agent Apprenticeship Learning (MixTURE) which enables robot teams to learn an expert's 's preferred strategy to act in environment. MiXTURE simultaneously learns end-to- end emergent communication for the robot team to enhance team coordination, without the need for human generated data or auxiliary reward function. To improve the quality of the learned inter-agent communication protocol, we reduce the entropy of a generated message given joint states and actions by maximizing the Mutual Information (MI) between messages and joint states. We demonstrate through empirical evaluation and a human subject experiment that our LfD-based MiXTURE outperforms RL based methods due to reward function independence and low sample complexity. Furthermore, MiXTURE significantly alleviates the human demonstrators' workload and time required to provide demonstrations, increases system usability, and improves overall collaboration performance of the robot team. Our key contributions are as follows:
1. We propose the MiXTURE framework for learning robot teaming strategies from human expert demonstrations while simultaneously learning inter-agent communication through online interactions during training, without the need for expert data or an auxiliary reward.
2. We develop an MI maximization- based emergent communication learning model which reduces the entropy of a generated message for an agent given joint state-observations.
3. We evaluate MiXTURE on real, diverse human genea-geee data, collected a human- subject user study, and show that, in complex multi-agent domain with heterogeneous tasks, we are able to achieve ~ 42% ~ 77% higher performance and a significantly lower sample complexity. We also show that using MiXTURE significantly improves workload and system usability relative to a benchmark MA-LiD framework. To best of our knowledge, this is the first work to train a MA-LiD framework on real human data.
4. We investigate the effects of demonstrating both environment actions and communication actions on a human expert's 's workload, demonstration quality, and system usability score. Our results show that a high-workload demonstration process in classic MA-LID approaches

---

significantly (p < .001) reduces an expert's demonstration quality (measured by perfor- mance) and the system's 's usability score. MiXTURE significantly improves these results; increasing a human's performance and experience engaging in MA-L.fD.
2 Related Work
The literature for Multi-Agent LfD (MA-L.f) primarily aims address the complexity of simul- taneously training multiple agents under coordinated [33 58
68 28 and uncoordinated tasks 66 69 40 77]. In [61] 8] the MA-LfD problem is reduced into single-agent problem by mak- ing the assumption that all agents share the same dynamics, observation spaces, and model parameters. In [4], coordinated multi-agent IL approach is proposed which learns a latent coordination model along with the individual agent policies. In [60] the single-agent GAIL framework is extended for multi-agent scenarios along with a practical actor-critic method for multi-agent imitation. Similarly, in [71] the AIRL method was extended to the multi-agent settings. In [62] scalable multi-agent LfD approach is proposed where a model- based heuristic method for automated swarm reorganization is leveraged to improve multi-agent task allocation problem. In 4 an expert feedback- system is developed address multi- aggntt path-finding problem. In 64 authors create an advising system to incorporate sub-optimal model-based heuristic policies to help improve MARL performance. Other prior work focused on learning human profile/behavior models for improved MA-LiD [45] [5). More recently, Hoque et al. (26) proposed Fleet-DA formalizing interactive fleet learning setting, in which multiple robots interactively query and learn from multiple human supervisors.
46
57
Nevertheless, applicability of these prior works in the collaborative multi-agent problems is consider- ably limited since none of these works explicitly address the inter-agent communication in dynamic and partially observable domains where agents not only need to take task-related actions, but also need communicate and share information for coordination. Additionally, none of these prior work leverage real human-genee generated data for training to evaluate the approach against heterogeneity and diversity human data. Our work addresses these limitations by eliminating the requirement for an expert to demonstrate a communication strategy. Instead, the human expert only needs to teach the robot team how to complete a task through demonstrations, and the robots will automatically learn a communication strategy that aligns with the expert's 's demonstrations. We also collect real human data to evaluate our method's ability to cope with variations in demonstration styles and strategies.
3 Problem Formulation: General MA-LfD with Heterogeneous Agents
We ground our problem formulation in a Markov Game (MG) 36 generalized to include partial observability and heterogeneous agents. We define a set of heterogeneous agents in a composite robot team (i.c., composed of different classes of robots) as agents that can have arbitrarily different state-, observation-, and action-spaces. The agents can also have different task objectives which, when enacted in coordination, enable the team to achieve a shared overarching mission. Accordingly, we Joec, r, T, 7). C is set of all available agent classes in the composite robot team and the index e C denotes the agent class. . Eieecy (())) is the total number of collaborating agents where (1 represents
define our generic MG as a 9-tuple: (C,N, (S(c) Joec, Je(C) (2(c)) Joec, fo(e)
the number of agents in class c. (S(e))ee and {A((e)) Jeee are discrete joint sets of state- and action-spaces, respectively. {S2()) is the joint set of observation-spaces, including class-specific observations. Agents of the same class have identical A, and J2. 7 E [0, 1) the temporal discount factor for each unit of time and T is the state transition probability density function. At each timestep. 1, each agent, j. of the e-th class can receive partial observation of¹ E 22(c) according to some class-specific observation function (Olo) JoEC : of ~ (-18). Next, each agent, j. of class c, takes an action, a;" forming a joint action vector à = (az" a........ ....... aa²)
When agents take the joint action in the joint state and depending on the next stint- they receive an immediate reward, r(8, a) E IR, shared by all agents regardless of classes. Each agent, j. of a class, achieves its own objective by sampling actions from stochastic policy T The objective of each agent is then to maximize the team return (expected sum of discounted rewards), i.e., Ex - 7'r
3

---

To directly learn the human's 's preferred strategy and resolve the reward specification problems 2 posed by RL, we leverage a demonstration dataset, D. provided by an expert, rather than the ground truth reward signal r employed in MARL.. Unlike 60 or [67], we do not assume multiple human experts in our MG to avoid the need for further coordination amongst the experts, which can be (
time consuming and expensive. D is set of trajectories {rī) 13=1 where 7j = {(o]" a )T= is an expert trajectory collected by sampling ai ~ Tg(a? (ōt) in which Te is the expert policy and o is the joint observation that the expert has access to at time t. We further assume that D contains the entire supervision to the learning algorithm (i.e., no online interactions during training). We build the MiXTURE architecture on the generative adversarial training [21]. Our distributed GAIL objective underlying MiXTURE is shown in Eq. where D/ is local discriminator that classifies expert and policy trajectories for agent j of class c, and T. (ry) is the parameterized policy of agent j of class c.
pg)))) (a,a)) 1g (c>) (ō, a)] -E
log D (c) (o.a)
(I)
T~x (0,a)~~
| and at optimality, the distribution of generated state-action pairs by &े should match the distribution of demonstrated state-action pairs.
According to 21, under the GAIL objective in Eq.
4 Mixed-Initiative Multi-Agent Apprenticeship Learning
Motivation and Problem Overview Consider a generic composite team of robots including agents with heterogeneous characteristics and task objectives. Without loss of generality, consider robot team composed of perception-only and action-only robots. Under our problem formulation in Section 3 collaborate perception robots and action robots create two separate classes of agents that need to overarching mission. For instance, in application of wildfire fighting, robots of the perception class (e.g., quadcopters) need to search an environment for firespots, while the action robots (e.g., fire-extigguishnng ground robots) who cannot sense the environment are required to extinguish the firespots found by the perception robots [54] 55 52 3. Note that neither of the robot classes are capable of accomplishing the task without the other class.
To teach a collaborative policy to such a robot team, one can leverage demonstrations from a team of humans where each member is an expert. Using a team of human experts, however, poses further challenges: (1) simultancous access to several human experts is expensive and can be time consuming, and (2) a communication strategy (e.g.. natural language) among the human demonstrators is required for coordination, which can be challenging to translate to robot domain due to ambiguity, colloquialisms, and context-dependent use 33 23 Additionally, humans' communications might not make sense to the robot agents as humans could be unaware of all agents' full local state spaces.
Alternatively, we can leverage the demonstrations from a single human with domain-knowledd, regarding the entire mission objective. For example, trainer/coach can play a simulated game of firefighting using the aforementioned perception and action robots and provide expert demonstrations for how to efficiently distribute agents and prioritize tasks for searching the environment and putting out the fire. The challenge of using single human expert, however, is that in this case the human would also need to demonstrate communication-actio broadcast at each state) on top of the environment-actioon this end, humans would have to create and maintain a theory-of-Mine (ToM) of each agent under this increased action-space dimensionality, which significantly increases workload [5 I7].
(i.e., what information should an agent (i.e., moving around or dousing fire). To
To resolve this problem, we propose taking separate initiatives for teaching the robots in the team how to operate (environment actions, dt, per observation, o₂) and how to communicate (commu- nication message, 2t, per state, 3t) such that a human expert would only be required to provide environment-actin action demonstrations and the robot team would automatically infer on their own suitable communication policy for the underlying expert demonstrations. We call our approach Mixed-Initiative Multi-Agent Apprenticeship Learning (MixTURE).
MiXTURE Architecture - The proposed MiXTURE architecture for human training of robot teams is shown in Fig. []] At each timestep, each agent generates an embedding, representing agent's ' belief space, from its local observation. To handle agents' partial observability, the local observation embeddings are then passed into local recurrent policies for each agent. Each GRU policy receives the preprocessed features from the local observations as well as its own hidden state from previous timesteps. Therefore, the policy output depends only on the history of local observations and actions.

---

Forward Model
a2
Reverse Model
Mutual Information Maximization
Zij
Attentional
Comm. Processing
Comm. Pracessing
(a, xy1
Comm.
FC
FCJ
MSE Loss
For 1, maximice Ml vin::
ajzj
(a. xx))
ZFz
max MI MI 6)
Recurrent Pollcy
Recurrent Policy GRU:
Preprocessing
Policy Policy Section
GRU: R
Local
Optimization Module
Discriminators
embedding
embedding
(o.ag)
Difo o
Yop
Optimization (PPO)
Prepracessing
Preprocessing
BCE Loss
FC
FC}
(
D o2
Agent
Agent
1j
Expert Sample: Dg (0c.
The proposed MiXTURE architecture for two-agent example scenario. At each timestep, each agent generates an embedding from its local observation, which then passed into local recurrent policies for each agent. To learn from the expert demonstrations stored in the dataset Dg. we build a distributed multi-agent GAIL architecture. We enable inter-agent communication by adding attentional communication module enhanced by MI maximization reverse model.
Figure 1:
To learn from the human expert demonstrations stored in the dataset DE. as shown in Fig. I build distributed multi-agent GAIL architecture where each agent is equipped with a parametrized discriminator, Dg. The discriminators between state-action pair samples from the expert dataset and those generated by the generator (i.e., local policies). The output of the discriminators are treated local rewards, which are then combined to encourage collaboration and teaming behaviour among agents [31]. To learn the agent policies through this shared reward signal, we leverage the Proximal Policy Optimization (PPO) 49
trained via Binary Cross Entropy (BCE) loss distinguish
We enable inter-agent communication by adding an attentional communication module in which each agent equipped with fully-connected network that processes aetion-embeeddings generated by the recurrent policies of an agent and those generated by its local teammates (i.e., messages, 2) to output an action-decision. We enable this message-ppssiinggg by creating differentiable communication channels among agents. To maintain locality, these communication channels can be leveraged locally (i.e., communication graph where edges only exist when robots spatially within close proximity). For an agent, i, the action embedddin messages received from teammate, j. are weighted by some learned attention coefficients, a'jt to assign message importance. Therefore, the input message for agent at time, f, can be computed
m ii = Exe(" Oze2je where Ni represents the neighbors, j, of agent i at time t. Here, a:j are the learned attention coefficients that are computed via O₁ = softmax, (a aeeeee/h,,, (wh,, || m)))))) In this equation, Wat are the learnable weights of the attention network, || represents concatenation, o is an activation function nonlinearity, and represents the hidden states. The Softmax function is used to normalize the coefficients across all neighbors j. Such attentional communication can enhance the action-decision quality, particularly with increased number of agents when the states may significantly vary in different parts of the environment [6 Messages in our communication module are entirely learned via backpropagation.
Mutual Information Maximization-Basse Differentiable Communication A challenge with the communication model learned via the described end-to-end differentiable channels that the distribution of the messages broadcasted by agent, i, given the state-observations,, p(zig jö), can have a high variance. The desired behavior, instead, is that the agents employ a cohesive communication strategy in which an agent sends consistent message when it observes relatively similar states.
To resolve this issue, we propose maximizing the Mutual Information (MI) between an agent's outgoing message and the joint otste-obeervattios. MI is measure of the reduction in entropy of a probability distribution, X, given another probability distribution, Y, such that T(X;Y) ((X)= H ((X where H (X) denotes the entropy of X and H (X(Y) is the conditional entropy
5

---

of X given Y B33. In our work, by maximizing the MI between the distribution of an agent's message, p(zzj o). and the joint observations, we reduce the entropy over messages and encourage the communication model to be more consistent. Maximizing the MI in this way encourages žij to correlate with features within the observation distribution (i.e., mode discovery)
13
Unfortunately, a direct MI Maximization (MIM) between the message distributions and joint obser- vations as formulated above, [Zyz 6) (2gg) H (2tg) lo), is intractable as requires access to the true posterior, p( (Zzg k)) Therefore, in keeping with prior work, we rely on the Evidence Lower Bound (ELBO) of the MI instead. As shown in prior work [3. by minimizing an MSE loss between a sample from the current message embedding and the approximate posterior, modeled as normal distribution with constant variance, equivalent to maximizing the likelihood of the posterior.
In practice we build a distributed reverse model in the MiXTURE architecture (shown in Fig. D to accommodate the mentioned MSE loss. The distributed reverse model for each agent has access to the local received messages as well as the global joint-actions taken by all agents such that the optimization results in a communication entropy-reduction mechanism at the team level. Since actions and generated messages are functions of the state-observations, each reverse model, θ, can take in the joint actions, a, and all received messages, Zji, estimate the outgoing message for agent, Big (a, =fz). The policy and reverse models are trained together in an end-to-end fashion to minimize the message reconstruction error as LMIM=: = t E |a, zja) - Zzyll² 112
i, as of
We note that while the addition of the MI maximization loss would make the overall loss function more complex, we believe there are further benefits that could be achieved via training the MIM reverse model, besides enhancing agent certainty in choosing actions. Particularly, we believe our MIM reverse model, when tuned well, has the potential to roughly cluster message embeddings based on observation-action pairs which can provide further useful information on the learned communication protocol for task and further insight into interpreting the learned communication protocols.
Training and Execution We build the MiXTURE framework in a Centralized Training for De- centralized Execution (CTDE) paradigm 19 accommodate for the global joint-action inputs to the MI maximizing reverse models during training. We note that, the MIM reverse model is only used during training and cut during the execution, and therefore, the learned policies can be executed fully decentralized. To optimize the policies based on the reward signal generated by the discriminators, we leverage the PPO algorithm 44). Moreover, enhance and stabilize the training
we propose combining an offline BC loss, LBC=. = - En (a," oo 24.), with the online GAIL loss. As shown by prior work [18), through this combination, the offline BC helps preserving ground knowledge that should be respected during training, while the online part helps with learning of new information encountered during execution. As such, putting together
distributed GAIL loss in Eq. [ the standard clipped PPO loss 44). the offline BC loss, and the MIM loss introduced above, we present the full loss expression to train the MiXTURE architecture as in Eq. [ where Çr and {DE are minibatches of trajectory segments belonging to current policy, ग, and demonstration dataset, DE. respectively, N is the total number of agents and À is tunable scaling parameter.
N
(ST (DL) =
Σ (gyo (age:
(2)
5 Evaluation
We break the problem of evaluating our proposed architecture for teaching multi-agent coordination policies to (heterogeneous) robot teams into three research questions (RQ):
RQ1 Can MiXTURE learn useful multi-agent coordination strategies from synthetic data (e.g, models of human experts Oz-of- wizard [63 ?
RQ2 Can MiXTURE learn from diverse data generated by real human experts?
RQ3 How challenging is it for human expert to provide multi-agent demonstrations and does MiXTURE alleviate these challenges (comparing workload and system usability)?
we selected three multi- agent domains that are partially observable and require collaboration among heterogeneous agents (see Section 3). Please refer to the supplementary material for more details about the environments.
Environments - In keeping with prior work in MARL and MA-LfD 60 54

---

Predator-Pres Prey 10x10
Predator-Capture-Prg 10x10
FireCommander 10x10
episode length
75
75
50
50
60
25
25
40
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
timesteps
1e5
timesteps
le5
timesteps
1e5
MARL
MA-GAIL
MA-GAIL
MiXTURE
Figure 2: Evaluation results for MixTURE and the baselines in the medium case on synthetic dataset. MixTURE outperforms all baselines in both sample complexity and performance at convergence.
1. Predator-Prey (PP) [59] the goal in this homogeneous (i.e., same class agents) domain is for N predator agents with limited vision to find a stationary prey and move to its location.
2. Cecdator-Capture-Pree (PCP) [54]
new class of capture-agents, are introduced to PP. In this heterogeneous domain, the goal of predator agents is the same, while capture agents must move to the prey location and capture it, without having any observation inputs.
3. FireComander (FC) 14 56]:
two classes of robots, perception and action robots, are required to collaborate to extinguish propagating firespots. Similar to the PCP domain, perception robots search the domain find hidden firespots. and action robots with no observation must rely on communication to know where to put out the firespots using an extra action when on fire. Unlike the PCP, in this complex domain firespots randomly spread over time and thus, the team must continue until all firespots are found and extinguished.
To investigate our RQ1, we employ variety of baselines, described below, all of which utilize the combined offline BC and online loss training scheme. To collect the synthetic dataset, the expert heuristic (Appendix 4) for environment-actions is a near-optimal search algorithm and for the communication, it an anticipatory observation sharing mechanism inspired by prior work
Baselines
MARL
49
MA-PPO optimizing both environment and communication policies.
BC+DC
66
MiXTURE ablation trained only via offline BC loss with diff. communication.
NC MA-GAIL
660
Non-communicative ablation of the MA-GAIL 60 framework.
MA-GAIL
60
Full MA-GAIL. trained on full dataset w/ demonstrated communication.
5.1
Human-Subject Experiment: Conditions and Procedure
To investigate RQ2 and RQ3, conducted IRB-approved human-subject user study in the FireCommander domain. For more experiment details please refer to the supplementary material.
Domain, Setup, and Procedure- Our experiments were conducted in the context of a simulated multi-robot task, leveraging the FireCommander (FC) domain under different difficulty levels and modes. Depending the level of difficulty, there can be multiple initial firespots, hidden from the human, that propagate randomly based on a fixed wall-clock rate. The human expert was responsible to strategically move the simulated robots to find and extinguish all firespots as fast as possible. The human subject was shown a performance score at the end of each round, computed based on existing, found, and extinguished fires. After briefing, each subject began by filling in a pre-questionnaire form (demographic and prior videogame experience) followed by reading through a series of detailed game instructions. To minimize the learning effect, subjects were allowed to practice all conditions until they felt comfortable. Next, each subject played six different rounds of the game (i.c., two conditions and three difficulty levels) in a randomly selected order. The demonstration data was fully stored to be later used for training. Finally, each subject was asked to fill some post- mett-mmmaeeent forms.
Participants We recruited 55 participants (mean age 25.0 2.67: 34.5% female). All participants were recruited through on- university-camppe campus advertisement and were trained equally for the task.
Independent Variables and Conditions In this study, we seek determine if MiXTURE can learn multi-agent collaborative policies from diverse human generated data without demonstrated communication and we compare its performance against the MA-GAIL (60] with expert demonstrated
7

---

Number of steps taken win the game (lower is better).
Predator-Prey
Predator- Predator-Capturee Capture-
FireCommander
Heu. Diff.
easy
medium
hard
easy
medium
hard
easy
medium
hard
MARI
15.57
17.93
47.40
23.38
56.04
79.76
78.77
79.09
80.00
BC+DC
V
11.84
21.28
46.16
15.38
61.54
79.75
38.95
44.70
72.87
NC AAM--AIL
17.48
60.38
79.52
27.60
77.77
80.00
76.49
79.47
79.98
MA-GAIL
11.68
20.39
49.08
16.43
44.84
79.86
77.46
79.47
80.00
MiXTURE
11.16
13.15
28.73
13.08
17.27
36.41
22.31
34.82
56.49
Table 1: Full evaluation results for all methods and all difficulty settings on synthetic dataset. Heu. and Diff. indicate models access to heuristic differentiable communication, respectively.
Easy
Medium
Hord
8D
8D
72
72
64
5G
66
56
60
eptsod
54
48
24
8
16
0.0
1.0
timesteps
le6
tmeestpp
146
timesteps
MARL
NC MA-GAIL
MA-GAIL
MisTURE
BC+
Figure 3: Evaluation results for MiXTURE and MA-GAIL 660 on real human data. From left to right, the figures present test results for easy, medium, and hard scenarios with one initial firespot.
communication. We examine the performance across three levels of difficulty, i.e., easy, medium, and hard, which represent game complexity. As such, we utilize a 2 x 3 within buthin-subjects experiment design varying across two abstractions: (1) only demonstrating environment actions for each robot each time step (i.e., noComm condition). 2) demonstrating both environment actions and communication actions for each agent each time step ((i.e, WithComm condition)).
Metries - To evaluate our RQ2 and RQ3, the demonstration data collected from the human subjects used to train MixTURE and the benchmark MA- GAIL [60], respectively. The algorithms are compared in terms the learned policy performance (i.e., number of steps taken to win the game, where lower is better). Additionally, address our RQ3, we leverage the NASA-TLX Workload Survey [24] and the System Usability Scale (SUS) [
for both noComm and withCommm conditions
5.2 Results and Discussion
RQ1: Evaluation on Synthetic Dataset- We perform our evaluations across three different difficulty levels: (1) easy (5 x 5 domain, 3 robots), (2) medium (10 x 10 domain, 6 robots), and (3) hard (20 x 20 domain, 10 robots). Table [ presents the full evaluation results for collaborative policies learned by MiXTURE and the baselines in terms of number of steps taken to win the game (lower is better). The test results show the mean performance values calculated over ten trials of running the best training models. As shown, MiXTURE achieves significant improvement over all baselines and all domains. Additionally, the learning curves in Fig. 2 (medium case), show significant improvement in sample complexity. We believe our model provides a strong step towards learning collaborative policies in multi-robot systems by setting new SOTA in complex heterogeneous tasks.
We also performed several ablation studies, such as investigating scalability and effects of the MIM reverse model for message reconstruction, discriminator architecture, and the combined offline BC loss. Due to space constraints, all ablation results are available in the supplementary material.
RQ2: Training and Evaluation Results on Human-Subject Dataset - We train MiXTURE and MA-GAIL (0)) (and other baselines) on data collected in our human-subjee user experiment to investigate RQ2. The results are presented in Fig. Please see the Appendix for environment
8

---

details. As shown, MiXTURE can learn gighh-ualtty multi-robot collaboration policies from diverse numan-generated demonstrations. This while purely relying on human demonstrations [60], does not succeed in this task. We hypothesize that key point underlying MiXTURE's success despite diversity and heterogeneity in human data is that the MIM-based differentiable communication channels provide the model with ability to reason about the underlying human demonstrations and cope with trajectory distribution through automatically finding a suitable communication protocol. This result also shows that demonstrated communication policy by human expert is not enough to efficiently coordinate the robot team in complex domains as the human is unaware all agents' local states, which leads to the failure of existing MA-LfD frameworks such as the MA-GAIL (60]. We hypothesize that MA-GAIL trained on demonstrated communications fails because: (1) humans are not fully aware of all agents' full local state spaces and therefore, the demonstrated communication could be severely inefficient and sub-optimal. MiXTURE instead learns an efficient communication policy automatically through gradient updates; (2) The laborious task of providing both environment and communication actions for the human demonstrators significantly deteriorates the quality of demonstrated policies, which can be confirmed by our subject-study results in next section.
We believe that this strong result shows great potential for the MiXTURE model to efficiently teach multi-agent coordination and collaboration policies to robot teams through human demonstrations.
RQ3: Statistical Analysis We investigate our RQ3 by quantifying the workload and SUS measures reported by the human subjects. We hypothesize that:
H1 Demonstrating both an environment- and a communication-actinnn strategy for the robot team increases the human expert's 's workload and decreases the system's usability score.
H2
Demonstrating both an environment-action and a communication-action strategy for the robot team negatively affects human performance and the demonstration quality.
HI: We test for normality and homoscedasticity and do not reject the null hypothesis in either case, using Shapiro-Wilt Wiilk (p > 0.32 and > 0.96) and Levene's (p > 0.39 and p > 0.09) tests for workload and SUS, respectively. For workload, we perform paired t-test and find that using the MiXTURE model w/o communication demonstration was rated statistically signif- icantly lower than using MA-GAIL w/ demonstrated communication by the expert (p 0.001) on NASA-TLX workload scale. For system usability, using a similar a paired t-test we find that using the MixTURE model w/o communication demonstration led statistically significantly higher SUS than using MA-GAIL w/ demonstrated communication by the expert (p < 0.001). As shown in Fig. f relaxing the need for demonstrating a communication strategy reduced the humans' workload by 44.3% and increased the systems' usability scale by 46.1% 1% in our experiment.
H2: We examine: (I) human's performance score in the game, (2) total tasks completed (i.e., fires extinguished), and (3) average demonstra- tion time per step. We apply a paired samples Wilcoxon test and confirm that (see Fig. 5 from left to right), relaxing the need for demonstrating a communication strategy through MiXTURE leads to achieving significantly higher perfor- mance score (p < 0.001) by the human, bet- ter ability to scale to more complex scenarios with more tasks (p- < 0.001), and significantly lower demonstration time per step (p < 0.001).
120
100
80
Scores
60
40
20
noComm
withComm
Note that, the middle plot in Fig. 5 shows more tasks completed (i.c., more firespots killed) for withComm condition under easy and moderate scenarios, which indicates worse human performance in this conditions since existence of more firespots means inefficiency in extinguishing the fire before it spreads too large. On the other hand, under the noComm condition, humans can easily scale to larger domain sizes and more initial firespots. Note that, lower tasks completed for withComm condition under the hard case is attributed to failing the task and losing the game (i.e., hard game cut-off when score drops below 50).
Workload
SUS
Figure 4: Workload and SUS results for HI.
Further Discussions and Limitations Demonstrating multi-agent strategies can be considered a highly involved and high-woog workload task, which in turn can affect human's 's situational awareness and

---

noComm condition
withComm condition
Aug. Time por Step (secands))
Total firespots
Eosy
Moderañe
Hard
Moderate
Easy
Moderate
Hard
Figure 5: Objective human performance results supporting the H2. In summary, using the MiXTURE framework leads to significantly better human performance and a lower demonstration time.
optimality of demonstrations. We posit that human's 's decision decision-making for one agent is influenced by knowledge of other agents' observations. However, we note that, classic MA-LfD methods, such as MA-GAIL, necessitate awareness all robots' states and observations and they require providing both environment and communication actions the same time. MiXTURE, we relax this assumption by tasking the human to only provide environment actions for robots one at a time. Our real human subject study demonstrates this relaxation's feasibility, enabling successful demonstrations provided by humans for large and heterogeneous teams of simulated robots. In practice, one can use existing multi-agent datasets (e.g., collaborative assembly ROBOTURK [39] 67 or Meta's STARDATA for StarCraft II (35)) readily create demonstration dataset using existing tools (c.g.. ROS, Gazebo). MixTURE's s contribution comes in at this stage, where we can learn collaborative multi- agent policies from such datasets.
The addition of the MIM loss Eq. can complicate the tuning process. While we never observed any performance decay as a result of the MIM loss, tuning the MIM loss to achieve consistent and significant performance improvement seems to challenging. Finally, MiXTURE currently does not account for demonstration sub-optimality. An interesting future direction is then to modify the MiXTURE architecture to address suboptimal human demonstrations.
Conclusion We proposed the MiXTURE model to learn multi-agent collaborative policies for robot team, directly from human expert demonstrations. Using our method, a human expert can teach the robot team how to accomplish a task collaboratively via demonstrations and the will automatically reason over and learn a communication strategy suitable for the underlying demonstrations. The learned communication helps the robot team to deal with the partial observability. reasoning about decisionsss to best respond to teammates' policies, and alleviate the effects of environment non non-staiionarity. We provided several empirical and experimental results, confirming MixTURE's strong ability to learn from expert heuristics and real diverse human generated data.
Broader Impact
Our experiments show that multi-agent learning from demonstration can be enabled for teams of cooperating heterogeneous robots via direct human teaching. This greatly assists robot collaboration through human domain knowledge and provides a strong next-step towards human-robot teaming. We note that our research could be applied for good or otherwise, particularly an adversarial setting. Nonetheless, we believe democratizing this knowledge is for the benefit of society.
Acknowledgments
This work was supported by the Naval Research Laboratory (NRL) under grant number N00173- 21-1-G009 and grant number N00014-22-1-2882. The authors also would like to thank Manisha Natarajan for being sounding board and guiding statistical analysis.
References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), page 1, 2004.
[2] David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
10

---

[3] Ian F Akyildiz and Ismail H Kasimoglu. Wireless sensor and actor networks: research challenges. Ad hoc networks, 2(4):351-367, 2004.
[4] Ahmed Alagha, Rabeb Mizouni, Jamal Bentahar, Hadi Otrok, and Shakti Singh. Multi-agent deep reinforcement learning with demonstration cloning for target localization. IEEE Internet of Things Journal, 2023.
[5] Feryal Behbahani, Kyriacos Shiarlis, Xi Chen, Vitaly Kurin, Sudhanshu Kasewa, Ciprian Stirbu, Joao Gomes, Supratik Paul, Frans A Oliehoek, Joao Messias, et al. Learning from demonstration the wild. In Proceedings of the International Conference on Robotics and Automation (ICRA), pages 775-781. IEEE, 2019.
[6] Sage Bergerson. Multi-agent inverse reinforcement learning: Suboptimal demonstrations and alternative solution concepts. arXiv preprint arXiv:2109.01178 2021.
[7] Matteo Bettini, Ajay Shankar, and Amanda Prorok. Heterogeneous multi-robot reinforcement learning. arXiv preprint arXiv:2301.077 0001777.77. 2023.
[8] Raunak P Bhattacharyya, Derek Phillips, Blake Wulfe, Jeremy Morton, Alex Kuefler, and Mykel J Kochenderfer. Multi-agen imitation learning for driving simulation. In Proceedings of the JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1534-1339 IEEE, 2018.
[9] John Brooke et al. Sus-a quick and dirty usability scale. Usability evaluation in industry. 189(194):4-7, 1996.
[10] Abhizna Butchibabu. Anticipatory communication strategies for human robot team coordination. PhD thesis, Massachusetts Institute of Technology, 2016.
[11] Abhizna Butchibabu, Christopher Sparano-Huiban, Liz Sonenberg, and Julie Shah. Implicit coordination strategies for effective team communication. Human Factors, 58(4):595-610, 2016.
[12] Letian Chen, Rohan Paleja, Muyleng Ghuy, and Matthew Gombolay. Joint goal and strategy inference across heterogencous demonstrators via reward network distillation. In Proceedings of the ACM/IEEE International Conference Human-Robo Interaction (HRI), pages 659-668, 2020.
[13] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 29, 2016.
[14] Jack Clark and Dario Amodei. Faulty reward functions in the wild. Internet: https://blog. openai. hewdll---rrwar----nittoooo 2016.
[15] Mary L Cummings and Carl Nehme. Modeling the impact of workload in network centric supervisory control settings. In Neurocognitive and Physiological Factors During High-tempo Operations, pages 23 41. CRC Press, 2018
[16] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In Proceedings of the International Conference on Machine Learning (ICML), pages 1538-1546. PMLR, 2019.
[17] Birsen Donmez, Carl Nehme, and Mary L Cummings. Modeling workload impact in multiple un- manned vehicle supervisory control. IEEE Transactions on Systems, Man, and Cybernetics-Par A: Systems and Humans, 40(6): 1 180-1190, 190,, 2010.
[18] Lydia Fischer, Barbara Hammer, and Heiko Wersing. Combining offline and online classifiers for life-long learning. Proceedings of the International Joint Conference on Neural Networks (UCNN), pages 1-8. IEEE, 2015.
[19] Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurlPS), 29, 2016.
11

---

[20] Avinash Gautam and Sudeept Mohan. A review of research in multi-robot systems. In Proceedings the IEEE 7th International Conference on Industrial and Information Systems (ICIIS), pages 1-5. IEEE, 2012.
[21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, (3(11):139-11 2020.
[22] Nate Gruver, Jiaming Song, Mykel J Kochenderfer, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning with latent variables. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 1855-1857,
2020.
[23] Ibrahim A Hameed. Using natural language processing (nlp) for designing socially intelli- gent robots. In Proceedings of the Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), pages 268-269) IEEE, 2016.
[24] Sandra Hart and Lowell E Staveland. Development of nasa-tix (task load index): Results of empirical and theoretical research. In Advances in Psychology, volume 52, pages 139-183. Elsevier, 1988.
[25] Guy Hoffman and Cynthia Breazeal. Collaboration in human-robot teams. In Proceedings of the AIAA 1st Intelligent Systems Technical Conference (IntelliSys), page 6434, 2004.
[26] Ryan Hoque, Lawrence Yunliang Chen, Satvik Sharma, Karthik Dharmarajan, Brijen Thanan- jeyan, Pieter Abbeel, and Ken Goldberg. Fleet-dagger: Interactive robot fleet learning with scalable human supervision. In Proceedings of the Conference on Robot Learning (CoRL), pages 368-380. PMLR, 2023.
[27] Junling Hu, Michael P Wellman, et al. Multiagent reinforcement learning: theoretical frame- work and an algorithm. Proceedings of the International Conference on Machine Learning (ICML), volume 98, pages 242-250, 1998.
[28] Wonseok Jeon, Paul Barde, Derek Nowrouzezahraai, and Joelle Pineau. Scalable multi-agent inverse reinforcement learning via actor-attentionn-rrtii. arXiv preprint 10555 2020.
[29] Sham Machandranath Kakade. On the sample complexity of reinforcement learning. University of London, University College London (United Kingdom), 2003.
[30] Diksha Khurana, Aditya Koli, Kiran Khatter, and Sukhdev Singh. Natural language processing: State of the art, current trends and challenges. Multimedia Tools and Applications, pages 1-32, 2022.
[31] Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning. Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[32] Sachin G Konan, Esmaeil Seraj, and Matthew Gombolay. Iterated reasoning with mutual infor- mation in cooperative and byzantine decentralized teaming. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[33] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E, 69661163,,, 2004.
[34] Hoang Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 1995-2003. PMIR, 2017.
[35] Zeming Lin, Jonas Gehring, Vasil Khalidov, and Gabriel Synnaeve. Stardata: A starcraft research dataset. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AAAI), volume 13, pages 50-56, 2017.
12

---

[36] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning, In Machine Learning Proceedings, pages 157-163. Elsevier, 1994.
[37] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compeiiix environments. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
[38] Jean MacMillan, Elliot E Entin, and Daniel Serfaty. Communication overhead: The hidden cost of team cognition. 2004.
[39] Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. Proceedings of the Conference on Robot Learning (CoRL), pages 879-893 PMLR, 2018.
[40] Negar Mehr, Mingyu Wang, Maulik Bhatt, and Mac Schwager. Maximum-entropy multi-agent dynamic games: Forward and inverse solutions. IEEE Transactions on Robotics, 2023.
[41] Douglas De Rizzo Meneghetti and Reinaldo Augusto da Costa Bianchi. Towards hetero- geneous multi-agent reinforcement learning with graph neural networks, arXiv preprint arXiv:2009. 131.911166611 2020.
[42] Manisha Natarajan, Esmaeil Seraj. Batuhan Altundas, Rohan Paleja, Sean Ye, Letian Chen, Reed Jensen, Kimberlee Chestnut Chang, and Matthew Gombolay. Human-roba teaming: Grand challenges. Current Robotics Reports, pages 1-20, 2023.
[43] Sriraam Natarajan, Gautam Kunapuli, Kshitij Judah, Prasad Tadepalli, Kristian Kersting, and Jude Shavlik. Multi-agent inverse reinforcement learning. Proceedings of the Ninth International Conference Machine Learning and Applications (ICMLA), pages 395-400. IEEE, 2010.
[44] Rohan Paleja, Andrew Silva, Letian Chen, and Matthew Gombolay. Interpretable and person- alized apprenticeship scheduling: Learning interpretable scheduling policies from heteroge- neous user demonstrations. In Proceedings of the Advances Neural Information Processing Systems (NeurIPS), 33:6417-6428, 2020.
[45] Georgios Th Papadopoulos. Asterios Leonidis. Margherita Antona, and Constantine Stephani- dis. User profile-driven large-scale multi-agent learning from demonstration in federated human-robot collaborative environments. In Proceedings of the Human-Commpn Interaction. Technological Innovation: Thematic Area (HCI), pages 548-563. Springer, 2022.
[46] Peixi Peng, Junliang Xing, and Lili Cao. Hybrid learning for multi-agent cooperation with sub-optimal demonstrations. In In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3037-3043, 2021.
[47] Lindsay Sanneman and Julie Shah. Transparent value alignment. In Proceedings of the ACM/IEEE International Conference on Human-Roba Robot Interaction (HRI), pages 557-560, 2023.
[48] Stefan Schaal. Learning from demonstration. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 9, 1996.
[49] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint T.XXv:1707706333 2017.
[50] Esmaeil Seraj. Embodied team intelligence in multi-robot systems. AAMAS 2022 Doctoral Consortium, 2022.
[51] Esmaeil Seraj. Embodied, intelligent communication for multi-agent cooperation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 37, pages 10135-161366 2023.
[52] Esmaeil Seraj. Letian Chen, and Matthew C Gombolay. A hierarchical coordination framework for joint perception-action tasks in composite robot teams. IEEE Transactions on Roboties, 2021.
13

---

[53] Esmaeil Seraj. Andrew Silva, and Matthew Gombolay. Multi-uav planning for cooperative wildfire coverage and tracking with quality-of- se-vvcee guarantees. Journal of Autonomous Agents and Multi-Agent Systems, 36(2):39, 2022.
[54] Esmaeil Seraj, Zheyuan Wang. Rohan Paleja. Daniel Martin, Matthew Sklar, Anirudh Patel, and Matthew Gombolay. Learning efficient diverse communication for cooperative heterogeneous teaming. Proceedings of the st International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 1173 1 1182, 2022.
[55] Esmaeil Seraj, Zheyuan Wang, Rohan Paleja, Matthew Sklar, Anirudh Patel, and Matthew Gombolay. Heterogeneous graph attention networks for learning diverse communication. arXiv preprint (..Xxx:2108.09550 2021.
[56] Esmaeil Seraj. Xiyang Wu, and Matthew Gombolay. Firecommander: An interactive, probabilis- tic multi-agent environment for heterogeneous robot teams. arXiv preprint vrXiv:2011.00165 2020.
[57] Andy Shih, Stefano Ermon, and Dorsa Sadigh. Conditional imitation learning for multi-agent games. In Proceedings of the 17th ACM/IEEE International Conference on Human- Robot Interaction (HRI), pages 166-175. IEEE, 2022.
[58] Marco AC Simöes, Robson Marinho da Silva, and Tatiane Nogueira. A dataset schema for cooperative learning from demonstration in multi-robot systems. Journal of Intelligent & Robotic Systems, 99:589-608, 2020.
[59]
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when communicate at scale in multiagent cooperative and competitive tasks, arXiv preprint arXiv: 181.00955. 2018. 88
[60] Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative ad- versarial imitation learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
[61] Adrian Šošic, Wasiur R KhudaBukhsh, Abdelhak M Zoubir, and Heinz Koeppl. Inverse rein- forcement learning in swarm systems. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 1413-1421, 2017.
[62] William Squires and Sean Luke. Scalable heterogeneous multiagent learning from demonstra- tion. In Proceedings of the International Conference on Practical Applications of Agents and Multi-Agent Systems, pages 264-277. 264-ne7.Springer, 2020.
[63] Aaron Steinfeld, Odest Chadwicke Jenkins, and Brian Scassellati. The oz of wizard: simulating the human for interaction research. In Proceedings of the ACM/IEEE International Conference on Human Robot Interaction (HRI), pages 101-108, 2009.
[64] Sriram Ganapathi Subramanian, Matthew E Taylor, Kate Larson, and Mark Crowley. Multi- agent advisor q-leamiig. Journal of Artificial Intelligence Research, 74:1-74, 2022.
[65] Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropa- gation. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 29, 2016.
[66] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint MtXiv:1805.01955 2018
[67] Albert Tung, Josiah Wong. Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Learning multi-arm manipulation through collaborative teleoperation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 9212-9219. IEEE, 2021.
[68] Hongwei Wang. Lantao Yu, Zhangjie Cao, and Stefano Ermon. Multi-agent imitation learning with copulas. In Machine Learning and Knowledge Discovery in Databases, pages 139-156. Springer, 2021.
14

---

[69] Xingyu Wang and Diego Klabjan. Competitive multi- aalti- inverse reinforcement learning with sub-optimal demonstrations. In Proceedings of the International Conference on Machine Learning (ICML), pages 5143-5151. PMLR, 2018.
[70] Fan Yang, Alina Vereshchaka, Changyou Chen, and Wen Dong. Bayesian multi-type mean field multi-agent imitation learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 33:2469-242 2478, 2020.
[71] Lantao Yu, Jiaming Song, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 7194-7201. PMLR, 2019,
15


Reviews:
[{"review_id": "KdOfbfJuwu", "reviewer": "Reviewer_ZoNx", "summary": "This work proposes a Multi-Agent Learning from Demonstration approach for learning multi-agent policies for collaborative tasks. The approach learns from human demonstrations of the joint policy, but does not require demonstrations of inter-agent communication, which is learned during training via online interaction. Results are presented on learning policies from both synthetic and real human data. A human study is performed demonstrating the increase in performance and decrease in workload for the proposed method, compared to methods that require demonstration of inter-agent communication.", "questions": "In addition to the points described in the Weaknesses section, I would appreciate clarification on the following:\n- Could the authors clarify / provide more details on how the communication graph is generated? Do edges in the graph only exist when robots are spatially within close proximity (and if so, why is this a useful property) or do edges exist between every pair of agents?\n- How many trials are shown in Figures 2 and 3?\n- What is the process for hyperparameter tuning for the proposed method and the baselines?\n- What is the intuition for maximizing the MI between the agent's message and the joint observation (including observations that the agent does not have access to) as opposed to the agent's individual observation history?", "limitations": "- The authors note that the method does not currently account for suboptimal demonstrations. If the method is particularly sensitive to how the various losses are weighed, this may be an additional limitation in terms of the amount of tuning required.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper relaxes a limiting assumption made by some prior works that a human expert provides demonstrations for both environment actions and communication actions, which can be taxing for a human to actually provide. In doing so, this work strikes a balance between prior RL and IL approaches for learning the action and communication policies.\n- The paper trains policies on both synthetic and human data. MixTURE is seemingly quite effective on learning from real human data.\n- The paper empirically verifies the motivating claim about the effect of removing communication actions from a human expert's workload, via a human study on 55 users.\n- A new \"mixed discriminator\" is proposed, where the discriminator is given global observations but only local actions. Per the discussion in Appendix 5.2, this choice exhibits improvements on the decentralized and centralized discriminators from MA-GAIL.\n- The work is well-written and clear. Source code is provided for reproducibility.", "weaknesses": "- The training objective has multiple moving parts (GAIL, PPO, BC, and MIM) which could potentially make training difficult; some additional discussion on this and how sensitive the method is to different weights on the losses would be helpful.\n- Contribution 2 is the mutual information maximization-based communication learning model. On the 5x5 PP and PCP environements, the MIM does not have a significant effect. In the FC environment, it moderately increases sample efficiency. Have ablations been run on the larger/more complex environments? This may provide more robust support for the claim that MI maximization is \"helpful for domains with increased task complexity,\" and in general may more strongly justify the inclusion of the MI objective, which otherwise complicates the training procedure.\n- Related to the above, in Section 5.1 the authors conjecture that the MIM is a key point in dealing with diversity in the human data, i.e., that it \"provides the model with the ability to reason about the underlying human demonstrations and cope with the trajectory distribution through automatically finding a suitable communication protocol.\" It would be valuable to discuss what communication protocol is learned in these settings (if it is interpretable) and why it so strongly outperforms MA-GAIL trained on the dataset with demonstrated communication actions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and positive feedback. Please find below a point-by-point response to your comments and questions:\n\n### **Weaknesses**:\n\n●\t**Sensitivity to Different Moving Objective Parts**: The initial two components of the objective – the GAIL loss and the PPO loss – are fundamental to our GAIL-based MA-LfD architecture. The remaining two parts – offline BC loss and MIM loss – can be regarded as dynamic components, both of which are studied in our supplementary results in Sections 5.3 and 5.4. Our experiments showed that: (1) the addition of the offline BC loss always helps improve model performance, and (2) the MIM loss seems more effective in intricate or larger domains with greater agent counts, yet adjusting its weight for substantial gains can be slightly challenging. For other loss-part weights we mostly stayed consistent with prior work and did not observe any significant trends of sensitivity to weight balance. We will provide this discussion in the appendix. \n\n●\t**MIM Ablation Results**: Yes, we ran MIM ablations in larger and more complex domains. As stated above, while we never observed any performance decay, tuning the MIM loss to achieve consistent and significant performance improvement seems to be challenging. As such, more comprehensive experiments are required to make a conclusion, which are being designed for our next step. Nevertheless, we believe there are further benefits that could be achieved via training the MIM reverse model. Particularly, we believe our MIM reverse model has the potential to roughly cluster message embeddings based on observation-action pairs which can provide further useful information and insight into the learned communication protocol for a task. We will amend the claim as suggested and add the further discussion in the camera-ready version.\n\n●\t**Reason for Significantly Outperforming MA-GAIL**: Thanks. That is a great point. The goal of the MIM in the reverse model is to improve the communication policy distribution for an agent by reducing the entropy in message distribution post-observation (i.e., observing a state and receiving a message), thereby enhancing agent certainty in choosing actions. We believe ultimately, the MIM reverse model, when tuned well enough, should be able to roughly cluster message embeddings (i.e., reduced entropy) based on observation-action pairs, which can provide further insight into interpreting the learned communication protocols. Investigating the interpretability of such learned protocols is an interesting avenue of future research that we are considering. We hypothesize that MA-GAIL trained on demonstrated communications fails because of the following reasons: (1) humans are not fully aware of all agents’ full local state spaces and therefore, the demonstrated communication could be severely inefficient and sub-optimal. MixTURE instead learns an efficient communication policy automatically through gradient updates; (2) The laborious task of providing both environment and communication actions for the human demonstrators significantly deteriorates the quality of demonstrated policies, which can be confirmed by our subject-study results on page 9. We will provide a summary of this discussion in Section 5.2 for the camera-ready version.\n\n### **Questions**:\n\n●\t**Communication Graph**: A spatial proximity-based communication graph is useful for large scale applications and domains where robots have limited communication ranges and can only communicate within a certain radius. While we built the MixTURE architecture to be localized to accommodate such proximity-based limited communication, our initial proof-of-concept experiments utilized extended communication ranges to cover the domain. Future extensions will encompass ablation results for shorter communication distances\n\n●\t**Number of Trials in Figure 2 and 3**: Figures 2 and 3 are showing results across ten trials. This is stated in Section 5.2, under RQ1.\n\n●\t**Hyperparameter Tuning Process**: We conducted extensive empirical hyperparameter tuning. As stated in Section 6 of the appendix, Table 4, the values in brackets represent variables which we swept over via grid search. Values were swept over simultaneously to a resolution of 2 steps per decade (i.e., logarithmic scale, where, for instance, [10^-1.5, 10^0] means that the values tested were {10^-1.5, 10^-1.0, 10^-0.5, 10^0.0}. Moreover, each baseline has been separately tuned for all experiments.  \n    \n●\t**Intuition Behind Including Joint Observations in MIM**: By providing the joint observations during centralized training, intuitively we intend to learn the communication policy such that the optimization of the MIM model results in a message distribution entropy-reduction at the team level. In this way, as discussed above, the model can potentially relate observation-action pairs to the globally sent/received messages to learn unified policies at the team level.\n\n### **Limitations**:\n●\tThank you. That is correct. We will add a further statement regarding the challenges of tuning the MIM loss to achieve improved performance as we observed in our experiments.\n\nThanks again for your time and constructive comments and questions. Your suggestions will be duly integrated into the paper's camera-ready version, reflecting our commitment to addressing your concerns comprehensively. We value your time, and we hope that we have addressed all your questions satisfactorily. If so, we would greatly appreciate it if you could please consider increasing your score. Thank you."}, {"review_id": "ReQ0cB3UsC", "reviewer": "Reviewer_N6Tr", "summary": "This paper proposes a learning framework for multi-agent coordination using communication from human expert demonstrations: Mixed-Initiative Multi-Agent Apprenticeship Learning (MixTURE). A human expert teaches a team of robots to collaborate on a task by demonstrating actions for every robot, while the robot team learns a emergent communication protocol that encourages sending information about the joint observation. An attentional communication module and a mutual information maximization reverse model  are used. The framework is evaluated on three domains with different levels of difficulty and complexity. They also conduct a human-subject user study to collect real human data and assess the usability of their framework. ", "questions": "- How does the MixTURE framework scale to larger environments, more complex tasks or more agents? \n- What is the computational cost and communication cost of the framework?\n", "limitations": "Not adequately.\n- need to mention the strong assumptions on the expert's joint observation and action space\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a interesting framework that leverages both human demonstrations and emergent communication to learn effective coordination policies among several robots. \n- A comprehensive evaluation of their method on synthetic and real human data is conducted. ", "weaknesses": "- The proposed framework assumes that the human expert has access to the aggregated / joint observation of all the robots at each time step and the human expert can command every robot's action simultaneously at each time step. However, this assumption is rather unrealistic for real-world robot teams or robot teams in simulation. \n- The synthetic environments that the framework is evaluated on is not large-scale enough compared to common testing environments for multi-agent reinforcement learning like SMACv2 (_SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning. Ellis, Benjamin and Moalla, Skander and Samvelyan, Mikayel and Sun, Mingfei and Mahajan, Anuj and Foerster, Jakob N. and Whiteson, Shimon._)\n- In the experimental results section, no qualitative analysis or visualization of the learned policies and communication protocols are provided. It would useful to observe how robots coordinate and communicate in different settings. \n- An expert heuristic is used to collect the expert demonstration dataset. However, heuristics can struggle to produce optimal demonstrations for more difficult environments. Then algorithms of multi-agent reinforcement learning still need to be used at the first place to collect the expert demonstration dataset. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and positive feedback. Please find below a point-by-point response to your comments and questions:\n\n### **Weaknesses**:\n\n●\t**Assumption Regarding Human Expert’s Access to the Joint Observation of All Robots**: We concur that a human's decision-making for one agent is influenced by knowledge of other agents' observations. However, we respectfully disagree that this assumption is too strong. Please note that, classic MA-LfD methods such as MA-GAIL, necessitate awareness all robots' states and observations AND they require providing both env. and communication actions at the same time. In MixTURE, we in fact relax a portion of this assumption by tasking the human to only provide env. actions for robots one at a time, to improve feasibility. Our real human subject study demonstrates this relaxation's feasibility, enabling successful demonstrations provided by humans for teams of simulated robots with up to 10 robots from 2 diverse classes. In practice, one can use existing multi-agent datasets (e.g., collaborative assembly in ROBOTURK (https://roboturk.stanford.edu) or Meta's STARDATA for StarCraft II) or readily create a demonstration dataset using existing tools (e.g., ROS, Gazebo). MixTURE’s contribution comes in at this stage, where we can learn collaborative multi-agent policies from this dataset.\n\n●\t**Environment Scale**: Respectfully, we disagree. The domain complexity only minimally impacts our proof-of-concept here. In fact, increasing complexity motivates MixTURE's application even more to reduce human involvement as scale grows. Even in domains like SMACv2, which humans expertly play, we can amass an expert dataset (as done before in AlphaStar) for MixTURE's training. Additionally, we believe the complexity of our tested domains may have not been fully perceived by the reviewer. Not only is the FC domain very challenging even for humans to master (i.e., based on the statistical analysis of our large subject study and making expert observations on 55 humans), but also, they are challenging to learn by SOTA MARL methods [46]. Notably, as reported by authors of [46] the achieved SOTA MARL performance in medium scenario of FC was about 60 steps (MixTURE achieves 34.8 steps), while all prominent MARL baselines tested completely failed to learn in the hard scenario (MixTURE learns and converges to 56.5 steps).\n\n●\t**Qualitative Analysis and Visualization of the Policies**: We appreciate this suggestion. We will include additional policy visualizations in our supplementary results. The interpretation of communication protocols will require further exploration through specialized interpretability-oriented experiments, an avenue we're considering for advancing our work.\n\n●\t**Using MARL for Collecting Expert Demonstration Dataset**: We believe there is a misunderstanding that can be addressed. The fact that heuristics can struggle in more complex domains is in fact one of the reasons we are proposing the MixTURE framework, that is, to relax the need for such heuristics and instead, relay on a human expert’s domain knowledge. Please note that our expert demonstrations were not only produced by heuristics. **We collected two sets of datasets**: (1) a heuristic dataset, and (2) a human dataset. **All methods, including the proposed method and the baselines have been trained on both datasets**, meaning that, regardless of how the dataset was collected (heuristic or real human) our method outperforms the baselines. Finally, MARL can also significantly suffer with increased domain complexity and can be very challenging and time-consuming to train. The efficiency of expert-generated data outpaces creating and tuning a MARL for similar expertise.\n\n### **Questions**:\n\n●\t**How does MixTURE Scale**: We conducted rigorous scalability analysis, covering various domain sizes (dimensions), team sizes (number of robots), team compositions (robots per class), and tasks (number of initial firespots which propagate and grow over time). Overall, our results underscore MixTURE's robust scalability and efficiency in adapting to all cases. Moreover, our human subject study substantiates that MixTURE empowers human experts as well to scale proficiently across larger domains, teams, and tasks, showcasing enhanced policy performance and demonstration efficiency. These results can be found in Section 5 (Table1, Figures 2-5) as well as Section 5, Sub-section 5-1: Scalability (Figure 6), in appendix.\n\n●\t**Computational and Communication Cost of the Framework**: Our computational resources for training MixTURE are listed in Appendix, Section 6. Moreover, the attention-based communication technically makes the time complexity O(N^2) in the number of agents, but practical implementation and optimization involve minimal additional inference layers. Therefore, we believe the overall additional computational cost is small, especially, in high-dimensional environments, policy and value networks bear the majority of the computation load. We will add this discussion to the Section 6 in appendix.\n\n### **Limitations**:\n\n●\tWhile it is technically unavoidable to provide multi-agent demonstrations without having humans informing their decisions based on robots  observations, to the reviewer’s point, we will add this discussion to the paper as suggested. Please note that this also applies to existing MA-LfD work. Additionally, respectfully, we disagree that knowing the robot teams' action-space is a limitation. This is a conventional and reasonable aspect in the LfD domain.\n\nThanks again for your time and constructive comments and questions. Your suggestions will be duly integrated into the paper's camera-ready version, reflecting our commitment to addressing your concerns comprehensively. We value your time, and we hope that we have addressed all your questions satisfactorily. If so, we would greatly appreciate it if you could please consider increasing your score. Thank you."}, {"review_id": "3Bz5JkqmA0", "reviewer": "Reviewer_fNhW", "summary": "This paper proposes a Mixed-Initiative Multi-Agent Apprenticeship Learning (MixTURE) framework to teach a team of agents using demonstrations provided by individual human experts. It learns both a cooperative policy for each agent and the inter-agent communication policy for each agent. The proposed MixTURE, including the framework and the communication learning model, is the main contribution of the paper. Human subject studies on the proposed framework can also be viewed as a contribution.", "questions": "- Both terms robots and agents are used in the paper. What are their differences?\n- Just curious, where does the name MixTURE (especially the TRUE) come from? \n", "limitations": "No negative societal impact of the work is perceived. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Evaluating the framework with statistical analysis/comparison and human subject studies is a strength.\n+ Enabling multi-robot learning from demonstrations by an individual expert has the potential of improving multi-robot learning efficiency and human knowledge transfer to robot teams.\n", "weaknesses": "-  The problem of learning communications is not well justified or formulated. For example, what are assumptions on the protocol and constraints for the multi-robot communication (broadcasting or message passing)? As an example, connected autonomous vehicles use a broadcasting mechanism and follow a standard data format for communication. Why to use a differential communication channel? Are there examples of multi-robot systems using a differential channel for communication?\n\n- If communication is learned based on a mutual information metric in the proposed framework, how does learning communication benefit from human demonstrations? Learning communication seems to be a separate problem from LfD in this work.\n\n- The communication learning module needs the global joint action-observation to compute each agent’s outcome message. How is the joint action-observation information obtained during decentralized execution?\n\n- How the approach can be generalized to realistic multirobot settings and environments? In the experiments, agents have discrete states and actions, and the environment is a grid world. These  experiment settings do not well support the argument of human training of “robot” teams.  \n\n- The proposed framework follows a centralized training decentralized execution paradigm, but does not review related work on this paradigm or discuss its pros and cons. Especially, how important is multi-robot communication in centralized training decentralized execution methods?\n\n- Existing works on learning communications (e.g., when, what, who and how to communicate between multiple agents) are not reviewed. \n\n- As an opinion, advantages of using an individual human expert to provide demonstrations for a team of agents are not convincing, especially for in decentralized multi-robot systems. The general goal of LfD is to transfer human knowledge to robots/agents. Humans behave differently when working alone or with other teammates. In addition, when a single human expert controls a team of agents to perform a task, the expert still provides a sequence of decisions for different agents. However, decentralized teaming involves concurrent decisions, may have conflicts, and has tightly-coupled teaming activities (e.g., two robots carrying an object). All the above properties of teaming cannot be demonstrated by a single expert.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and positive feedback. Please find below a point-by-point response to your comments and questions:\n\n### **Weaknesses**:\n\n●\t**Justifying the Learned Communication**: Please note that MixTURE operates under a CTDE paradigm: the differentiable communication channels are only used during training to learn a communication policy via gradient updates. Post-training, there is no more need for the “differentiable” channels in real systems. For instance, robots in a real system can use common wireless channels to send/receive messages generated by the learned communication policy. Also, please see the following quotes from the paper regarding the justification of the problem. The necessity of communication is described in paragraph 3, lines 42-44. The need to hand-engineer a communication protocol for a team of robots for each and every domain leaves much to be desired, leading researchers to make efforts to learn such communication protocols end-to-end.  In MA-LfD, humans can't feasibly provide both environment and communication action demos (paragraph 4, lines 48-61). Additionally, we motivated and described the communication problem in Section 4, lines 153 – 183 as well as on page 5, starting from line 214, where we also mentioned that **each agent will broadcast a message to the rest of the team at each time step** (line 215). Further details regarding the domains are also provided in the appendix, sections 2 and 3.\n\n●\t**How Does Learning Communication Benefit from Human Demonstrations**: We believe there might be a misunderstanding. Respectfully, we disagree that learning communication vs. learning from demonstration (LfD) are separate problems in our formulation. The communication policy is learned automatically and simultaneously while the robot team is being trained via LfD on human demonstrations for environment actions. The learned communication is automatically adjusted to fit the provided human demonstrations through gradient updates in LfD optimization. The structure of our discriminator (global observations with local actions) and the MIM objective are specifically designed to address the problem space of both LfD and the learned communications.\n\n●\t **Joint Action-Observation Information During Execution**: Please note that, as stated on page 6, lines 240-244, access to joint action-observation info in MIM reverse model is for training, not execution. Learned policies are fully decentralized during execution, removing the MIM model. In training access to global info is possible due to the CTDE paradigm.\n\n●\t**Generalization to Real Robots**: Generalization to continuous spaces requires a very slight modification to our PPO optimization (i.e., outputting a vector for the mean and standard deviation for a multi-dimensional action distribution, instead of discrete actions). This feature is already supported in our codebase for our future experiments. Discrete domains were chosen for our experiments here for consistency with prior work on this topic (referenced in paper). The MIM reverse model can remain unchanged in continuous spaces. While it is an interesting next-step to extend MixTURE’s case-studies, doing so is mostly a matter of engineering rather than a proof-of-concept. In this work, for the first time, we are building a foundation to learn multi-robot policies directly from human generated trajectories and we provide extensive set of results to demonstrate the feasibility on synthetic and real human data.\n\n●\t**Importance of Communication in CTDE**: Communication is vital in CTDE methods during centralized training for optimizing collective behavior. During this phase, robots share experiences and data with the central controller, enabling the learning of complex team-level strategies, supporting coordination, and enabling adaptation to dynamic environments. Once the training is complete, the robots execute the learned strategy in a fully decentralized manner during the execution phase while benefiting from the knowledge acquired during centralized training. We will add this discussion in the paper as well as an overview of recent literature on CTDE methods.\n\n●\t**Review on Prior Work for Learned Communication**: Thanks. Due to space limitations, we focused on providing the literature on prior MA-LfD work as they relate to the core of our algorithm. We will include a review of prior work on MARL and learned communication in the appendix for the final version.\n\n●\t**Single vs Multiple Experts**: Using multiple humans requires further communication and coordination among humans which can be challenging to translate to robot domains. Additionally, deciding the optimal number of human experts needed for a task raises further complications, while accessing multiple human experts can be expensive and time consuming. A single human expert can generate action demonstrations for each robot one at a time (as in our setup) which resolves the mentioned tightly coupled scenarios.\n\n### **Questions**:\n\n●\t**Terms Robots and Agents**: Thanks. We use the terms robots and agents interchangeably to generalize MixTURE’s applicability. By ‘agent’ we are mainly referring to a ‘robot agent’. We will add this clarification to the paper.\n\n●\t**The Name MixTURE**: We created the name MixTURE by taking letters from the title as follows: **Mixed**-ini**T**iative m**U**lti-agent app**R**enticeship l**E**arning. Rather than putting together the first letters, we tried to create an abbreviation that fits the core ideology of our algorithm.\n\nThanks again for your time and constructive comments and questions. Your suggestions will be duly integrated into the paper's camera-ready version, reflecting our commitment to addressing your concerns comprehensively. We value your time, and we hope that we have addressed all your questions satisfactorily. If so, we would greatly appreciate it if you could please consider increasing your score. Thank you.\n"}, {"review_id": "WP7AxltfJR", "reviewer": "Reviewer_BRdM", "summary": "The paper introduces a model, called MixTURE, for learning multi-agent collaborative policies from human demonstrations, addressing the challenges of coordinating heterogeneous agents in complex tasks. MixTURE leverages a mutual information maximization-based differentiable communication module to reason about and adapt to diverse human demonstrations. The authors evaluate MixTURE through synthetic and human-subject experiments in the FireCommander (FC) domain. In the synthetic evaluations, MixTURE outperforms baselines, achieving significant improvements across difficulty levels and setting a new state-of-the-art in learning collaborative policies for complex tasks.\nIn the human-subject experiments, the proposed approach demonstrates its ability to learn high-quality collaboration policies from diverse human-generated demonstrations, surpassing frameworks that solely rely on demonstrated communication. The model's ability to reason about human demonstrations and adapt to trajectory distributions is highlighted as a key factor in its success. The statistical analysis supports the claims made by the authors. It shows that relaxing the need for demonstrating communication reduces the workload for human experts and improves system usability. Furthermore, the results indicate that MixTURE achieves higher performance scores, better scalability to complex scenarios, and lower demonstration time per step compared to frameworks with demonstrated communication.\nOverall, the results support the effectiveness of MixTURE in learning collaborative policies, its ability to leverage human demonstrations, and its potential to enhance human-robot collaboration in complex multi-agent systems.\n", "questions": "Can you provide more insights into the generalization capabilities of the MixTURE model? How would it perform in unseen environments or more complex tasks? Or when it’s trained on a lower-difficulty level but tested on a harder one?\n\nCould you discuss how / whether sub-optimality of human demonstrations might be detected? \n", "limitations": "Are there specific scenarios or conditions where the MixTURE model might struggle or fail to learn effective communication strategies?\n\nAs stated previously, a discussion on whether using MixTURE for continuous (state/action) problems is not trivial would be useful.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality:\nThe paper introduces a novel approach by explicitly addressing the challenge of inter-agent communication in dynamic and partially observable domains.\nThe use of a differentiable communication module based on Mutual Information (MI) is a creative and original contribution.\nThe paper's focus on learning communication strategies from human demonstrations and addressing heterogeneity and diversity in human data sets it apart from previous methods.\nThe elimination of the need for an expert to demonstrate a communication strategy and allowing robots to automatically learn communication protocols is a useful idea.\n\n- Quality:\nThe technical details and algorithms are thoroughly explained, ensuring reproducibility and soundness of the proposed approach.\nThe paper demonstrates a fair experimental methodology, conducting evaluations in both synthetic and human-subject environments.\nThe inclusion of ablation studies and statistical analysis enhances the evaluation and supports the provided claims.\n\n- Clarity:\nThe paper is well-structured and organized, with clear sections that guide the reader.\nThe authors provide comprehensive explanations of the methodology, algorithms, and experimental setup.\nThe use of figures, tables, and visualizations aids in understanding the concepts and results.\nThe paper uses appropriate terminology and notation, making it accessible to readers familiar with the field.\n\n- Significance:\nThe paper addresses a significant research gap by explicitly considering the challenge of inter-agent communication in collaborative multi-agent problems.\nThe proposed approach has practical implications for teaching multi-agent coordination, as it eliminates the need for explicit communication demonstrations from experts.\nThe use of real human-generated data demonstrates the model's ability to cope with variations in demonstration styles and strategies, increasing its applicability in real-world scenarios.\nThe experimental results support the claims and show the effectiveness of the proposed MixTURE model, contributing to the advancement of Multi-Agent Learning from Demonstrations (MA-LfD).\n", "weaknesses": "A discussion on how MixTURE could be used for continuous (state/action) problems would be helpful.\n\nIt’s also unclear whether the proposed approach would be generalizable to other domains.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and positive feedback. Please find below a point-by-point response to your comments and questions:\n\n### **Weaknesses**:\n\n●\t**Discussion on MixTURE’s Applicability to Continuous Domains**: Generalization to continuous state and action spaces will require a very slight modification to our PPO optimization to be extended to the continuous settings (i.e., outputting a vector for the mean and standard deviation for a multi-dimensional action distribution, instead of discrete actions). This feature is already supported in our codebase for our future experiments. We chose the discrete domains for our experiments here to be consistent with prior work on this topic (referenced in paper). The MIM reverse model can remain unchanged in continuous spaces for reconstructing an agent’s output message.\n\n●\t**Generalization to Other Domains**: We investigated MixTURE’s generalizability in our problem formulation by considering a generic form of Markov Games to include both partial observability and agent heterogeneity. This will ensure MixTURE’s applicability to a wide range of domains and real-world scenarios. In future extensions of this work, we intend to include more environments (such as StarCraft II via Meta's STARDATA public dataset) as well as investigating applicability to collaborative team of robotic arms using existing datasets, such as the ROBOTURK (https://roboturk.stanford.edu).\n\n### **Questions**:\n\n●\t**Insights into Generalizability of MixTURE**: These are interesting questions that require further ablation studies and investigations. Theoretically, the learned multi-agent policies are not expected to directly perform well in unseen environments or harder domains when trained on lower-difficulty levels. Instead, the learned policies from a lower-level difficulty domain can perhaps provide a warm-start for the robot team to further interact with the environment and improve their policies in a new, unseen domain or a harder difficulty level (e.g., via RL, assuming there are no human data available for the unseen domain or domain with harder level of difficulty).\n\n●\t**Detecting Human Sub-optimality**: Typically, it is challenging to determine human sub-optimality without ground truth optimal solution. However, to evaluate the quality of demonstrations one can collect user feedback, perform trajectory analysis, or get expert review. To handle human sub-optimality however, these steps may not always be required. For instance, one can integrate existing methods of learning from sub-optimal demonstrations, such as [1].\n\n[1] Zhu, Zhuangdi, et al. \"Learning sparse rewarded tasks from sub-optimal demonstrations.\" arXiv preprint arXiv:2004.00530 (2020).\n\n### **Limitations**:\n\n●\t**Scenarios where MixTURE Might Fail**: Generally, MixTURE will not be applicable if there isn’t a good way to collect human expert or heuristic demonstrations for a multi-agent scenario (also applies to other MA-LfD prior work). We believe there are not many such cases as in most applications, either direct human demonstrations can be collected, or an expert heuristic can be designed to generate demonstrations. Additionally, we have not tested MixTURE in mixed collaborative-competitive domains. Nevertheless, we are optimistic that MixTURE can still learn a good policy in a mixed domain, as long as high-quality demonstrations are available for the cooperative team against the competition.\n\n●\t**MixTURE in Continuous Domains**: Thanks. As discussed above, generalization to continuous state and action spaces will require a very slight modification to our PPO optimization to be extended to the continuous settings while the MIM reverse model can remain unchanged. This feature is already supported in our codebase for our future experiments.\n\nThanks again for your time and constructive comments and questions. Your suggestions will be duly integrated into the paper's camera-ready version, reflecting our commitment to addressing your concerns comprehensively. We value your time, and we hope that we have addressed all your questions satisfactorily. If so, we would greatly appreciate it if you could please consider increasing your score. Thank you.\n"}, {"review_id": "hnP92H937y", "reviewer": "Reviewer_dfhv", "summary": "The paper introduces a novel learning approach designed for robot teams to acquire a preferred policy for collaborative task completion using human expert-generated data. Additionally, the method enables the robots to gain an understanding of the theory of mind of each individual agent within the team.\n\n\n\n\n", "questions": "My questions are listed in the weakness, i hope the authors can address them.", "limitations": "Yes, the authors have addressed all the limitations.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper presents a learning framework that enables robots to acquire teaming strategies through human expert demonstrations, while simultaneously learning the theory of mind between agents. The effectiveness of the proposed approach is demonstrated on synthetic and real data collected from humans. The method outperforms other baselines in accomplishing the defined task.", "weaknesses": "\n1.I agree that incorporating learning communication policies is crucial for both multi-agent learning from demonstrations (MA-LfD) and multi-agent reinforcement learning (MARL). However, the tasks presented in the paper may be considered relatively simple, which might not fully showcase the author's capabilities. It would be beneficial if the author could demonstrate the proposed methods in more complex 3D environments such as ITHOR or Habitat, or even leverage visual-based theory of mind datasets like \"Learning triadic belief dynamics in nonverbal communication from videos\" or \"BOSS: A Benchmark for Human Belief Prediction in Object-context Scenarios.\"\n2.I'm curious if the author has conducted a performance comparison between a single human expert and multiple experts (N) for each agent. Benchmarking the differences in performance could provide valuable insights into the impact of expert diversity on the proposed approach.\n3.Could the author elaborate on the rationale behind selecting GRU (Gated Recurrent Unit) over a transformer for the proposed MixTURE architecture? Understanding the specific reasons behind this choice would enhance the clarity and comprehensiveness of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and positive feedback. Please find below a point-by-point response to your comments and questions:\n\n### **Weaknesses**:\n\n●\t**Complexity of Tasks**: First, we believe the complexity of our tested domains, particularly in hard scenarios of the PCP and FC (i.e., 20x20 domain, 10 agents including 2 heterogeneous classes of robots, and several propagating fires) may have not been fully perceived by the reviewer. Not only are these domains very challenging even for humans to master (i.e., based on the statistical analysis of our large subject study and making expert observations on 55 humans), but also, they are challenging to learn by SOTA MARL methods. Please note that, several prior works on MARL, including [46], have attempted to learn the hard scenario of FC and failed or learned a very sub-optimal policy. For instance, authors of [46] in a later work stated that the SOTA performance they achieved in medium (i.e., 10x10) scenario of FC was about 60 steps (MixTURE achieves 34.8 steps), while all popular MARL baselines completely failed to learn in hard (i.e., 20x20) scenario (MixTURE learns and converges to 56.5 steps). Second, we believe the complexity of the domain has little effect on our proof-of-concept here. As mentioned by the reviewer, while fully showcasing MixTURE’s capability to learn from expert demonstrations in other complex domains can be interesting, we note that even in the domains that we tested here, none of the baselines can directly learn from human generated data while MixTURE can learn high-quality multi-agent policies. In this work, for the first time, we are building a foundation to learn multi-robot policies directly from human generated trajectories and we provide extensive set of results to demonstrate the feasibility on synthetic and real human data.\n\n●\t**Human Data Diversity (Multiple Humans) for Each Robot**: We note that, including multiple human demonstrators for each data collection task is a challenging setup and perhaps a separate research question to be extensively studied in a future step. Using multiple humans require further communication and coordination among humans which can be challenging to translate to robot domains. Additionally, deciding the optimal number of human experts needed for a task raises further complications and challenges, while accessing multiple human experts can be expensive and time consuming.\n\n●\t**Choosing GRU vs. Transformer**: There are several reasons for choosing GRUs over Transformers. Please note that our policies are recurrent in time, i.e., non-Markovian to handle partial observability in the environment. We reiterate that the GRU integrates information across time rather than over agents. A vanilla transformer-based architecture to handle this would require either storing data about all states seen throughout an episode or have a hard limit on how many steps back an agent can remember. The computational complexity would also become quadratic in the number of timesteps. Finally, GRUs are simpler and more light-weight in terms of memory and computation. We note that, eventually, the goal is to deploy and execute such learned policies in real-world and on physical robots with limited computation and memory resources. Therefore, it is important to design light-weight models that could be more practical on real robotic systems.\n\nThanks again for your time and constructive comments and questions. Your suggestions will be duly integrated into the paper's camera-ready version, reflecting our commitment to addressing your concerns comprehensively. We value your time, and we hope that we have addressed all your questions satisfactorily. If so, we would greatly appreciate it if you could please consider increasing your scores. Thank you.\n"}]

Author rebuttal:

Meta-review: This paper proposes MixTURE, which enables a team of robots to learn simultaneously a policy from human demonstration, and emergent inter-agent communications. All the reviewers agree that the idea is original, and the presentation is clear. Some concerns were raised in the reviews, including strong assumptions, simple experiments, and needing more ablations. Most of these concerns were sufficiently addressed in the rebuttal. After discussion, most of the reviewers voted for acceptance. Please incorporate reviewers' suggestions, and the discussions in the rebuttal into the final version of this paper.
Decision: Accept (poster)