SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In practice, we build a distributed reverse model in the MixTURE architecture (shown in Fig. 1) to accommodate the mentioned MSE loss. The distributed reverse model for each agent has access to the local received messages as well as the global joint-actions taken by all agents such that the optimization results in a communication entropy-reduction mechanism at the team level. Since actions and generated messages are functions of the state-observations, each reverse model,  $\Theta$ , can take in the joint actions,  $\bar{a}$ , and all received messages,  $z_{ji}$ , to estimate the outgoing message for agent,  $i$ , as  $\hat{z}_{ij}^{t} = \Theta_{i}(\bar{a}, z_{ji}^{t})$ . The policy and reverse models are trained together in an end-to-end fashion to minimize the message reconstruction error as  $\mathcal{L}_{\mathrm{MIM}^{\mathrm{c}}i} = \frac{1}{N} \sum_{i=1}^{N} \|\Theta_{i}(\bar{a}, z_{ji}) - z_{ij}\|^2$ .
We note that while the addition of the MI maximization loss would make the overall loss function more complex, we believe there are further benefits that could be achieved via training the MIM reverse model, besides enhancing agent certainty in choosing actions. Particularly, we believe our MIM reverse model, when tuned well, has the potential to roughly cluster message embeddings based on observation-action pairs which can provide further useful information on the learned communication protocol for a task and further insight into interpreting the learned communication protocols.
Training and Execution - We build the MixTURE framework in a Centralized Training for Decentralized Execution (CTDE) paradigm [19] to accommodate for the global joint-action inputs to the MI maximizing reverse models during training. We note that, the MIM reverse model is only used during training and is cut during the execution, and therefore, the learned policies can be executed fully decentralized. To optimize the policies based on the reward signal generated by the discriminators, we leverage the PPO algorithm [49]. Moreover, to enhance and stabilize the training we propose combining an offline BC loss,  $\mathcal{L}_{\mathrm{BC}^c i} = -\frac{1}{N}\sum_{i=1}^{N}\pi_i(a_t^{c_i}|o_t^{c_i},z_{ji}^t)$ , with the online GAIL loss. As shown by prior work [18], through this combination, the offline BC helps preserving ground knowledge that should be respected during training, while the online part helps with learning of new information encountered during execution. As such, putting together our distributed GAIL loss in Eq. 1, the standard clipped PPO loss [49], the offline BC loss, and the MIM loss introduced above, we present the full loss expression to train the MixTURE architecture as in Eq. 2, where  $\zeta_{\pi}$  and  $\zeta_{DE}$  are minibatches of trajectory segments belonging to current policy,  $\pi$ , and demonstration dataset,  $D_E$ , respectively,  $\mathcal{N}$  is the total number of agents and  $\lambda$  is a tunable scaling parameter.
$$
\mathcal {L} _ {\text {t o t a l}} \left(\zeta_ {\pi}, \zeta_ {D _ {E}}\right) = \sum_ {i = 1} ^ {\mathcal {N}} \left(\mathcal {L} _ {\mathcal {D} _ {\theta} ^ {(c _ {i})}} \left(\zeta_ {\pi}, \zeta_ {D _ {E}}\right) + \mathcal {L} _ {\mathrm {P P O} ^ {(c _ {i})}} \left(\zeta_ {\pi}\right)\right) + \lambda_ {\mathrm {B C}} \mathcal {L} _ {\mathrm {B C} ^ {(c _ {i})}} \left(\zeta_ {D _ {E}}\right) + \lambda_ {\mathrm {M I M}} \mathcal {L} _ {\mathrm {M I M} ^ {(c _ {i})}} \left(\zeta_ {\pi}\right) \tag {2}
$$
We break the problem of evaluating our proposed architecture for teaching multi-agent coordination policies to (heterogeneous) robot teams into three research questions (RQ):
RQ1 Can MixTURE learn useful multi-agent coordination strategies from synthetic data (e.g, models of human experts / Oz-of-Wizard [63])?
RQ2 Can MixTURE learn from diverse data generated by real human experts?
RQ3 How challenging is it for a human expert to provide multi-agent demonstrations and does MixTURE alleviate these challenges (comparing workload and system usability)?
**Environments - In keeping with prior work in MARL and MA-LfD [60, 54], we selected three multiagent domains that are partially observable and require collaboration among heterogeneous agents (see Section 3). Please refer to the supplementary material for more details about the environments.**
Figure 2: Evaluation results for MixTURE and the baselines in the medium case on synthetic dataset. MixTURE outperforms all baselines in both sample complexity and performance at convergence.
1. Predator-Prey (PP) [59]: the goal in this homogeneous (i.e., same class agents) domain is for  $\mathcal{N}$  predator agents with limited vision to find a stationary prey and move to its location.
2. Predator-Capture-Prey (PCP) [54]: a new class of capture-agents, are introduced to the PP. In this heterogeneous domain, the goal of predator agents is the same, while capture agents must move to the prey location and capture it, without having any observation inputs.
3. FireComander (FC) [54, 56]: two classes of robots, perception and action robots, are required to collaborate to extinguish propagating firespots. Similar to the PCP domain, perception robots search the domain to find hidden firespots, and action robots with no observation must rely on communication to know where to put out the firespots using an extra action when on a fire. Unlike the PCP, in this complex domain firespots randomly spread over time and thus, the team must continue until all firespots are found and extinguished.
Baselines - To investigate our RQ1, we employ a variety of baselines, described below, all of which utilize the combined offline BC and online loss training scheme. To collect the synthetic dataset, the expert heuristic (Appendix 4) for environment-actions is a near-optimal search algorithm and for the communication, it is an anticipatory observation sharing mechanism inspired by prior work [11, 10].
- MARL [49]: MA-PPO optimizing both environment-action and communication policies.
- BC+DC [66]: MixTURE ablation trained only via offline BC loss with diff. communication.
- NC MA-GAIL [60]: Non-communicative ablation of the MA-GAIL [60] framework.
- MA-GAIL [60]: Full MA-GAIL trained on full dataset w/ demonstrated communication.
5.1 Human-Subject Experiment: Conditions and Procedure
To investigate our RQ2 and RQ3, we conducted an IRB-approved human-subject user study in the FireCommander domain. For more experiment details please refer to the supplementary material.
Domain, Setup, and Procedure - Our experiments were conducted in the context of a simulated multi-robot task, leveraging the FireCommander (FC) domain under different difficulty levels and modes. Depending on the level of difficulty, there can be multiple initial firespots, hidden from the human, that propagate randomly based on a fixed wall-clock rate. The human expert was responsible to strategically move the simulated robots to find and extinguish all firespots as fast as possible. The human subject was shown a performance score at the end of each round, computed based on existing, found, and extinguished fires. After briefing, each subject began by filling in a pre-questionnaire form (demographic and prior videogame experience) followed by reading through a series of detailed game instructions. To minimize the learning effect, subjects were allowed to practice all conditions until they felt comfortable. Next, each subject played six different rounds of the game (i.e., two conditions and three difficulty levels) in a randomly selected order. The demonstration data was fully stored to be later used for training. Finally, each subject was asked to fill some post-measurement forms.
Participants - We recruited 55 participants (mean age  $= 25.0 \pm 2.67$ ;  $34.5\%$  female). All participants were recruited through on-university-campus advertisement and were trained equally for the task.
Independent Variables and Conditions - In this study, we seek to determine if MixTURE can learn multi-agent collaborative policies from diverse human generated data without demonstrated communication and we compare its performance against the MA-GAIL [60] with expert demonstrated
Number of steps taken to win the game (lower is better).
Heu.	Diff.	Predator-Prey	Predator-Capture-Prey	FireCommander
easy	medium	hard	easy	medium	hard	easy	medium	hard
MARL	✓	-	15.57	17.93	47.40	23.38	56.04	79.76	78.77	79.09	80.00
BC+DC	-	✓	11.84	21.28	46.16	15.38	61.54	79.75	38.95	44.70	72.87
NC MA-GAIL	-	-	17.48	60.38	79.52	27.60	77.77	80.00	76.49	79.47	79.98
MA-GAIL	✓	-	11.68	20.39	49.08	16.43	44.84	79.86	77.46	79.47	80.00
MixTURE	-	✓	11.16	13.15	28.73	13.08	17.27	36.41	22.31	34.82	56.49
Table 1: Full evaluation results for all methods and all difficulty settings on synthetic dataset. Heu. and Diff. indicate a models access to heuristic or differentiable communication, respectively.
Figure 3: Evaluation results for MixTURE and MA-GAIL [60] on real human data. From left to right, the figures present test results for easy, medium, and hard scenarios with one initial firespot.
communication. We examine the performance across three levels of difficulty, i.e., easy, medium, and hard, which represent game complexity. As such, we utilize a  $2 \times 3$  within-subjects experiment design varying across two abstractions: (1) only demonstrating environment actions for each robot at each time step (i.e., noComm condition), (2) demonstrating both environment actions and communication actions for each agent at each time step ((i.e., withComm condition)).
Metrics - To evaluate our RQ2 and RQ3, the demonstration data collected from the human subjects for both noComm and withComm conditions are used to train MixTURE and the benchmark MA-GAIL [60], respectively. The algorithms are compared in terms of the learned policy performance (i.e., number of steps taken to win the game, where lower is better). Additionally, to address our RQ3, we leverage the NASA-TLX Workload Survey [24] and the System Usability Scale (SUS) [9].
5.2 Results and Discussion
RQ1: Evaluation on Synthetic Dataset - We perform our evaluations across three different difficulty levels: (1) easy ( $5 \times 5$  domain, 3 robots), (2) medium ( $10 \times 10$  domain, 6 robots), and (3) hard ( $20 \times 20$  domain, 10 robots). Table 1 presents the full evaluation results for collaborative policies learned by MixTURE and the baselines in terms of number of steps taken to win the game (lower is better). The test results show the mean performance values calculated over ten trials of running the best training models. As shown, MixTURE achieves significant improvement over all baselines and in all domains. Additionally, the learning curves in Fig. 2 (medium case), show significant improvement in sample complexity. We believe our model provides a strong step towards learning collaborative policies in multi-robot systems by setting a new SOTA in complex heterogeneous tasks.
We also performed several ablation studies, such as investigating scalability and effects of the MIM reverse model for message reconstruction, discriminator architecture, and the combined offline BC loss. Due to space constraints, all ablation results are available in the supplementary material.
RQ2: Training and Evaluation Results on Human-Subject Dataset – We train MixTURE and MA-GAIL [60] (and other baselines) on data collected in our human-subject user experiment to investigate our RQ2. The results are presented in Fig. 3. Please see the Appendix for environment
details. As shown, MixTURE can learn high-quality multi-robot collaboration policies from diverse human-generated demonstrations. This is while purely relying on human demonstrations [60], does not succeed in this task. We hypothesize that a key point underlying MixTURE's success despite diversity and heterogeneity in human data is that the MIM-based differentiable communication channels provide the model with ability to reason about the underlying human demonstrations and cope with trajectory distribution through automatically finding a suitable communication protocol. This result also shows that a demonstrated communication policy by a human expert is not enough to efficiently coordinate the robot team in complex domains as the human is unaware of all agents' local states, which leads to the failure of existing MA-LfD frameworks such as the MA-GAIL [60]. We hypothesize that MA-GAIL trained on demonstrated communications fails because: (1) humans are not fully aware of all agents' full local state spaces and therefore, the demonstrated communication could be severely inefficient and sub-optimal. MixTURE instead learns an efficient communication policy automatically through gradient updates; (2) The laborious task of providing both environment and communication actions for the human demonstrators significantly deteriorates the quality of demonstrated policies, which can be confirmed by our subject-study results in next section.
We believe that this strong result shows great potential for the MixTURE model to efficiently teach multi-agent coordination and collaboration policies to robot teams through human demonstrations.
RQ3: Statistical Analysis - We investigate our RQ3 by quantifying the workload and SUS measures reported by the human subjects. We hypothesize that:
H1 Demonstrating both an environment-action and a communication-action strategy for the robot team increases the human expert's workload and decreases the system's usability score.
H2 Demonstrating both an environment-action and a communication-action strategy for the robot team negatively affects human performance and the demonstration quality.
H1: We test for normality and homoscedasticity and do not reject the null hypothesis in either case, using Shapiro-Wilk ( $p > 0.32$  and  $p > 0.96$ ) and Levene's ( $p > 0.39$  and  $p > 0.09$ ) tests for workload and SUS, respectively. For workload, we perform a paired t-test and find that using the MixTURE model w/o communication demonstration was rated statistically significantly lower than using MA-GAIL w/ demonstrated communication by the expert ( $p < 0.001$ ) on NASA-TLX workload scale. For system usability, using a similar a paired t-test we find that using the MixTURE model w/o communication demonstration led to a statistically significantly higher SUS than using MA-GAIL w/ demonstrated communication by the expert ( $p < 0.001$ ). As shown in Fig. 4, relaxing the need for demonstrating a communication strategy reduced the humans' workload by  $44.3\%$  and increased the systems' usability scale by  $46.1\%$  in our experiment.
H2: We examine: (1) human's performance score in the game, (2) total tasks completed (i.e., fires extinguished), and (3) average demonstration time per step. We apply a paired samples Wilcoxon test and confirm that (see Fig. 5, from left to right), relaxing the need for demonstrating a communication strategy through MixTURE leads to achieving significantly higher performance score  $(p < 0.001)$  by the human, better ability to scale to more complex scenarios with more tasks  $(p < 0.001)$ , and significantly lower demonstration time per step  $(p < 0.001)$ . Note that, the middle plot in Fig. 5 shows more tasks completed (i.e., more firespots killed) for withComm condition under easy and moderate
Figure 4: Workload and SUS results for H1.
scenarios, which indicates a worse human performance in this conditions since existence of more firespots means inefficiency in extinguishing the fire before it spreads too large. On the other hand, under the noComm condition, humans can easily scale to larger domain sizes and more initial firespots. Note that, lower tasks completed for withComm condition under the hard case is attributed to failing the task and losing the game (i.e., hard game cut-off when score drops below 50).
Further Discussions and Limitations - Demonstrating multi-agent strategies can be considered a highly involved and high-workload task, which in turn can affect a human's situational awareness and
Figure 5: Objective human performance results supporting the H2. In summary, using the MixTURE framework leads to significantly better human performance and a lower demonstration time.
optimality of demonstrations. We posit that a human's decision-making for one agent is influenced by knowledge of other agents' observations. However, we note that, classic MA-LfD methods, such as MA-GAIL, necessitate awareness all robots' states and observations and they require providing both environment and communication actions at the same time. In MixTURE, we relax this assumption by tasking the human to only provide environment actions for robots one at a time. Our real human subject study demonstrates this relaxation's feasibility, enabling successful demonstrations provided by humans for large and heterogeneous teams of simulated robots. In practice, one can use existing multi-agent datasets (e.g., collaborative assembly in ROBOTURK [39, 67] or Meta's STARDATA for StarCraft II [35]) or readily create a demonstration dataset using existing tools (e.g., ROS, Gazebo). MixTURE's contribution comes in at this stage, where we can learn collaborative multi-agent policies from such datasets.
The addition of the MIM loss to Eq. 2 can complicate the tuning process. While we never observed any performance decay as a result of the MIM loss, tuning the MIM loss to achieve consistent and significant performance improvement seems to be challenging. Finally, MixTURE currently does not account for demonstration sub-optimality. An interesting future direction is then to modify the MixTURE architecture to address suboptimal human demonstrations.
Conclusion - We proposed the MixTURE model to learn multi-agent collaborative policies for a robot team, directly from human expert demonstrations. Using our method, a human expert can teach the robot team how to accomplish a task collaboratively via demonstrations and the team will automatically reason over and learn a communication strategy suitable for the underlying demonstrations. The learned communication helps the robot team to deal with the partial observability, reasoning about action-decisions to best respond to teammates' policies, and alleviate the effects of environment non-stationarity. We provided several empirical and experimental results, confirming MixTURE's strong ability to learn from expert heuristics and real diverse human generated data.
Our experiments show that multi-agent learning from demonstration can be enabled for teams of cooperating heterogeneous robots via direct human teaching. This greatly assists robot collaboration through human domain knowledge and provides a strong next-step towards human-robot teaming. We note that our research could be applied for good or otherwise, particularly in an adversarial setting. Nonetheless, we believe democratizing this knowledge is for the benefit of society.
This work was supported by the Naval Research Laboratory (NRL) under grant number N00173-21-1-G009 and grant number N00014-22-1-2834. The authors also would like to thank Manisha Natarajan for being a sounding board and guiding our statistical analysis.
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), page 1, 2004.
[2] David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[69] Xingyu Wang and Diego Klabjan. Competitive multi-agent inverse reinforcement learning with sub-optimal demonstrations. In Proceedings of the International Conference on Machine Learning (ICML), pages 5143-5151. PMLR, 2018.
[70] Fan Yang, Alina Vereshchaka, Changyou Chen, and Wen Dong. Bayesian multi-type mean field multi-agent imitation learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 33:2469-2478, 2020.
[71] Lantao Yu, Jiaming Song, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 7194-7201. PMLR, 2019.