SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Esmaeil Seraj
eseraj3@gatech.edu
Mariah Schrum
mariahschrum@berkeley.edu
Jerry Xiong
jxiong60@gatech.edu
Matthew Gombolay
matthew.gombolay@cc.gatech.edu
Extending recent advances in Learning from Demonstration (LfD) frameworks to multi-robot settings poses critical challenges such as environment non-stationarity due to partial observability which is detrimental to the applicability of existing methods. Although prior work has shown that enabling communication among agents of a robot team can alleviate such issues, creating inter-agent communication under existing Multi-Agent LfD (MA-LfD) frameworks requires the human expert to provide demonstrations for both environment actions and communication actions, which necessitates an efficient communication strategy on a known message space. To address this problem, we propose Mixed-In initiative Multi-Agent Apprenticeship Learning (MixTURE). MixTURE enables robot teams to learn from a human expert-generated data a preferred policy to accomplish a collaborative task, while simultaneously learning emergent inter-agent communication to enhance team coordination. The key ingredient to MixTURE's success is automatically learning a communication policy, enhanced by a mutual-information maximizing reverse model that rationalizes the underlying expert demonstrations without the need for human generated data or an auxiliary reward function. MixTURE outperforms a variety of relevant baselines on diverse data generated by human experts in complex heterogeneous domains. MixTURE is the first MA-LfD framework to enable learning multi-robot collaborative policies directly from real human data, resulting in  $44\%$  less human workload, and  $46\%$  higher usability score.
In recent years, Multi-Agent Reinforcement Learning (MARL) has been predominantly used by researchers to optimize a reward signal and for learning multi-robot tasks. Nevertheless, RL generally suffers from key limitations such as difficulty in designing an expressive and suitable reward function for complex tasks [60, 2] which can lead to undesirable robot behavior [14, 60, 64], high sample complexity [29], and safety concerns due to direct robot-environment interactions for optimizing the policy [64]. These problems are further exacerbated in multi-robot scenarios where inter-robot interactions and environment dynamics can be more complex and task descriptions and objectives more ambiguous [60, 53, 52]. As such, accurate models of human strategies and behaviors achieved via imitation methods are increasingly important for safely and effectively deploying autonomous systems and aligning values motivating robot behaviors with human values [47, 22, 12].
Learning from Demonstration (LfD) attempts to learn the correct behavior (policy) from a set of expert-generated demonstrations rather than a reward function, which can result in lower sample
complexity and a learned policy that more closely reflects the human's preferred strategy [48, 47]. Unfortunately, extending existing single-agent LfD paradigms such as Behavioral Cloning (BC) [66] or Inverse Reinforcement Learning (IRL) [1] to multi-robot settings poses several challenges such as environment non-stationarity and existence of multiple equilibrium solutions (an agent's optimal policy depends on other agents' policies) [37, 27, 60]. One can adopt such frameworks directly in a centralized system. However, centralized systems are not scalable, are prone to single-node failure, and pose significant computation overhead, and therefore, decentralized approaches (e.g., limited-range communication and local computations) have been more desired in multi-robot systems [20].
Prior work has shown that enabling communication among agents of a robot team creates a shared mental model of joint action-spaces and therefore, allowing coordinated action decisions [38, 65, 55, 19, 51] to handle challenges such as partial observability and environment dynamicity [25, 54, 50]. Although LfD can resolve the high sample complexity and reward shaping problems in RL, the task of MA-LfD can be onerous for the humans as they have to control multiple robots and imagine and simulate an appropriate Theory-of-Mind to create a communication strategy for the robot team.
Introducing the inter-agent communication [32] under current state-of-the-art (SOTA) Multi-Agent LfD (MA-LfD) frameworks such as Multi-Agent Generative Adversarial Imitation Learning (MA-GAIL) [60] or Multi-Agent Adversarial IRL (MA-AIRL) [71] requires the human expert to provide demonstrations for both environment actions and communication actions. Such approaches assume that the human expert has access to an efficient communication strategy on a known finite message space in addition to a known strategy for taking environment actions [42]. This assumption, however, is invalid, since dynamic environments with multiple interacting agents will be too complex and unpredictable for humans to be able to develop cohesive and comprehensive inter-agent communication protocols. Even if such an efficient communication strategy exists, demonstrating both the task strategy as well as the communication strategy could potentially pose significant workload on the human expert, which in turn can affect the performance and quality of demonstrations. These issues become even more severe when the robot team is heterogeneous or of composite nature (i.e., agents with different observation- and action-spaces as well as different objectives) where agents must rely on communication to operate and fulfill their tasks correctly [54, 7, 41].
To address these challenges, we develop a distributed MA-LfD framework to efficiently incorporate a human expert's domain-knowledge of teaming strategies for collaborative robot teams and directly learn team coordination policies from expert human teachers. To this end, we propose Mixed-Initiative Multi-Agent Apprenticeship Learning (MixTURE) which enables robot teams to learn an expert's preferred strategy to act in an environment. MixTURE simultaneously learns end-to-end emergent communication for the robot team to enhance team coordination, without the need for human generated data or an auxiliary reward function. To improve the quality of the learned inter-agent communication protocol, we reduce the entropy of a generated message given joint states and actions by maximizing the Mutual Information (MI) between messages and joint states. We demonstrate through empirical evaluation and a human subject experiment that our LfD-based MixTURE outperforms RL based methods due to reward function independence and low sample complexity. Furthermore, MixTURE significantly alleviates the human demonstrators' workload and time required to provide demonstrations, increases system usability, and improves overall collaboration performance of the robot team. Our key contributions are as follows:
1. We propose the MixTURE framework for learning robot teaming strategies from human expert demonstrations while simultaneously learning inter-agent communication through online interactions during training, without the need for expert data or an auxiliary reward.
2. We develop an MI maximization-based emergent communication learning model which reduces the entropy of a generated message for an agent given joint state-observations.
3. We evaluate MixTURE on real, diverse human-generated data, collected in a human-subject user study, and show that, in a complex multi-agent domain with heterogeneous tasks, we are able to achieve  $\sim 42\% - \sim 77\%$  higher performance and a significantly lower sample complexity. We also show that using MixTURE significantly improves workload and system usability relative to a benchmark MA-LfD framework. To best of our knowledge, this is the first work to train a MA-LfD framework on real human data.
4. We investigate the effects of demonstrating both environment actions and communication actions on a human expert's workload, demonstration quality, and system usability score. Our results show that a high-workload demonstration process in classic MA-LfD approaches
significantly ( $p < .001$ ) reduces an expert's demonstration quality (measured by performance) and the system's usability score. MixTURE significantly improves these results; increasing a human's performance and experience engaging in MA-LfD.
The literature for Multi-Agent LfD (MA-LfD) primarily aims to address the complexity of simultaneously training multiple agents under coordinated [43, 58, 46, 57, 68, 28] and uncoordinated tasks [6, 69, 40, 70]. In [61, 8] the MA-LfD problem is reduced into a single-agent problem by making the assumption that all agents share the same dynamics, observation spaces, and model parameters. In [34], a coordinated multi-agent IL approach is proposed which learns a latent coordination model along with the individual agent policies. In [60] the single-agent GAIL framework is extended for multi-agent scenarios along with a practical actor-critic method for multi-agent imitation. Similarly, in [71] the AIRL method was extended to the multi-agent settings. In [62] a scalable multi-agent LfD approach is proposed where a model-based heuristic method for automated swarm reorganization is leveraged to improve multi-agent task allocation problem. In [4] an expert feedback-based system is developed to address multi-agent path-finding problem. In [64] authors create an advising system to incorporate sub-optimal model-based heuristic policies to help improve MARL performance. Other prior work focused on learning human profile/behavior models for improved MA-LfD [45, 5]. More recently, Hoque et al. [26] proposed Fleet-DAgger, formalizing interactive fleet learning setting, in which multiple robots interactively query and learn from multiple human supervisors.
Nevertheless, applicability of these prior works in the collaborative multi-agent problems is considerably limited since none of these works explicitly address the inter-agent communication in dynamic and partially observable domains where agents not only need to take task-related actions, but also need to communicate and share information for coordination. Additionally, none of these prior work leverage real human-generated data for training to evaluate the approach against heterogeneity and diversity in human data. Our work addresses these limitations by eliminating the requirement for an expert to demonstrate a communication strategy. Instead, the human expert only needs to teach the robot team how to complete a task through demonstrations, and the robots will automatically learn a communication strategy that aligns with the expert's demonstrations. We also collect real human data to evaluate our method's ability to cope with variations in demonstration styles and strategies.
3 Problem Formulation: General MA-LfD with Heterogeneous Agents
We ground our problem formulation in a Markov Game (MG) [36] generalized to include partial observability and heterogeneous agents. We define a set of heterogeneous agents in a composite robot team (i.e., composed of different classes of robots) as agents that can have arbitrarily different state-, observation-, and action-spaces. The agents can also have different task objectives which, when enacted in coordination, enable the team to achieve a shared overarching mission. Accordingly, we define our generic MG as a 9-tuple:  $\langle \mathcal{C},\mathcal{N},\{\mathcal{S}^{(c)}\}_{c\in \mathcal{C}},\{\mathcal{A}^{(c)}\}_{c\in \mathcal{C}},\{\Omega^{(c)}\}_{c\in \mathcal{C}},\{\mathcal{O}^{(c)}\}_{c\in \mathcal{C}},r,\mathcal{T},\gamma\rangle$ .  $\mathcal{C}$  is a set of all available agent classes in the composite robot team and the index  $c\in \mathcal{C}$  denotes the agent class.  $\mathcal{N} = \sum_{\langle c\in \mathcal{C}\rangle}N^{(c)}$  is the total number of collaborating agents where  $N^{(c)}$  represents the number of agents in class  $c$ .  $\{\mathcal{S}^{(c)}\}_{c\in \mathcal{C}}$  and  $\{\mathcal{A}^{(c)}\}_{c\in \mathcal{C}}$  are discrete joint sets of state- and action-spaces, respectively.  $\{\Omega^{(c)}\}_{c\in \mathcal{C}}$  is the joint set of observation-spaces, including class-specific observations. Agents of the same class have identical  $S$ ,  $\mathcal{A}$ , and  $\Omega$ .  $\gamma \in [0,1)$  is the temporal discount factor for each unit of time and  $\mathcal{T}$  is the state transition probability density function. At each timestep,  $t$ , each agent,  $j$ , of the  $c$ -th class can receive a partial observation  $o_t^{c_j} \in \Omega^{(c)}$  according to some class-specific observation function  $\{\mathcal{O}^{(c)}\}_{c\in \mathcal{C}}:o_t^{c_j} \sim \mathcal{O}^{(c)}(\cdot|\bar{s})$ . Next, each agent,  $j$ , of class  $c$ , takes an action,  $a_t^{c_j}$ , forming a joint action vector  $\bar{a} = (a_t^{1_1},a_t^{1_2},\dots,a_t^{c_1},\dots,a_t^{c_j})$ . When agents take the joint action  $\bar{a}$ , in the joint state  $\bar{s}$  and depending on the next joint-state, they receive an immediate reward,  $r(\bar{s},\bar{a}) \in \mathbb{R}$ , shared by all agents regardless of classes. Each agent,  $j$ , of a class,  $c$ , achieves its own objective by sampling actions from a stochastic policy  $\pi_j^{(c)}$ . The objective of each agent is then to maximize the team return (expected sum of discounted rewards), i.e.,  $\mathbb{E}_{\pi}\left[\sum_{t=0}^{T}\gamma^t r_t\right]$ .
To directly learn the human's preferred strategy and resolve the reward specification problems [2] posed by RL, we leverage a demonstration dataset,  $D$ , provided by an expert, rather than the ground truth reward signal  $r$  employed in MARL. Unlike [71, 60] or [67], we do not assume multiple human experts in our MG to avoid the need for further coordination amongst the experts, which can be time consuming and expensive.  $D$  is a set of trajectories  $\{\tau_j^c\}_{j=1}^{\mathcal{N}^{(c)}}$ , where  $\tau_j^c = \{(o_t^{c_j},a_t^{c_j})\}_{t=1}^T$  is an expert trajectory collected by sampling  $a_t^{c_j} \sim \pi_E(a_t^{c_j}|\bar{o}_t)$  in which  $\pi_E$  is the expert policy and  $\bar{o}_t$  is the joint observation that the expert has access to at time  $t$ . We further assume that  $D$  contains the entire supervision to the learning algorithm (i.e., no online interactions during training). We build the MixTURE architecture on the generative adversarial training [21]. Our distributed GAIL objective underlying MixTURE is shown in Eq. 1 where  $\mathcal{D}_{\theta}^{(c_j)}$  is a local discriminator that classifies expert and policy trajectories for agent  $j$  of a class  $c$ , and  $\pi_{\phi}^{(c_j)}$  is the parameterized policy of agent  $j$  of a class  $c$ .
$$
\mathcal {L} _ {\mathcal {D} _ {\theta} ^ {(c)}} = - \mathbb {E} _ {\tau \sim \pi_ {E}, (\bar {o}, \bar {a}) \sim \tau} \left[ \log \mathcal {D} _ {\theta} ^ {(c _ {j})} (\bar {o}, \bar {a}) \right] - \mathbb {E} _ {\tau \sim \pi_ {\phi} ^ {(c _ {j})}, (\bar {o}, \bar {a}) \sim \tau} \left[ \log \left(1 - \mathcal {D} _ {\theta} ^ {(c _ {j})} (\bar {o}, \bar {a})\right) \right] \tag {1}
$$
According to [21], under the GAIL objective in Eq. 1 and at optimality, the distribution of generated state-action pairs by  $\bar{\pi}_{\phi}$  should match the distribution of demonstrated state-action pairs.
4 Mixed-Initiative Multi-Agent Apprenticeship Learning
Motivation and Problem Overview - Consider a generic composite team of robots including agents with heterogeneous characteristics and task objectives. Without loss of generality, consider a robot team composed of perception-only and action-only robots. Under our problem formulation in Section 3, perception robots and action robots create two separate classes of agents that need to collaborate on an overarching mission. For instance, in an application of wildfire fighting, robots of the perception class (e.g., quadcopters) need to search an environment for firespots, while the action robots (e.g., fire-extinguishing ground robots) who cannot sense the environment are required to extinguish the firespots found by the perception robots [54, 55, 52, 3]. Note that neither of the robot classes are capable of accomplishing the task without the other class.
To teach a collaborative policy to such a robot team, one can leverage demonstrations from a team of humans where each member is an expert. Using a team of human experts, however, poses further challenges: (1) simultaneous access to several human experts is expensive and can be time consuming, and (2) a communication strategy (e.g., natural language) among the human demonstrators is required for coordination, which can be challenging to translate to robot domain due to ambiguity, colloquialisms, and context-dependent use [30, 23]. Additionally, humans' communications might not make sense to the robot agents as humans could be unaware of all agents' full local state spaces.
Alternatively, we can leverage the demonstrations from a single human with domain-knowledge regarding the entire mission objective. For example, a trainer/coach can play a simulated game of firefighting using the aforementioned perception and action robots and provide expert demonstrations for how to efficiently distribute agents and prioritize tasks for searching the environment and putting out the fire. The challenge of using a single human expert, however, is that in this case the human would also need to demonstrate communication-actions (i.e., what information should an agent broadcast at each state) on top of the environment-actions (i.e., moving around or dousing fire). To this end, humans would have to create and maintain a Theory-of-Mind (ToM) of each agent under this increased action-space dimensionality, which significantly increases workload [15, 17].
To resolve this problem, we propose taking separate initiatives for teaching the robots in the team how to operate (environment actions,  $a_{t}$ , per observation,  $o_{t}$ ) and how to communicate (communication message,  $z_{t}$ , per state,  $s_{t}$ ) such that a human expert would only be required to provide environment-action demonstrations and the robot team would automatically infer on their own a suitable communication policy for the underlying expert demonstrations. We call our approach Mixed-Initiative Multi-Agent Apprenticeship Learning (MixTURE).
MixTURE Architecture - The proposed MixTURE architecture for human training of robot teams is shown in Fig. 1. At each timestep, each agent generates an embedding, representing agent's belief space, from its local observation. To handle agents' partial observability, the local observation embeddings are then passed into local recurrent policies for each agent. Each GRU policy receives the preprocessed features from the local observations as well as its own hidden state from previous timesteps. Therefore, the policy output depends only on the history of local observations and actions.
Figure 1: The proposed MixTURE architecture for a two-agent example scenario. At each timestep, each agent generates an embedding from its local observation, which is then passed into local recurrent policies for each agent. To learn from the expert demonstrations stored in the dataset  $D_{E}$ , we build a distributed multi-agent GAIL architecture. We enable inter-agent communication by adding an attentional communication module enhanced by a MI maximization reverse model.
To learn from the human expert demonstrations stored in the dataset  $D_{E}$ , as shown in Fig. 1, we build a distributed multi-agent GAIL architecture where each agent is equipped with a parametrized discriminator,  $\mathcal{D}_{\theta}$ . The discriminators are trained via a Binary Cross Entropy (BCE) loss to distinguish between state-action pair samples from the expert dataset and those generated by the generator (i.e., local policies). The output of the discriminators are treated as local rewards, which are then combined to encourage collaboration and teaming behaviour among agents [31]. To learn the agent policies through this shared reward signal, we leverage the Proximal Policy Optimization (PPO) [49].
We enable inter-agent communication by adding an attentional communication module in which each agent is equipped with a fully-connected network that processes action-embeddings generated by the recurrent policies of an agent and those generated by its local teammates (i.e., messages,  $z$ ) to output an action-decision. We enable this message-passing by creating differentiable communication channels among agents. To maintain locality, these communication channels can be leveraged locally (i.e., a communication graph where edges only exist when robots are spatially within close proximity). For an agent,  $i$ , the action-embedding messages received from a teammate,  $j$ , are weighted by some learned attention coefficients,  $\alpha_{ji}$ , to assign message importance. Therefore, the input message for agent  $i$  at time  $t$ , can be computed as  $m_t^{j\rightarrow i} = \sum_{j = 1}^{\mathcal{N}_t(i)}\alpha_{ji}z_{ji}^t$  where  $\mathcal{N}_t(i)$  represents the neighbors,  $j$ , of agent  $i$  at time  $t$ . Here,  $\alpha_{ji}$  are the learned attention coefficients that are computed via  $\alpha_{ji} = \mathrm{softmax}_j(\sigma (\bar{W}_{att}[\omega \bar{h}_i\parallel \bar{m}^{j\rightarrow i}]))$ . In this equation,  $\bar{W}_{att}$  are the learnable weights of the attention network,  $\parallel$  represents concatenation,  $\sigma$  is an activation function nonlinearity, and  $\bar{h}$  represents the hidden states. The Softmax function is used to normalize the coefficients across all neighbors  $j$ . Such attentional communication can enhance the action-decision quality, particularly with increased number of agents or when the states may significantly vary in different parts of the environment [16]. Messages in our communication module are entirely learned via backpropagation.
Mutual Information Maximization-Based Differentiable Communication – A challenge with the communication model learned via the described end-to-end differentiable channels is that the distribution of the messages broadcasted by an agent,  $i$ , given the state-observations,  $\rho(z_{ij} | \bar{o})$ , can have a high variance. The desired behavior, instead, is that the agents employ a cohesive communication strategy in which an agent sends a consistent message when it observes relatively similar states.
To resolve this issue, we propose maximizing the Mutual Information (MI) between an agent's outgoing message and the joint state-observations. MI is a measure of the reduction in entropy of a probability distribution,  $X$ , given another probability distribution,  $Y$ , such that  $I(X;Y) = H(X) - H(X|Y)$ , where  $H(X)$  denotes the entropy of  $X$  and  $H(X|Y)$  is the conditional entropy
of  $X$  given  $Y$  [33]. In our work, by maximizing the MI between the distribution of an agent's message,  $\rho(z_{ij}|\bar{o})$ , and the joint observations, we reduce the entropy over messages and encourage the communication model to be more consistent. Maximizing the MI in this way encourages  $z_{ij}$  to correlate with features within the observation distribution (i.e., mode discovery) [13, 44].
Unfortunately, a direct MI Maximization (MIM) between the message distributions and joint observations as formulated above,  $I(z_{ij};\bar{o}) = H(z_{ij}) - H(z_{ij}|\bar{o})$ , is intractable as it requires access to the true posterior,  $\rho (z_{ij}|\bar{o})$ . Therefore, in keeping with prior work, we rely on the Evidence Lower Bound (ELBO) of the MI instead. As shown in prior work [13], by minimizing an MSE loss between a sample from the current message embedding and the approximate posterior, modeled as a normal distribution with constant variance, is equivalent to maximizing the likelihood of the posterior.