Generalization of Heterogeneous Multi-Robot Policies via Awareness and Communication of Capabilities
Authors: Pierce Howell
Max Rudolph
Reza Joseph Torbati
Kevin Fu
Harish Ravichandar
Venue: CoRL 2023 Poster (2023)
Keywords: Heterogeneity
Multi-Robot Teaming
Generalization

Abstract: Recent advances in multi-agent reinforcement learning (MARL) are enabling impressive coordination in heterogeneous multi-robot teams. However, existing approaches often overlook the challenge of generalizing learned policies to teams of new compositions, sizes, and robots. While such generalization might not be important in teams of virtual agents that can retrain policies on-demand, it is pivotal in multi-robot systems that are deployed in the real-world and must readily adapt to inevitable changes. As such, multi-robot policies must remain robust to team changes -- an ability we call adaptive teaming. In this work, we investigate if awareness and communication of robot capabilities can provide such generalization by conducting detailed experiments involving an established multi-robot test bed. We demonstrate that shared decentralized policies, that enable robots to be both aware of and communicate their capabilities, can achieve adaptive teaming by implicitly capturing the fundamental relationship between collective capabilities and effective coordination. Videos of trained policies can be viewed at https://sites.google.com/view/cap-comm .

Generalization of Heterogeneous Multi-Robot Policies via Awareness and Communication of Capabilities'
Pierce Howell' Max Rudolpha Reza Torbati', Kevin Fu', Harish Ravichandar!
GGeorgaa Institute of Technology. PUniversity of Texas at Austin Email: heere.hoowell hwwelleggtecc..... edu
Abstract: Recent advances in multi-agent reinforcement learning (MARL) are enabling impressive coordination in heterogeneous multi-robot teams. However, existing approaches often overlook the challenge of generalizing learned policies to teams of new compositions, sizes, and robots. While such generalization might not be important in teams of virtual agents that can retrain policies on-demand, it is pivotal in multi-robot systems that are deployed in the real-world and must read- ily adapt to inevitable changes. As such, multi-robot policies must remain robust team changes- an ability we call adaptive teaming. In this work, investigate if awareness and communication of robot capabilities can provide such general- ization by conducting detailed experiments involving an established multi-robot test bed. We demonstrate that shared decentralized policies, that enable robots to be both aware of and communicate their capabilities, can achieve adaptive team- ing by implicitly capturing the fundamental relationship between collective capa- bilities and effective coordination. Videos of trained policies can be viewed attps://sites httollll.ss.oggle..../eeooooooes com/view/ cap-comm
Keywords: Heterogeneity, Multi-Rons Roboot Teaming, Generalization
Introduction
Heterogeneous robot teams have the potential to address complex real-world challenges that arise in wide range of domains, such as precision agriculture, defense, warehouse automation, supply chain optimization, and environmental monitoring. However, a key hurdle in realizing such potential is the challenge of ensuring effective communication, coordination, and control.
Existing approaches address the challenges of multi-robot systems can be crudely categorized into two groups. First, classical approaches use well-understood controllers with simple local interaction rules, giving rise to complex global emergent behavior [1]. Indeed, such controllers have proven ex- traordinarily useful in diverse domains. However, designing them requires both significant technical expertise and considerable domain knowledge. Second, recent learning- based approaches alleviate the need for expertise and domain knowledge by leveraging advances in learning frameworks and computational resources. Learning has been successful many domains, such as video games [2], autonomous driving [3], disaster response [4], and manufacturing [5].
However, learning approaches are not without their fair share of limitations. First, the majority of ex- isting methods focus on homogencous teams and, as such, cannot handle heterogeneous multi-robot teams. Second, and more importantly, even existing methods designed for heterogencous teams are often solely concerned with the challenge of learning to coordinate given team, entirely ignoring the challenge of generalizing the learned behavior to new teams. Given the potentially prohibitive cost of retraining coordination policies after deployment in real-world settings, it is imperative that multi-robot policies generalize learned behaviors to inevitable changes to the team.
In this work, we focus on the challenge of generalizing multi-robot policies to team changes. In particular, focus generalization of trained policies to teams of new compositions, sizes, and robots that are not encountered in training (see Fig. 1). We refer to such generalization as adaptive teaming, wherein the learning policy can readily handle changes the team without additional
"Equal Contribution. i This work was supported in part by the Army Research Lab under Grants W91 INF 17-2-0181 and W91 20--002-00
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

Rewerds Observations
Jonst Actions
Phase Train heterogenous
Training set
Robot state
Observation
Graph Network
Shared Action Network
Capabllities
Decentralized copability-bureee bused Policy
Phase 2: Generalize to new teams
Locol Observations
Joint Actions
Test wet
Figure 1: We investigate the role capability awareness and communication generalizing decentralized heterogencous multi-robot coordination policies to teams of new composition, size, and robots.
training. To this end, we need policies that reason about how group of diverse robots can collectively achieve common goal, without assigning rigid specialized roles to individual robots.
We investigate the role of robot capabilitiess in generalization to new teams. Our key insight is that adaptive teaming requires the understanding of how team's 's diverse capabilities combine to dic- tate the behavior of individual robots. For instance, consider an autonomous heterogeneous team responding to multiple concurrent wildfires. Effective coordination in such situations requires rea- soning about the opportunities and constraints introduced by the robots" individual and relative ca- pabilities, such as speed, water capacity, and battery range. general, robots must learn how their individual capabilities relate to those of others to determine their role in achieving shared objectives.
We develop policy architecture that can explicitly reason about robot capabilities when select- ing actions. Our architecture has four key properties: i) capability awareness: our design enables actions be conditioned on continuous capabilities addition observations, ii) capability com- munication: we leverage graph networks to learn how robots must communicate their capabilities iii) robor-agnostic: we utilize parameter sharing and learn policies that are not tied to individual robots, and iv) decentralized:: our trained policies can be deployed in decentralized manner. To- gether, these four properties provide the potential to generalize to new teams. One can view this design as an extension of agent identification techniques [6] to the metric space of capabilities. As such, capabilities do not merely serve to distinguish between agents during training enable be- havioral heterogeneity [7], but also to provide more general means encode how individual and relative capabilities influence collective behavior.
We evaluate the utility of capability awareness and communication in two heterogeneous multi- tasks in sim and real. Our results reveal that both awareness and communication of capabilities can enable adaptive teaming, outperforming policies that lack either one or both of these features in terms of average returns and task -sse-ipec metrics. Further, capability- -based policies achieve superior zero- shot generalization than existing agent identification-barl techniques, while ensuring comparable performance on the training set.
Related Work
Learning for multi-robot teams: Recent advances deep learning are providing promising ap- proaches that circumvent the challenges associated with classical control of multi-robot systems. Multi-agent reinforcement learning (MARL), in particular, has been shown to be capable of solving wide variety of tasks, including simple tasks in the multi-agent particle environments (MPE) [8], complex tasks under partial observability [9], coordinating an arbitrary number of agents in video games [2], and effective predictive modeling of multi-agent systems [10]. These approaches
driven by popular MARL algorithms like QMIX [1 [11], MADDPG [8], and MAPPO [12] nontrivial extensions of their single agent counterparts DQN [13], DDPG [14], and PPO [15], respectively. We adopt a PPO-based learning framework given its proven benefits despite its simplicity [12]. Cen- tralized training, decentralized execution (CTDE) is commonly used framework in which decen-
tralized agents learn to take actions based on local observations while a centralized critic provides

---

feedback based on global information [16, 17]. We use the CTDE paradigm as it lends itself natu- rally multi-robot teams since observation and communication are often restricted. However, its important to note that our approach is agnostic to the specific learning algorithm.
Learning for heterogeneous teams: Many MARL. algorithms were originally designed for use in homogeneous multi-agent teams. However, truly homogeneous multi-robot teams are rare except because of manufacturing differences, wear and tear, or task requirements, Most real-world multi- robot problems such as search & rescue, agriculture, and surveillance require a diverse set of ca- pabilities aggregated from heterogeneous robots [18-20]. While many MARL approaches consider heterogeneity, they either tend to focus on differences in behavior exhibited by physically identi- cal robots [21]. or identical behavior exhibited by peysically-differe
robots [22, 23]. A common strategy used to elicit heterogeneous behavior from shared models is referred to as agent identifi- cation or behavioral typing, which the agents' observations are appended with an agent-specific
index [24, 25]. While these methods have been shown to he highly effective, recent investigations have revealed issues with scalability [26), and robustness to observation noise [7]. While capability- awareness is similar in spirit existing identification-batt
techniques, it does not require assigning
indices to individual robots and can thus generalize to teams with new robots. Further, most existing methods do not simultaneously handle teams with physical and behavioral differences. According to a recent heterogeneous multi-robot taxonomy [7], our work falls under the category consisting of physically -iffereett robots that differ in behavior, but share the same objective. Two recent ap- proaches belong to this same category [4, 7]. However, one is limited to a discrete set of robot types [4] and the other learned decentralized robot-specifie policies that cannot handle the addition of new robots and might not generalize to new compositions [7].
Generalization: For applications to real-world multi-robot systems, it is essential to consider the generalization capabilities of learned control policies. our formulation, there are two axes of gen- eralization in heterogeneous multi-robot teams: combinatorial generalization (new team sizes and new compositions of the same robots) and individual capability generalization (new robots). Prior works reliant on feed forward or recurrent networks tend to be limited to teams of static size [8, 27]. Combinatorial generalization for homogeneous teams can be achieved with graph network-basee policies [27, 28]. However, existing methods tend to struggle with generalization in the presence of heterogeneity [29]. While methods that employ agent identification [24, 25] might be able to achieve combinatorial generalization by reusing the IDss from training, it is unclear how they can handle new robots. In stark contrast, capability-basse policies are robot-agnostic and can take the
capabilities of the new robot as input feature determine its actions.
3 Capability Awareness and Communication for Adaptive Teaming
In this section, we first model heterogeneous teams and then introduce policy architectures that enable capability awareness and communication, along with the associated training pipeline.
3.1 Modeling Heterogeneous Multi-Robot Teams
We model teams of N heterogeneous robots as a graph G = (V,E), where each node i/ E V is a robot, and each edge Ezj = (1,,,,) E is communication link. We use 2, to denote the obser- vations of the ith robot, which includes its capabilities and its sensor readings of the environment. We assume that the robots' heterogeneity can be captured by their capabilities. We represent the capabilities of the ith robot by a real-valued vector E C C RC, where C is the C-dimensional space of all capabilities of the robots. An example of multi-diimeniinnl capability is vector with elements representing payload, speed, and sensing radius. When robot i does not possess the kth capability, the keth element of o is set to zero.
3.2 Problem Description
We interested in learning a decentralized control policy that can i) effectively coordinate team of heterogeneous robots to achieve the task objectives, and ii) generalize readily to teams of both novel compositions and novel robots that are encountered during training. Our problem can be viewed as a multi-agent reinforcement learning problem that can be formalized as a decentral- ized partially-observabeee Markov Decision Process (Dec-POMDP) [30]. We expand on the Dec- POMDP formulation to incorporate the capabilitiess of heterogeneous robots and arrive at the tuple
3

---

(.A,) (2,,,,,,, O) where D is the set of N robots, S is the set of global states, } set of action spaces across all robots, {2,} J is a set of joint observations across all robots, C is the multi-dimensional space of capabilities, R is the global reward function, and T and O are the joint state transition and observation models, respectively. Our objective is learn decentralized action policies that control each robot to maximize the expected return EI r'] over the task horizon T₁e The decentralized policy of the ith robot π (,, defines the probability that Robot i takes Action o, given its effective observation o;. The effective observation o; of the äth robot is a function of both its individual observation and that of others in its neighborhood.
3.3 Policy Architecture
To enable capability awareness and communication in multi-robot coordination policies, we de- signed a policy architecture that leverages graph convolutional networks (GCNs) since a plethora of recent approaches attest to their ability to learn effective communication protocols and enable decentralized decision-making in multi-agent teams [27]. Further, operations are local to nodes and can therefore generalize graphs of any topology (i.e., permutation invariance [31]). We illustrate our architecture in Fig. 1 and explain its components below. Specific details including the exact architecture and hyperparameters we used can be found in Appendix E.
We argue that the heterogeneity of robots can have a significant impact on how the team must coordinate to achieve task. Specifically, individual and collective capabilities can affect the roles the robots play within the team. For instance, consider heterogeneous mobile robot team responding to wildfire incidents multiple locations. To effectively respond, robot within the team must account for its speed and water capacity. Therefore, it is necessary that robots aware of their capabilities. To enable such awareness, we append each robot's ' capability vector ca to its observations before passing them along as node features to the graph network. This information will help each robot condition its actions not just observations, but also its capabilities.
Capability awareness:
Capability communication: In addition to awareness, communicating capability information can enable team to reason about how its collective capabilities impact task performance. Revisiting our wildfire example, robots in the response team can effectively coordinate their efforts by implicitly and dynamically taking on roles based on their relative speed and water capacity. But such complex decision making is only possible if robots communicate with each other about their capabilities. Each node in our GCN- based policy receives capability information along with the corresponding robot's s local observations so the learned communication protocol can help the team communicate and effeetivvly build representations of their collective capabilities.
Note that our policy is robot-agnostic and learns the implicit and interconnected relationships be- tween the observations, capabilities, and actions of all robots in the team. Further, as we demonstrate in our experiments, capability awareness and communication enables generalization to teams with new robot compositions, sizes, and even to entirely new robots as long as their capabilities belong to the same space of capabilities@ C.
3.4 Training Procedure
We utilize parameter sharing to train a single action policy that is shared by all of the robots. Pa- rameter sharing is known to improve learning efficiency by limiting the number of parameters that must be learned. More importantly, parameter sharing required for our problem so policies can transfer to new robots without the need for training new policies or assigning robots to already trained policies. Additionally, we believe that parameter sharing serves secondary role in learning generalizable strategies for efficient generalization. Sharing parameters enables the policy to learn generalized coordination strategies that, when conditioned on robot capabilities and local observa- tions, can be adapted to specific robots and contexts.
We employ centralized training, decentralized execution (CTDE) paradigm train the action policy. We apply an actor-critic model, and train using proximal policy optimization (PPO). The actor-critic model is composed of decentralized actor network (i.e., shared action policy) that maps robots observations to control actions, and a centralized critic network [12], which estimates the value of the team's 's current state based on centralized information about the environment and robots aggregated from individual observations. Finally, we trained our policies on multiple teams until they converged, with the teams changing every 10 episodes to stabilize training.

---

Experimental Design
We conducted detailed experiments to evaluate how capability awareness and communication impact generalization to: i) new team sizes and compositions, and ii) new robots with unseen capabilities.
Environments: We designed two heterogeneous multi-robot tasks for experimentation:
Heterogeneous Material Transport (HMT): team of robots with different material car- rying capacities for lumber and concrete (denoted by c E I2² for the ith robot) must transport materials from lumber and concrete depots to a construction site to fulfill a pre-specified quota while minimizing over-provisse wer--rooision. We implemented this environment as Multi-Particle Envi- ronment (MPE) [32] and leverage the infrastructure of EPyMARL [33].
Heterogeneous Sensor Network (HSN): A robot team must form a single fully-connected sensor network while maximizing the collective coverage area. The ith robot's s capability o; E IR corresponds to its sensing radius. We implemented this environment using the MARBLER [34] framework which enables hardware experimentation in the Robotarium [35], a well-established multi-robot test bed.
Policy architectures: In order to systematically examine the impact of capability awareness and communication, we consider the following policy architectures:
ID (MLP): Robot ID-based MLP
ID (GNN): Robot ID-based GNN
CA (MLP): Capability-aware MLP
CA (GNN): Capability-aware GNN without communication of capabilities,
CA+CC (GNN): Both capability awareness and communication.
The ID-based baselines stand in for SOTA approaches that employ behavioral typing to handle het- erogeneous teams [24, 25), and, such, question the need for capabilities. The MLP based base- lines help us investigate the need for communication. Finally, the CA (GNN) enables communication of observations but does not does not communicate capability information.
Metrics: For both environments, we compare the above policies using Average Return: the average joint reward received over the task horizon (higher is better). Additionally, use environment- specific metries. In HMT, we terminate the episodes when the quotas for both materials are met. Therefore, consider Average Steps taken to meet the quota (lower is better). For HSN, consider Painwise Overlap: sum pairwise overlapping area of robots' coverage areas (lower is better).
Training: For each environment, we used five teams with four robots each during training. We selected the training teams to ensure diverse compositions and degree of heterogeneity. For HSN, we sampled robots' sensing radius from the uniform distribution U0.2, 0.6). For HMT, we sampled robots" lumber and concrete carrying capacities from the uniform distribution (0, 1.0). We also assigned each robot a one-hot to train ID-based policies. We trained each policy with 3 random seeds. We resampled robot teams every 10 episodes to stabilize training.
5 Results
Below, we report i) performance the training team, ii) zero-shot generalization new teams, and iii) zero-shot generalization to new robots with unseen values of capabilities for each environment.
5.1
Heterogeneous Material Transport
We first focus on the Heterogeneous Material Transport (HMT) environment.
Performance on training set: To ensure considering capabilities does not negatively impact train- ing, we first evaluate trained policies on the training set in terms of average return and average steps (see Fig. 2). We find that all policies resulted in comparable average returns (Fig. 2 (a)). However, average number steps per episode better captures performance in HMT, since episodes termi- nate early when the quota is filled. Compared to agent-ID policies, capability-based policies took fewer steps per episode (Fig. 2 (b)) achieve comparable rewards, suggesting that reasoning about capabilities can improve task efficiency performance.
5

---

IDJMLP)
ID(GNN]
CA(MLP)
CA|GNN)
CAACCIGNN)
166
Anarge Raturn
130
(a) HMT: Average return
(b) HMT: Average steps
(c) HSW: Average return
(d) HSN: Overlap area
Figure 2: When evaluated on teams seen during training, capability-awar policies performed comparably to
ID- policics in terms of both average return (higher is better) and task-specifie metrics (lower is better).
IDIMLPI
IDINNN]
CA(MLP)
CAIGNN)
CA+CCIGNN]
100
100
80
Average Steps
90
Average Stegs
Averege
60
50
30
30
(a) 3 Robots
(b) 4 Robots
(c) 5 Robots
Figure 3: When generalizing to new team compositions and sizes in HMT, capability- policies consistently
outperformed ID-based policies in terms of average steps taken meet the quota (lower is better).
Zero-shot generalization to new team compositions and sizes: We next evaluated how trained policies generalize to team compositions and sizes not encountered during training. Fig. shows plots quantifying performance on new compositions with team sizes of 3, 4, and robots. To ensure fair comparison, we evaluated all policies on the same set of 100 teams by randomly sampling novel combinations
robots from the training set. We evaluated each policy on each test team across 10 episodes per seed. Given that this evaluation involved no new individual robots, we reused each robot's from the training set to facilitate the evaluation of ID-based policies.
We find that all capability-awan methods outperformed ID-based methods in return and task- specific metries. This is likely due capability methods' ability capture the relationship between robots' carrying capacities and the material quota. In contrast, ID-based methods must learn to implicitly reason about how much material each robot can carry. Interestingly, CA (MLP) sulted in fewer steps per episode (lowest mean and variance) across all team sizes, and outperformed both the other capability- based and communication enabled baselines: CA (GNN) and CA+CC (GNN). This suggests that mere awareness of capabilities is sufficient to perform well in the HMT environ- ment. Indeed, communication not as essential in this task as robots can directly observe relevant information (e.g. material demands) and implicitly coordinate as long they are aware of their own capabilities. It might be possible to further improve performance by learning better communi- cate, but that would be significantly more challenging since the task can be mostly solved without communication.
CA/MLP)
CA(GNN)
CA+CCIGNN)
100
100
90
90
80
Average Steps
90
70
Average
60
Merea
50
50
30
60
20
robots
(b) robots
(c) S robots
When generalizing new robots with unseen values for capabilities in HMT, policies that are only of capabilities (CA (MLP) and CA (GNN)) outperformed policies that also communicated capabilities (GANCC(GMN) in terms average number steps taken to transport the required material (lower is better).
Figure 4:

---

Zero-shot generalization to new robots: In Fig. 4, we show the policies' ability to generalize to teams composed of new robots. Since the robots' capabilities in this evaluation are different from those of the robots in the training set, we could not evaluate agent-II) methods since there is no trivial way to assign IDs to the new robots. These results clearly demonstrate that reasoning about capabilities can enable generalization to teams with entirely new robots. Further, we again see that capability awareness without communication sufficient to generalize in the HMT environment.
Additional results: We provide additional results for HMT by reporting more task-specific metrics and evaluations on significantly larger teams in Appendix A. The results on task-specific metrics further support the claim that capability-aweer aware methods generalize better than ID-based methods. We also find that these benefits extend teams consisting 8, 10, and 15 robots. Taken together, the above results suggest that reasoning about capabilities (rather than assigned IDs) improves adaptive teaming, likely due to the ability to map capabilities to implicit roles.
5.2 Heterogeneous Sensor Network
Below, we discuss results on the Heterogeneous Sensor Network (HSN) environment.
Performance on training set: In Fig. 2 (c) and (d), we report the performance of trained policies in HSN on teams in the training set in terms of average return and pairwise overlap. All policies except CA (MLP) performed comparably and were able to effectively learn to maximize expected returns, achieve fully connected sensor network, and minimize the pairwise overlap in coverage area. Further, CA+CC (GNN) and ID (CNN) perform similarly but marginally better than the other baselines. CA (MLP) 's suboptimal performance indicates that capability awareness in isolation with- out any communication hurts performance in the HSN task. Indeed, while it is possible to achieve good performance in HMT without communicating, HSN requires robots effectively communicate and reason about their neighbors' sensing radii in order to form effective networks.
Taken together, these results suggest that capability awareness and communication can lead to effec- tive training in heterogeneous teams. ID-based methods are able to perform at a similar level. This is to be expected given that we conducted these evaluations on the training and IDs are sufficient to implicitly assign roles and coordinate heterogeneous robots within known teams.
Zero-shot generalization to new feam compositions and sizes: In Fig. 5, we report the perfor- mance of the training policies in HSN when evaluated on teams of different compositions and sizes. We found that CA+CC (GNN) achieved the best average returns (highest mean and lowest variance) across all team sizes. Both ID-based methods (ID (MLP) and ID (GNN) resulted in lower returns compared to all three capability-awarene for the training set in Fig. 2, demonstrating that IDs alone might help train heterogeneous teams but tend to generalize poorly to new team compositions and sizes. This is likely because ID-based policies fail to reason about robot heterogeneity, and instead overfit the relationships between robot IDs and behavior in the training set. Further, CA+CC (GNN) in particular consistently outperformed all other policies across metrics and variations, suggesting that both capability awareness and com- munication necessary to enable generalization HSN.
baselines. Note that this is in stark contrast to the results
Zero-shot generalization to new robots:
We evaluated the trained policies' ability to generalize to teams of different sizes which are composed of entirely new robots whose sensing radii are different from those encountered in training. Similar to HMT, we cannot evaluate ID-based policies on teams
IDIMLP)
ID(GNN]
CA(MLP)
CAIGNN)
CA+CC[GNN]
-50
-100
-50
-100
Aecroge Setum
-200
1100
-150
-150
-20
-300
-250
-300
-350
-500
(a) 3 Robots
(b) 4 Robots
(c) 5 Robots
Figure 5: When generalizing to new team compositions and sizes in HSM, capability-based policies consistently outperformed ID- D-based baselines in terms of average return (higher is better). Further, combining awareness and communication of capabilities resulted in the best generalization performance.
7

---

CA/MLP)
CA|GNN)
CA+CCIGNN)
-50
-100
Average Resur
120
Neeege Retum
-150
-150
-20
-200
-000
-250
-350
420
(a) 3 Robots
(b) 4 Robots
(c) Robots
Figure 6: When generalizing teams comprised of new robots in HSN, combining awareness and communi- cation of capabilities (CANCC(GNN) achieves higher average returns than baselines that merely aware of capabilities, irrespective whether they communicate observations (CA (GNN)) or not (CA (MLP)).
with new robots since there is no obvious way to assign IDs to the new robots. In Fig. 6, we report the performance of all three capability- -based policies terms of average return. Both GNN-based policies (CA (GNN) and CA+CC (GNN)) considerably outperform the CA (MLP) policy, underscoring importance of communication in generalization to teams with new robots. However, we also see that communication of observations alone is insufficient, evidenced by the fact that CA+CC CA+CC(GNN) (which communicates both observations and capabilities) consistently outperforms (GNN) (which only communicates observations).
Real-robot demonstrations: We also deployed the trained policies of CA+CC (GNN), CA (MLP), and CA (GNN) on the physical Robotarium (see Section B.1 for further details and snapshots), Overall, we find that the benefits reported above extend physical robot teams. We find that CA+CC (GNN) and CA (CNN) policies generalize to physical robots and successfully build a sensor network while minimizing sensing overlap for teams of 3 and robots. The CA (MLP) policy resulted in signif- icantly worse performance, where robots' executed paths provoked significant engagement of the Robotarium's barrier certificates due to potential collisions.
Additional results: We provide additional results for HSN by reporting more task-specific metrics in Appendix B. Much like the results for HMT, the results on task-specific metries further support the claim that capability-aware methods show superior adaptive teaming ability compared with ID- based methods. The additional results also support our claim in this section that communication of capabilities is essential for success on this task.
6 Limitations
While our framework could reason about many different capabilities simultaneously, our experi- ments only involved variations in I-D and 2-D capabilities. We also only consider generalization to new values for capabilities; we do not consider generalization to new types of capabilities. Ad- ditionally, our work only considers the representation of robot's capabilities that we can quantify. Handling implicit capabilities and communication thereof may benefit from additional meta-learning mechanisms, uncovering a more general relationship between robots' learned behaviors and capa- bilities. Further, do not consider high-level planning and task-allocation and rely solely the learning framework to perform implicit assignments to sub-tasks within the macro task. Future work investigate appropriate abstractions and interfaces for considering both learning-based low-level policies and efficient algorithms for higher-level coordination. Lastly, we only consid- ered cullycconeected communication graphs in our evaluations for simplicity. While graph networks are known to effectively share local observations for global state estimations in partially-connected teams [27, 36), is unclear if such ability will translate the communication of capabilities.
7 Conclusion
We investigated the utility of awareness and communication of robot capabilities in the general- ization of heterogeneous multi-robot policies to new teams. We developed graph network-based decentralized policy architecture based on parameter sharing that enables robots to reason about and communicate their observations and capabilities to achieve adaptive teaming. Our detailed experi- ments involving two heterogeneous multi-robot tasks unambiguously illustrate the importance and the need for reasoning about capabilities opposed to agent IDs.

---

References
[1] J. Cortés and M. Egerstedt. Coordinated control of multi-robot systems: A survey. SICE Journal of Control, Measurement, and System Integration, 2017.
[2] B. Ellis, S. Moalla, M. Samvelyan, M. Sun, A. Mahajan, J. N. Foerster, and S. Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning, 2022.
[3] S. Shalev-Shwartz, S. Shammah, and A. Shashua. Safe, multi-agent, reinforcement learning for autonomous driving, 2016.
[4] E. Scraj. Z. Wang. R. Paleja, D. Martin, M. Sklar, A. Patel, and M. Gombolay. Learning efficient diverse communication for cooperative heterogeneous teaming. In Proceedings of the 21 International Conference Autonomous Agents and Multiagent Systems, 2022.
[5] Z. Wang, C. Liu, and M. Gombolay. Heterogeneous graph attention networks for scalable multi-robot scheduling with temporospatial constraints. Autonomous Robots, 2022.
[6] C. Li, T. Wang, C. Wu, Q. Zhao, J. Yang, and C. Zhang. Celebrating Diversity in Shared Multi-Agent Reinforcement Learning, 2021.
[7] M. Bettini, A. Shankar, and A. Prorok. Heterogeneous multi-robot reinforcement learning, 2023.
[8] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. Neural Information Processing Systems (NIPS), 2017.
[9] S. Omidshafiei, J. Pazis, Amato, J. P. How, and Vian. Deep decentralized multi-task multi- agent reinforcement learning under partial observability. In International Conference Machine Learning, 2017.
[10] Y. Hoshen. Vain: Attentional multi-agent predictive modeling. In Advances in Neural Infor- mation Processing Systems, 2017.
[11] T. Rashid, M. Samvelyan, C. S. de Witt, G. Farquhar, J. Foerster, and Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning, 2018.
[12] C. Yu, A. Velu, E. Vinitsky, Y. Wang, A. M. Bayen, and Y. Wu. The surprising effectiveness of MAPPO in cooperative, multi-agent games. CoRR, 2021.
[13] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning, 2013.
[14] P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning, 2019.
[15] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.
[16] V. Egorov and A. Shpilman. Scalable multi-agent model-based reinforcement learning, 2022.
[17] Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients, 2017.
[18] Dang, M. Tranzatto, S. Khattak, F. Mascarich, K. Alexis, and M. Hutter. Graph-based sub- terranean exploration path planning using aerial and legged robots. Journal of Field Robotics, 2020.
[19] T. Dang, M. Tranzatto, S. Khattak, F. Mascarich, K. Alexis, and M. Hutter. Graph-based sub- terranean exploration path planning using aerial and legged robots. Journal of Field Robotics, 2020.

---

[20] G. Gil, E. Casagrande, L. P. Cortés, and R. Verschae. Why the low adoption of robotics in the farms? challenges for the establishment of commercial agricultural robots. Smart Agricultural Technology, 2023.
[21] C. Li, T. Wang. C. Wu, Q Zhao, J. Yang, and C. Zhang. Celebrating diversity in shared multi- agent reinforcement learning. Advances in Neural Information Processing Systems, 2021.
[22] J. K. Terry, N. Grammel, S. Son, and B. Black. Parameter sharing for heterogeneous agents in multi-agent reinforcement learning, 2022.
[23] C. Wakilpoor, P. J. Martin, C. Rebhuhn, and A. Vu. Heterogeneous multi-agent reinforcement learning for unknown environment mapping. arXiv preprint wrXiv:2010.0266. 2020.
[24] J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 2016.
[25] K. Gupta, M. Egorov, and M. Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In Autonomous Agents and Multiagent Systems: AAMAS 2017 Work- shops, Best Papers, São Paulo, Brazil, May 8-12, 2017, Revised Selected Papers 16. Springer, 2017.
[26] Christianos, G. Papoudakis, M. A. Rahman, and S. V. Albrecht. Scaling multi-agent rein- forcement learning with selective parameter sharing. In International Conference on Machine Learning. PMLR, 2021.
[27] A. Agarwal, S. Kumar, and K. Sycara. Learning transferable cooperative behavior in multi- agent teams, 2019.
[28] Q. Li, W. Lin, Z. Liu, and A. Prorok. Message-aware graph attention networks for large-scale multi-robot path planning, 2021.
[29] A. Mahajan, M. Samvelyan, T. Gupta, B. Ellis, M. Sun, T. Rocktäschel, and S. Whiteson. Generalization in cooperative multi-agent systems, 2022.
[30] A. Oliehoek and C. Amato. A Concise Introduction Decentralized POMDPs. Springer- Briefs in Intelligent Systems, 2016.
[31] A. Khan, E. Tolstaya, A. Ribeiro, and V. Kumar. Graph policy gradients for large scale robot control, 2019.
[32] Mordatch and P. Abbeel. Emergence of grounded compositional language in multi-agent populations. CoRR, 2017. URL http://arxiv www.oogg/////oooooooooooooo... /103. 04908.
[33] G. Papoudakis, F. Christianos, L. Schäfer, and S. V. Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurlPS), 2021.
[34] R. Torbati, S. Lohiya, S. Singh, M. Nigam, and H. Ravichandar. Marbler: An open platform for standardized evaluation of multi-robot reinforcement learning algorithms, 2023.
[35] S. Wilson, P. Glotfelter, L. Wang, S. Mayya, G. Notomista, M. Mote, and M. Egerstedt. The robotarium: Globally impactful opportunities. challenges, and lessons learned in remote- access, distributed control of multirobot systems. IEEE Control Systems Magazine, 2020.
[36] Q. Li, F. Gama, A. Ribeiro, and A. Prorok. Graph neural networks for decentralized multi- robot path planning. CoRR, 2019.
[37] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms. 2017.
10

---

A Heterogeneous Material Transport (HMT) Additional Results
This section provides additional results for the HMT environment. Specifically. we provide additional task-specific metrics for the generalization experiments, and new generalization results for robot teams of significantly larger sizes (i.e. team sizes of 8, 10, and 15 robots).
The task-specific metrics defined below evaluate the rate which each policy contributes fulfilling the total quota, and the individual quotas for lumber and concrete:
% of episodes by which the total quota was filled.
of lumber quota remaining. % of concrete quota remaining.
For both generalization to new teams (see Fig. 7) and new robots (see Fig. 8), capability-aware methods filled the total quota in fewer episode steps compared to the ID-based methods, while gen- erally better preventing over-provisioning of both lumber and concrete. This result further supports our claim that capability awareness improves generalization performance. Observing Fig. 9, we find that these benefits of capability-basek policies extend to considerably larger team sizes.
IDIMLP) ID(GNN]
CA(MLP)
CA(GNN)
CA+CC(GNN)
Durlnng
Which Quata was Filled
50
% of Episodes During
Which Queta was Filled
50
% of Episodes During
Which Ovota was Filled
40
40
40
30
30
20
20
10
10
100
Episode
Episode
Episade Step
-10
-10
-10
-20
% Lumber Queta
-20
% Lumber Queta
-00
Acmaininn
Remaining
Remaining
Lumber
20
20
20
100
20
40
60
80
100
100
40
60
80
100
100
20
40
60
80
100
Episode Step
Episode Step
Episode Step
40
40
% Concrete Quota
-20
% Concrate Quota
-20
Romaining
Remaining
Remaining
Concrete
20
40
40
40
60
60
80
60
100
100
40
80
100
40
80
Episode Step
Step
Step
(a) 3 robots
(b) robots
(c) 5 robots
Figure 7: Policies with capability awareness outperform agent methods at meeting material quota with minimal number of steps when generalizing to new team compositions. sapability-wwarene methods without communication of capabilities (i.c. CA (MLP) & CA (GNN) outperform methods with capability communication for this task.
B Heterogeneous Sensor Network (HSN) Additional Results
This section provides additional results on training performance and ii) task-specific metrics.
The task-specific metrics for the HSM environment are the following:
Pairwise overlap: The sum of pairwise overlap in coverage area among robots (lower is better).

---

CA/MLP)
CA(GNN) CA+CCIGNN)
of Episodies During Which Filled
% of Episories During
Which Quata was Filled
Which Quata was
600
Episodes
20
20
20
20
10
10
40
80
100
20
80
100
80
Episode Step
Episode Step
Episade Step
% Lumber Queta
-20
% Lumber Queta
-20
Remaining
Remaining
Remaining
20
20
20
40
40
100
100
100
40
40
Episode Step
Episode Step
Episode Step
440
-40
-40
Concrese Quota
-20
Concrese Quata
-20
Concrete Queta Remaiming
-20
Remaining
Remaining
100
20
40
60
80
100
100
0
20
40
60
80
100
100
20
40
60
80
100
Episode Step
Episode Step
Episode
Step
robots
robots
(c) 5 robots
Figure 8: Policies without communication of capability-awarrness (i.e. CA (MLP) and CA (GNN)) outperformed the policy with communication of capabilities (CN+CC(GN)) on task-specific metrics when generalizing to new robots with capabilities not seen during training.
of fully connected teams (by episode step): Percentage of teams that managed to form a sensor network that connects all of the robots (higher is better).
In Fig. 12, report the training performance (i.e. training teams only) for each policy. The train- ing curve suggest that capability- awarre and ID-based methods perform comparably during learning. Notably, the communication models ID (GNN) and CA+CC (GNN) converge faster and achieve higher overall returns than other methods. This result suggests that communication between robots signifi- cantly assists in learning collaborative behavior.
Capability- aware policies again demonstrate superior performance when generalizing to new teams (see Fig. 10) and new robots (see Fig. 11) on task-specific metrics, highlighting the importance of capability awareness for generalization to robot teams with new robots, team sizes, and team com- positions. Notably, DA+CC(GNN) results in significantly lower pairwise overlap for robot teams of size and robots, and marginally lower pairwise overlap for robot teams size 5, compared to ID-based methods. This suggests the coammmiicationnnnaalee policy effectively learns to commu- nicate capabilitics and that such communication capabilities is essential for generalization in this task.
B.1 Images of Robotarium Experiments
In this section, we present visual representations derived from actual robot demonstrations of the trained capability-aware communication policy. Videos of the robot demonstrations can be found at: https: / / /sttts.tssseeeeeeeeeeeeeeeeeeees wwwgll..co/vvee////////o/ httss/ssttssglllleeeeeese
12

---

IDIMLP)
ID(GNN]
CA(MLP)
CA(GNN)
CA+CC[GNN)
100
100
100
90
80
80
Average
70
60
40
50
30
of Eoisodes Filled
60
% Epicodes During
Which Quela wes Blle
% of Eplsodes During
Which Queta wes Filled
50
40
30
20
20
10
20
40
60
80
100
20
40
60
80
100
20
40 Episode Step
60
80
100
Episode Step
Episode Step
40
-40
40
% Lumber Quata Remaining
-20
% Lumber Quota
-20
Romaining
Remaining
Lumber
40
80
100
60
100
100
40
80
100
100
20
40
80
100
Episode Step
Episode
Step
Episode
Step
% Concrete Quota
% Concrete Quota
-20
Romaining
Remaining
20
20
20
40
%
100
100
100
Episode Step
Episode
Episode Step
(a) robots
(b) 10 robots
(c) 15 robots
Figure 9: Experiments evaluating the generalization of policies to significantly larger teams (size 10, and 15) compared the training meeting the total material quota with minimal number of steps when generalizing to new, large team com- positions. Capability-awareeess methods without communication of capabilities (i.c. CA (MLP) & CA (GNN)) outperform methods with capability communication for this task.
(size 4). Policies with capability awareness outperform agent ID methods
13

---

IDIMLP)
ID(GNN]
CA(MLP]
CA(GNN)
CA+CCIGNN]
3.5
0.8
2oa
g2.
22.0
0.6
808
1.5
1.0
0.
0.0
0.0
0.0
1.0
1.0
1.0
Fuly Conoccted Reams
0.8
0.8
0.4
Fuly
0.2
0.2
0.2
Epsode Step
Episade
(a) 3 Robots
(b) 4 Robots
(c) Robats
Figure 10: Capability- hased policy architectures consistently outperform ID-based haselines both in terms of average return and task performance metries when generalizing team compositions and sizes. Further, combining awareness and communication of capabilities results the best generalization performance.
CA(MLP)
CA[GNN]
CA+CCIGNN)
1.50
rawato Overap Arca
2.0
1.00
4.5
0.75
us0
0.2
2025
8.5
0.0
0.00
1.0
1.0
1.0
Full Connectedd Thams
8..
di Tully Connected
0.6
0.4
0.4
0.4
0.2
0.2
0.
Ep Episode Stop
Edisode Stap
Episode Stap
(a) 3 Robots
(b) 4 Robots
Robots
Figure 11: Policy architecture that combines awareness and communication (CA+CC of capabilities out- performs both other baselines (CA (GNN) in terms of % fully connected and CA (MLP) in terms of average return and pairwise overlap) when generalizing teams comprised of new robots.
14

---

IDIMLPI
IDIGNN]
CAIMLP)
CA(GNN)
CAACCIGNN)
of Fully Connected Teams
-800
1000
-1700
1200
IXGNN
0.2
CAIMUF:
GAIGNNI
CCIGNNI
0.0
95
2.0
2.5
2.0
10
20
30
40
50
60
80
Environment Steps
Episode Step
Figure 12: For the HSN environment, capability- policies perform comparably to ID-based policies in terms of training efficiency (first) and terms of task-specific metrics when evaluating the trained policy on the training set.
(a) Beginning of episode (3 robots).
(b) End of episnde (3 robots).
(c) Beginning of episode (4 robots).
(d) End of episode (4 robots).
Figure 13: Demonstrations of (ANCC(GNN) policy deployed real robot teams in the Robotarium testbed for google. www.ooll.../oooooooooooooooooooo www/viee/eeeeeeooooooooe/.aeeeee cap- comm for videos deployment to the Robotar-
the HSN task. See https://sites jum.
15

---

C Environment Specifications
C.1 Heterogeneous Material Transport
This section describes additional details about the heterogeneous material transport (HMT) environ- ment.
Gainda 13
Lonker beleoerd tucber qurls: 2.9/6
treceste delivard concreta cots: 1339999006599719
Concrete Depot
Lumber Depot
Construction
Site
Figure 14: and red, which represents its bias towards its carrying capacity for either lumber (red) or concrete (blue). The objective of the team is to fill the lumber and concrete quota at the construction site without delivering excess.
the Heterogemcous Material Transport (HMT) environment, each agent's 's color is a mixture blue
The lumber and concrete quota limit for the HMT environment randomly initialized an integer value between (0.5 x agents) and (2.0 x n agents).
Robots have five available actions: they can move left, right, up, down, stop. At the beginning of each episode, all of the robots begin at a random position in the construction site zone (see Fig. 14). The observation space for robot is the combination of the robot's 's state and the environment' 's state: specifically it is composed of the robot's position, velocity, amount lumber and concrete it's carrying, and its distance to each depot, the total lumber quota, the total concrete quota, the total amount lumber delivered, and the total amount of concrete delivered. Robots' observations do not contain state information about other robots. Finally, append the robot's unique ID for the ID baseline methods and the robot's maximum lumber and concrete carrying capacity for the CA methods to the robots' observations.
The total reward for each robot HMT computed by summing the individual rewards of each robot. Robots are rewarded when they make progress in meeting the lumber and concrete quotas and are penalized when they exceed the quota. If a robot enters the lumber depot or concrete depot, and the robot is empty (i.e. not loaded with any lumber concrete), and the quota has yet to be filled, then the robot is rewarded with pickup reward of 0.25. If the robot is loaded with material, then robot is rewarded penalized when it drops off the material at the consiruction site. Specifically, when a robot delivers a material and the quota for that material has yet to be filled, then the robot is rewarded with positive dropoff reward of 0.75. However, if the robot delivers a material and goes over the quota, then the robot penalized with a negative surplus penalty reward proportional to the amount of surplus: -0.10 surplus material. Finally, robots received a small time penalty of -0.055 for each episode step in which the total quota is not filled; this promotes the robots to finish the task quickly possible.
C.2 Heterogeneous Sensor Network
This section describes additional details about the heterogeneous sensor network (HSN) environment.
16

---

(a) Our agents running in simulation using the MAR- BLER framework.
(b) Our agents running in the physical Robotarium.
Robots have five available actions: they can move left, right, up, down, stop. After selecting an action, the robots move in their selected direction for slightly less than a second before selecting action. The robots start random locations least 30cm apart from each other, move at ~21cm/second, and utilize barrier certificates [35] that takes effect at 17cm away to ensure they do not collide when running in the physical Robotarium.
The reward from the heterogeneous sensor network environment is a shared reward. We describe the reward below:
D(i,)) = (Ip(t) p()1l (c + c.)
+ [D(i,j) + 0.05, if Di, <
r(i,5)
-1.1 [D(i,)) 0.05, otherwise
N R = Σ r(,))
where and j are robots, p(i) the position of robot e; the (capability) sensing radius robot i, and R is the cumulative team reward shared by all the robots. The above reward is designed to reward the team when robots connect their sensing regions while minimizing overlap so as to maximize the total sensing area.
D Training and Evaluation Specifications
This section describes the design of the training teams and the sampling of evaluation teams for both environments. To learn generalized coordination behavior, the training teams were required to be diverse in terms of composition and capture the underlying distribution of robot capabilities.
D.1 Heterogeneous Material Transport
Training Team Number (concrete capacity, lumber capacity)
(0.9.0.1), (0.7,0.3), (0.0,,
(0.9, 0.1), (0.7,0.3), (0.0, 1.0), (0.2,0.8)
(0.8,0.2), (0.3, 0.. (0..0.,))
(1.0,000) (0.0, (0.1, (0.3,
(0.6, (0.3) (0.7, 0.3), (0.0, 1.0)
Table Training teams used for the HMT task (five teams of four robots each).
D.2 Heterogeneous Sensor Network
To design these training teams, we first binned robot capabilities into small, medium, and large sensing radii with bin ranges [0.2m, 0.33m), [0.33m, 0.46ml], and [0.46m, 0.60m) respectively. We then generated all possible combinations with replacements for teams composed of four robots of small, medium, um, and large robots for a total of 15 teams. Fach robot assigned to one of the bins small, medium, and large had its capability (i.c. sensing radius) uniformly sampled within the bin
17

---

range. This resulted in 15 total teams, for which we hand-selected 5 sufficiently diverse teams to be the training teams. The resulting training teams are given in Table 2.
Training Team Number
Robot Sensing Radii (m)
(0.2191), (0.2944), (0.2608), (0.2660))) (0.3668)
(0.2746), (0.2746) (0. (0.5756)
(0.3178), (0.3467). (0.5317), (0.6073)
(0.2007), (0.5722), (0.5153), (0. (0.4622)
(0.4487), (0.5526), (0.5826), (0.58343)
Table 2: Training teams used for the HSN task (live teams of four robots each).
The evaluation robot teams were sampled differently for the different experimental evaluations per- formed. In the training evaluation experiment, the teams were the same the training teams in Table 2. Teams for the generalization experiment to new team compositions, but not new robots, sampled randomly from the 20 robots from the training teams (with replacement). Each robot from the pool of 20 robots was sampled with equal probability. In contrast, teams for the gener- alization experiment to new robots were generated by randomly sampling new robots, where each robot's sensing radius was sampled from uniform distribution independently U.0.2m, 0.6m). For the two generalization experiments, 100 total teams on the same set of sampled teams by fixing the pseudo random number generator's 's seed.
sampled. Each algorithm was evaluated
We first focus on the training curves and subsequent evaluations conducted the training set, without considering generalization. The goal of this experiment is to ensure introducing capabilities does not negatively impact training. The learning curves in terms of average return are shown in Fig. 12. All models achieved convergence within 20 million environment steps, with ID (GNN) and CA+CO exhibited both the fastest convergence and the highest returns. These results suggest that communication of individual robot features, whether based on IDs or capabilities, improves learning efficiency and performance for heterogeneous coordination.
E
Policy Details
E.1 Graph Neural Networks
We employ graph convolutional network (GCN) architecture for the decentralized policy n; which enables robots to communicate for coordination according the robot communication graph G.
GCN is composed L layers graph convolutions, followed by non- non-linearity. In this work, we consider single graph convolution layer applied to node is given by
4 =a
Σ
so(h
JEN()
where (1(11) E FR ( the node feature of node j.. (1 Uz) E E} are nodes j connected i, do is node feature transformation function with parameters non-linearity (e.g. Relu), and hg E G is the output node feature.
E.2 Policy Architectures
Each of the graph neural networks in the GNN-based policy architectures evaluated are composed of an input encoder network, a message passing network, and an action output network. The encoder network is a 2-layer MLP with hidden dimensions of size 64. For the message passing network, single graph convolution layer composed of 2-layer MLPs with ReLU non-linear activations. The action output network is additionally 2-layer MLP with hidden dimensions of size 64. The learning rate is 0.005.
MLP (ID)/LL (CA): The MLP architectures compose of a 4-yer multi-layer perceptron with 64 hidden units each layer and ReLU non-linearities.
CA (GNN)/CA+CO) (GNN) (INN))) Fach of the graph neural networks compose of an input "encoder" network, message passing network, and an action output network. The encoder network and the
18

---

action output network are multi-layer perceptrons with hidden layers of size 64, ReLU non-linear activations, and with one and two hidden layers respectively. The message passing network is graph convolution layer wherein the linear transformation of node features (i.e. observations) is done by a 2-layer MLP with ReLUU non-linear activations and 64 dimensional hidden units, followed by summation of the transformed neighboring node features. The ouptut node features a concatenated with the output feature from the encoder network. This concatenated features the input to the two out action network. The CANG))) network doesn't communicate the robot's capabilities with the graph convolution layers. Rather, the capabilities are appended to the output of the encoder network and output of node features of the graph convolution layer just before the the action network. Thus, the the action network is the only part of this model that is conditioned on robot capabilities.
E.3 Policy Training Hyper parameters
We detail the hyperparameters used train each of the policies using proximal policy optimization (PPO) [37] in Table 3.
Hyperparameter
Value
Action Selection (Training)
soft action selection
Action Selection (Testing)
hard action selection
Critic Network Update Interval
200 steps
Learning Rate
0.0005
Entropy Coefficient
0.01
Epochs
Clip
0.2
Q Function Steps
Buffer Length
Number of training steps
x 10 (HMT), 20
10t (HSN)
Table 3: Hyperparameters used to train each of the policies with PPO.
19


Reviews:
[{"review_id": "h4GSR9sI2L", "reviewer": "Reviewer_jEiU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper investigates whether communication of robot capabilities can improve the generalization of RL-based multi-robot policies to different robot-team sizes and robot-type compositions.  To this end, the paper develops a decentralized policy architecture with parameter sharing that allows robots to use their capabilities as implicit role indicators within the multi-robot team.  The role of the policy is to enable the robots to reason about individual robots' capabilities and team capabilities. The proposed approach is evaluated in both simulator and hardware experiments of active sensing tasks with multiple robots.", "summary_of_recommendation": "The paper proposes an excellent problem and a method that appears promising. In my opinion, the current version of the paper would benefit from additional experiments, across a wider range of team sizes and tasks, to better support the paper's conclusions about the generalization capacity of the proposed algorithm.\n\nUPDATE: In light of the experimental results provided during the rebuttal process, which they show that both capability and communication awareness may not necessarily increase perfromance, I keep the \"Weak Accept\" recommendation.  I look forward to future iterations of the proposed algorithm.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please see all my comments in the weakness section above, repeated here:\n\n\n-  I would elaborate in the introduction what is the role of the task at hand in the learning process, and its impact on the generalization capacity of the learned policies:  How does the generalization capacity get affected when the task changes, for example, from collaborative mapping to multi-target tracking?  How do the robots can \"link\" the current awareness of the robots' capabilities to the needed capability compositions for the new task?\n\nAnswering the above questions may be outside the scope of the paper.  If this is the case, then I would state early on in the paper (or in the limitations) that the focus of the paper is to achieve generalization given a task, and not across tasks.  I believe such a clarification will help readers to understand the paper's scope.\n\n-  In my opinion, additional experiments, across a wider range of team sizes and tasks, could strengthen the paper's conclusions about the generalization capacity of the proposed algorithm.  In more detail:\n\n(i) Fig. 3, which is used to conclude that awareness and communication of capabilities do not hurt performance, is based on experiments over a single task that involves 4 robots only.  In my opinion, additional experiments, across different tasks and more robots, may be needed to substantiate the conclusion that awareness and communication of capabilities do not hurt performance.  Or, could the paper be revised to elaborate on why the presented set of experiments is enough to support the conclusion?\n\n(ii) Fig. 4 demonstrates a nice generalization to robot teams of 3, 4, and 5 robots, given training on teams of 4 robots.  In my opinion, a wider range of team sizes could be used to demonstrate the generalization capacity of the proposed method (e.g., up to 10 or more robots per team).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n\n-  The paper identifies a learning problem of interest to the CoRL community. \n-  The proposed method is tested on hardware, using the Robotarium multi-robot test bed.\n-  The proposed method is observed to outperform current methods that are capabilities *un*aware.\n\nWeaknesses\n\n-  I would elaborate in the introduction what is the role of the task at hand in the learning process, and its impact on the generalization capacity of the learned policies:  How does the generalization capacity get affected when the task changes, for example, from collaborative mapping to multi-target tracking?  How do the robots can \"link\" the current awareness of the robots' capabilities to the needed capability compositions for the new task?\n\nAnswering the above questions may be outside the scope of the paper.  If this is the case, then I would state early on in the paper (or in the limitations) that the focus of the paper is to achieve generalization given a task, and not across tasks.  I believe such a clarification will help readers to understand the paper's scope.\n\n-  In my opinion, additional experiments, across a wider range of team sizes and tasks, could strengthen the paper's conclusions about the generalization capacity of the proposed algorithm.  In more detail:\n\n(i) Fig. 3, which is used to conclude that awareness and communication of capabilities do not hurt performance, is based on experiments over a single task that involves 4 robots only.  In my opinion, additional experiments, across different tasks and more robots, may be needed to substantiate the conclusion that awareness and communication of capabilities do not hurt performance.  Or, could the paper be revised to elaborate on why the presented set of experiments is enough to support the conclusion?\n\n(ii) Fig. 4 demonstrates a nice generalization to robot teams of 3, 4, and 5 robots, given training on teams of 4 robots.  In my opinion, a wider range of team sizes could be used to demonstrate the generalization capacity of the proposed method (e.g., up to 10 or more robots per team).", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "btrTv-Wd4bI", "reviewer": "Reviewer_TgTL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper tackles the problem of training policies that control multiple robots simultaneously, and is specifically interested in the case of 0-shot generalization to novel robot team configurations. The proposed approach aims to use awareness and communication of robot capabilities in order to learn a more robust approach. The paper proposes to use a graph network where each node is a robot and each edge is a communication link. The robot's observation are augmented with a vector of its capabilities (payload, sensor, etc). The method is evaluated in sim and real robotarium task where the robots must form a dense sensor network, taking into account each robot's sensor radius. The results show that the graph network and capability communication help the policy generalise to new robot configurations", "summary_of_recommendation": "This is an interesting paper that investigates how enabling robots to use and communicate their capabilities enables generalization to novel robot teams. The evaluation is on a fairly limited set of environments, but the results pan out.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- The graph edges are \"communication links\". What does this mean in practice? How does performance scale with graph connectivity?\n\n\nNits:\n- Figure 3 is difficult to read. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The paper is written clearly.\n- The experiment baselines are thorough\n- The proposed method is sound, interesting, and to my knowledge this is a novel application to multi-robot policies. \n\nWeaknesses:\n- The results are only on one kind of task, where only one of the capabilities matters. While this is a good proof of concept, it would be more impactful to push the method further with more complex task/capability relationships.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "WO1YvbJc_U", "reviewer": "Reviewer_FZZg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper focuses on improving the generalization of multi-agent RL in terms of different team compositions, team sizes, and different robot capabilities. The author proposes to include robots’ capability in the observation space and use graph networks to enhance communication. The experiments show that the proposed method is effective in improving the generalization of the baselines.", "summary_of_recommendation": "The paper presents a method for improving generalization in multi-agent RL. The contribution is relatively minor but it does show improvement against reasonable baselines. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please improve the figures and include additional experiments on more than one dimension of robot capabilities.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n* The paper is overall clearly written.\n* Improving generalization across heterogeneous robots is an important topic.\n* The experiments are pretty comprehensive, including real robot results from Robotarium.\n\nWeaknesses:\n* As mentioned in the limitation section, only one dimension of robot capability is tested. \n* The figures in the experiments need improvement - see below for details. \n  * The legends and labels in Figure 3,4,5 are way too small.\n  * How is the variance/std calculated in the figures?\n  * Figures should be placed closer to the text that describes the corresponding figures.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "LWB1U1TEqA", "reviewer": "Reviewer_foRa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a new algorithm for multi-agent reinforcement learning. The authors propose adding the “capability” of each robot as additional information in MARL, so that the system has capability awareness (each robot knows its own capability) and capability communication (each robot knows about the capabilities of the team). The authors show that this approach significantly improves team performance in a real-world multi-robot test bed (the Robotarium) when generalizing to different team compositions, robot capabilities, and numbers of robots.", "summary_of_recommendation": "**Post-Rebuttal Update**: I am keeping my score as a Weak Accept. The authors' rebuttal acknowledges the lack of algorithmic novelty w.r.t. augmented state information.\n\nThe paper is well-executed, well-written, and technically sound, but I would like the authors to clarify how their approach differs from augmented state information.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please see weaknesses above, in particular the first bullet point.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths: \n- The paper is very well-written and exceptionally clear.\n- The motivation is compelling (generalizing to heterogeneity in multi-robot teams without re-training).\n- The evaluation is thorough, with systematic investigation of the impact on training and generalization at test-time and a thorough set of baselines.\n- Quantitative results are compelling, showing clear benefits of the proposed method.\n- Qualitative results are compelling, showing real robots performing the task in accordance with intuition (maximizing surface area while maintaining connection).\n\nWeaknesses:\n- The main weakness is that the proposed approach has limited technical novelty. If I understand correctly, one could argue that the algorithm is equivalent to simply augmenting the robot state with the “capability” information. It is then not surprising that critical state information (the sensing radius in the experiment) is instrumental in solving the multi-agent task, and that the learned policy can generalize to different values of the augmented state. Treating this information as “capability” rather than state seems like mere semantics.\n- The task is relatively simple in nature and in a single domain. As the authors acknowledge in the limitations, they only test the case where the capability vector is a scalar. The robots are also all identical in the experiment (in terms of hardware rather than capability).\n- It seems that while the policy is able to generalize to new *values* of capabilities, it’s unable to generalize to entirely new capabilities, since the capability space C must be known beforehand.\n\nMinor notes:\n- “Heterogeneity” is later defined as variation in robot behavior and hardware, but it could be helpful to define this term upfront in the introduction.\n- Some additional experiments that are not necessary but could be helpful include (1) ablations justifying the CTDE paradigm and (2) testing the impact of variations in the communication graph structure.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the original submission, the rebuttal by the authors and the discussion that followed, the reviewers agree that this paper should be accepted to CoRL. 
To the authors: great job on writing a strong rebuttal and engaging with the reviewers during the discussion period! Please address the remaining comments in the camera-ready version of the paper.
Decision: Accept (Poster)