EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents

Authors: Junting Chen, Checheng Yu, Xunzhe Zhou, Tianqi Xu, Yao Mu, Mengkang Hu, Wenqi Shao, Yikai Wang, Guohao Li, Lin Shao
Primary area: applications to robotics, autonomy, planning
Keywords: Embodied Artificial Intelligence, LLM Multi-agent System, Multi-robot System, Task Planning

Abstract:
Heterogeneous multi-robot systems (HMRS) have emerged as a powerful ap-
proach for tackling complex tasks that single robots cannot manage alone. Current
large-language-model-based multi-agent systems (LLM-based MAS) have shown
success in areas like software development and operating systems, but applying
these systems to robot control presents unique challenges. In particular, the ca-
pabilities of each agent in a multi-robot system are inherently tied to the physical
composition of the robots, rather than predefined roles. To address this issue,
we introduce a novel multi-agent framework designed to enable effective collab-
oration among heterogeneous robots with varying embodiments and capabilities,
along with a new benchmark named Habitat-MAS. One of our key designs is
Robot Resume: Instead of adopting human-designed role play, we propose a self-
prompted approach, where agents comprehend robot URDF files and call robot
kinematics tools to generate descriptions of their physics capabilities to guide
their behavior in task planning and action execution. The Habitat-MAS bench-
mark is designed to assess how a multi-agent framework handles tasks that require
embodiment-aware reasoning, which includes 1) manipulation, 2) perception, 3)
navigation, and 4) comprehensive multi-floor object rearrangement. The experi-
mental results indicate that the robot’s resume and the hierarchical design of our
multi-agent system are essential for the effective operation of the heterogeneous
multi-robot system within this intricate problem context.

Decision: accept (poster)
Reviewer scores: 6, 8, 8, 5

Reviews and discussion:
<thread>
creation date:
1729382107938

thread_type:
official_review

<thread_object>
creator: reviewer_xzuu

<content>
summary:
This paper proposed a heterogeneous multi-robot system framework to enable effective collaboration among heterogeneous robots with varying embodiments and capabilities. However, from my perspective, the topic of this paper is too big, making it more like a technical instruction or report rather than a research paper. Moreover,  many details are not clearly explained, such as low-level planning and control (collision avoiding), mid-level decision-making, and high-level learning. Moreover, the communication graph is an important part of the MAS design, which is also not mentioned in the framework. I suggest the authors reorganize the paper, abstract the model, and then discuss it from theoretical and practical perspectives.

strengths:
This paper introduced a heterogeneous multi-robot system framework to enable effective collaboration among heterogeneous robots with varying embodiments and capabilities. It implements the proposed LLM-based multi-agent framework in indoor household environments with four tasks. The paper presents the LLM-based MAS framework conducting embodiment-aware reasoning in simulation.

weaknesses:
From my perspective, the topic of this paper is too big, making it more like a technical instruction or report rather than a research paper. Many details are not clearly explained, such as low-level planning and control (collision avoiding), mid-level decision-making, and high-level learning. Moreover, the communication graph is an important part of the MAS design, which is also not mentioned in the framework. Furthermore, several robots' plannings have apparent errors in the simulation video. For example, at the 16 seconds of the video, the drone does not follow the routine but goes through the wall directly, etc. I hope the authors can improve it and consider more details about different modules in the framework design.

questions:
I suggest the authors reorganize the paper, abstract the model and ideas, and then discuss it from theoretical and practical perspectives, 
such as how to formularize the capabilities of heterogeneous multi-robot systems to satisfy various tasks' requirements, how to define the specific roles and relationships in the group, how to organize their behaviors or strategies to optimize system performances, etc.  

I suggest the authors check and refer the paper as below:

1) Yang, Q., & Parasuraman, R. Hierarchical needs based self-adaptive framework for cooperative multi-robot system. In 2020 ieee international conference on systems, man, and cybernetics (smc) (pp. 2991–2998). IEEE.

2) H. Hamann and H. Wo ̈rn, “A framework of space–time continuous models for algorithm design in swarm robotics,” Swarm Intelligence, vol. 2, no. 2-4, pp. 209–239, 2008.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730350244067

thread_type:
official_review

<thread_object>
creator: reviewer_jjgy

<content>
summary:
The paper presents an interesting approach to LLM-based multi-robot systems where robot roles are not predefined but are instead determined dynamically based on the robots' physical capabilities. The authors introduce the concept of a "robot resume," generated from robot URDF files and kinematic data, to guide task planning and integrate role flexibility within the model.

This concept is very compelling; however, my main concern is the additional degree(s) of freedom introduced into the decision-making and planning. In systems with predefined/assumed heterogeneous roles, the answer to “who does what” is partially predetermined, simplifying the planning process. Here, this isn’t the case, and the current centralized hierarchical LLM-based planning and task assignment system becomes significantly constrained. With the current design, the centralized planner and task assignment modules will face considerable burden in optimizing plans and assignments, particularly as domain complexity increases or when dealing with similar or more universally capable robots. I suggest comparing your approach against systems with predefined roles to illustrate any potential advantages or limitations. Additionally, you can provide a quantitative analysis of how your system's performance scales with increasing numbers of robots, robot classes, or task complexity (see [1] for example).

[1] Seraj et al. "Learning efficient diverse communication for cooperative heterogeneous teaming", AAMAS 2022

One potential solution to mitigate this issue could involve incorporating prior knowledge to streamline planning and decision-making. For example, a standard quadcopter lacks object manipulation capabilities, so it can be automatically excluded from relevant tasks, reducing the computational load. Such prior knowledge could be represented as a context/class vector for a contextual classification. Additionally, other factors such as robot availability (i.e., whether a robot is still executing a prior task) could contribute to a truly asynchronous multi-robot system.

I also have a few questions:
- Can you discuss the modularity of this approach? For instance, can you describe the process for adding a new capability like battery life to the robot resume and how it would be integrated into the task planning and allocation process. This would give a clearer picture of the system's modularity and extensibility.
- Do LLM-based planners consider assignment risks, and if so, how can these risks be leveraged in the decision-making process?
- It remains unclear how the central planner optimizes "which robot does what." For example, how does the system handle scenarios with (1) multiple equally capable robots, (2) differently capable robots that can all execute a task, or (3) universally capable robots? Does the LLM-based “discussion” incorporate strategic negotiation or cognitive hierarchy mechanisms (e.g., k-level thinking) to optimize plans and assignments? Can you provide a specific example scenario for each of these cases and explain how the system would handle the allocation in each case?
- Given the dynamic nature of the environment, assuming full observability is unrealistic. However, it is also unclear how partial observability is handled in this system. Can you discuss how your system might be adapted to handle partial observability, and what challenges you anticipate in doing so?

Additionally, I suggest including a section on Heterogeneous Multi-Robot Systems literature rather than the current Multi-Robot Systems section. Given the context of this work, it would be more impactful to emphasize on the heterogeneity in collaborative multi-robot teams, categorizing and introducing previous approaches such as MARL (heterogeneous multi-robot communication and coordination), Multi-Agent Apprenticeship Learning (learning heterogeneous multi-robot policies from human demonstrations), trait-based heterogenous multi-robot planning, and other non-LLM-based strategies. Below are a list of papers to begin with:

[1] "Learning efficient diverse communication for cooperative heterogeneous teaming", AAMAS 2022

[2] "Heterogeneous Multi-Robot Reinforcement Learning", AAMAS 2022

[3] "STRATA: A Unified Framework for Task Assignments in Large Teams of Heterogeneous Agents", JAAMAS 2020

[4] "Mixed-initiative multiagent apprenticeship learning for human training of robot teams", NeurIPS 2023

[5] "Heterogeneous policy networks for composite robot team communication and coordination", T-RO 2024

Lastly, a minor but important point: the paper contains numerous grammatical and verbal errors. A thorough review is recommended. 

Overall, I'm happy with the contributions made in this paper and vote for a weak accept. I'd be happy to raise my score given mine and my fellow reviewers' comments are adequately addressed. Thanks!

strengths:
-- See above.

weaknesses:
-- See above

questions:
-- See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730516827325

thread_type:
official_review

<thread_object>
creator: reviewer_79k2

<content>
summary:
This paper introduces a novel framework for controlling heterogeneous multi-agent collaboration, named EMOS. This framework integrates recent advancements in large language models (LLMs) and multimodal models, enabling intelligent cooperation among multiple robots. Unlike previous works, which primarily relied on role-playing approaches, this multi-agent framework innovatively proposes a robot resume based on the robot's URDF files. This enhancement allows for a more precise description of the robots' capabilities, thereby facilitating more efficient communication and collaboration among them. Additionally, the authors present a new benchmark named Habitat-MAS. This benchmark validates the framework's effectiveness. The paper also provides several specific experimental results, which have been published on a dedicated website.

strengths:
Robots’ resumes are generated by the LLM based on urdf files, which enhance the communication among robots. This paper more effectively leverages the intelligence of LLMs in addressing the challenges of heterogeneous multi-robot collaboration. LLM can not only play a planner role but also promote the communication effectiveness among robots.

weaknesses:
The experimental section of the paper requires additional enhancements. While the authors conducted ablation studies on their framework and provided detailed results, there is a notable absence of comparisons with similar frameworks.

questions:
1）In the domain of heterogeneous multi-agents, how does the proposed framework's use of robot resumes improve performance compared to traditional role-playing methods? Moreover, does the framework consistently outperform others across various scenarios and different robot configurations? 
2）All robot are completely interconnected, and they are collaborative under complete information. Can the proposed framework be extended to scenarios where there is a game relationship between heterogeneous individuals.
3)  What is the scalability of the framework? How does the increase in the scale of collaborative robots affect the success rate of tasks?
4） The format of resumes is artificially defined. Will the format of resumes affect the efficiency of robot assistance? How to generate a better formatted resume?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730572076622

thread_type:
official_review

<thread_object>
creator: reviewer_5wca

<content>
summary:
This paper presents the Embodiment-Aware Heterogeneous Multi-Robot Operating System (EMOS), an LLM-driven, multi-agent system designed to manage diverse robots in complex household tasks.

strengths:
The paper is well-written and explores an interesting application.  In essence, it describes a system with multiple robots, each equipped with an LLM and distinct capabilities. The robots communicate to determine task distribution, using robot resumes to identify which robot is best suited for each task. Through hierarchical and decentralized planning, they collaborate to complete complex tasks in a shared environment, leveraging spatial reasoning and embodiment awareness to enhance coordination.
1) The main contribution appears to center around how questions are passed to the pre-trained LLM for task allocation in a multi-robot system.  Authors proposed a robot resume method, that creates a dynamic capability profile for each robot. This profile includes an understanding of each robot's URDF files, enabling the system to call upon robot kinematics tools to generate detailed descriptions of their physics capabilities.
2) Habitat-MAS simulation: It is designed for multi-agent systems, facilitating complex task management using heterogeneous team of robots.

weaknesses:
Presenting results is not a contribution. Instead, it serves as a validation that the proposed methodology works as intended and is better than baselines.

questions:
I don't have any questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732369335487

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Update of Revision V1

comment:
Dear reviewers,

We greatly appreciate all the constructive comments from the reviewers. We have revised our paper according to the suggestions. The updates are as follows:
- We reorganized the `Section 2. Related Works` and their relations to our paper as suggested by **Reviewer jjgY** and **Reviewer xZUU**. Specifically, we divided the "Multi-Agent System" subsection into two sections "LLM-Based Multi-Agent System" and "Heterogeneous Multi-Agent Learning". we also updated the "multi-robot systems" section with new references. All the mentioned related works by these two reviewers have been added to the reference list. 
- We added `Appendix A.2 MULTI-AGENT SYSTEM DESIGN AND COMMUNICATION` according to the request by **Reviewer xZUU** to add the agents' communication graph
- We added `Appendix B.4 DETAILED IMPLEMENTATION OF ROBOT LOW-LEVEL CONTROL IN BENCHMARK` according to the request by **Reviewer xZUU**. 
- We added `Appendix C.5 EXTRA EXPERIMENT ON FORMAT OF ROBOT RESUME` according to the question by **Reviewer 79K2**

We have highlighted all the revised/added parts with blue color for the reviewers' convenience. We will remove the colorization in the formal version. We are also looking forward to reviewers' feedback on this revision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes EMOS, a framework that uses large language models (LLMs) to dynamically coordinate the behavior of teams of heterogeneous robots according to their physical design. EMOS generates a "robot resume" based upon a robot's kinematics (as determined by its URDF) that it then uses to determine their roles. Additionally, the paper proposes a new benchmark (Habitat-MAS) for evaluating multi-agent coordination for tasks that involve manipulation, navigation, and object rearrangement within a multi-floor building.

The paper was reviewed by four referees who largely agreed on the paper's strengths an weaknesses. Among the strengths, several reviewers appreciated the use of LLMs to dynamically allocate tasks across robot teams based upon their kinematics, as opposed to the more traditional approach of pre-defining their roles. At the same time, several reviewers shared the concern that the paper read more like a project report that described a particular prompting strategy, as opposed to a research paper, and that it lacked several key details about the framework. In their response to the reviewers, the authors addressed the need for more detailed explanations of EMOS, however they did not speak to the concern about the presentation style. Meanwhile, at least two reviewers emphasized the need to compare against traditional role playing approaches, while one asked for an evaluation of how the method scales as the number of robots increases and the tasks become more complex. The authors clarified that the initial submission already provided a comparison to role playing approaches, while they provided an additional analysis of scalability during the rebuttal period.