Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence
Authors: Weize Chen
Ziming You
Ran Li
yitong guan
Chen Qian
Chenyang Zhao
Cheng Yang
Ruobing Xie
Zhiyuan Liu
Maosong Sun
Venue: ICLR 2025 Spotlight (2025)
Keywords: llm agent
multi-agent
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose IoA, a novel framework inspired by the Internet for effective collaboration among diverse LLM agents. IoA enables autonomous conversation flow, integration of heterogeneous agents, etc. It outperforms SoTA baselines in various tasks.

Abstract: The rapid advancement of large language models (LLMs) has paved the way for the development of highly capable autonomous agents. However, existing multi-agent frameworks often struggle with integrating diverse capable third-party agents due to reliance on agents defined within their own ecosystems. They also face challenges in simulating distributed environments, as most frameworks are limited to single-device setups. Furthermore, these frameworks often rely on hard-coded communication pipelines, limiting their adaptability to dynamic task requirements. Inspired by the concept of the Internet, we propose the Internet of Agents (IoA), a novel framework that addresses these limitations by providing a flexible and scalable platform for LLM-based multi-agent collaboration. IoA introduces an agent integration protocol, an instant-messaging-like architecture design, and dynamic mechanisms for agent teaming and conversation flow control. Through extensive experiments on general assistant tasks, embodied AI tasks, and retrieval-augmented generation benchmarks, we demonstrate that IoA consistently outperforms state-of-the-art baselines, showcasing its ability to facilitate effective collaboration among heterogeneous agents. IoA represents a step towards linking diverse agents in an Internet-like environment, where agents can seamlessly collaborate to achieve greater intelligence and capabilities. We will release our code to facilitate further research.

Published as a conference paper at ICLR 2025
INTERNET OF AGENTS: WEAVING A WEB OF HET-
EROGENEOUS AGENTS FOR COLLABORATIVE INTEL- LIGENCE
Weize Chen'*, Ziming You Raan Li Yitong Guan Chen Qian', Chenyang Zhao' Cheng Yang", Ruobing Xie Zhiyuan Liu MMasong Sun! Tsinghua University, 2 Peking University 3 Beijing University Posts and Telecommunications Tencent chenwz21emails t....siiig tswnggua....... edu. en, ran. fan.11572482ggmal... {2 (zimingyou, 210 2101210206/estu.pka.
en, liuzy@tsinghuu edu.cn
ABSTRACT
The rapid advancement of large language models (LLMMs) has paved the way for the development of highly capable autonomous agents. However, existing multi-agent frameworks often struggle with integrating diverse capable third-party agents due to reliance agents defined within their ecosystems. They also face challenges in simulating distributed environments, as most frameworks are limited to single-device setups. Furthermore, these frameworks often rely hard-coded communication pipelines, limiting their adaptability to dynamic task requirements. Inspired by the concept of the Internet, propose the Internet of Agents (IoA), novel framework that addresses these limitations by providing a flexible and scalable platform for LLM-based multi-agent collaboration. IoA in- troduces an agent integration protocol, an mestatt-essaging-iee sign, and dynamic mechanisms for agent teaming and conversation flow control. Through extensive experiments general assistant tasks, embodied AI tasks, and aetrieval-ugmmented generation benchmarks, we demonstrate that IoA consistently outperforms state-of-the-an baselines, showcasing its ability to facilitate effective collaboration among heterogeneous agents. IoA represents step towards link- ing diverse agents in miterne-llike environment, where agents seamlessly collaborate to achieve greater intelligence and capabilities!
architecture de-
INTRODUCTION
The Internet has revolutionized collaboration and knowledge-sharing, connecting people with di- verse skills and backgrounds from around the world. This global network has enabled remarkable projects like Wikipedia and the development of Linux, achievements that would have been impos- sible for any single individual. By fostering collaboration, the Internet pushes the boundaries of human achievement, turning the once-impossible into reality.
The success of the Internet in enabling human collaboration raises an intriguing question: can we create a similar platform to facilitate collaboration among autonomous agents? With the rapid ad- vancements LLMs (OpenAl, 2023 Reid al.) 2024). we now have autonomous agents capable of achieving near-human performance
a wide range of tasks. These LLMM-bssed agents have demonstrated the ability to break down complex tasks
executable steps, leverage various tools, and learn from feedback and experience (Qin et al.) 2023 Wang et al.) 2023c Shinn et al.) 2023 Qian et al. 2023b). As the capabilities of these agents continue to grow, and with an increasing
number of third-party agents with diverse skills consistently emerging (Chase) 2022 Team 2023
Significant Gravitas 2023 Open Interpreter (003). it crucial to explore how we can effectively and efficiently orchestrate their collaboration, just the Internet has done for humans.
Equul Contribution.
R Corresponding author.
httpp:://giii thu.. com www.:///w.pp..////e/ee/om

---

Published as a conference paper at ICLR 2025
To address this challenge, we propose the concept of the Internet of Agents (IoA), a general frame- work for agent communication and collaboration inspired by the Internet. IoA aims to address three fundamental limitations of existing multi-agent systems (MAS) (Chen et al. 2023 Wu et al. 2023 Hong et al. 2023] Qian et al. 2023a): (1) Ecosystem Isolation: Most frameworks only con- sider agents defined within their own ecosystems, potentially blocking the integration of various third-party agents and limiting the diversity of agent capabilities and the platform's 's generality; (2) Single-Device Simulation: Nearly all MAS simulate MAS on single device, which differs signif- icantly from real-world scenarios where agents could be distributed across multiple devices located in different places; (3) Rigid Communication and Coordination: The communication process, agent grouping, and state transitions are mostly hard-coded, whereas in real life, humans decide on teammates based on the task at hand and dynamically switch between collaboration states.
To overcome these limitations, we propose agent integration protocol that enables different third- party agents running on different devices to be seamlessly integrated into the framework and collabo- rate effectively. Additionally, we introduce an mnstant-messaging--e -ap--al---ii---iieeeeaaa framework that facilitates agent discovery and dynamic teaming. By autonomously searching for potential agents capable of handling the tasks at hand, agents dynamically decide form different teams and communicate within various group chats. Inspired by Speech Act Theory (Searle) 1969). and its application in conventional MAS (Finin al., 1994, Labrou et al. 1999). within each group chat, abstract out several conversation states and provide a flexible and general finite-state machine mechanism to support autonomous state transition for agents.
We demonstrate the effectiveness of IoA through extensive experiments and comparisons with state- of-the-art autonomous agents. By integrating AutoGPT (Significant Gravitas 2023) and Open Inter- preter (Open Interpreter] 2022), we show that IoA achieves a win rate of 66 to 76% in open-domain task evaluations when compared to these agents individually. Furthermore, with only a few basic ReAct agents integrated, IoA outperforms previous works on the GAIA benchmark (Mialon et al.) 2023). In the aegrentll-ugmeented generation (RAG) question-answeriine domain, our framework substantially surpasses existing methods, with GPT-3355 implementation achieving perfor- mance close to or even exceeding GPT-4, and effectively surpassing previous MAS.
The impressive performance of IoA across various domains highlights the potential of this paradigm for autonomous agents. As smaller LLMs continue to advance (Mesnard et al. 2024; Hu et al.) 2024 Abdin et al. 2024), running agents on personal computer even mobile device is becoming increasingly feasible. This trend opens up new opportunities for deploying MAS in real-world
scenarios, where agents can be distributed across multiple devices and collaborate to solve complex problems. We believe that by further exploring and refining the loA paradigm, more sophisticated and adaptable MAS can be developed, ultimately pushing the boundaries of what autonomous agents can achieve in problem-solving and recision-making
FRAMEWORK DESIGN AND KEY MECHANISMS OF IoA
2.1 OVERVIEW OF IoA
platform that enables seamless communication and collaboration among diverse autonomous agents. Inspired by the concept of Internet, IoA ad- dresses three fundamental challenges in MAS (Chen et al. 2023 Wu et al. 2023 Qian et al. (022a)): (1) Distributed agent collaboration: Unlike traditional Trameworks that simulate MAS on a sin- gle device, IoA supports agents distributed across multiple devices and locations. (2) Dynamic and adaptive communication: IoA implements mechanisms for autonomous team formation and conversation flow control, allowing agents to adapt their collaboration strategies based on task re- quirements and ongoing progress. (3) Integration of heterogeneous agents: IoA provides a flexi- ble protocol for integrating various third-party agents, expanding the diversity of agent capabilities within the system. To better illustrate the difference between IoA and other MAS, see Appendix A for comparative analysis of key aspects of different MAS.
IoA is designed as an instant-messsging-app
At its core, IoA consists of two main components: the server and the client. The server acts as a central hub, managing agent registration, discovery, and message routing. It enables agents with varying capabilities to find each other and initiate communication. The client, on the other hand, serves wrapper for individual agents, providing them with the necessary communication func-
2

---

Published as a conference paper at ICLR 2025
Client Side
Server Side
Agent Message Protocol
Interaction Layer
Team Formation Block
Commuricatin Block
Message Routing
Group Setup
Agent Query
Data Layer
Agont Contact
Block
Group Block
Session Management Block
Ageni Registry Block
Agent Register
Foundation Layer
Data Infra Block
Data Infta Black
Agent integration Block
Security Block
Network Intra Block
Network Infra Block
Figure 1:
The illustration on the conceptual layered architecture on the design of IoA.
tionalities and adapting them to the specified protocol. IoA employs a layered architecture BBass
al. 1999) for both the server and client components, comprising three layers:
Interaction Layer: Facilitates team formation and agent communication.
Data Layer: Manages information related agents, group chats, and tasks. Foundation Layer: Provides essential infrastructure for agent integration, data management, and network communication.
These layers work together to facilitate agent collaboration through the network. In the following subsections, we will go through the IoA's 's architecture and design.
2.2 ARCHITECTURE OF IOA
The layered architecture IoA is designed to support scalable, flexible, and efficient multi-agent collaboration. This architecture enables clear separation of concerns and facilitates the integration of diverse agents and functionalities (Fig. T).
2.2. SERVER ARCHITECTURE
The server acts as the central hub of IoA, facilitating agent discovery, group formation, and mes- sage routing. architecture consists of three layers. manages high-level interactions between agents and the system. It encompasses the Agent Query Block for enabling agents to search for other agents based on specific characteristics, the Group Setup Block for facilitating the creation and management of group chats, and the Message Routing Block for ensuring efficient and accurate routing of messages between agents and group chats. The Data Layer Serves the information backbone, it handles the storage and management of critical system information. The Agent Registry Block maintains a comprehensive database of registered agents, including their capabilities and current status, similar service discovery in distributed sys- tems (Meshkova et al. [2008; Netflix.. Mcanwhile, the Session Management Block manages active connections and ensures continuous communication between the server and connected clients. The Foundation Layer provides the essential infrastructure for the server's operations. It encompasses the Data Infrastructure Block for handling data persistence and retrieval, the Network Infrastructure Block for managing network communications, and the Security Block for implementing authentica- tion, authorization, and other security measures to maintain system integrity.
top level, the Interaction Layer
2.2.2 CLIENT ARCHITECTURE
The client component of IoA serves as a wrapper for individual agents, providing them with the nec- essary interfaces to communicate within the system. Its architecture mirrors that of the server with three layers. The Interaction Layer manages the agent's interactions within the system. The Team Formation Block implements the logic for identifying suitable collaborators and forming teams for the task at hand, similar coalition formation in conventional multi-agent research (Rahwan al.) 2009). Complementing this, the Communication Block manages the agent's participation in group chats and handles message processing. The Data Layer maintains local data relevant the agent's 's operations. It includes the Agent Contact Block for storing information about other agents the cur- rent agent has interacted with, the Group Info Block for maintaining details about ongoing group
3

---

Published as a conference paper at ICLR 2025
chats and collaborations, and the Task Management Block for tracking the status and progress of tasks assigned to the agent. The Foundation Layer provides the basic functionalities for the client's operations. The Agent Integration Block defines the protocols and interfaces for integrating third- party agents into the loA ecosystem. Alongside this, the Data Infrastructure Block handles local data storage and retrieval, while the Network Infrastructure Block manages network communica- tions with the server.
This layered architecture enables IoA to support wide range of agent types and collaboration sce- narios. By providing a clear separation of concerns and well-defined interfaces between layers, the architecture facilitates the integration of diverse agents and allows for future extensibility. Further- more, this design supports the key mechanisms of IoA, as will be introduced in the next section.
2.3 KEY MECHANISMS
The effectiveness of IoA relies several key mechanisms that enable seamless collaboration among diverse agents. These mechanisms work concert to facilitate agent integration, team formation, task allocation, and structured communication. We detail these critical components in this section, with an example walk-through provided in Appendix B for better understanding.
2.3.1 AGENT REGISTRATION AND DISCOVERY
To enable collaboration among distributed agents with heterogeneous architectures, tools, and en- vironments, we propose the agent registration and discovery mechanism. This mechanism forms the foundation for collaborative interactions within IoA, enabling the integration of diverse agents into the system and facilitating their discovery on the online server by other agents for potential collaboration through the network.
When a new agent joins IoA, its client wrapper registers with the server by providing a comprehensive description of its capabilities, skills, and areas of expertise. This de- scription, denoted as d; for an agent c, is stored in the Agent Registry Block of the server's Data Layer. Formally, we represent the set of all registered agents as C {c1, C2, J. where each ea is associated with its description dj.
Agent Registration:
The agent discovery function leverages the information stored in the Agent Reg- istry from the online server to enable agents to find suitable collaborators for specific tasks. Agents can use the searchacl cintt tool provided by the server's Agent Query Block to search for other agents based desired characteristies or capabilities. Formally, let La [4, 1,.,2,...44 4x] be list of desired characteristics generated by an agent seeking collaborators. The search el ient function can be represented The function returns subset of clients CJ C C. whose descriptions d, match the desired character- isties in L.. The matching process between Ca and d; can be implemented with various semantic matching techniques (Robertson & Zaragoza, 2009 KKrrukhhn et al. 2020). It ensures that agents with relevant capabilities can be discovered even if their descriptions do not exactly match the query.
Agent Discovery:
search_cl c__ ient La - P(C), where P(C) denotes the power set of C.
2.3.2 AUTONOMOUS NESTED TEAM FORMATION
The autonomous nested team formation mechanism enables dynamic and flexible combinations of agents based on task requirements, allowing for the creation of nested sub-teams for complex tasks.
When a client G E C is assigned task f, it initiates the team formation process using two essential tools provided by the server: search. wwweeh.clieee client and launch group chat. The client's LLM decides which tool to call based on the task and the current set of discovered clients. If more collaborators are needed, it calls search_se int with appropriate characteristics. Once suitable collaborators are found, it calls launch.group.chat to initiate a new group chat g E G. where G is the space of all group chats.
Team Formation Process:
group
Nested Team Structure: The nested team formation allows for hierarchical structure of teams and sub-teams. Let Go E G be the initial group chat for task t. During the execution of t, client c is assigned with sub-task t; (the task assignment mechanism will be introduced in Section2.3..4). and it identifies ty requires additional expertise, c; is allowed to search for appropriate agents again and initiate new sub-group chat gi G. This process can continue recursively,
4

---

Published as a conference paper at ICLR 2025
forming tree-like structure of group chats. Formally, can define function h 9 P(g) that maps a group chat to its set of sub-group chats. The nested structure can be represented as: ......99 Pin and
h(g) {g1. 92, .... 9m h(g.) {ga. 9:2.
Communication Complexity: The nested team formation mechanism can theoretically reduce
Ovenal Godl: Craate Conpraletthre
Market Andlysa Repert for ihone Agantyy
Che
the communication complexity in large agent
Googengant.
teams in terms of communication channels. As-
sume that the minimal agent setting capable of completing given task is g with lgl mem- bers. Consider two types of team structures: a fully connected team structure and a nested team structure (h(g) (91.92,.... 9m where i 9j = g. In the first scenario, communica- tion channels (connected edges) are (g) (gb,9))00 < i,j S m) with (g)(g)
Sab Harket Research
Data Andylli
Collection
Bak-Tock Ropert Wrilin
Aalgges
Aesignee
Audigne diPleperinn isgepgr Nached tear Formaliont X
Marted Teare Pormationita
Ovensi God: Harkat
Research end Data Collectio
Ohd
Agests
Gooplegget:
tak-Toch Competiter Andlyia
Sub Tesl-e Castumer Realysis
(g)n(g) = -11 For the Nested Team Structure, it necessarily follows that Guested gg Cau(gg)) I (g)( Fig. 2 illustrates an example of the nested team formation pro- cess. In this example, the initial group chat go spawns three sub-group chats 91 92 and Ja for specific sub-tasks during the discussion. 91 further creates two sub-group chats Ø21 and J22 for more specialized sub-task.
Tasks
Agolts
Scodiageer
Agents
Marset//SAgent
Maclod Tean Formationt X
Natted Toan formutionn X
Figure 2: An example of nested team formation mechanism. The process is simplified for clarity.
2.3.3 AUTONOMOUS CONVERSATION FLOW CONTROL
Effective communication is crucial for successful collaboration among autonomous agents. Inspired by Speech Act Theory (Austin 1975 Searle 1969) and its applications conventional MAS (Finin et al., 1994 Labrou et al.) 1999,, we introduce an autonomous conversation flow control mechanism IoA. This mechanism enables agents coordinate their communication and maintain a structured dialogue, enhancing the efficiency and effectiveness of their collaboration.
Sequential Speaking Mechanism: To manage potential conflicts and ensure clear communication, IoA adopts sequential speaking mechanism where only one agent is permitted to speak at time. This approach, while simple, provides a foundation for more sophisticated conversation manage- ment when combined with the following dynamic features.
Finite State Machine for Group Chat States: We formalize the conversation flow as finite state machine M = (S, M, 6, so, F), where:
= {8d, &s; Sa; Sp; 5c] is the set of states representing discussion, synchronous task assignment, asynchronous task assignment, pause & trigger, and conclusion, respectively.
the state transition decision space.
S: S x Σ S is the transition function mapping the current state and the transition decision made by LLMs to the next state.
50 5, the initial state, representing the start of the conversation in the discussion phase.
F' = { is the set of final states, containing only the conclusion state.
Figure illustrates the state transitions in the conversa- tion flow. Each state corresponds to different phases of the collaboration process: (1) Discussion (sa): Agents engage in general dialogue, exchange idcas, and clarify task requirements. (2) Synchronous task assignment (sx): Tasks are assigned to specific agents, pausing the group chat until completion (Section 2.3.4). (3) Asynchronous task assignment (sa): Tasks are assigned without inter- rupting the ongoing discussion (Section 2.3.4). (4) Pause & trigger (sp): The group chat is paused, waiting for the completion of specified asynchronous tasks. (5) Conchi-
Nested Formation
Nested Formation
Formation
Asyne Task Assignment
Sync Task Assignment
Discussion
Conclusion
Trigger
Group Chatl Stotes
End
Figure The state transition diagram.
sion (s.): Marks the end of the collaboration, prompting final summary. These states align with
5

---

Published as a conference paper at ICLR 2025
the speech acts in Speech Act Theory, e.g., assertives (discussion), directives (task assignment), commissives (pause & trigger), and declarations (conclusion) (Searle 1976..
Autonomous State Transitions and Next Speaker Selection: Recent studies have demonstrated the efficacy of LLMs in autonomously managing state transitions within predefined state spaces (Liu & Shuai, 2023 Wu al. 2024a), with state machines often enhancing overall system perfor- mance (Li et al. 2024). In IoA, the LLM within each client determines state transitions and se- lects the subsequent speaker. Let M, be the of messages exchanged up time step t. We define the decision function of the LLM as: JIAM Mr x S - S x C, where S is the set of states and C is the set of clients. The next state $1+1 and the next speaker C++1 are determined as: ($1+1, C4+1) = [LM, st). This decision- decision-making process considers factors such as the comple- tion of assigned tasks, the need for further discussion, and the overall goals of the collaboration. The autonomous selection of the next speaker ensures that the most relevant agents are involved at appropriate times, promoting efficient information exchange and problem-solving.
By implementing this autonomous conversation flow control mechanism, IoA enables structured and efficient communication among agents, facilitating more effective problem-solving and decision- making in complex multi-agent scenarios.
2.3.4 TASK ASSIGNMENT AND EXECUTION
The task assignment and execution mechanism in IoA is designed to distribute work efficiently among agents and manage the execution of both simple and complex tasks. This mechanism works in concert with the team formation and conversation flow control mechanisms ensure effective collaboration and task completion.
Task Representation: In IoA, a task E 'T is represented as a tuple (d,, where d, is the task description and S, = {81,
$2,, ) {2, is the set of sub-tasks that can decomposed into. Initially, S, may be empty, with sub-tasks being identified dynamically during the collaboration process.
Task Allocation: Task allocation loA oceurs within the context of group chats and is closely tied to the conversation flow control mechanism. There two types of task allocation:
Synchronous Task Allocation: When the group chat enters the synchronous task assignment state sa, tasks are allocated to specific agents, and the group chat is paused until the tasks are completed.
2. Asynchronous Task Allocation: In the asynchronous task assignment state Sa, tasks are allocated without interrupting the ongoing discussion. This allows for parallel execution of tasks.
Formally, we define task allocation function a T x P(C), which maps task and group chat to a subset of clients responsible for executing the task.
Task Execution: Once task is allocated, the responsible agent(s) begin execution. For integrated third-party agents, task execution is handled through the Agent Integration Block in the client's Foundation Layer. This block provides standardized interface for task execution, typically in the form: run String + TaskID, where the input is the task description, and the output is a unique identifier for the task.
Upon completion of a task or sub-task, the responsible agent(s) report back to the group chat. In the case of synchronous tasks, this triggers the resumption of the group chat. For asynchronous tasks, the completion is noted, and any relevant information is shared with the group.
The pause trigger state Sy in the conversation flow control mechanism plays a crucial role in managing the completion of multiple asynchronous tasks. It allows the group chat to wait for the completion of specified asynchronous tasks before proceeding, ensuring that all necessary informa- tion is available for subsequent stages of the collaboration.
3 EXPERIMENTS
To demonstrate the effectiveness and versatility of IoA in integrating heterogeneous agents, conduct comprehensive experiments across diverse tasks. These experiments are designed to show- case different aspects of agent heterogeneity: tool variability (Section 3.T), architectural diversity

---

Published as a conference paper at ICLR 2025
Table
The performance on the validation set of GATA benchmark.
Models
Agent Type
Level 1
Level
Level 3
Overall
GPT-4
15.09
2.33
0.00
6.06
GPT-4-Turbo
20.75
5.81
0.00
9.70
AutoGPT-4 (Significant Gravitas
2023
13.21
0.00
3.85
4.85
GPT-4 Plugins (Mialon et al. 2023
30.30
9.70
0.00
14.60
FRIDAY
(wu et al. 2024b
45.28
34.88
11.54
34.55
AutoGen
Wu et al.
2023
54.72
38.37
1 11.54
39.39
IoA
50.94
40.70
15.38
40.00
and varied knowledge bases Due to space limit, we place the analysis on the average cost, nested team forma- tion precision and communication case study in Appendices [nd D Our objective is twofold: (1) to illustrate IoA's proficiency in facilitating collaboration among heterogeneous agents, and (2) to highlight its adaptability across various problem domains. We present our experimental results and offer comparative analyses between IoA and state-of-the-art (SoTA) approaches for each task cate- gory. not specified, we use GPT-4-1 106-preview model in our experiments. The prompts within IoA are kept the same across different tasks, and specifically tuned for a certain task.
Seection
3.2
disparate observation and action spaces (Section 3.3).
(Section 3.4
3.1 HETEROGENEOUS TOOLS: GAIA BENCHMARK
To evaluate IoA's capability in integrating agents with heterogeneous tools, we employ the GAIA benchmark [Mialon et al.) 2023). This benchmark comprises a diverse set real-world questions designed to assess agent system's proficiency in solving complex tasks through the synergistic application of multiple skills, including natural language understanding, reasoning, and external knowledge integration. The benchmark's three-tiered difficulty structure provides robust testbed for evaluating the capability of agent systems.
We instantiate IoA with four basic ReAct agents (Yao al. 2023) each equipped with a distinct tool: web browser, a code interpreter, a Wikidata searcher, and a You'Tube video transcript downloader. This configuration allows to assess IoA't 's ability orchestrate col- laboration among agents with heterogeneous tools. We benchmark IoA against several SoTA agent systems, evaluating performance across all three difficulty levels of GAIA. Detailed implementation specifics are provided in Appendix 1F..1 1.
Experimental Setups:
The results, presented in Table I demonstrate IoA's 's superior performance across the GAIA benchmark. Despite utilizing only basic ReAct agents, IoA achieves the highest overall performance, surpassing all other approaches. Notably, IoA excels in the more challenging Level 2 and Level 3 tasks, which require advanced reasoning and intricate collaboration.
Results and Analysis:
Compared to AutoGen, IoA demonstrates superior performance two out three difficulty levels. This superiority can be attributed to IoA's 's collaboration mechanisms and the flexibility of integrating agents with different tools, while in AutoGen, only one agent utilizes different tools, and other agents act feedback providers. IoA enables adaptive team composition and efficient sub-task execution, leading to enhanced performance on complex, multi- multi-faccted problems.
These results highlight IoA's effectiveness world, multi-step problems. By providing a flexible and efficient platform for agent collaboration, IoA enables basic agents achieve SoTA performance, outperforming more sophisticated standalone agents and representative MAS.
orchestrator for diverse agents in solving real-
3.2 HETEROGENEOLS ARCHITECTURE: OPEN-ENDED INSTRUCTION BENCHMARK
To evaluate IoA's capability in integrating and orchestrating agents with heterogeneous architec- tures, we develop a comprehensive benchmark comprising 153 open-ended instructions with self- instruct (Wang et al. 2023e). This benchmark spans four diverse categories: search & report, cod- ing, mathematics, and life assistance. Unlike the GAIA benchmark, which primarily focuses doesstion-nsseeeee tasks with deterministic answers, our curated benchmark incorporates a higher proportion of non-QA tasks requiring generative responses. This design choice aims to better reflect

---

Published as a conference paper at ICLR 2025
loA vs. Autogpt
loA vs. Open Interpreter
IoA vs. AutoGen
Math
18.7%
83.0%
Coding
3%
13.3%
3%
16.7%
50,
3%%
13.5%
.%%%%
Assictance
Life
56,
26,00 0%
17.1%
63.4%
19.0%
17.1%
43.9%
38.%
17.1%
search
04.0%
11.5%
57.3%
19.2%
13.5%
Performace
Overall
78.5%
17.0%
63.4%
18.3%
18.3%
53.6%
25.6%
20.9%
Parcentage (%6
Figure 4: Comparison of win rates on the open-ended instruction benchmark between IoA, Auto- red loss rate.
GPT, Open Interpreter and AutoGen. Green win rate; yellow tie rate;
Table 2: Average success rate and the number of steps on different tasks from RoCoBench.
Model
Metric
Cabinet
Sweep
Sandwich
Sort
Rope
Central Plan
Success #Step
0.90
1.00
0.96
0.70
0.50
(oracle)
4.0
8.4
8.8
8.6
2.3
Roco Dialog
Success
0.75
0.70
0.70
0.70
0.70
#Step
4.7
7.9
9.1
5.4
2.4
Success
1.00
0.80
1.00
1.00
0.70
loA
#Step
4.6
8.5
8.9
5.8
2.6
the diverse nature of real-world challenges that agent systems are expected to address. The curation process is claborated at Appendix(F...2
Experimental Sctups: We integrate two SoTA third-party agents with distinct architectures: Au- toGPT (Significant Gravitas 2023) and Open Interpreter (Open Interpreter (023). into IoA. The integration process, detailed Appendix F.4.2. demonstrates loA's versatility in accommodating agents with divergent internal structures and operational paradigms. This configuration allows us to assess IoA's 's efficacy in facilitating collaboration among independently developed agents with heterogeneous architectures.
impartial judge, choice supported by previ- ous research demonstrating high agreement between GPT models and human evaluators assessing response quality (Chiang et al., 2023 Zheng et al. 2023ai Chan al. 2023). To mitigate potential order-induced biases, we implement robust evaluation approach following ZZhnng et al. (2023a), where the order of responses is alternated in the prompt. A "win" is only declared when one com- petitor is consistently judged superior across both orderings.
For evaluation, employ GPT-4- 106-pre
Results and Analysis: As shown Fig. 4 IoA consistently outperforms both individual agents across all task categories, and is also generally better than AutoGen. Overall, IoA achieves a re- markable win rate of 76.5% against AutoGPT and 63.4% against Open Interpreter. These results underscore IoA's 's proficiency in efficiently gathering and synthesizing information, well as its effectiveness facilitating collaborative problem-solving across diverse domains. Additionally, we demonstrate in Appendix C that fine-tuning Llama 3 8B (Meta 2024) on the communica- tion trajectories collected from various tasks enables it to serve as the communication layer LLM, outperforming standalone agents and remaining comparable to AutoGen powered by GPT-4.
The demonstrated capability of loA to seamlessly integrate and orchestrate agents with heteroge- neous architectures enables the hamess of the strengths of diverse, independently developed agents, making it possible to create more versatile and capable agent systems. Recent studies on scaling laws in MAS (Qian al. 2024) indicate that MAS performance improves as the number of agents increases. By design, loA provides a robust foundation for integrating a greater variety of agents, potentially leading better scaling laws.
3.3 HETEROGENBOUG OBSERVATION AND ACTION SPACE: EMBODIED AGENT TASKS
To evaluate loA's 's efficacy in orchestrating agents with heterogeneous observation and action spaces, we conduct experiments using RoCoBench (Mandi et al. 2023), a benchmark for assessing collabo- ration and communication capabilities of embodied agents. RoCoBench comprises six collaborative
8

---

Published as a conference paper at ICLR 2025
tasks requiring two three agents with partial, often distinct action spaces observations to coop- erate towards a common objective.
Experimental Setups:
We benchmark IoA against two baselines established by Mandi et al. (2023)
(1) Central Plan, a centralized agent with complete environmental information and control, and (2) Roco Dialog, specialized MAS designed for this task. Given that RoCoBench requires agents to output action plans in a specific format rather than interact with tools, we adapt IoA to this scenario without integrating external agents. Instead, we provide environmental observations to two IoA clients and extract their action plans from their discussion. This setup allows us to evaluate IoA's ability to manage agents with heterogeneous observation and action spaces. Detailed implementa- tion specifics are available in Appendix [F4.3] To ensure a fair comparison, we conduct 10 runs for both loA and Roco Dialog for each task, reporting average success rates and steps taken. Results for Central Plan are sourced directly from Mandi et al. (2023). Note that the Pack Grocery task is omitted due to implementation errors in the benchmark release.
As shown in Table 2 IoA outperforms Roco Dialog in four out of five tasks in terms success rate, despite not being specifically optimized for embodied tasks. IoA achieves perfect scores on the Cabinet, Sandwich, and Sort tasks, demonstrating the robustness of its communication and collaboration mechanisms enabling embodied agents with heterogeneous observation and action spaces work synergistically towards common goals. Notably, IoA's 's suc- cess rates are superior or comparable to Central Plan across tasks, although it generally requires slightly more decision steps for task completion. Given that IoA is general multi-agent framework not specifically designed for embodied AI tasks, the marginal increase in step count is a reasonable trade-off for its versatility and effectiveness.
Results and Analysis:
3.4 HETEROGENROLS KNOWLEDGE: RETRIEVAI-AUGMENTT GENERATION
To evaluate IoA's efficacy in orchestrating agents with heterogeneous knowledge, we conduct ex- periments retrieval-auge -aggmented generation (RAG) tasks (Lewis et al. 2021). RAG tasks present unique challenge where agents must retrieve relevant information from diverse sources and collab- orate to synthesize accurate responses, making them an ideal testbed for assessing IoA's 's ability to manage knowledge heterogeneity and facilitate effective inter-agent communication.
Experimental Setups: We implement loA with GPT-3.5-turbo-0125 as the core language model, following Apollo's Oracle (Wang et al. (0026)). To evaluate knowledge heterogeneity and its im- pact, we design three scenarios: (I) Heterogeneous Knowledge: Two clients access different idence pools (Wikipedia/Google), testing IoA's ability to manage knowledge heterogeneity. (2) Homogeneous Knowledge (2 Agents): Two clients access both pools, serving as control to isolate heterogeneity effects. (3) Homogeneous Knowledge (3 Agents): Three clients access both pools, assessing scalability and knowledge redundancy trade-off'.
This design allows us to disentangle the effects of knowledge heterogeneity from agent count and knowledge redundancy. We evaluate across four datasets: TriviaQA (Joshi et al., 2017). Natural Questions (NQ) [Kwiatkowski et al. 2019). HotpotQA (Yang et al. 2018), and 2 2WikiMultHHoQAA (2WMHQA) (Ho et al. 2020), using 250 randomly sampled question-and answer pairs from each. Im- plementation details are in Appendix F.4.4
Results and Analysis: Table 3 demonstrates IoA's remarkable performance across all datasets, often surpassing or matching GPT-4 despite being based on GPT-3.5. On two out of four tasks, IoA's 's heterogeneous knowledge scenario outperforms homogeneous Apollo's 's Oracle, showeasing IoA effectiveness in managing knowledge diversity. This configuration achieves the best performance on NQ and competitive results on other datasets, often outperforming single-model approaches and specialized frameworks like Apollo's Oracle.
We also conduct experiments in homogencous settings. IoA with 3 agents achieves the best overall performance, outperforming all baselines on TriviaQA and showing competitive results on other datasets. Interestingly, the 2-agent homogeneous scenario outperforms the 3-agent setup on Hot- potQA and 2WMHQA, suggesting that optimal agent configuration may be task-dependent. These results not only validate IoA's 's effectiveness in RAG tasks but also highlight its potential as a versatile platform for managing both heterogeneous and homogeneous knowledge in MAS.

---

Published as a conference paper at ICLR 2025
Model
TriviaQA
NQ
HotpotQA
2WMHIQA
Overall
GPT 4
0.902
0.692
0,566
0.284
0.611
GPT 3.5 Turho
0.778
0.532
0.384
0.210
0.476
Zero-Shot CoT (Wei al.. 2022
0.772
0.588
0.410
0.190
0.490
Self Consistency Wang of al. 2023d
0.818
0.622
0.408
0.206
0,514
Rellxion (Shinn el al. 2023)
0.762
0.586
0.378
0.254
0.495
Multi-Agent Debate Du et al. 2023
0.798
0.648
0,394
0.186
0,507
Multi-Agent Debate2 Liang al. 2023
0.756
0.576
0,450
0.334
0.529
Apollo's Oracle (Homogeneous)
0.834
0.662
0.542
0.350
0.597
IoA 2 Agents (Heterogeneous)
0,803
0.708
0,478
0,449
0,610
IoA 2 Agents (Homogenenus)
0.820
0.671
0,586
0.530
0.652
IoA + 3 Agents (Homogeneous)
0.908
0.682
0.575
0,519
0.671
Table Model Accuracy for RAG task. IoA (GPT-3.5) matches or exceeds GPT-4 across all tasks. Best results are in bold; second-best (excluding GPT-4) are underlined. Heterogeneous refers to agents with different evidence pools, while Homogeneous means all agents share all evidence pools.
4 RELATED WORK
LLM-based Agents Recent advancements in LLMs, such as GPT (OpenAl 2023 Claude (An-
thropic 2024) and Gemini (Reid et al. 2024). have led to the development highly capable Al agents, which can engage in natural language interactions and perform a wide range of tasks. To enhance the capabilities of LLM-based agents, researchers have explored the integration of exter-
nal tools and knowledge sources (Nakano al. 2021 Yao et al. 2023 Schick al. 2023 Shen jet al., (023), enabling agents to access and utilize relevant information beyond their pre-trained knowledge. The various agents have demonstrated significant progress in wide range of domains, including operating system interactions, software engineering, and general Al applications. For in- stance, OS-Copilot facilitates generalist interactions across web browsers and code terminals (Wul iet al. 2024b), while OpenDevin focuses on autonomous software development tasks such as coding and debugging (OpenDevin Team 2024). Other notable developments include XAgent for complex task solving (Team] (2023) and Voyager (Wang et al. (003a), an open-ended embodied agent lever- aging LLMs for Minecraft game-playing These advancements have laid the foundation for more sophisticated and versatile LLLM-based agents, capable of autonomous task execution.
LLM-based Multi-Agent Systems Building upon the success of individual LLM-based agents, researchers have begun to explore the potential of multi-agent systems composed of these agents. Early works demonstrated the feasibility of using LLMs to simulate multi-agent interactions and emergent behaviors (Park et al. 2023). Since then, various approaches have been proposed to en- able effective collaboration and communication among LLM-based agents. Frameworks such as Agent Verse (Chen et al. 2023) and AutoGin (Wu et al. 2023) provide the necessary infrastructure for agent collaboration. In software development, multi-agen systems like ChatDev (Qian et al.) 2023a), MetaGPT (Hong et al. 2023) have shown promising results in automating coding, testing, and debugging processes. Despite these advancements, significant limitations remain, such as the lack of support for integrating diverse third-party agents, the inability to support distributed multi- agent systems, and the reliance hard-codes communication protocols and state transitions. IoA aims to address these limitations and provide a more flexible and scalable platform for LLM-bbsed multi-agent collaboration, paving the way for more advanced and practical systems that tackle complex real-world problems effectively.
5 CONCLUSION
We introduced IoA, novel LLM--bssed multi-agent collaboration framework inspired by the Inter- net. It overcomes limitations of existing frameworks by offering a flexible, scalable platform for integrating diverse third-party agents, enabling distributed collaboration, and introducing dynamic teaming and conversation control. Extensive experiments on various benchmarks demonstrate IoA's 'ss effectiveness in facilitating heterogeneous agent collaboration, consistently outperforming SoTA baselines. We believe loA will serve as a foundation for future research, enabling the integration of independently developed agents and advancing multi-agent systems.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work was supported by the National Key R&D Program of China (No.202220011331))) the Postdoctoral Fellowship Program of CPSF under Grant Number G2B20230348, the Young Elite Scientists Sponsorship Program by CAST (2023QNRC001) and Tencent Rhino-Bird Focused Re- search Program.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorek, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen. Vishrav Chaudhary, Parul Chopra, Allie Del Giomn,, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee. Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mi- tra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward. Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang. Ziyi Yang. Donghan Yu, Chengruidong Zhang. Cyril Zhang. Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model lo- cally on your phone. CoRR, abs/2404.14219, 2024. doi: (044550/ARXIV.244444222 URL
2404 14219
LangChain AI. Langgraph: A knowledge graph tool for langchain agents. https://github com/ Com/langehain- a htt///aaggggappp 2023.
Anthropic. Introducing the next generation of claude, 2024. URL http:::/ww..nttrro.tt. htwphh/wswclaaae familamlll Accessed: 2024-06-14.
John Langshaw Austin. How to do things with words, volume 88. Oxford university press, 1975.
Leonard J. Bass, Paul C. Clements, and Rick Kazman. Software architecture in practice. SEI series 1999. ISBN 978-0-201-19930-1
in software engineering. Ceddison-eesle-Lonee
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better based evaluators through multi-agent debate. CoRR, abs/2308.0F 072001, 2023. doi: 1048550/ARXXV2238800222
URL https:://o..cz///////
48550/a / 2.308. 07201 07201
Harrison Chase. LangChain, October 2022. URL htt.:t/p//gghhub...mm/agggg//o//
langchain
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representa- tions, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL
https:
CrewAllnc. crewAl. bttps://github.ce wwt.om//rewwll.. crewAI 2024.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. CoRR, abs/2305.14322 143255 2023. doi: 10.48550/ARXII22305..24o 14325. URL hatpss: : 0/tp://o.......500.0///// xxg/10..4550/arX55
2305. 14325

---

Published as a conference paper at ICLR 2025
Timothy W. Finin, Richard Fritzson, Donald P. McKay, and Robin McEntire. KQML an agent communication language. In Proceedings of the Third International Conference on In- formation and Knowledge Management (CIKM (9)), Gaithersburg. Maryland. USA, Novem- ber 29 - December 2. 1994, pp. 456-463. ACM, 1994. https: //do: .rgg/ 10. 1145/ 91246. 1913222
doi: 00.1145/19124..11122 URL
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps, 2020.
Sirui Hong, Xiawu Zheng. Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. Metagpt: Meta programming for multi-agent collaborative framework. CoRR, abs/2308.0033 2023. doi: 0044850/AXXV.2238000055 URL 0 3rg//10.4855//arxiv
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng. Yewei Fang, Yuxiang Huang. Weilin Zhao, Xinrong Zhang, Zhen Leng Thai, Kai Zhang. Chongyi Wang. Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies. CoRR, abs/2404.0633)9 063955 2024. doi: 10.48550/ARXIV.
2404.06395. URL http:::tt//ooooo otg/10. ott..//./...../...11....... 2404. 06395
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answer- ing. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 6769 6781. Association for Computational Linguistics, 2020. doi: URL http:pp://ooo..rr//.....155/////
N0.18653/1120200EMMNN
emn emr.p-maain 550
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wet Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Trans- actions of the Association for Computational Linguistics, 7:453-466 2019. URL. https: 7/api. semant cscholar. www.arrrrg/oooooooooeee
1911
Yannis Labrou, Tim Finin, and Yun Peng. Agent communication languages: the current landscape. IEEE Intell. Syst., 14(2):45-52, 1999. doi: 10. 10.1109/5254.75761. (ttL:tts::///// /doii. org/
10. 1109/5254. 757631
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: communicative agents for "mind" exploration of large language model society. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL htL htthttpp://ppprrsse http://s/.ppppsss.p......cpppppp httpappee..lllesppper/2222333/ 1 http:///eee..11l222eeeee2525333. ottp:///aa..aaa...o.oaaaaaaaaaaa
Abst wow.eettttcoooeeenee..ccc lttm
Zelong Li, Wenyue Hua, Hao Wang, He Zhu, and Yongfeng Zhang. Formal-llm: Integrating formal language and natural language for controllable llm-based agents. CoRR, abs/2402.007998, 2024. doi: 1044550/ARXI2.242000000 00798
12

---

Published as a conference paper at ICLR 2025
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang. Yujiu Yang. Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi- agent debate. CoRR, abs/2305. 19118..9911.... 2023. doi: 10.4850/ARXIV23351111188 URL https:
//doi org/ 10. 001111.4845555 48550/arXiv 2305. 19118
Jia Liu and Jie Shuai. Smot: Think in state machine. CoRR, abs/2322.122445333333 2023. doi: 10.48550/
1...IV.2112.144455 /wt.://.../ett..o/////4.5.////22
Zhao Mandi, Shreeya Jain, and Shuran Song. Roco: Dialectic multi-robot collaboration with large
language models. CoRR, abs/2307.047m7 04738, 2023. doi: 10.48550/ARXXIV22307..00
URL
https://don htt.:/ooro......////rr.....
48550/ arxiv 2307 04738
Elena Meshkova, Janne Rithijärvi, Marina Petrova, and Petri Mähönen. A survey on resource dis- covery mechanisms, peer-to-peer and service discovery frameworks. Comput. Networks, 52(11); 2097-2128, 2008. doi: 10.1016/J. COWNET..0.....8.8 00.00000.0........... URL https: org/10.
1016/1. comnet.. 2008 03.005.00
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari. Charline Le Lan, Christopher A. Choquette- Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Cristian Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, Ian Tenney, Ivan Gr- ishchenko, Jacob Austin, James Keeling, Jane Labanowski. Jean can-ttatiste Lespiau, Jeff Stanway, Jenny Brenman, Jeremy Chen, Johan Ferret, Justin Chiu, and et al. Gemma: Open models based gemini research and technology. CoRR, abs/24433.08299 2024. doi: 1044855//RXIV..400.3 08295. URL
Meta. Llama 3 model card. 2024. URL https://github.com blob/main/M00B. MARADMOODEELLADDDD.... mdl
1achi-11ammaaaaa ama /mmmmmmmmmmm/////mmaaaaaaam33m3m
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA; a benchmark for general Al assistants. CoRR, abs/2311. 2023. doi: 10.
48550/ARXIV:231 12983. CRLLhttps://do..com wwt..o/g/...........
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders. Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted quwp:aissddddeeesttisonsseeiiiie with human feedback. CoRR, abs/21 12.09332, 2021. URL. https:///axxivoorr/abbs/ 09332
Netflix. Eureka: Aws service registry for resilient mid-tier load balancing and failover. https //github..cmm//etfiiii...
Accessed: 2024-06-26.
Open Interpreter. Open Interpreter, 2023. 2023. URL httttpp://ggttuu..oom//////eee
spen-interprreeer
OpenAl. GPT-G techncclll report, CoRR, abt.2223333.087772222200 doi: 1048555/ARXXV2220000777.
URL [tRL:tttpp:///oo.o//0000.... wtt.://.....rr000......522222.02
OpenDevin Team. OpenDevin: An Open Platform for AI Software Developers as Generalist Agents.
https://githine www.///peeneeei//ppeeeeeeeee 2024. Accessed: ENTER THE DATE
ACCESSED THE PROJECT.
Joon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Sean Follmer, Jeff Han. Jürgen Steimle, and Nathalie Henry Riche (eds.), Proceedings of the 36th Anmual ACM Symposium User Interface Software and Technology. UIST 2023, San Francisco, CA, USA, 29 October 2023- November 2023 pp. 2:1-2:22 ACM, 2023, doi: 10. 10.1145/3586183 3606763. URL wttpthttt//cto..... :/doi .rgg 0ttpr//.....11.55111
13

---

Published as a conference paper at ICLR 2025
Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. CoRR, abs/2307.07794, 2023a.
doi: 10.48550/ARXIV.23300079999 /
07924
Chen Qian, Yufan Dang, Jiahao Li, Wei Liu, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Experiential co-learning of software-developing agents. CoRR, abs/2312.17022 2023b. doi: 1044550/ARXI22312211100 17025
Chen Qian, Zihao Xie. Yifei Wang, Wei Liu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling sungellangugge-moddl-baasss multi-agent collaboration. CoRR, abs/2406.07155 07155, 2024. doi: 10.4850/ARXXV.2246.00717 URL httLhtts: org/ 10.48550/arX 2406. 07155
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. CoRR, abs/2307.16789 2023. doi: 1048550AARXI.2337.66788 URL htLhttps:
//doi.org/10 48550/ 16789
Talal Rahwan, Sarvapali D. Ramchurn, Nicholas R. Jemnings, and Andrea Giovannucci. An anytime algorithm for optimal coalition structure generation. J. Artif. Intell. Res., 34:521-567, 2009. doi:
10.1613//AIR.26955 URL httLhhtts:///..ccoo org/ 161.11111//arrr 2695
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean- Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, loannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross Mcllroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ay- oub, Megha Goel, Clemens Meyer, Gregory Thornoo,, Zhen Yang, Henryk Michalewski, Za- heer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403.055300, 2024. doi: 1044550ARXXI22400.05533 URL
https: /10 48550/ arxiv. 2403..40... 05530
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. 3(4):333 2009. doi: 00.156//150000009, URL https:
//doi..og//0. 1ee.15561111 1111//1101/000000
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Ham- bro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Lan- guage models can teach themselves to use tools. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neu- ral Information Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL
Abst Cowprracttt-oooooeeeeeeee html]
John R. Searle. Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press, 1969.
John R Searle. A classification of illocutionary acts Language in society, 5(1):1-23, 1976.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hug- ginggpt: Solving AI tasks with chatgpt and its friends in hugging face. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurIPS 2023, New Orleans, LA. USA, December 10 16, 2023, 2023. URL http: /ttt:://paperee http:/paprrsaaaaaaa nips http:pp..ccppppppe http://pp....pprao/aaaaaaaaaaaaa
html
14

---

Published as a conference paper at ICLR 2025
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: language agents with verbal reinforcement learning. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- fion Processing Systemy 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, nips. wttpsss.ccppppppppppppppp
2023, 2023. URL
Abstract Coeettooooeeeeeettnm
Significant Gravitas.
AutoGPT,
2023.
URL https://github. com/
Signi iomiifican ficant Gravitas/Autoce:
XAgent Team. Xagent: An autonomous agent for complex task solving, 2023.
Lewis Tunstall, Edward Beeching. Nathan Lambert, Nazneen Rajani, Shengyi Huang. Kashif Rasul, Alvaro Bartolome, Alexander M. Rush, and Thomas Wolf. The Alignment Handbook. URL
Guanzhi Wang. Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. CoRR, abs/2305. 16291, 2023a. doi: 10,48550/RRXIV22005. 16291. Utthttps: //oii. ht/ddoi.org org/
48.50 /a//550//rxiv... arXiv.
Haotian Wang, Xiyuan Du, Weijiang Yu, Qianglong Chen, Kun Zhu, Zheng Chu, Lian Yan, and Yi Guan. Apollo's oracle: Retrieval-augmented reasoning in multi-agent debates. ArXiv,
wtt./ooros/oo.oo//oo 2023b. URL lttpss: semanticscholese http://sschooaa.oooor/rropppaaaa
266149845
Xingyao Wang. Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: evaluating Iims in multi-turn interaction with tools and language feedback. CoRR, 1b6/2309.10691, 2023c. doi: 10.48550/ARXIV2309.0000 48550/arXiv 2309 1691
URL/httss: htttt:://doii http://ooo..r//...////
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net, 2023d. URL htthhttts///ppeeeeeee adf?id-1PLINIMNN
net/
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 13484-1333 -3508. Association for Computational Linguistics, 2023e. doi:
URL attpps://oo.... 100 10..0..833/
v v1/2023. acl -ooellooonn long 754
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho. and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 - De- cember 9, 2022, 2022. URL http:ttttppp/aaer..nttt nip. http://ppapeee...ss...cpaa.eeeee files/paper/2022 Caw.aatttoooffeeeeeeee html
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li. Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. CoRR, awt/2308.08115555 2023. doi: 10.48550/ARXIV.3300. 08155. htRl.htts:://// http://oo.org///// www.orgg110........ 48550/ arri 2308 08155
Yiran Wu, Tianwei Yue, Shaokun Zhang, Chi Wang, and Qingyun Wu. Stateflow: Enhancing LLM tosk-solving through state-driven workflows. CoRR, abs/2403.11222. 2024a. doi: 10.48550/ ARXIV.2403.11333 11322. URL //do .org /0rg/10.cr
www.:iwww........... 11322
15

---

Published as a conference paper at ICLR 2025
Zhiyong Wu, Chengeheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-
improvement. CoRR, abs/2402.07456, 2024b. doi: 10.4555/AR21122400074466.
URL
httpps:/ddot wtw.ooooorg/1...00000.. /tt///50///////./// 21t.:/500000...222......... 07456
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering, 2018.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. The Eleventh International Conference Learning Representations, ICLR 2023, Kigali. Rwanda, May 1-5, 2023. OpenRe- view.net, 2023. URL
net net/ ?id=WE vluYUL-X
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 nips wwtpaper_aaeceeeeeeeee Shaper_filess fww.fflleepeeeeeeee htteee//pppr//223.3.
16, 2023, 2023a. URL ha5133511112333000/2225555522223 -t Datasets and
Serccmarks.html.
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023b.
16

---

Published as a conference paper at ICLR 2025
Table 4: Comparison of Key Features Across Different Frameworks.
Candidates should be pre-
specified in the team expansion from AutoGen.
Key Features
IoA
AutoGen
MetaGPT
AgentVerse
Crew AI
LangGraph
CAMEL
Heterngeneity
X
Distributed
/
x
X
Auto Teaming
X
X
V
X
X
Team Expansion
X
X
X
X
X
Asyne Tasks
X
X
Flow Control
Autonomous Pro-Defined Pre-Defined Pre-Defined Pre-Defined Supervisor-Managee Tum- nan-Taking
A COMPARATIVE ANALYSIS OF MULTI-AGENT SYSTEM FRAMEWORKS
To contextualize IoA's capabilities within the broader landscape of multi-agent systems, including AutoGen (Wu al. 2023). AgentVerse (Chen et al. 2023), MetaGPT (Hong al. 2023), Crew Al [CrewAllnc] (024), LangGraph (AI 2023) and CAMEL (Li et al. 2023). we conducted a compar- ative analysis of key features across different frameworks, as summarized in Table 4 This analysis reveals IoA's 's unique position in supporting agent heterogeneity and distributed collaboration, fea- tures not fully realized other prominent frameworks. While some frameworks like Agent and AutoGen offer partial support for autonomous team formation and dynamic expansion respec- tively, IoA integrates these capabilities more comprehensively, allowing for flexible team structures without pre-specified constraints.
IoA's 's autonomous conversation flow control further distinguishes it from other frameworks that rely on pre-defined, user-managed, turn-taking approaches. This autonomy, combined with support for asynchronous task execution (a feature shared with MetaGPT and Crew AI), enables IoA to handle complex, multi-faceted projects with greater adaptability. The unique combination of these features in IoA heterogeneity. distributed collaboration, autonomous teaming and flow control, and asynchronous task Socliinn-ppsiitonss versatile solution for diverse multi-agent collab- oration scenarios.
B PUTTING IT ALL TOGETHER: A WALKTHROUGH OF IOA IN ACTION
To illustrate the integrated functionality of IoA, Fig. we present an example walkthrough of the system with an illustrative complex task: writing research paper on the Internet of Agents. Initially, client C1. an AI research specialist trained additionally on AI academic paper, engages the Team Formation Block, utilizing the search.client function with a list of keywords [Internet, Multi-Agent System Specialist, Paper Writing, LLM Expert). The server returns a set of matched clients {c2, C22 ca. ),), from which forms group Pu with members {c1, C2 cy) via launch. www.cc.hggr.....oo writing.
grat chat, where 02 has access to scholarly databases and c3 specializes in academic
Upon the formation of group chat fo. all clients transition to the Communication Block for go. where the autonomous conversation flow control mechanism, implemented finite state machine, guides the collaboration. The process begins with brainstorming in the discussion state (sæ), progressing to task assignment states (5,: 8.) where agents allocated specific responsibilities. For instance, C2 is tasked with conducting literature review using its access to scholarly resources. The nested team formation mechanism is demonstrated when e identifies a need for specialized PDF expertise. This prompts 22 to initiate a sub-group formation process, resulting in the creation of sub-group yi with new agent Cs a PDF expert. Throughout the process, the conversation alternates between discussion (s.) and asynchronous task assignment message protocol ensures efficient communication, enabling the exchange of ideas, citations, and draft segments across the nested group structure.
(sa) states, facilitating parallel work assigned tasks. The
In the final integration phase, the group enters a synchronous task assignment state (s,) for collab- orative editing and refinement, demonstrating IoA's 's capacity for coordinating intensive, real-time collaboration among multiple agents. The process concludes with a transition to the conclusion state (s.), where final review is conducted and the paper is prepared for submission.
17

---

Published as a conference paper at ICLR 2025
en
Al Researcher Agent
Google Scholar Agent:
c: Academic Writing Agent
PDF Agent
Gool
Output
Write research paper the Internet of Agents
Abstract: The rapid advancement of LLMs has paved the way development highly capable autonomous agents
Formation
Nasted Team Formation caarchcllemt(((PP)
swwrch narchhcitentif ( (Pager Weiing H
Lamnch group ctat: (ewera))
Researcher Agent
Google Scholar Agent
Chat Group Sahup
Server
Group v.
Teok Execution
Generate Message
yield
omit N turns
sub-group
result
content: team! Let'
Paper Powerhouse /.
LitReviewBot
es
collaborute paper about
Internet of Agents..
Let's collaborate paper Internet of Agents..
thare! Ready tockle our literature review?
sender: Researcher Agant message__tyee type: discussion
Google Scholor Agent, you give literahure review on the recent multi-agent system... (message_type sync__tssk_assigl
Hello Google Scholar
next_ next__ppeker:-
Agent: Absolutely, let's list topics first. Sub-group
Autonomous Conversation Flow Control
Figure 5: An example walkthrough of the major components of IoA.
loA(Liama 3 vs. Autogpt
IoA(Liama 3 BB)
Interpreter
[oA[Liama BB) AutoGen
Math
73.3%
20.0%
7%
20.0%
3%
40.0%
Category
Coding
73.3%
20.0%
43.3%
33.3%
50,
13.3%
36.7%
Assietance
Life
59.5%
25.0%
14.6%
08.33 37.
22.0%
41.5%
26.5%
31.7%
Search
9%
19.2%
55.0%
25.0%
19.2%
48.17 13.1%
25.0%
25.9%
Performance
Overall
70.6%
21.5%
64.9
26.9%
19.8%
44.4%
22.9%
82.7%
60
Parcentage (%)
Figure 6: Comparison of win rates on the open-ended instruction benchmark between IoA, Auto- GPT, Open Interpreter and AutoGen.
C LLAMA AS THE COMMUNICATION LLM IN OPEN-ENDED INSTRUCTION BENCHMARK
While we have shown that GPT-4 can have remarkable performance in facilitating agent communi- cation when they are used as the communication LLM, their deployment can be expensive and may raise privacy concerns. By fine-tuning a smaller, locally deployable model to be communication LLM, we aim to create more efficient and potentially more tailored communication layer for IoA. To explore the potential of using locally deployable models for the communication layer in IoA, we conducted an experiment involving fine-tuning Ilama 3 8B (Meta] 2024) on the communication trajectories collected from various tasks presented in our paper.
C.I FINE-TUNING PROCESS
We fine-tuned the Llama 3 8B model using the collected communication trajectories. Since IoA heavily relies on JSON format output, we fine-tune the model to generate responses following the specification of IoA. We directly use the training recipesf] provided by the alignment handbook (Tunstall et al.), and only change the model and data related hyper-parameters, and leave others unchanged.
zephyr- 7 http:////baaaa/aa///a/aaaagg////
18

---

Published as a conference paper at ICLR 2025
C.2 EXPERIMENTAL SETUP
After fine-tuning, we deployed the Llama 3 8B model as the communication LLM in IoA evaluated its performance on the open-ended instruction benchmark, comparing to: (1) Standalone Open Interpreter (2) Standalone AutoGPT (3) AutoGen, all the baselines are powered by GPT-4.
5 We then
C.3 RESULTS
The results presented in 6 demonnttrate that loA, utilizing a fine-tuned Llama 3 8B, consistently outperforms standalone agents such as AutoGPT and Open Interpreter across task categories. Specifically, loA achieves notable overall win rate of 70.6% against AutoGPT and 54.9% against Open Interpreter. Furthermore, when compared to AutoGen, IoA remains competitive, illustrat- ing that fine-tuned smaller models can serve as highly effective communication agents within IoA, enhancing its practicality.
These findings underscore the effectiveness of fine-tuning Llama 8B for communication in multi- agent coordination and show that a fine-tuned smaller model can successfully function as the com- munication L.M,, effectively coordinating diverse agents within the system. IoA, incorporating smaller model, either matches or surpasses other agent systems in most tasks, suggesting the poten- tial for more efficient and locally deployable solutions.
D TEAM FORMATION PRECISION
To evaluate the precision of IoA's autonomous team formation mechanism, we developed a bench- mark using GPT-4, comprising 625 diverse tasks paired with 1500 dummy agent profiles. This simulated environment allows us to assess the accuracy of both regular and nested team formation in large-scale setting. Detailed data construction processes available in Appendix H
Experimental Design: We evaluate two distinct scenarios: regular team formation and nested team formation. For regular team formation, each task is associated with or more suitable agent profiles generated by GPT. For nested team formation, that can or cannot be completed by the initially formed team, if not, an additional agent profile capa- ble of addressing this subtask is generated. We evaluate whether the team can correctly decide when to enter the nested team formation stage, and evaluate the precision of the nested team formation.
generate subtask for each original task
We assess both settings using four metrics: Top@ I and Top@ 10 recall rates, Mean Rank (MR), and Mean Reciprocal Rank (MRR). Top@1 1 measures exact matches, while Top@ 10 accounts for semantic similarity, considering an agent as recalled a recruited agent is among the top 10 most similar to a labeled agent. MR and MRR provide insights into the ranking quality of retrieved agents.
Results and Analysis: Table 5 presents the performance of both team formation mechanisms, each evaluated on its own specific dataset and setting. In the reg- ular team formation scenario, which as- sesses the ability to form initial teams for given tasks, we observe a Top@ recall of 41.4% and Top@ 10 recall of 64.9%. This indicates that the mechanism can ex- actly match the labeled agents 41.4% of the time, and when considering semantic similarity, the retrieved agent fall into the top 10 similar agents to the labeled agent for 64.9% of the time. The Mean Rank (MR) of 27.4 and Mean Reciprocal Rank (MRR) of 50.1% suggest that, on average, relevant agents are ranked within the top 30 results, with tendency towards high ranking.
Table 5: Performance Team Formation Mechanisms. Regular denotes the initial team formation setting, and Nested denotes the nested team formation mechanism.
Top 1t
Top@ 101
MR.
MRR1
Regular Nested 59.7%
41.4%
64.9%
27.4
50.1%
81.8%
10.6
66.5%
For the nested team formation scenario, which evaluates the mechanism's 's performance in a setting where subtasks may emerge requiring additional expertise, we see a Top@ recall of 59.7% and a Top@ 10 recall of 81.8% The MR of 10.6 and MRR of 66.5% indicate that relevant agents are
Jllma 3 is only used communicate. The task execution agents such as AutoGPT are still powered by GPT-4.
19

---

Published as a conference paper at ICLR 2025
typically found within the top 11 results, with a strong tendency towards very high rankings. These metrics suggest effective performance in this more dynamic setting.
These results demonstrate IoA's capability to form precise teams in both initial task allocation and in scenarios where task requirements may evolve. The high recall rates, especially with similarity matching @Top@ 10), are crucial for addressing complex tasks that require diverse specialized skills.
E COST AND SUB-OPTIMAL COMMUNICATION PATTERN ANALYSIS
To evaluate the economic feasibility and potential for optimization the IoA, we conduct cost analysis the open-ended instruction benchmark (Section 3.2), where AutoGPT and Open In- terpreter are integrated. We compare the average cost per task for these agents when operating individually and when integrated into the IoA.
As shown in T when integrated into IoA, the costs of both agents are decreased due to the task decomposition for each task. However, the IoA introduces an additional communica- tion cost of $0.53 per task, resulting in an over- all cost of $0.99.
Table 6: Cost analysis of standalone agents and loA-integrated agents on the open-ended instruc- tion benchmark.
Setting
Cost per Task
AutoGPT (Standalone)
$0.39
During our analysis, we observed unexpected and suboptimal communication patterns that contributed to the high communication cost. One notable pattern was the repetition of infor- mation, where the LLMs in the clients would repeat or rephrase previous chats from them- selves others, leading progress. This phenomenon was particularly prevalent after several asynchronous task as- signments. Although each task assignment did not require immediate waiting, as the conversation progressed, new decisions had be made based on the conclusions from previously assigned and not yet completed asynchronous tasks. Despite providing the client L.MMs with the option switch the group chat state to pause & trigger, they sometimes fail to switch, as illustrated in Fig. 7 This drawback in LLM is also observed in other
Open Interpreter (Standalone)
$0.16
AutoGPT (in IoA)
$0.33
Open Interpreter (in IoA)
$0.13
IoA Communication
$0.53
IoA Communication (Dedup.)
$0.28
stagnation in
loA Overall
$0.99
IoA Overall (Dedup.)
$0.74
multi-agent work (Li et al. 2023: Mandi et al. 2023
To quantify the impact of this suboptimal communication pattern, we manually removed the repetitions and recal- culated the token numbers and corresponding costs. Sur- prisingly, this resulted in a nearly 50% reduction in com- munication costs, shown in the "Dedup." rows of Ta- ble 6 This finding aligns with observations from other multi-agent communication frameworks, suggesting that while modern LLMs are well-aligned to be effective chat- bot assistants, they may not be optimally aligned be efficient communicating agents. Agents should not only complete the given tasks accurately but also communicate effectively with others, understanding conversation states and making proper decisions. This insight raises new research questions regarding the agent align- ment of LLMs and highlights the need for further investigation in this area.
Math Masters
Given the current situation, need our strategy for achieving the goal Since
Given the challlnges we've encountered with writing python code, It seems we need to explore athernative methods. Since
A. Simply Rephrase!
Figure 7: An example of the suboptimal pattern.
Despite the current cost overhead and suboptimal communication patterns, the IoA demonstrates significant potential for enabling effective collaboration among heterogeneous agents. By addressing these challenges through prompt optimization, protocol refinement, and the development of more sophisticated frameworks under the concept of IoA, we believe that the cost of communication can be significantly reduced. As research progresses, IoA and similar approaches will become increasingly attractive and economically viable solutions for complex multi-agent systems.
20

---

Published as a conference paper at ICLR 2025
F IMPLEMENTATION DETAILS OF 10A
In this appendix, we provide a comprehensive overview of the implementation details for each mod- ule in the client and server layers of IoA.
F.1 MESSAGE PROTOCOL
The effectiveness of the autonomous nested team formation and conversation flow control mech- anisms in IoA relies on a comprehensive message protocol. This protocol enables seamless com- munication and collaboration among agents by encapsulating all necessary information required for various mechanisms to function properly.
Protocol Overview and Key Fields ity and flexibility,, facilitating effective multi-agent collaboration. The protocol consists of two main components: a header and a payload.
The agent message protocol in IoA designed for extensibil-
The header contains essential metadata about the message, ensuring correct addressing and process- ing by receiving agents. Key fields in the header include:
sender: The unique identifier of the agent sending the message.
group_id: The identifier the group chat to which the message belongs.
The payload carries the main content of the message, varying by message type. It can include:
message type: Indicates the purpose of the message (c.g., discussion, task assignment, pause & trigger). next.speaker: The identifier(s) of the agent(s) expected to respond.
This structure contains other fields to support the diverse functionalities of IoA effectively. A de- tailed explanation and example of the message protocol can be found in Appendix F.I
To ensure seamless communication and coordination, both the client and server components of IoA implement the message protocol. When a client sends a message, encodes it according to the protocol and transmits it to the server. The server parses the message, extracts relevant information from the header, and routes it to the appropriate group chat based on the group-id. Upon receiving message, the client decodes it and processes it accordingly. This consistent implementation ensures that all agents can understand and respond to messages correctly, regardless of their roles or tasks, maintaining coherent and efficient collaboration process.
Details We provide overview of key fields within the protocol in Fig. 8 It consists of several fields that cater to the specific requirements of various mechanisms within the framework.
Header sender: str
Autonomous Team Formation
goal: str
state:
team_members: list(str] team_up_deppth
comm_id:
Firstly, the protocol includes the following header for all message types:
max_ max_turns: int
Pause Trigger triggers: list(str)
Task Assignment
sender (str): The name or unique identifier of the agent sending the message.
Discussion
task_id: str task_desc: str
content: str type: enum next next_speeker:: list(str)
state (enum): The current state of the group chat associated with the message, which can be either team formation or communication.
cos__concuuiion: str absk_abstactt:
comm_id (str): The unique identifier of the group chat to which the message belongs.
Figure 8: Fields in the IoA message pro- tocol.
To support the autonomous team formation mechanism, the protocol incorporates the following fields:
goal (str): The objective or task that the current group chat aims to accomplish.
team_members (list[str]): The names or unique identifiers of the agents required for the current
group chat.
team up depth (int): The depth of the current nested team formation, used to determine the maximum allowed depth has been reached.
21

---

Published as a conference paper at ICLR 2025
max.turns (int): The maximum number of discussion turns allowed for the current group chat. exceeded, the group chat will be forced into the conclusion phase.
For facilitating the discussion phase, the protocol includes the following fields:
content (str); The actual content of the current message.
type (enum): Specifies the next dialogue state, which can be discussion, task assignment, conclusion.
next _setkks (list[str]): The name(s) unique identifier(s) of the agent(s) expected to speak next. In the discussion state, next next. speaker is limited to single agent, while in the task assignment state, it can include multiple agents, indicating that the current message contains multiple task assignments.
To support the task assignment mechanism, the protocol incorporates the following fields:
task_id (str): The automatically generated unique identifier for the current task. task_desc (str): The description of the task assigned the client, extracted from the chat. task.conclusion (str): The conclusion or result provided by the client after completing the assigned task.
task abstract (str): A concise summary of the completed task.
Lastly, to support the pause & trigger mechanism, the protocol includes the following field:
triggers (list[str]): A list of task IDs that require trigger to be set.
By adhering to this comprehensive agent message protocol for sending and receiving messages, clients within IoA can effectively achieve autonomous team formation and conversation flow con- trol. The protocol ensures that all necessary information is communicated among agents, enabling seamless collaboration and coordination in various task scenarios.
F.2
CLIENT
The client component of IoA plays crucial role in enabling the integration and collaboration of heterogeneous agents. It consists of three layers: the Foundation Layer, the Data Layer, and the Interaction Layer. Each layer comprises several modules that work together to facilitate efficient communication, data management, and agent coordination. this subsection, we provide detailed overview of the implementation of each module within the client's 's layers.
F.2.1 FOUNDATION LAYER
Network Infrastructure Module In loA, all clients maintain a persistent connection to the server using the WebSocket protocol, similar an instant messaging application. When a client sends message, it is transmitted to the server, which parses the comm id field in the message and forwards it the other clients in the corresponding group chat via their respective WebSocket connections. The real-time nature of WebSocket ensures that messages are delivered promptly, enabling clients to receive and respond to messages without delay.
Data Infrastructure Module To support the data storage and retrieval requirements of the upper- level Data Layer modules, we employ SQLite as the primary database solution. SQLite provides a lightweight and efficient means of persisting and accessing data related to agent contacts, group information, and task management. By leveraging SQL ite, the client can store and retrieve informa- tion about encountered agents, group chat details, and task assignments, ensuring data consistency and availability throughout the collaboration process.
Agent Integration Module The Agent Integration Module defines the protocol that third-party agents must adhere to in order to seamlessly integrate with IoA. Currently, the agent integration pro- tocol in loA requires agents to implement function def run (task_desc: str) -> str, which accepts a task description as input and returns a summary the task completion. This simple yet effective protocol allows diverse agents to be incorporated into the framework, enabling them to contribute their unique capabilities to the collaboration process. As IoA evolves, the integration protocol can be extended to support more advanced functionalities and interaction patterns.
22

---

Published as a conference paper at ICLR 2025
F.2.2 DATA LAYER
Agent Contact Module The Agent Contact Module is responsible for maintaining record of the clients that the current client has previously collaborated with. It stores information such the names and descriptions of these clients, providing a valuable reference for future collaborations. The module aims support the client in evaluating and storing collaboration outcomes after each task, allowing it to make informed decisions when forming teams for subsequent tasks. During the team formation process, the information stored in this module is included in the prompt assist the client in selecting the most suitable partners based on prior experiences.
Group Info Module The Group Info Module manages all group chat-related information, includ- ing the following fields:
comm_id (str): The unique identifier of the group chat.
goal (str): The objective or task that the group chat aims to accomplish.
t team memmmmbeers (str): The list of agents participating in the group chat.
state (str): The current state the group chat (e.g.. team formation, discussion, task assignment, conclusion).
conclusion (str - -None:: The final outcome or conclusion reached by the group chat.
team.up.depth (int): The depth of the nested team formation within the group chat. max_turns (int): The maximum number of communication turns allowed the group chat.
By organizing and persisting this information, the Group Info Module enables clients to maintain a coherent view of the ongoing collaborations and their progress.
Task Management Module The Task Management Module responsible for storing and tracking the tasks assigned within each group chat. It maintains the following fields for each task:
t ask_id (str): The unique identifier of the task.
task-desc (str): The detailed description of the task.
task.abstract (str): concise summary of the task.
assi gree (str): The agent assigned to complete the task.
status (enum): The current status of the task (e.g., pending, in progress, completed).
conclusion (str - None): The final result or outcome of the task.
By keeping track task-related information, the Task Management Module enables clients to mon- itor the progress of assigned tasks and ensures that all task-related data is readily available for refer- ence and decision-making purposes.
F.2.3 INTERACTION LAYER
Team Formation Module briefly introduced in Section 2.3.2 when a client receives
task, it is equipped with two essential tools: search agent (desc: 1ist [str]) -> list [agent:] and launch. group chat (team members: list [str] None) -> comm_id. The client must decide whether to utilize the search_agent tool to find agents on the server that match the specified description, or to directly call the launch.group.che. tool based the discovered agents and historical collaboration information. If the client invokes saunch.group.... chat without specifying any agents, it implies that the task will be completed by single agent. To prevent infinite loops, loA imposes limit on the maximum number of tool calls, set to 10 by default. If the client reaches this limit without successfully launching a group chat, is forced to invoke the 1 launch group chat tool to initiate the collaboration process.
23

---

Published as a conference paper at ICLR 2025
Communication Module The Communication Module handles the core functionalities of mes- sage generation and message reception. When client generates a message, IoA processes ac- cording the agent message protocol. If the message type is conclusion, the client enters the conclusion phase, where it provides a final answer to the group chat goal based on the accumulated chat records and task completion information. In the of pause trigger message, the framework prompts the client to generate the task IDs that require triggers and broadcasts them to all group members. For cii scuss scussion or task assignment messages, they are directly broadcast to all participants in the group chat.
Upon receiving a message, the client parses it according to the agent message protocol. If the next speaker field does not include the current client, the message is simply added to the group chat history. However, if the client designated as the next speaker, it must take appropriate actions based on the message type. For di scussicsnn on messages, the client generates a response to con- tinue the conversation. In the case of sync or asyne task assi assigt messages, the client extracts its assigned task from the chat record, summarizes it, and specifies the relevant information to be passed to the integrated agent. The agent then executes the task based on the summarized description and relevant chat messages, returning the result upon completion. If the message type is pause & trigger, the client updates the corresponding task triggers in the Task Management Module.
The Communication Module, in conjunction with the other modules in the Interaction Layer and Data Layer, enables seamless and structured collaboration among agents. By adhering to the well- defined agent message protocol and leveraging the functionalities provided by the various modules, clients can effectively participate discussions, assign tasks, and coordinate their actions achieve the desired goals.
F.3 SERVER
The server component ToA serves as the central hub for agent coordination, communication, and management. It comprises three layers: the Foundation Layer, the Data Layer, and the Interaction Layer. Each layer contains modules that work together to facilitate agent registration, discovery, and message routing. In this subsection, we provide detailed description of the implementation of each module within the server's 's layers.
F.3.1 FOUNDATION LAYER
Network Infrastructure Module and Data Infrastructure Module The Network Infrastructure Module and Data Infrastructure Module in the server are largely similar to their counterparts in the client. However, the server's Data Infrastructure Module incorporates the use of the Milvus vector database to support the construction and maintenance of the Agent Registry. Milvus enables efficient similarity search and retrieval of agent information based on their characteristics, allowing the server to provide clients with the functionality discover and match agents effectively.
Security Module While the Security Module not extensively utilized in the current implementa- tion of loA, we acknowledge its crucial role in ensuring the integrity and reliability of the framework in real-world deployments. This module responsible for verifying and controlling the integration of third-pan -art-party agents into the clients, preventing malicious agents from compromising the entire framework. As loA evolves, the Security Module will be enhanced to provide robust authentica- tion, authorization, and monitoring mechanisms, safeguarding the collaborative environment from potential security threats.
F.3.2
DATA LAYER
Agent Registry Module The Agent Registry Module maintains a comprehensive record of all clients integrated into the server. When client connects to the server, it is required to provide a detailed description of the integrated agent, including its name and capability description. This information is stored in the Agent Registry, enabling similarity matching based on agent character- istics. The Agent Registry serves as a central repository for agent information, facilitating agent discovery and team formation processes.
24

---

Published as a conference paper at ICLR 2025
Session Management Module The Session Management Module is responsible for managing the WebSocket connections of all online agents and keeping track of the group chats they participate in. It maintains mapping between agents and their respective WebSocket connections, as well as the associations between agents and group chats. When a client sends a message, the Session Manage- ment Module ensures that the message is properly routed all clients involved in the corresponding group chat, guaranteeing reliable and efficient communication within the collaborative environment.
F.3.3 INTERACTION LAYER
Agent Query Module The Agent Query Module handles incoming requests from clients seeking discover and match agents based specific characteristics. Upon receiving a query request, the module converts the provided characteristics into vector representations and performs similarity matching against the agents stored in the Agent Registry. The implementation of this module can vary depending on the specific requirements and scalability needs of the framework. For instance, techniques such as BM25 or other information retrieval methods can be employed enhance the matching process and improve the relevance of the returned agent results.
Group Setup Module The Group Setup Module is responsible for handling client requests to create new group chats. When client submits a request set up a group chat, specifying the desired team members, the Group Setup Module processes the request and initializes a new group chat instance. It assigns unique comm_id to the newly created group chat and notifies all participating clients about their inclusion in the chat. The Group Setup Module works in conjunction with the Session Management Module ensure that the necessary WebSocket connections and mappings are established for efficient communication within the group chat.
Message Routing Module The Message Routing Module plays a critical role in facilitating com- munication between clients within group chats. When client sends a message, the Message Routing Module receives the message and parses it according to the agent message protocol. Based on the comm id specified in the message, the module identifies the corresponding group chat and forwards the message to all clients associated with that chat. The Message Routing Module leverages the in- formation maintained by the Session Management Module ensure accurate and timely delivery of messages to the intended recipients.
The server component of IoA, with its carefully designed modules and interactions, provides robust and efficient infrastructure for agent coordination, communication, and management. By leveraging the capabilities of the Foundation Layer, Data Layer, and Interaction Layer, the server enables seamless agent discovery, team formation, and message exchange, fostering a collaborative environment where diverse agents can work together achieve common goals.
As IoA continues to evolve, the server component will be further enhanced to incorporate advanced features such as load balaneing, fault tolerance, and scalability, ensuring that the framework can handle the growing demands of real-world multi-agent systems. Additionally, the Security Module will be strengthened to provide comprehensive security measures, safeguarding the integrity and confidentiality of agent interactions within the framework.
F.4 IMPLEMENTATION DETAILS OF DIFFERENT EXPERIMENTS
In this section, we provide an overview of the implementation details for each experiment conducted to evaluate the performance of loA.
F.4.1 GAIA
For the GAIA benchmark, loA integrated four ReAct agents: Web Browser, Code Executor, YouTube Transcript Downloader, and Wikidata Searcher. The tools provided to Web Browser and Code Executor agents are adapted from the AutoGen framework with minor modifications to en- sure compatibility with loA. To address the You/Tube-related tasks in GAIA, we develop YouTube video transcript downloader based on PyTube For videos without readily available transcripts, the tool employs the Whisper model to transcribe spoken language into text. Similarly, we adapt the
https://gi tww..httu..oomm http:://i.ttabb.comm/ptttt//tttt
25

---

Published as a conference paper at ICLR 2025
Please complete function according comment. def minimum Time(grid: List[List(int]])
country, cities connected by one-way roads. known that from any city, (possibly passing through cities) leading the capital. Prove it's possible choose one a way chosen roads lead directly or indirectly capital.
grid consisting of non-negative integers where grid[row[[col] represents the minimum required visit cell (row, col), which means you can visit cell (row, col) only when the you visit is greater or equal to grid[rowl[soo).
given
city
Math
second, you must move to any jdcecttt cell the four directions: up, down, left, and right. Each move you make takes second.
standing in top-left of matrix
Review three smartphone models (Apple iPhone 13, Samsung Galaxy 522, and Google
Return the minimum time required which you can visit the tom--ihghhh cell of the matrix. If you cannot visit the bottom-right cell, then return -1.
Pixel 6) based on camera quality, battery life, user interface, and price to decide the best buy.
Exomple
Search & Report:
Input: grld 11011322055112.51443.....)
Output:
Explanation: One the paths that we can take is the following:
at 0, we are on the cell (0,0).
am 35-year-old software engineer who is vegan and looking to optimize for a balanced diet containing 2500 calories per day. Create personalized weekly meal plan for me. Include three meals and two snacks per day, paylng close attention to Incorporating a variety of protein sources to meet daily protein needs. Provide a detailed grocery list that organizes ingredients by aisle for standard grocery store layout.
[-]]
Constraints:
[-]
After you complete the function, display the content of the script as res.py directly.
Coding
Life Assistant
Figure 9: Example instructions from different categories in our open-ended instruction benchmark
Wikidata tool from Langchair fit the IoA ecosystem. These adaptations showcases key feature of loA: when a task requires specific tool, it can be easily integrated into the system through its implementation and agent adaptation, enabling it to participate in task completion.
Due to budget constraints, we conduct performance testing on the GAIA validation set. Despite this limitation, the results provide valuable insights into the effectiveness of IoA in handling complex, multi-step tasks.
F.4.2 OPEN-ENDED INSTRUCTION BENCHMARK
To create diverse and challenging benchmark for evaluating the performance of IoA on open-ended tasks, we construct a set of 1.53 instructions spanning four categories: search & report, coding, math, and life assistance. The benchmark construction process involved three main steps:
First, we select the instructions based on the real-world complex tasks used by XAgent (Team] 20233 These instructions were categorized into the four aforementioned groups Second, to increase the diversity of the benchmark, we manually create an additional 10 complex tasks. Finally, we use the Instruunt method (Wang et al. 2023e) to generate approximately 200 instructions, using the previously selected instructions as seeds. After manual screening and modification, we obtained the additional 94 instructions, resulting in a total of 153 tasks. The benchmark eventually consists of 52 search & report tasks, 30 coding tasks, 30 math tasks, and 41 life assistance tasks. By incorporating a diverse set of open-ended instructions, this benchmark allows for comprehensive evaluation of the performance and versatility of IoA handling wide range of real-world scenarios. We show one example instruction for each category in Fig.
Evaluation Methodology. For loA, we consider the final conclusion generated by the agents as the final answer. However, since AutoGPT (Significant Gravitas 2023) and Open Interpreter (Open
26

---

Published as a conference paper at ICLR 2025
(a) Cabinet
(b) Sweep
(c) Sandwich
(d) Sort
(c) Rope
Figure 10: The different environments in RocoBench.
Interpreter 2023) complete tasks in multiple steps and do not inherently generate a conclusion, we prompted them to provide detailed conclusion as the final answer after task completion.
Inspired by the pairwise comparison evaluation method used in MT-Bench (Zheng et al. 2023b), we employ GPT-4 evaluate the responses of IoA against AutoGPT and Open Interpreter. To mitigate potential biases introduced by the order of the responses, we alternate the order of the two responses when presenting them to GPT-4 for evaluation. A result is counted it is consistently determined to be superior to its competitor in both orderings. In cases where the performance inconsistent across the orderings, the result is considered draw.
win for system only when
F.4.3 EMBODIED AGENT TASKS
For the RocoBench experiments, adhere the original paper's methodology, which relies discussions and parsing specific formatted strings from the discussion results to determine the em- bodied agent's actions, rather than using agents call tools directly. We implement two clients that communicate without integrated agents, requiring them to output strings in the RocoBench format the conclusion stage. These strings are then parsed and used interact with the environment us- ing RocoBench's 's predefined parsing functions. This approach serves validation of IoA's A's client implementation and communication mechanism design.
To accommodate the varying requirements of different tasks in RocoBench, we adopt task-specing settings. For the Sort, Sandwich, and Sweep tasks, which exhibit strong interdependencies between steps, we retained the chat history and continued each new action discussion based on the previous group chat. In contrast, for the Cabinet and Rope tasks, where the steps were less interdependent, we initiated new group chat for each action to optimize costs. Other settings remained consistent with the Roco Dialog baseline.
F.4.4 RETRIEVAL-AUGMENTE GENERATION
For the retrieval-augmentte generation (RAG) cneestion-asswett task, we follow the settings out- lined in Apollo's 's Oracle. We provide agents with two evidence pools: one derived from Wikipedia and the other from Google. For Wikipedia, we utilize Pyserini's pre-built index of Wikipedia content up to January 20, 2021, retrieving the top 10 most relevant results for each query. For Google, we di- rectly access the Google Search API, returning the top most relevant results each query. These tools were made available the client-side LLMs, enabling them to query relevant information during discussions and ultimately provide well rell-informed answers.
To evaluate the performance of IoA the RAG task, randomly sample 500 entries from the validation or test sets of the four datasets. After the model generates answers, we employ GPT-4 provide GPT-4 with the dataset answers and the model's
for answer evaluation. Specifically, answers, requiring it to output its reasoning in a Chain Thought (CoT) manner before providing final correctness judgment.
G VISUALIZATION OF ROCOBENCH
We provide the visualization of RocoBench at Fig. 10 The cabinet task requires three agents to collaborate: two agents open and hold the cabinet door while the third agent retrieves two cups from inside the cabinet and places them onto coasters that match the color of the cups. The sweep task
27

---

Published as a conference paper at ICLR 2025
involves two agents coordinating their actions: one agent controls broom to sweep cubes, while the other agent holds bucket to collect the cubes, and finally, they dump all the cubes into dustbin. In the sandwich task, two agents work together pick up ingredients and stack them according to a given recipe. The sort task requires three agents to place three cubes onto coasters with matching colors. Since each agent can only reach limited area, they must coordinate their movements. Lastly, the rope task involves agents moving a rope into bracket. They must communicate effectively to decide the correct path for maneuvering the rope.
H SIMULATED ENVIRONMENT FOR TEAM FORMATION EVALUATION
H.1 REGULAR TEAM FORMATION SIMULATED ENVIRONMENT CONSTRUCTION
To construct simulated environment for evaluating the regular ,tam formation mechanism, employ GPT-4-1106 -06-preveww to generate a diverse set of tasks and agents. The dataset construction process involved the following steps:
1. Task Generation:
Using ChatGPT-4, generate 399 distinct categories of theme keywords, covering various domains such as sports, lifestyle, and entertainment.
From these categories, we randomly select 25 themes and task GPT-4 with generating task descriptions related at least four themes from the selected set, thus obtaining task that require diverse agents with different capabilities.
Task descriptions generated in JSON format using the GPT-4 API, ensuring structured and consistent representation.
2. Agent Generation:
After generating the tasks, for each task, we again prompt GPT-4 construct least two agents with varying capabilities for the given task, including the name of the agent, the type the agent and the description of the agent.
The agent profile format is designed to align with the server-side agent registry, facil- itating seamless integration and interaction within IoA.
example of a generated task description in JSON format is follows:
"task_idk id" "xxx",
"task_dee boask_descrrpttinn" "Develop a mobile app that helps users plan and manage their personal finance, including budgeting, expense tracking, and investment suggestions.
Similarly, an example of an agent profile in JSON format is:
"agent_name": "FinanceGuru",
"agent_type"; "Thing Assitant"
"asent_descriptioonn "FinanceGuru is highly skilled agent specializing in personal finance management. It has extensive knowledge of budgeting techniques, expense tracking tools, and investment strategies. FinanceGuru can provide personalia i recommendations based user's financial goals and risk tolerance."
A complete example with agent profiles and task description in JSON format is:
"agents""
28

---

Published as a conference paper at ICLR 2025
"agent_name :ame": "BeautyRottheekssstttt""" "agent_type"; "Thing Assistant", "agent_deacriptioon" "This agent specializes in grooming and beauty routines. beauty tips and tutorials for efficient makeup application based on the user's facial features, skin type, and preferences. It suggests makeup looks that align with weather conditions and the user's daily agenda The assistant can interface with smart mirrors, makeup organizers, and tutorials for streamlined morning routine."
is designed to offer personall ized
7
Dogent_namme" "IanguageCoachhssssttttttt"
"agent_type": "Human Assistant", "agent_descriptionn" "This is an educational aide focused on facilitating language learning sessions. assesses the user's s current language proficiency, learning style, and daily schedule allocate an optimal one-hour learning window. The agent customizes lesson plans, integrates with language learning apps or platforms, and can organize virtual interactions with native speakers for immersive learning experiences."
1I
13
"agent_name name "BcoCuisinekssisttan"" "agent_type"; "Thing Assistant",
15
"agent_deeccription": "EcoCuisinedi stant is dedicated to healthy meal planning and anvironmntall consciousness: It suggests simple, nutritious dinner recipes based on dietary needs, kitchen inventory, and prep time constraints.
interfaces with smart kitchen appliances to guide the cooking process and monitors waste to teach and reinforce correct recycling habits, ensuring a minimized environmental impact."
17
l.
19 "task_ "ess__deccription" "I am looking to create a daily routine that incorporates applying makeup efficiently in the morning, spending an hour learning a new language, preparing a simple and healthy dinner, and correctly recycling the waste generated throughout the day."
H.2 NESTED TEAM FORMATION SIMULATED ENVIRONMENT CONSTRUCTION
In similarly way, in order to construct simulated environment for evaluating the nested team formation mechanism, we also employ GPT-4-1 106 106-preview to generate two diverse sets tasks and agents. The dataset construction process involved the following steps:
Sub-tasks Completed by Existing Agents:
Su-btask Generation:
Based the dataset that have constructed for regular team formation,
randomly select 300 sets as the original dataset.
- For tasks in the original dataset, we prompt GPT-4 to construct a sub-task that can be completed by existing agent, with the agent being selected by GPT-4.
29

---

Published as a conference paper at ICLR 2025
Sub-task description generated in JSON format using the GPT-4 API with the existing agent, ensuring structured and consistent representation.
2. Sub-tasks Completed by Additional Agent:
Sub-task and Agent Generation:
After generating the sub-tasks for exiting agent, take the rest of sets the
another original dataset.
The difference for sub-task completed by existing agent that we prompt GPT-4 construct sub-task requiring very specific expertise.
Meanwhile, we also prompt GPT-4 to construct agent with distinct capabilities compared to the existing agents to complete the generated sub-task, including the name of the agent, the type of the agent and the description of the agent.
Sub-task description and additional agent generated in JSON format using the GPT-4 API ensuring a structured and consistent representation.
An example of a generated sub-task description with existing agent in JSON format is as follows:
"additional_subtas""" {
"task_dee deask_deccription": "Develop comprehensive marketing plan highlighting the business's s commi oment to sustainability, including strategies for podcast promotion, brand awareness, and customer engagement.", "agent":
"agent_name "MrketingStrateiiss"
"agent_type"; "Human Assistant",
"agent__dascription": "Critical to the success of the sustatnabillyy-foccedd business, this agent in charge of advertising campaigns, social media presence, and public relations. With a strong emphasis on the company' eco-friendly values, it develops targeted marketing strategies to reach a wider audience, creating a strong brand identity around sustainability. The agent also handles analytics, gauging the effectivenese of marketing efforts and adjusting tactics optimize outreach and customer engagement."
}.
"agents":
10
"agent_name"; "SustainabillityyEucaao "agent_type"; "Human Assistant", "agent_descriptionn "This agent is specialized in creating, curating, and disseminating information about sustainable living. It is responsible for researching various subjects related to sustainability, structuring podcast content, interviewing experts, and sharing practical tips on incorporating eco reo-friendly practices into daily life. The agent will also engage the audience through various channels, answer listener queries, and promote discussion on sustairability."
12
13
14
16
"agent_name": "EcoDesigner",
17
"agent_type"; "Human Assistant",
"agent__deccristionn: "Tasked with the creation of custom eco -friendly products, this agent has expertise in sustainable design practices and materials. It collaborates with customers to understand their needs and preferences, and uses innovative methods to craft
30

---

Published as a conference paper at ICLR 2025
personalized, environmentally responsible goods while maintaining aesthetic and functional standards. Additionally, the agent works closely with suppliers to ensure the sustainability and ethical sourcing of raw materials.
19
"agent_name", "Marketingstrategiist"
"agent_typp": "Human Assistant",
"agent_descriptionn "Critical to the success of the sustal business, this agent in charge of advertising campaigns, social media presence, and public relations. With a strong emphasis on the company' eco-friendie values, it develops targeted marketing strategies to reach a wider audience, creating a strong brand identity around sustainability. The agent also handles analytics, gauging the effectiveness of marketing efforts and adjusting tactics optimize outreach and customer engagement."
25 },
"task_ deassk_eescrrptiion "I want to start a business that focuses on sustainable living. The business will include podcast series on how to incorporate sustainability into daily life and crafting custom eco- cco-friendly products for customers."
27
Similarly, an example of generated sub-task description with additional agent in JSON format is:
"additional. subtask";
"additional_subtask"
3
beaas_ddsccrittiii
"Implement advanced custom animations and interactive elements to enhance the visual appeal of the personal website, particularly for the graphic design portfolio section. This includes creating dynamic, c, engaging animations that showcase the artist's skills and bring the homepage to life, as well as ensuring cross browser compatibility and responsiveness on various devices.",
"agent": {
"agent_name": "AnimationExpert""
"agent_type"; "Thing Assistant",
"agent_description"" "Animation innxpett is highly specialized virtual assistant dedicated to creating sophisticated web animations and interactive experiences. It is equipped with state-of-the-art tools and knowledge of the latest animation libraries like CSAP, Three. and WebGL. This agent analyzes the existing style and content of the website to develop tailored, eye-catching animations that complement the graphical elements without compromis sing website performance. It ensures compatibility with all major browsers and devices and works seamlessly with responsive design principles to deliver a consistent. experience across all user interfaces."
8
10
"agents"; [
12
"osent_name": "Weblessigerrksaistant" pogent_type". "Human Assi stant",
31

---

Published as a conference paper at ICLR 2025
sngent_descriptiors "This agent specializes in web design and user experience. It assists in creating a visually appealing and intuitive hemepage layout that effectively showcases the portfolio of graphic design work. It will help organize content in cohesive manner, using best web design practices to emphasize the most compelling pieces. This assistant can also suggest and implement design elements that reflect personal style and artistic sensibility."
15
16
17
"agent_name": "ContentStrategiissi htt.straattissssssssssssstt
18
"agent_type"; "Human Assistant",
"ayent_Gescrittonn" "This agent focuses content creation and management. It supports in putting together the fashion and style blog posts by helping to curate topics, edit posts for clarity and brand consistency, and integrate them into the website. content is strategically placed for optimal engagement, incorporating SEO best practices to increase visibility and draw in more visitors interested in fashion and style
ensures that the blog
20
21
"agent_name": "Bhotorapttyhhocaasssssssaa"""""
23
"agent_type": "Thing Assistant", "agent__escriptionn "This agent is tailored to enhance the presentation of photography work on the website. Equipped with image organizing and editing software integration capabilities, it can help sort and select the best photographs to feature. It will ensure that the images are displayed in high quality and that the loading speed is optimized for user convenience. This assistant will also provide options for interactive image qalleries that enable visitors to view the work in detail."
26
"task "task_ description": "I want to create personal website that showcases my portfolio of graphic design work, my fashion and style blog posts, and my photography. Please provide instructions on how to design the layout for my homepage that effectively incorporates all three aspects."
28
By generating couple of diverse of tasks and agents, create comprehensive simulated environment for evaluating the regular team formation mechanism and the nested team formation mechanism. This environment enables priate teams to complete task requirements, addressing the limitations of existing benchmarks in providing suitable large-scale agent evaluation scenarios.
assess the effectiveness of IoA in assembling appro-
EXAMPLE RESPONSES AND TRAJECTORIES
In this section, we present several complete trajectories of IoA (Figs. and 12). and comparison of responses from different frameworks on different tasks in open-ended instruction benchmark (Figs. to 16
32

---

Published as a conference paper at ICLR 2025
Team
Geol Somele
Univee
opper doy Inkede
The fortic the arille Fnd spen tnuir ahat NASA nuiler wark sead & Arted oupurted by"
Marsage Types:
Scaation
reportet The
Asysc Tasi Asiigmmnt
Quean piodo
Tusk Assigmanti
Sani
Wechvmarrpe gpd
Infom Task Propreau
ppen furter moreeree
Infomm Teak Aasuh
searh Diaar of Adleve
Var
Canclack Diccuaion
ArAdl Agel
Fulf douari huparrry Trumplle "ie oa boad by
2011
L/ sccurtty "ally
god
berts Tiae cole por sigen ole sitee si's
Triggar
pulled york
Figure 11: A complete trajectory of IoA on GAIA benchmark.
33

---

Published as a conference paper at ICLR 2025
Tears
doel
slon nq macoeis a rhai autty for youhhly peoaiar cacitall sare party" Ale. Fra aupert aacital rou are
Fodd cana toracingg dea relunda sirthalp Mlactiinn
Message Types!
Discuesion
Anigmennt
Aesgreant
sarig hopart the wnicie the hodey
ieterve deper The seaey dude buirsh Avidle
beiing bert
ragoring
Pasoc
Coen Invperter
Iefom Task Progras
compert and an
Inform Tear Read?
Disoustion
Dse Seopert
phore
An
of quala poole andyy
your dhaty frose
Noyoe and
Copeder rodes repringgt of colpen Pore cnltira File mesto vall ctltecien
Figure 12: A complete trajectory of IoA on open-ended instruction benchmark.
34

---

Published as a conference paper at ICLR 2025
Instruction from Gategory Coding
besign a solenoid using copper wire that con produce magnetic field strength of 0:01 Tesls when a current of 2A is passed. betail the number of turrs required and the length of the solenoid.
IoA
AutoSPT
To design a selonedd using copper eine that con procuce e negnetic field streegth of 0,1 Teale wher cament of 24 peamed tire calceloeed the rumber of hurss per tiugtt (0) and
(N0
to opprasinntll bnyy We chare proctical sstenels leigt mether, neeating cupper wre (Comeel Ardwer).
stronght ju ir vocuum perraspiltyy (en x 30n-7 TeVA) ie the number of tome per usit we
lesgth,
and
3979
29, orzer produce the speeil and nagnetic lieid stringth siing copper vire prances les bring the Savart lay and Ansere's lax, accourting for the pernest illty free space and the specific current spol fed.
"Finiisn twesore The tack has been cosplated ucresstally. The dectgned
leagth meters
Next we congaled lan notanatee ragu ned for coneructin thi rolonid and drotted battc
construch guidic The masernda include
fessback: you should suer areze and provide a cony led and conslete cont lustun in text assigned you, there Yas should consiting the information gatherss and as do not use any wib or coee
Okpper eine The grage shoud be condiic of com myinn a corment The longth neqiied
form filles the fieal
answer the previeus
2 Come naterall a suriible naturill in sach soft man recammended for bet magnetic
poreasiilty The disaster
one sheuld neughty aqui to diseeher wine
Tris on for wins Palying divonide (PVC) or in equiatien tealinig noneiaal it eaggected rehee
expecter oper rrinn teeperihuues and be capuble of celivrrigg
Power supply The Ti oureen softly.
should have
6.1 acters to produce segietie tield streigth of 0.81 Testa wher a carnent ot passed shrough coppen nan This contiguration based on inutations using tundamental slectronagnttinn principles, consibering por pormealility of free space.
app
The conetraction go de inckude des the fel fellwwing stepe:
Prepare the came cut langth macussary and enauee is free rust, pa cocting.
delee the exeect sumteer of wine tume per wheer and wark the wre bosted on the tottl fures
Noxt Commard: finishi reasones The task has been completed successfully. The eeters in orter to produce the speciated eegretic field streegth uitig given sagreto-ttaris priseip the meiulinng the Biar-dauart aspore's low, The dctalled information about this design, ancludirg the turns nearest whele rumber, his poon comsited and presented firal *tmeeyy-)) 1)
3 Regin us ncing the copper wine rightlly and evenly around the core. fellwwngg the morks for tarms tape othestre.
Afteen wine har baen wound. recein place
pelority
copper wire
design was
Teat the in by ftumiig ng the power eupply and using ga mourmneeen ineass ne the "esis taryet curnent achieve the desined
mogrehic stronghs
cal cofurrrtett m stranght.
schety pressutions muet be tollowes while conetracting and tecting my the oslenoid Thile compnenesivee ope each ensurcl ea des yn and conaimaction aelkoeed that monts the requi
The task has been complated successtatly. The desagred as selnnsd requires spprocinatelly 75/578 turns (Wheng Araver! stong length 8.1 aeters in megratic
with the given current, This design was derived based stantord eajrets- static provetple en, ancludiig Bist-Savart this length or a.2 meters and roussing off the total, turns to the neerest stote sumaary.
and (epere's lax. choosing
including
Open Interpreter
invelvd sdens wring wine thet could product rrsyetic fildd strongth of 0.c0 Thdl what carnent of EA is posred. To icuatily the requiernunt har this mugeetic generated oy selewwif
conpltedd
serup. anglied
Rosed thes cel celcelon on using the foreul lo for the mayneiic field iedda seloved fara da splenad ength 0.1 metera (50 canfimehers, you aodd rezare szarsurnntll 598 tume (Camest field
caloulating
provices poramehers,
requie
opproe monely 8979 hames of aire netor to penerste the decired mayeric field stranghh assuredd selenadd angth of one meter, this trundiste to tand reugtly 3979 for
Ampares passed through in This caka leter canamal deel candmora witt anform vagretic suiched i'n soschic
leng lighlly rumber
your design cospinonts
the the langth, the diamaner of the es aesid (smsumdd to oa and the formulo the cncuee crance (we), 3 caloulated then the rendl longit of the supper wire roquied to make
regnetic Fisl shroegth 0.00 with salrned can dusigeclvnt with ford oppoosmatil 3979 tumsi of coppor oppros mihely 625 iathers long.
took medie crcation modifiaaiin any files, hanoe there ane no print neaat.
Figure 13: Responses from different frameworks on coding task.
35

---

Published as a conference paper at ICLR 2025
Instruction from Category Life Assistance
I'm planning surprise birthday party for my best friend. Can you help me find list of populor cocktoils to serve at the party? Also, reed some suggestions for cocktail recipes that are easy make. Acditionally, find some interesting news articles related to birthday celebrations.
IoA
Auto6PT
read, fiinf fenones tiimkddy fartc bet' amaud The user internupted the octisn with the followeng feeebook "Ner you shoold summarizs and provicc ca camplied complete conclatoo tout
Cemanbb compreheutv
you in
dimnhday party
for yoar hest frient
find find sraver pleasc pron than sut You should focss on compling the information gothenel and any
shore
sase Pepular Cockin te Serve the Party
"Majjns" refreshig rg mix
run, time jaice wint lenves, suger, sodo
veher
"Merger hott A clessic codktall mode with togale, line juice, and omarge kigu served with
Qlick of fist popular cockislle to nenvo the porty.
"CGomepsittunn a chic Mend vodks, orenberry jucc. rriple see, and Frestlly squeezed on "ma Furvicees"h sinsle yet amphiriictted boarton whindey, rager bithere,
For the aur purpite diythday party, hone ane some popalor and ecayy to- moke cocknal rediee
pase
(the supportiona one practical for planning the party.)
o hest of ciima ried
jurnahhed with
aqud Compart,
Mt rido ruir looas liss rigar and line jucce And and top with sode water. Soreis with line
slice min songg
ewe Evsy-te- masoo Cocittl Red piet
Rub the riy of the oleer with the time toe to make the suft stikk to it. Shoke the other
Trigle Selt for
sice
smeoth and serve gariated with slice of a nesppe
fresh line juice with ioc. thes stren ino chillet glast and parnic with lime wheel
in sPrpy Calodyen Bleet
-rwhite Rusdaven ce ports wethe I part Kobis, and parn beoey creon or whote milk
old sesieeedd yas
plneaylle
with ice
part
Stake ingned arte
spring sodo feasisting sbout birtddaye that cauld fur d disuusiio paints dur my the party.
Tha longe pitcher, mbe wine, broiddy and anarge ge juice Add dieas tratt and retrigorcte, Add
Coomwee from Vadka, Crumbery jeice Trnge slass
fy
into les cocitsl glas.
Lime jase. Lime slice for gorwsh
sedo.
date Interest News drt clep Bekned to Bir rhudy Coldrenies
day Portym- unioss cstetrotiee for clocks who emerce every 37 yeare. (Read more ht//tt////weee
(The AutodPT respende provides fas inctnng bor birthday facts inteed of the regasited intenaling newt artictie niteeed to birm calbr eroore)
aheettreal 2021/ Siiss a bi-iidby parry)
Resase" (Poad more, (el / hhmn hun any foowea/fd.
bockgrabbee later be leors orediol
1. The world's is largeat ba brtdday cake was created 1089, ws phining 125 225 pounds and ssed
1o/ has bone- lime rnggenn and daytedy pornee)
Coldratias" university coldimotion
sulten Brussi hastee the world's
riiday Party 50th ba barhhcy
voowaye coion and morks for attondese Bood seme]
1990, morting $17 i wallin The eteation included pols extch and noccert by Michool
Dutipp///
wwtp///rckkaaeeeeeeeeeeeaaa /ewl///e66777/ hrppy thdoy coldratte the univeraltyo-
The rest comman bernheay the United stana Sephember pes, This teand beliened to
"Luss Duy ćo cokdrationn" Cheative way lep day bebses te celdrate ther specil day, ind nncicing
earthe prior, perk
conceptir,
matorical sources, tradition brtdday Grouter They used casilees rokes make then sloe like the moee, a twittte Arteess
Accending
with the Ancient
Dimpes/ /oew oer 2014/02 11t/1//99/0////a/aeeo oow loop 2024)
"Foy Jonas 32ard Barthday Ooleoratiz"" endiincturn
medel
In Wommon, evenyhody odlateatte maeir birthday os New Year's Dey, siio known Ter, regar clen the actual day they were bar. This lstivee colddration refiests the cuftinal
jonss berthday coents fechune ovar ding--o of oaguan lapsc mackl)
and ingpenng artidees you're ne well ecuipedd to the ou foegerssle surpiie dertidy party for yoar beef frivnd, Enjoy the
With the cartned lot of populor cadetale, eay make
The Hanyy Birtlddg ceng. one of the most frees estly sang Erglish sings, was originslly companed by Aerry
tkase to eake
nimianty party rpvearcbl event Dijos!
Open Interpreter
fendhiinsseen Completed
birthdaye?)
allow to sammerios the infermation:
Congleted took provided cocktoll recipe fun focts about birrhdays.
compltted liet of the most popuiar sodinal's wilcch may creat inckuck of your bust friend's
parmy
Exprooes
2. dle frund seee only recke celktall resipes for cenveence Here it brief carmory of whit you wight consiter
AutoGen
You celectice populor cocatels enly to recipes your fweed's
aye Seates whidley.
seneorrs Nirters gereiteed cherry or
surprise party.
"Wagite incrrssinn
juice
"nCaaaicc mas champsgnc or apar dinng wine, Cranpe germahed with erange aloce
Corepp theeth ohe bieed vedko, line Caletroaa crmmerry
"Wag no-n white run, Lime juic tipkk ayrap, 6-3 mun leowe, 3oda wcher Colodi White Creams of socanat, hreppple juis, the juss gormates with pinazoplc
" "Mangenho"" The stanti consinch rereed glass
juice, strwed aoth-
wndye
"Nogrs oie.. sopharticheed this of gin, Compant and sosett vermouth, ettimad over ioe
"Maacow Mare"* Ginger bear, Vodka, Lieea juice
"*Deiqinee to ve Line plic. Singde svmp
Edkedd Tend Tacc" vodke, div, Taculk, Rue, Triple Line jaice Golo, Tes
5. "oai Paal ceasth A rich musture of bzarten, Angasmars bitture, augar cubs, and water.
genvdtedd with on snonge twist and chenny
Plesse
depe
These czoktala are popdar choieer that sater to ver city of tarier and ane obe relerively simple presort.
seeded.
Roge
duditie ie terrsday colebrmmens rnturted search pet
coleb-etiens you
and prowie you instination event
stories reloted birthday
crticka
perties that shane
(The kes populer cocitola is and suppestora for cadinall recier lack dencil, Last but not loart, the the fiwd of
(The of popular cscittila auaperiioa for codina neapee lock deteil Lean not loart the repjoses is eiciin the thet article)) )
ieggone meiting files geneerted
With this intermmtion you're well pood to shrow memondble ln and anjoyotll birthday parmy for your best friand, Cheers and happy adetroieg!
Figure 14: Responses from different frameworks on life assistance task.
36

---

Published as a conference paper at ICLR 2025
Instruction from Category Math
Two players turns choosing numbers from the set (1.3.3...103 number, chosen, connot be chosen again. The ends when the chasen numbers least 55, player picks the number, pushing the total to Determine if there's winning strategy for second player describe
Auto6PT
diccation
the find
felliar:
In the game where two alayere take tuma cheesing numbers from the set (1 2., 12l whosut repetitis fona
thersfors, numbers will chocen by the and of the geme. 5 ncc there are eves ber tumm (10 in
besed that the mambere
neing sirerag
player, Recoaniig and gone analytte satereaaed the eacand player witning appraach, No Purmen
All complened The Fythen contires atrihhoy
chelose seaving ing thet the raeaiing ursseee conett osherne by the fivet player in oucha sscand player ahwape shat ofter hhe fearth tum (the eighth furs). the the chooce nambors less tean eeerg bost poorss meatted fiod timm of 55, this winning the
Io nammary, the second gi leyar by symehully by so actingg rambere hurporene to the tinet playen's sleeys weroove- the pame pasitiss they wimiing fimt rinal wirving strotigg
with aprinal
denamei
Open Interpeeter
AutoGen
tokn numbere from sel D.E.B. 100 with the rale that number once choses cannol chesen sgan. The game unds rumbers lest player sku number
the sinuloted
the of resch sncrd thi lotal
ooing rush sytvsilly centrolling the poos the come, making atrchege cheica, the first playe con extre they winking goss (Wreeg).
The sexdetien casll 5. 10) wits
The first player has weiiing the game share player tume cheesing
By sinulsting this grew using loy soi appronch, played the grew outerntionll where both playera chese
number which didn's the fstol sum cocoed
sinalstion
reseved roand
always wine
if boith plyees play opteealy, The alrology narka by han checsing the anallics sumber gii player reach 52 first, hemos
way.
Tharefors, the wining atroneay for the first player always chease the amailaat rumber to (Wrong))
wherewere tereite ar crestel during the cncsution this thenefore I hope tnation
simulation provides unower
195k.
Figure 15: Responses from different frameworks on math task.
37

---

Published as a conference paper at ICLR 2025
Instruction from Category Search Report
Anelyze. the role of the character Togo in Shakespeare's "Othello", Quste three sigrificont monologues of Togo and interpret their meoning in the context of the play.
IoA
AutoSPT
shahepuunc''s "'thhle"
onigeritt whose orhave un the centrol conficr of the play. Thrugh sir curning and decant lups rechemmnne the dowefel Oriclls, Weer Viniis other chaneshera ahand wwitting
Proceedin with Toge's monvlgga onelytid bosed on my exitting kinuladiee
comples, indluding ont line and delp seoned moice fewords de these he feel hire wooe
Hore are three signifiont morelgues of Toge, whick are piaotall to understoncing his rele the
Friih oitirressee "as onan is expected to provile the une yos witheut exccating any farthnr knowisziee bask and stternuthed octicn with folkiing feodhand: form the Final
already fur sation file puripuatiios disiiiin returne The you ehas sumoor the procsto
to mait mindlggae Zega nesile tir supliieeee rature to Roderign Hi daciones than he not
opperrs indlcating his decestive mecks spenarch pr c foood
tide ondissed
word whick
you if thene are any fike in the find craverr ploase pr then out You should food compling ing the irfermation gottiened and une any wes cook commands""
arcand him for gais The sendagac andersconss the thems deception and be rapal otherrng
2 tind what's be thes that soye play vilise (Act 2. Some 3)
Toge's signtfoon moniigg es discuss tee mesoning sigrffiaace sithen Othelle".
Lage's mompuuation Othell panagrin icn deowttd dippbyin in menolagu gaor and
Proumshonce, bags sugertt that by deriing a plat to male Oftile jedous he dang whit
procephios
Next Gremand fivdhiressenn Procceding with Zege 's men logue une jos busted oniting Incelsige.
coleaiiigg imeriiona and fudl the
orgaenet dig
teecharou plat.
(Th teak war praneturaal halted before complete annwer wor provided Upan revinuing ondlydi wis parforsed)
you
speak
ward (Aet 5. Scene 2)
En rall traningae Eape rot discias watheer even planu andovel
aresafiatt silence open caphured contranted
the ait behropol.
Io Tage's i shardayyinn aacy he has becicelll deceived mooolgge
charecters and the cullence ing ened questions cbeat tree reture and nativel ona
In sommary, those monslogues noto Machivvllion compl exipyxiity Lage's chomacter. Through mhrass jeolury, desiprion, batroyel
pertnat deven by doop osataal vakee und curning anclec He bitter movelogus boy tore the intriatt Jostiyeson play' a plen und Officie.
tregery
inaggral shomaster particulrryy
tregedl that befaila the
AutoGen
Open Interpreter
Zage a central chereste- sholespean's "Ohelli' serves sa the ontopenat. Ha rale decertie
tare, E can provide zoep let sumeary and condiator of the toak, Hoverrr es there more to tike priet
expirshinn
like
The wighhtt tho play Teyo is pa os carnnn ext es iikes fiyen ahe or cecheeeeee the cownffcl cendls athers. Hore are three 2200 sageficut more saus slnng with interpnctotion
deceett ond nomoed anderthuiing of fume totere are A ruesing phretey of onl mecter
mon pultiion Tego
somratie
keetng tregic canc kiles
eige
ply feutible
1, Soone 3: "Mintal A right monalogua
eige Traar vani canilegac Highlee Zegi'
In earty manclipe, Faos medie conapt sima, indiating that belkwess Otf lo while plottngg oo kis. Thi speech expars logo doal city and beto up Hes
Act Soure "Virnuel rig"t The ane le resecl zage's a omrica view wertec as more
oil
lon feverrde deciprias oociines their desines
3. doees 2: Marclagas about repuractinn
by foete niw ne strsssttt eega-ni wiited hit cosetteee contrel over other claracters
Sivee This cattinn Tea''s sonpelrre Ontul i for Deodenani
decsption
ecnly tan Ahrough he spesk lights of reputation ho men pule on rdles heaw by on linn the good
that
giole
spccessful mores
hes
Act 3.
dicee you are newd swe't calkeey
craatooned shove of conecince, and depth of hit munde ie prowest
ongarty. Th reveste inhenert iel
otis prowess he the bone or to tax Caund's a deraged rep ropuratii creanic a nift qutlishte Tay stweteic ky dillityy so the Hear frode are and in orheringe
(The secard and thend panto avide apenric sugeficunt mone spues)
ondyying
(The hun cortinn fachal error the line 10 you are well hused new" sapea in Aet 2, Stuee revering the theeeee' achvett. rotser keow feall
decept theter countment,
palstion brbdddy belke in the dissei lncc human vetu highigh fer role at the bolves frigety
anephiam feeords viruue tage's calouloned
resuting in this helluc enterr)
hark innead rulied inhunel rocaledoee
This divs Eas''s chonecter arovides insiat the pulst machon urar that propal the
Jape's a rondegue serve up windevs into he catculeing mg wind, reved his ph shapppph and thoughts lie ectice
dramchc sorrstiv "Oweel""" lage momceo yue serveci os we baws wnoh v mind prosenting dii my monipuisted
praciaios
netivas inclding persinel pradgea and jesluay whinh matcch lead tregedy, Throug mas deprint
calliquee Gholkopeare
dinger
saception.
hew they commboune to the play dyanic cetorattoon of the her dapects human rature.
Figure 16: Responses from different frameworks on search and report task.
38


Reviews:
[{"review_id": "ckOkUIzZ7C", "reviewer": "Reviewer_gNTA", "summary": "This paper proposes the \"Internet of Agents\" (IoA) framework to facilitate enhanced collaboration among autonomous agents using large language models (LLMs). Inspired by the Internet’s structure, IoA aims to address the integration challenges seen in current multi-agent systems (MAS). This is achieved through three key innovations: a protocol for integrating diverse agents, a layered architecture resembling an instant-messaging framework, and mechanisms for dynamic team formation and conversation flow. Through a series of experiments, the paper claims that IoA enables more effective collaboration than existing MAS frameworks, particularly in domains like retrieval-augmented generation (RAG) and embodied AI tasks.", "questions": "- How does IoA handle latency and data synchronization when agents are distributed across devices in real time?\n\n- How does the protocol for agent integration address security, especially for third-party agents that may have differing trust levels? Could the authors expand on IoA’s security measures, especially regarding agent registration and message routing?\n\n- Can the FSM for conversation flow control be adapted or scaled effectively if used with a larger number of agents than tested in the experiments? How adaptable is the finite-state machine-based flow control to tasks requiring high real-time interaction or rapid state changes?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Innovative Conceptual Framework: The IoA draws a compelling parallel between the functionality of MAS and the Internet, introducing layered architecture and team formation protocols that offer adaptability and scalability.\n\n- Comprehensive Experiments: The paper provides a well-rounded experimental design, including benchmarks that demonstrate the system’s effectiveness across multiple task categories.\n\n- Task-Specific Improvements: IoA demonstrates improved performance in benchmarks like the GAIA and RAG, highlighting its potential advantage over existing MAS in facilitating agent cooperation and integrating heterogeneous tools.", "weaknesses": "- Lack of Detailed Evaluation on Real-World Applications: While the experimental benchmarks are extensive, the paper lacks real-world application scenarios that demonstrate the IoA’s effectiveness in real world environments that require distributed settings. It is not unclear why GAIA benchmarks would need agents over internet instead of a simple central multi-agent system.\n\n- Scalability and Performance Analysis: The paper lacks analysis on IoA’s computational overhead and network latency issues, especially when scaled to large numbers of agents across different devices distributed settings. An understanding of these scalability and performance trade-offs is critical.\n\n- Vagueness in Mechanisms for Conversation Flow Control: The conversation flow control relies on LLMs and a finite state machine (FSM) for transitions, but the scalability of this approach when coordinating many situations with many states is not well justified. It would be great to have some ablation studies.\n\n- Limited Insight into Integration of Third-Party Agents: The implementation of heterogeneous agent integration is described conceptually, but details on handling diverse architectures and security considerations when integrating third-party agents are lacking.\n\n- Limited Novel Research Contributions: The paper’s focus on architecture and engineering limits its contribution to ML research. While the framework offers novel infrastructure, the lack of new research methodologies or performance optimizations in agent interactions might reduce its appeal in ML-centric venues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yTQeVHAzsP", "reviewer": "Reviewer_kxKm", "summary": "This paper introduces the Internet of Agents (IoA), a novel framework for LLM-based multi-agent collaboration, inspired by the concept of the Internet. The framework aims to facilitate collaboration among autonomous agents by addressing the limitations of existing multi-agent frameworks, including ecosystem isolation, single-device simulation, and rigid communication and coordination. IoA consists of two core components: the server and the client. The server acts as a central hub controller, while the client serves as a wrapper for individual agents, enabling the necessary communications and protocols. The IoA architecture has three layers, i.e., interaction, data, and foundation layers, they collectively accelerate agent collaboration through the network. Experiments conducted in various settings demonstrate the effectiveness and versatility of IoA in facilitating efficient collaboration among heterogeneous agents.", "questions": "It would be beneficial to provide additional examples related to the constructed benchmark and to illustrate the specific steps undertaken by IoA.\n\nGPT-4 might occasionally give higher evaluation scores or exhibit overconfidence. Could you discuss any potential biases or limitations associated with using this evaluation method in your benchmarks and whether you considered alternative evaluation approaches.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is well-written and organized, with the key components of IoA clearly illustrated and easy to follow.\n\nThe experiments are comprehensive, assessing IoA's capability to integrate agents with heterogeneous tools on the GAIA benchmark. It introduces a robust benchmark comprising 153 open-ended instructions spanning four diverse categories. The results illustrate IoA's effectiveness in integrating, orchestrating, and enabling independently developed agents with heterogeneous architectures. Further experiments with RoCoBench highlight IoA's ability to enhance the communication and collaboration capabilities of embodied agents. \nMoreover, in the RAG question-answering domain, IoA also surpasses existing methods.", "weaknesses": "In Section 3.2, while the paper constructs 153 open-ended instructions spanning four categories—search & report, coding, math, and life assistant—there is a lack of sufficient or list of examples (only one figure) to illustrate whether the questions are complex and require multi-step reasoning, or if they can be directly solved.  Therefore, the authors could include an appendix with a representative sample of tasks & solutions from each category, or provide a link to the full benchmark.\n\nAdditionally, the paper primarily employs GPT-4 as an impartial judge of response quality, but IoA might require more steps than competitors such as AutoGen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XGX3HkRolX", "reviewer": "Reviewer_3Ebv", "summary": "This paper introduces the Internet of Agents framework for multi-agent collaboration, where agents from different developers communicate via an instant-messaging-like architecture and work in teams.\nCommunication is implemented by abstracting several conversational states, and implementing a finite-state machine to support state transition for agents.\nThe set of states includes 5 states, representing discussion, synchronous task assignment, asynchronous task assignment, pause & trigger, and conclusion, so and most of the \"work\" happens in the discussion state (this was not very clear to me).   Each agent has a LLM-based component which determines state transitions and selects the subsequent speaker.\nThe framework is evaluated using experiments with three types of tasks where teams of agents communicating using Internet of Agents framework are compared to existing multi-agent systems.", "questions": "How where these three specific benchmarks chosen?\n\nThe workflow of the conversational state machine is unclear to me. How are the tasks broken down into components? \n\nEach agent has a LLM component that manages state transitions, and which other components must agents have, to be part of this framework? \n\nAre only LLM-based agents able to participate in the framework?\n\nIs it possible to present results (e.g. Table 3) as a bar plots with confidence intervals? If not, why?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "Inventing a \"language\" that can enable information transfer between different versions of AI agents is an exciting research question. It is certainly a useful idea, as proliferation of AI agents comes with a proliferation of agent versions and brands. At this stage, the question is still rhetorical rather than practical, as it took decades to build countless networking protocols for wireless routers and phones, like SIP, H323, TCP, UDP,.... In the current work the authors touch upon a novel question of introducing such as language, and take a step toward considering what would this language look like.", "weaknesses": "I feel like the authors miss the opportunity to focus the argument as a rhetorical/philosophical consideration to build an analogy with internet communication protocols.\nGiven the novelty of the issue, the system is still a toy and the experimental evaluation does not look as convincing as the conceptual notion of the potential that such a framework could be. \n\nPart of this general presentation issue, I find the Figure 1 to be much too cluttered. Maybe one way to approach this is to use an open source diagram creation tool, like draw.io, to make a clean arrows and boxes diagram with minimal text, showing how the various components interact. \n\nThe paper could benefit from a few task workflow examples.\n\nOverall all these comments are referring to one common point, which is lack of focus and clarity. My impression from this paper is that this is a worthy subject, but it is difficult to assess (rather than infer) what was done.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HEYlIhyUkY", "reviewer": "Reviewer_78P6", "summary": "This paper introduces the Internet of Agents (IoA), a framework for enabling distributed collaboration among heterogeneous LLM-based agents. The key innovations include: \n(1) a protocol for distributed agent collaboration: a multi-layered architecture that enables agents to operate across different devices and locations, where each agent maintains its own \"WebSocket\" connection to a central server that handles message routing and coordination.\n\n(2) an instant-messaging-like architecture for agent discovery and teaming: agents can search for collaborators based on capability descriptions and form nested teams, implemented through a central registry that maintains agent capabilities and allows semantic matching for team formation\n\n(3) dynamic mechanisms for conversation flow control: a finite state machine approach that manages conversation states (discussion, task assignment etc.,), where LLMs autonomously decide state transitions and next speakers to maintain structured dialogue\n\n(4) a protocol for integrating third-party agents: a standardized interface that requires agents to implement a simple `run()` function accepting task descriptions and returning results, allowing agents built under different protocols (AutoGPT or OpenInterpreter) to be wrapped and integrated into the framework.\n\nIoA (GPT3.5) is evaluated across GAIA benchmark, open-ended instructions, embodied AI tasks, and retrieval-augmented generation, and IoA consistently outperformed other baselines on GAIA, open-ended instruction benchmark and matches or exceeds GPT4 performance on IoA.", "questions": "How does performance scale with increasing number of agents?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The architecture is novel and the design is well-motivated which address three key issues in typical multi-agent systems (MAS). \n2. The technical foundation is clear as there is rigorous formalization of key mechanisms and detailed ablation studies showing importance of each component\n3. The evaluation is comprehensive as it include a variety of benchmarks testing different aspects of agent heterogeneity along with strong empirical results across all tasks. (GPT3.5-based IoA agents outperforming GPT4 baselines)", "weaknesses": "One major concern about this framework is the cost (compute/financial) as Table 6 showed IoA is much more expensive than other multi-agent frameworks like AutoGPT and Open Interpreterm which might limit deployment of this frame at scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yUD8z3Ykpm", "reviewer": "Reviewer_LshR", "summary": "The authors propose the Internet of Agents (IoA), a novel framework for enabling collaboration between heterogeneous AI agents distributed across different devices. Drawing inspiration from the Internet's architecture, IoA introduces key mechanisms including agent registration/discovery, nested team formation, conversation flow control using finite state machines, and flexible task assignment. The framework addresses limitations of existing multi-agent systems like ecosystem isolation, single-device constraints, and rigid communication pipelines.", "questions": "- Really cool that you drew inspiration from Speech Act Theory for this work. Have you considered what Rational Speech Act has to offer in this case? Would be neat to see it leveraged to deal with things like recursive reasoning (theory of mind), own uncertainty and uncertainty of other agents, etc.\n- How does the system handle potential deadlocks in the conversation state machine? Are there specific examples of potential deadlock scenarios you've considered? Can you describe any mechanisms or safeguards implemented to prevent or resolve such deadlocks in the conversation flow?\n- What mechanisms ensure fair resource allocation when multiple agent teams are operating simultaneously?\n- How does the framework maintain consistency when agents have conflicting knowledge or goals?\n- Could the authors elaborate on the scalability limits of the nested team formation approach?\n- What strategies are employed to prevent malicious agents from disrupting team collaboration? Is that just offloaded to the Security Module?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The authors present a well-motivated and timely contribution given the rapid advancement of autonomous agents and the need for frameworks to orchestrate their collaboration\n- The layered architecture design is elegant and comprehensive, with clear separation between interaction, data, and foundation layers for both server and client components\n- The experimental evaluation is thorough, testing the framework across diverse scenarios including general assistance, embodied AI tasks, and retrieval-augmented generation\n- The nested team formation mechanism is theoretically sound, with formal analysis showing reduced communication complexity compared to fully connected structures\n- The finite state machine approach to conversation control is well-grounded in Speech Act Theory and provides a principled way to manage agent interactions", "weaknesses": "- While the authors demonstrate strong empirical results, the theoretical analysis of the framework's properties (e.g., convergence guarantees, scalability limits) could be strengthened\n- The evaluation focuses primarily on task performance metrics. Additional analysis of system properties like latency, resource usage, and communication overhead would be valuable (I do see some of this in Appendix E)\n- The authors could expand on potential failure modes and mitigation strategies, particularly for scenarios with unreliable network connections or agent failures\n- The security implications of allowing distributed agents to collaborate deserve deeper treatment, including potential vulnerabilities and safeguards. I appreciate that the authors included the Security Module in the schematic, but think having a bit more discussion of it (in the appendix) could be beneficial", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)