AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant
Authors: Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xie, Zhiyong Wu
Accepted at: acl25

AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant

Chengyou Jia1,2, Minnan Luo1,4✉, Zhuohang Dang1,5, Qiushi Sun2,3, Fangzhi Xu1,2, Junlin Hu2, Tianbao Xie3, Zhiyong Wu2✉
1School of Computer Science and Technology, Xi’an Jiaotong University
2Shanghai Artificial Intelligence Laboratory  3The University of Hong Kong
4Ministry of Education Key Laboratory of Intelligent Networks and Network Security, China
5Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, China
cp3jia@stu.xjtu.edu.cn  minnluo@xjtu.edu.cn  wuzhiyong@pjlab.org.cn

Abstract

Digital agents capable of automating complex computer tasks have attracted considerable attention. However, existing agent methods exhibit deficiencies in their generalization and specialization capabilities, especially in handling open-ended computer tasks in real-world environments. Inspired by the rich functionality of the App store, we present AgentStore, a scalable platform designed to dynamically integrate heterogeneous agents for automating computer tasks. AgentStore allows the system to continuously enrich its capabilities and adapt to rapidly evolving operating systems. Additionally, we propose a novel core MetaAgent with the AgentToken strategy to efficiently manage diverse agents and utilize their specialized and generalist abilities for both domain-specific and system-wide tasks. Extensive experiments on three interactive real-world benchmarks demonstrate that AgentStore significantly expands the capability boundaries of agent systems in both generalization and specialization, underscoring its potential for developing the specialized generalist 1 computer assistant.

1 Introduction

The continual evolution of computer Operating Systems (OS), along with proliferating applications, has transformed how people work and live. This transformation goes beyond daily life like shopping and gaming, encompassing professional works such as writing in Office or editing in Photoshop. However, this increased functionality comes with a steep learning curve, often burdening users. As a result, autonomous computer assistants—once limited to fiction like JARVIS in Iron Man or MOSS in Wandering Earth—have become a concrete pursuit, attracting great interest from researchers.

Advancements in Multimodal Large Language Models (MLLMs) (OpenAI, 2023; Reid et al., 2024), are gradually turning this vision into reality. However, real-world OS environments encompass a diverse array of open-ended computer tasks, each with inherent requirements for capabilities across multi-dimensions (Xie et al., 2024), posing substantial challenges to existing methods. Specifically, “Task_1” in Figure 1 illustrates that many computer tasks necessitate specific knowledge and operations. In such scenarios, existing generalist agents (Wu et al., 2024; Tan et al., 2024) often underperform due to their lack of these specialized abilities. Conversely, specialized agents, despite excelling at specific tasks within single domains like tabular data processing (Li et al., 2024; Chen et al., 2024a) or web browsing (Zhou et al., 2023; Deng et al., 2024), struggle to perform when confronted with more integrated, system-wide tasks like “Task_2” in Figure 1. This heterogeneous demand for capabilities presents a challenge for existing single generalist or specialized agents.

We attribute this dilemma to overlooking a key factor behind the success of modern operating systems: App store2 which continuously expands the range of functionalities beyond the core OS itself. Correspondingly, we argue that specialized generalist computer agents should possess this characteristic, evolving to grow heterogeneous abilities and autonomously handle an increasingly diverse range of tasks. To substantiate this, we propose AgentStore, a flexible and scalable platform for dynamically integrating various heterogeneous agents to independently or collaboratively automate OS tasks (illustrated on the right in Figure 1).

We develop a prototype of AgentStore, establishing an integration protocol and implementing over 20 computer OS agents and 10 mobile OS agents

1The concept of “Specialized Generalist” refers to an AI system that excels in specific tasks while still maintaining broad general capabilities (Zhang et al., 2024).
2In this paper, App store not only refers to App Store for Apple but all similar platforms. See the concept in App store.
Task_1: In a new sheet with 4 headers “Year”, “CA changes”, “FA changes”, and “OA changes”, calculate the annual changes for the Current Assets, Fixed Assets, and Other Assets columns.

Task_2: Find the daily paper and take down the meta information of papers on 1st March, 2024 in the opened .ppts file. Please conform to the format and complete others.

Figure 1: Task examples illustrate that diverse open-ended tasks require a combination of generalization and specialization capabilities. The right part provides a simple overview of specific steps.

with diverse functionalities. Based on this foundation, the main challenge is efficiently managing the rapidly growing and increasingly large number of agents, which overwhelms traditional management methods, such as In-Context Learning (ICL; Dong et al., 2022) and full Fine-Tuning (FT; Qin et al., 2023). To this issue, we introduce a novel MLLM-based MetaAgent with AgentToken strategy, to select the most suitable agent(s) to complete tasks. Each integrated agent in AgentStore is denoted as a learnable token embedding in MetaAgent’s architecture like a word token embedding. During inference, MetaAgent activates the corresponding agent to execute the task when an agent token is predicted. Innovatively, we enhance this approach by shifting from single-token (Hao et al., 2024) to multi-token prediction, allowing MetaAgent to predict and coordinate multiple agents for collaborative task execution. Additionally, we propose an automated process with self-instruct for tuning AgentToken without relying on manual data.

We validate the effectiveness of AgentStore across three real-world dynamic benchmarks involving both computer and mobile OS platforms. On the highly challenging OSWorld benchmark, AgentStore achieves a success rate of 23.85%, more than doubling the performance of the previous generalist agent (11.21%) (Xie et al., 2024). On a novel benchmark OSWorld-Multi for evaluating agent collaboration, AgentStore significantly improves the execution success rate from 6.93% to 22.77%, and achieves strong performance across all metrics. These results highlight the importance of scalable agent integration in expanding the system’s capabilities. Similar outcomes are observed when evaluating AgentStore in mobile environments. Additionally, we also demonstrate the advantage of AgentToken in comparison to other strategies, highlighting its efficiency in training and its effectiveness in dynamic management.

2 Related Work

LLM-based Agents. Recent advancements in (M)LLMs (OpenAI, 2023; Reid et al., 2024) have led to the development of highly capable AI agents, applied across various domains, including robotics (Driess et al., 2023), software development (Wang et al., 2024), and beyond. A rapidly growing research field among these is automating interactions with computer environments to solve complex tasks. Early work primarily focused on specific scenarios, such as web manipulation (Yao et al., 2022; Deng et al., 2024; Cheng et al., 2024), command-line coding (Sun et al., 2024), and gaming (Wang et al., 2023a). Following this, recent methods (Wu et al., 2024; Tan et al., 2024) have started exploring general-purpose computer agents capable of interacting with diverse components in OS. Unfortunately, both of these struggle with open-ended tasks in real environments, exposing limitations in their generalization and specialization capabilities.

Multi-Agent Systems. Recently, various approaches (Park et al., 2023; Sun et al., 2023; Hong et al., 2023) have been proposed to facilitate effective collaboration among multi-agents to overcome hallucinations. Advanced frameworks like AutoGen (Wu et al., 2023) and Dylan (Liu et al., 2023) have demonstrated flexible agent orchestration capabilities, primarily focusing on tool-based
task execution with structured workflows. Similarly, Camel and the concurrent IoA explore agent integration for specialized tasks. While these approaches have shown promising results in domains such as automating coding, their application to OS environments presents unique challenges. First, OS tasks demand dynamic integration across heterogeneous applications with varying operational constraints and system-level interactions. Second, OS automation requires specialized capabilities across diverse system operations, from low-level file management to application-specific automation. These unique challenges motivate AgentStore’s design to expand multi-agent capabilities specifically for real-world OS applications.

3 AgentStore

As illustrated in Figure 2, AgentStore consists of two main components: AgentPool and MetaAgent. The AgentPool (in Section 3.1) stores feature-specific agents with distinct functionalities and defines the integration protocol for adding new agents. The MetaAgent (in Section 3.2 and 3.3) selects the most suitable agent(s) from AgentPool to independently or collaboratively complete tasks.

3.1 AgentPool

The AgentPool is a collection of all available agents within AgentStore. When the developer creates a new agent and seeks to integrate it into AgentStore, it is essential to register the agent’s information in a standardized format. To ensure consistency in the integration process, we establish an integration protocol. During enrolling, the developer completes a predefined form outlining the agent’s capabilities, limitations, applications, and demonstrations of its functionality (in Figure 2). Formally, the set of all enrolled agents is represented as \( \mathcal{A} = \{(a_1, d_1), (a_2, d_2), ..., (a_n, d_n)\} \), where the completed form for each agent \( a_i \) constitutes a document \( d_i \). To build the prototype of AgentStore, we create 20 computer agents and 10 mobile agents to handle tasks on their respective platforms. We employ both manual and automated agent generation.

Manual Generation for Computer: Due to operation inconsistencies among computer apps, achieving uniform automated agent generation remains a significant challenge. Consequently, we manually designed computer agents tailored to common applications. These agents are evolved by injecting domain-specific knowledge documents into GUI-based MMAgent (Xie et al., 2024) and CLI-based FridayAgent (Wu et al., 2024). These agents range from unimodal to multimodal, from open-source to closed-source models, and from GUI to CLI. This heterogeneous combination provides a solid foundation to validate the effectiveness of the AgentStore concept. The details of these agents are presented in Appendix A and B.

Automated Generation for Mobile: In contrast, the operations of mobile apps are entirely GUI-based with a more uniform design, enabling us to automate the generation of domain-specific agents. Following the approach of APPAgent (Yang et al., 2023), we initially develop a general GUI-based APPAgent. Subsequently, we guide it with self-exploration to navigate and interact with nine common mobile apps, ultimately evolving it into specialized agents for mobile platforms. This approach minimizes manual effort and ensures the efficient creation of tailored agents for mobile applications.

3.2 MetaAgent with AgentToken

We employ the powerful open-source MLLM as the foundation for our MetaAgent \( M \). This enables it to process multi-modal information covering task descriptions and OS states. Since the number of agents in AgentStore dynamically grows and reaches a large scale, common methods like In-Context Learning (ICL) (Chase, 2022; Li et al., 2023) and Fine-Tuning (FT) (Qin et al., 2023) become impractical due to the excessive context length and the high cost of retraining, respectively.

To address these, we propose the AgentToken strategy. Inspired by ToolkenGPT (Hao et al., 2024), which captures tool semantics using special tokens, AgentToken extends this concept by encoding enrolled agents as special tokens in the MetaAgent’s vocabulary. Specifically, the agent tokens are parameterized as an embedding matrix \( W_{\mathcal{A}} \in R^{|\mathcal{A}| \times d} \) and appended to the original word token head \( W_v \in R^{|\mathcal{V}| \times d} \). Assuming the agent tokens \( W_{\mathcal{A}} \) have been trained and available (as described in Section 3.3), the concatenated result forms the new language modeling head of MetaAgent. In this way, MetaAgent predicts the next token with the following probability:

\[
P_M(t_i|t_{<i}) = \text{softmax}([W_v; W_{\mathcal{A}}] \cdot h_{i-1}),
\]

where the next token can be either a word token or an agent token, i.e., \( t_i \in \mathcal{V} \cup \mathcal{A} \). The operation \( [:] \) denotes concatenation, and \( h_{i-1} \in R^d \) represents
the last hidden state. In this context, AgentToken enables MetaAgent with two primary functions:
MetaAgent as Router: Following the above manner, the most probable next token is obtained by maximizing the conditional probability:

\[
t_i^* = \arg\max_{t \in \mathcal{V} \cup \mathcal{A}} (P_M(t_i|t_{<i})) .
\]

Once an agent token is predicted, i.e., \( t_i^* \in \mathcal{A} \), the MetaAgent halts decoding, and the corresponding agent is invoked to execute the task. As illustrated in Figure 2, the above method enables MetaAgent to act as an efficient router, predicting the most appropriate agent when a single agent is sufficient.
MetaAgent as Hash Manager: We discover that, although each agent token is trained on individual tasks, they exhibit generalization capabilities for complex, collaborative tasks. Specifically, when a task requires multiple agents, the trained agent tokens often appear among the top candidates in the next token predictions. This observation led us to enhance this approach by shifting from single-token to multi-token prediction:

\[
T_i^* = \operatorname{TopK}_{t \in \mathcal{A}} (P_M(t_i|t_{<i}), K),
\]

where TopK(\cdot) is a function that returns the set of \( K \) tokens from the vocabulary \( \mathcal{A} \) that have the highest probabilities. These predicted tokens represent the \( K \) agents most relevant to this task. The MetaAgent then switches to Manager mode by using a new prompt consisting of in-context documents for these selected agents, outlining how to generate subtasks for the complex task and assign them to the corresponding agents. This narrows the management scope to a few selected agents, leaving ample context space for detailed documentation of these fixed agents. This design shares similarities with hashing methods (Aggarwal and Verma, 2015), which convert inputs of arbitrary size into fixed-size outputs to facilitate retrieval.

![The illustration on the main components in AgentStore.](page_184_134_1227_362.png)

Figure 2: The illustration on the main components in AgentStore.

3.3 Train AgentToken with SELF-INSTRUCT
The embedding \( W_A \) corresponding to agent tokens are the only tunable parameters, introducing minimal additional training overhead. However, training these agent tokens requires a number of agent demonstrations that consist of the task descriptions and initial OS states. The corresponding token demonstrations were pre-collected for training in previous efforts (Hao et al., 2024; Chai et al., 2024). However, this strategy is not applicable in our scenario, as developers only provide a document about the agent, and it is unrealistic to expect them to supply massive demonstrations. Therefore, we propose an automated process with self-instruct (Wang et al., 2023b) for tuning these tokens.

The overall process follows an iterative algorithm to guide the generation of extra demonstrations, beginning with a limited set of original demonstrations \( S_i = \{(y_k)\}_{k=1}^{n_i} \) and the agent description \( c_i \) provided in document \( d_i \). Specifically, we first prompt MetaAgent with existing demonstrations and agent descriptions:

\[
S'_i = M(S_i, c_i),
\]

where MetaAgent \( M \) is expected to produce the new set of demonstrations \( S'_i \). Following this, to ensure the quality of the generated outputs, we apply BERTScore (Zhang et al., 2019) to all newly generated outputs \( y' \in S'_i \), ensuring both consistency and diversity. Specifically, we use a greedy algorithm (see Appendix D) to iteratively filter elements from \( S'_i \), resulting in a refined set \( S''_i \subseteq S'_i \). The new set satisfies the following conditions:

\[
\tau_1 \leq \text{BETRScore}(y_k, y_j) \leq \tau_2,
\]
\[
\forall y_k, y_j \in S_i \cup S''_i \text{ and } k \neq j,
\]

where BETRScore(\cdot) represents the similarity between two demonstrations, with imposing a lower bound \( \tau_1 \) to avoid overly irrelevant outputs and \( \tau_2 \)
ensuring diversity among them. In this way, we filter the generated data, and the refined set is merged, i.e., \( S_i = S_i \cup S_i^{new} \). The entire process is an automated iterative bootstrapping. MetaAgent further generates additional examples until sufficient demonstrations to meet the training requirements.

Training with self-generated data: During training, each task description and initial state in \( S_i \) serve as the prefix, and a special agent token <Agent_i> is appended as the ground truth for the next token prediction. The training objective is:

\[
\mathcal{L}(W_{\mathcal{A}}) = \sum_i \sum_{y_j \in S_i} - \log P(<\text{Agent}_i|y_j),
\]

the embedding \( W_{\mathcal{A}} \) represents the only tunable parameters for all agents \( \mathcal{A} \) in AgentPool. Notably, this training paradigm simply introduces additional tokens to the MetaAgent. The original language generation of the MLLM remains entirely unaffected when agent tokens are masked. This guarantees that ICL methods can be invoked seamlessly.

4 Experiments

To assess the effectiveness and versatility of AgentStore, we conduct comprehensive experiments across various tasks. We demonstrate its outstanding performance on three real-world OS benchmarks (in Section 4.1), followed by an in-depth analysis of agent integration and AgentToken (in Section 4.2 and 4.3). We also provide running cases for qualitative analysis (in Section 4.4).

Settings We employ InternVL2-8B (Chen et al., 2024b) as the base model. Additionally, details regarding the Agents in the AgentPool can be found in Appendix A, along with the threshold selection for \( \tau_1 \) and \( \tau_2 \) in Appendix D. We generate 100 examples for each agent using self-instruct for token training. We also apply BertScore filtering to exclude any overlap with test tasks, thereby preventing data leakage. The AdamW optimizer is used with a learning rate of 4e-5 and a weight decay of 1.0, for a total of 10 training epochs. When executing the Hash Manager, \( K \) is set to 5.

4.1 Main Results on Real-world Benchmarks

4.1.1 Computer OS Assistant: OSWorld

OSWorld (Xie et al., 2024) provides a scalable and real environment for evaluating computer agents, encompassing 369 tasks involving real web and desktop applications across open domains. As one of the most realistic and challenging benchmarks, OSWorld is ideal for capturing the diversity and complexity of computer tasks. Details of OSWorld are provided in the Appendix E.

Table 1 presents the performance comparison between our approach and previous SoTA generalist agents on OSworld. While more advanced base models can improve performance (e.g., GPT-4o outperforming GogVLM in CogAgent (Hong et al., 2024)), even the best base models still face significant challenges. Notably, these methods exhibit not only overall weak performance but also significant disparities and weaknesses in specific task categories, despite using the same base models. For instance, MMAgent (Xie et al., 2024) and CRADLE (Tan et al., 2024) struggle with calculation tasks due to their lack of knowledge and operational capability in Excel, while Friday (Wu et al., 2024) and Open-Interpreter (Open-interpreter, 2024), CLI-based agents, fails to execute GUI operation effectively in tasks, e.g., Chrome or Thunderbird.

In contrast, AgentStore overcomes these limitations by integrating various specialized agents, each proficient in specific software and operations (the base model “Hybrid” indicates that these agents are built on different base models, as detailed in Appendix A). “AgentStore(GT)” in Table 1 refers to each task being assigned to the most suitable agents, representing the upper bound of performance for the current AgentStore implementation. As shown, using specialized agents to handle tasks in their respective domains consistently outperforms generalist agents, with no significant performance shortcomings in almost all domains. This underscores the importance of various capabilities. Furthermore, when different methods are used to manage task allocation, all approaches outperform previous single-agent systems. AgentToken (AT) demonstrates the best performance. We will elaborate on this in Section 4.3.

Cost Analysis: While AgentStore’s deployment introduces some overhead, it is able to assign specialized agents to complete tasks with fewer steps and lower costs. We detail this in Appendix C.

4.1.2 Agent Collaboration: OSWorld-Multi

We develop a novel benchmark OSWorld-Multi based on OSWorld, comprising over 100 diverse tasks that necessitate collaboration among multiple agents. This newly proposed benchmark allows us to assess the accuracy of both task decomposition and subtasks handling in a real-world environment.
Table 1: Detailed success rates of previous methods and AgentStore on OSWorld. Methods marked with “*” represent our re-implementation of the corresponding agents. Additionally, due to the significant overlap between the OS and Workflow domains in the original division, we have merged these two domains into “OS*”.

<table>
  <tr>
    <th rowspan="2">Agent</th>
    <th rowspan="2">Base</th>
    <th colspan="9">Success Rate (%)</th>
  </tr>
  <tr>
    <th>OS*</th>
    <th>Calc</th>
    <th>Impress</th>
    <th>Writer</th>
    <th>VLC</th>
    <th>TB</th>
    <th>Chrome</th>
    <th>VSC</th>
    <th>GIMP</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>CogAgent</td>
    <td>GogVLM</td>
    <td>1.60</td>
    <td>2.17</td>
    <td>0.00</td>
    <td>4.35</td>
    <td>6.53</td>
    <td>0.00</td>
    <td>2.17</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>1.32</td>
  </tr>
  <tr>
    <td>MMAgent</td>
    <td>GPT-4o</td>
    <td>14.44</td>
    <td>4.26</td>
    <td>6.81</td>
    <td>8.70</td>
    <td>9.50</td>
    <td>6.67</td>
    <td>15.22</td>
    <td>30.43</td>
    <td>0.00</td>
    <td>11.21</td>
  </tr>
  <tr>
    <td>CRADLE</td>
    <td>GPT-4o</td>
    <td>8.00</td>
    <td>0.00</td>
    <td>4.65</td>
    <td>8.70</td>
    <td>6.53</td>
    <td>0.00</td>
    <td>8.70</td>
    <td>0.00</td>
    <td>38.46</td>
    <td>7.81</td>
  </tr>
  <tr>
    <td>Friday*</td>
    <td>GPT-4o</td>
    <td>15.20</td>
    <td>25.50</td>
    <td>0.00</td>
    <td>21.73</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>17.39</td>
    <td>15.38</td>
    <td>11.11</td>
  </tr>
  <tr>
    <td>Open-Inter*</td>
    <td>GPT-4o</td>
    <td>12.80</td>
    <td>12.76</td>
    <td>0.00</td>
    <td>13.04</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>17.39</td>
    <td>15.38</td>
    <td>8.94</td>
  </tr>
  <tr>
    <td><b>AgentStore(GT)</b></td>
    <td>Hybrid</td>
    <td>20.00</td>
    <td>36.17</td>
    <td>10.63</td>
    <td>47.83</td>
    <td>47.06</td>
    <td>40.00</td>
    <td>34.78</td>
    <td>47.82</td>
    <td>38.46</td>
    <td>29.54</td>
  </tr>
  <tr>
    <td>AgentStore(ICL)</td>
    <td>Hybrid</td>
    <td>9.60</td>
    <td>0.00</td>
    <td>2.13</td>
    <td>4.34</td>
    <td>35.29</td>
    <td>33.33</td>
    <td>30.43</td>
    <td>30.43</td>
    <td>15.38</td>
    <td>13.55</td>
  </tr>
  <tr>
    <td>AgentStore(FT)</td>
    <td>Hybrid</td>
    <td>8.80</td>
    <td>27.65</td>
    <td>4.26</td>
    <td>13.04</td>
    <td>41.17</td>
    <td>40.00</td>
    <td>34.78</td>
    <td>8.60</td>
    <td>15.38</td>
    <td>17.34</td>
  </tr>
  <tr>
    <td><b>AgentStore(AT)</b></td>
    <td>Hybrid</td>
    <td>13.86</td>
    <td>31.91</td>
    <td>8.51</td>
    <td>39.13</td>
    <td>47.06</td>
    <td>40.00</td>
    <td>32.61</td>
    <td>39.13</td>
    <td>30.77</td>
    <td>23.85</td>
  </tr>
</table>

Table 2: Performance comparison On OSworld-Multi.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Agent Match</th>
    <th colspan="2">Subtask Acc</th>
    <th colspan="2">Execution Acc</th>
  </tr>
  <tr>
    <th>Match</th>
    <th>Acc</th>
    <th>Acc</th>
    <th>Acc</th>
  </tr>
  <tr>
    <td>MMAgent</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>6.93%</td>
  </tr>
  <tr>
    <td>AgentStore(FT)</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>AgentStore(ICL)</td>
    <td>24.75%</td>
    <td>40.00%</td>
    <td>9.90%</td>
    <td></td>
  </tr>
  <tr>
    <td>AgentStore(ICL*)</td>
    <td>28.71%</td>
    <td>51.72%</td>
    <td>14.85%</td>
    <td></td>
  </tr>
  <tr>
    <td><b>AgentStore(AT)</b></td>
    <td><b>36.63%</b></td>
    <td><b>62.16%</b></td>
    <td><b>22.77%</b></td>
    <td></td>
  </tr>
</table>

Additionally, we propose three metrics: Agent-Match, SubtaskAcc, and ExecutionAcc, which respectively measure multi-agent prediction accuracy, subtask decomposition accuracy, and execution success rate. Detailed benchmark constructions and metric descriptions are provided in Appendix F.

As shown in Table 2, single-agent MMAgent, which lacks Agent prediction and task decomposition, performs poorly on this benchmark. The Fine-Tuning method is not applicable in this scenario due to the infinite combinations of agents, making it impossible to pre-organize the necessary data for training. Moreover, while the ICL methods function to a certain extent, even with advanced commercial models (with ICL* indicating the use of GPT-4o), the constraints of overly long contexts and vast combinatorial spaces result in subpar outcomes. In contrast, AgentToken leverages its inherent task awareness, significantly narrowing the scope to a few selected agents. It demonstrates excellent performance across all metrics.

4.1.3 Mobile OS Assistant: APPAgent
We also employ the APPAgent (Yang et al., 2023) benchmark to validate that AgentStore can generalize to mobile OS platforms. It consists of 9 popular mobile applications, each serving distinct purposes and collectively forming 45 tasks.

Table 3 compares the performance of AgentStore with a single general agent in (Yang et al., 2023). As shown, the performance of the generalist agent, lacking specific knowledge of each app, is subpar across many applications, even when utilizing the strongest base model. In contrast, AgentStore constructs dedicated agents tailored to their respective applications, effectively addressing performance deficiencies in certain apps and demonstrating a significant performance improvement from 26.7% to 57.8%. This underscores the applicability of the AgentStore concept to other OS platforms.

Figure 3: The performance curve as the number of agents increases, with the y-axis representing the success rate (%) on OSWorld and the horizontal x-axis representing the number of agents.

4.2 Analysis of Agent Quantity and Diversity
To comprehensively analyze the advantages of scalable integration, we further explore the impact of the quantity and diversity of integrated agents within AgentStore on performance. To ensure thoroughness, we analyze AgentStore starting from a generalist MMAgent and incrementally add agents
Table 3: Success rates of generalist agents and AgentStore. Methods marked with “*” indicate the re-implementation of the single agent in APPAgent without app-specific knowledge. Due to differences between the original paper and the public benchmark, the results vary. Additionally, while Appagent also generated app-specific agents, it did not integrate them, instead only evaluating individual apps, and thus it is not included in the comparison.

<table>
  <tr>
    <th rowspan="2">Agent</th>
    <th rowspan="2">Base</th>
    <th colspan="9">Success Rate (%)</th>
  </tr>
  <tr>
    <th>Maps</th>
    <th>X</th>
    <th>TG</th>
    <th>Temu</th>
    <th>YT</th>
    <th>Spotify</th>
    <th>Yelp</th>
    <th>Gmail</th>
    <th>Clock</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>APPAgent-Single*</td>
    <td>Qwen-VL</td>
    <td>20.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>20.0</td>
    <td>4.4</td>
  </tr>
  <tr>
    <td>APPAgent-Single*</td>
    <td>GPT-4o</td>
    <td>60.0</td>
    <td>20.0</td>
    <td>20.0</td>
    <td>0.0</td>
    <td>40.0</td>
    <td>20.0</td>
    <td>20.0</td>
    <td>20.0</td>
    <td>40.0</td>
    <td>26.7</td>
  </tr>
  <tr>
    <td>AgentStore(GT)</td>
    <td>GPT-4o</td>
    <td>80.0</td>
    <td>60.0</td>
    <td>40.0</td>
    <td>40.0</td>
    <td>60.0</td>
    <td>80.0</td>
    <td>80.0</td>
    <td>60.0</td>
    <td>60.0</td>
    <td>66.7</td>
  </tr>
  <tr>
    <td>AgentStore(AT)</td>
    <td>GPT-4o</td>
    <td>80.0</td>
    <td>40.0</td>
    <td>40.0</td>
    <td>40.0</td>
    <td>60.0</td>
    <td>60.0</td>
    <td>80.0</td>
    <td>60.0</td>
    <td>60.0</td>
    <td>57.8</td>
  </tr>
</table>

Table 4: Routing success rates of different strategies for enabling MetaAgent as the router.

<table>
  <tr>
    <th rowspan="2">Agent</th>
    <th rowspan="2">Base</th>
    <th colspan="10">Success Rate (%)</th>
  </tr>
  <tr>
    <th>OS</th>
    <th>Calc</th>
    <th>Impress</th>
    <th>Writer</th>
    <th>VLC</th>
    <th>TB</th>
    <th>Chrome</th>
    <th>VSC</th>
    <th>GIMP</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>ICL</td>
    <td>GPT-4o</td>
    <td>58.33</td>
    <td>14.89</td>
    <td>12.77</td>
    <td>13.04</td>
    <td>88.24</td>
    <td>100</td>
    <td>97.83</td>
    <td>60.87</td>
    <td>53.85</td>
    <td>49.63</td>
  </tr>
  <tr>
    <td>ICL</td>
    <td>InternVL</td>
    <td>37.50</td>
    <td>6.38</td>
    <td>21.28</td>
    <td>8.70</td>
    <td>35.29</td>
    <td>33.33</td>
    <td>52.17</td>
    <td>30.43</td>
    <td>30.77</td>
    <td>41.57</td>
  </tr>
  <tr>
    <td>FT-LoRA</td>
    <td>InternVL</td>
    <td>50.00</td>
    <td>74.47</td>
    <td>55.32</td>
    <td>13.04</td>
    <td>88.23</td>
    <td>100</td>
    <td>89.13</td>
    <td>30.43</td>
    <td>34.61</td>
    <td>60.82</td>
  </tr>
  <tr>
    <td><b>AgentToken</b></td>
    <td>InternVL</td>
    <td>75.00</td>
    <td>80.85</td>
    <td>72.34</td>
    <td>43.47</td>
    <td>100</td>
    <td>100</td>
    <td>95.65</td>
    <td>91.30</td>
    <td>73.08</td>
    <td>80.60</td>
  </tr>
</table>

Table 5: Efficiency comparison.

<table>
  <tr>
    <th>Method</th>
    <th>Params</th>
    <th>Memory</th>
    <th>Time</th>
  </tr>
  <tr>
    <td>FT-Full</td>
    <td>7.78B</td>
    <td>>80G</td>
    <td>-</td>
  </tr>
  <tr>
    <td>FT-Pt</td>
    <td>86K</td>
    <td>26G</td>
    <td>-</td>
  </tr>
  <tr>
    <td>FT-LoRA</td>
    <td>38M</td>
    <td>28G</td>
    <td>2.5 hours</td>
  </tr>
  <tr>
    <td><b>AgentToken</b></td>
    <td><b>86K</b></td>
    <td><b>17G</b></td>
    <td><b>0.2 hours</b></td>
  </tr>
</table>

in AgentPool to compare their effects.

We employ two strategies for adding agents: one involves randomly selecting agents to incrementally add to the AgentPool, while the other categorizes agents into GUI and CLI types, starting with one type before supplementing with the other. As shown in Figure 3, performance gradually increases with the growing number of agents, confirming the performance benefits of scalable integration within AgentStore. Additionally, we observe differences between the two strategies: random selection maintains a consistent mix of agent types, leading to a more stable growth. In contrast, adding agents of only one type causes the growth rate to slow over time, but this is mitigated when the other type is introduced (the x-axis reaches 11). This highlights the crucial role of agent diversity, demonstrating the importance of integrating heterogeneous agents.

4.3 Analysis of AgentToken

In this section, extensive experiments demonstrate the advantages of AgentToken in terms of effectiveness, efficiency, and low data requirements.

Effectiveness  As shown in Table 4, ICL methods perform poorly as routers, even when using advanced models like GPT-4o. This confirms our assertion that relying on simple descriptions and few-shot demonstrations to master new agents can be challenging. In contrast, other tuning methods show some improvement by training on more task demonstrations. However, these methods are highly dependent on the quantity of data, while their overall performance improvement remains marginal. In comparison, our AgentToken overcomes these challenges, requiring only minimal data to efficiently train the corresponding tokens.

Efficiency  In Table 5, we compared the efficiency of the AgentToken with other efficient-tuning methods, i.e., prompt tuning (Pt) and adapter tuning (LoRA), focusing on the number of trainable parameters, memory requirements, and training time on the same A100 device. Results indicate that AgentToken is the most efficient across all dimensions, requiring the least amount of parameters and memory with the shortest training duration. Specifically, because AgentToken eliminates the need for gradients to flow through the main body of MLLM, training time is significantly reduced, and the process becomes more stable. Conversely, full fine-tuning and prompt tuning fail to converge properly due to their sensitivity to data.

Data Requirement  Generally, the larger and higher-quality the demonstration set \( S_t \), the more beneficial it is for training AgentToken. However,
Task-1: Set up to forward every email received by anonym-x2024@outlook.com in the future to anonym-x2024@gmail.com.

Step1: click(file_x, filters_y)
# Click on ["Manage message filters"]

Step2: click(view_x, new_y)
# Click on "New..." to create a new filter

Step3: type(enterForward to Gmail) ... click(choose_x, choose_y), type(enter"anonym-x2024@gmail.com")
# Ensure the filter is enabled and close the window

Task-2 : In a new sheet with "Year", "CA changes", "FA changes", and "OA changes", calculate the annual changes for the Current, Fixed, and Other Assets columns.

Step 2: create new sheet and headers

SheetAgent init_state
Step 1: install and locate
pip install openpyxl && ls | grep ".xlsx"
Successfully install openpyxl /home/user/SmallBalanceSheet.xlsx

Step 3: insert table for the required data
from openpyxl import load_workbook
file_path = '/home/user/.../Sheet.xlsx'
load_workbook(file_path)[sheet_name] ...
wb.create_sheet(title='Sheet2')
ws_new.append(headers), ws_new.append(data)
# Successfully run

final_state # Successfully execute

from openpyxl import load_workbook
original = load_workbook(file_path).active
for row in range(2, original.max_row+1):
    ea_current = original.cell(row=row, column=5).value
    ea_previous = original.cell(row=row-1, column=5).value
    ea_change = (ea_current-ea_previous)
    ws_new.cell(row=row, column=6).value = ea_change
# Save the workbook

Figure 4: Specific steps involved in executing three tasks mentioned in the qualitative analysis.

![Three task execution steps with screenshots and code examples](page_180_97_1082_377.png)

Figure 5: The accuracy curves with increasing training data corresponding to one agent. The x-axis represents the demonstration set size corresponding to each agent. The left y-axis represents the routing accuracy while the right y-axis indicates the executing accuracy.

![Accuracy curves plot](page_180_573_1082_377.png)

in practical scenarios, manually acquiring a large volume of high-quality demonstrations poses significant challenges. The proposed automated process can mitigate this issue by generating data automatically; nevertheless, the scope of the generated data remains relatively limited (Shumailov et al., 2024). Fortunately, AgentToken can still be effectively trained due to its small parameter size and stable training process. As shown in Figure 5, when the demonstration set size reaches 100, a satisfactory accuracy rate can be achieved.

Task-3: Could you assist me in boosting the contrast of my photo in the desktop and then insert it into the opening document at the point of the cursor?

Step 1: install and locate
apt-get install -y imagemagick & & ls ~/Desktop
Successfully install imagemagick - Desktop/berries.png

Step 2: boosting the contrast
convert ~/Desktop/berries.png -contrast ~/Desktop/berries_contrast.png
Writer -- # Successfully execute

Step 1: click(word_x, word_y)
Step 2: hotkey("ctrl", "v")
Step 3: hotkey("ctrl", "s")

Figure 4: Specific steps involved in executing three tasks mentioned in the qualitative analysis.

4.4 Qualitative Analysis

In Figure 4, we provide running cases to illustrate how AgentStore enhances the overall system’s capability. In Task-1, the agent is tasked with setting up automatic email forwarding, which involves frequent GUI interactions and requires understanding of Thunderbird settings. MetaAgent assigns the specialized MailAgent to handle the task which navigates the exact steps to configure the forwarding settings. In particular, during the Step3, it executes a sequence of actions to accurately fill out the required forms and options, showcasing its advanced understanding capabilities within the mail domain. Similarly, in Task-2, which requires complex processing of a spreadsheet, MetaAgent selects the SheetAgent from the AgentPool to handle the task, avoiding overly complex GUI interactions. SheetAgent possesses knowledge of “openpyxl” and a deep understanding of the steps needed to manipulate sheets, efficiently completing this task. In addition, Task-3 illustrates a system-wide task that requires collaboration among multiple agents. MetaAgent successfully decomposes the task into subtasks and assigns the appropriate agents to complete each one. These cases highlight AgentStore’s specialized generalist abilities in handling both domain-specific and system-wide tasks.
5 Conclusion

In this paper, we introduce AgentStore, a flexible and scalable platform for dynamically integrating heterogeneous agents to independently or collaboratively complete OS tasks. Furthermore, we propose MetaAgent with the AgentToken strategy to achieve efficient management of the growing number of agents. Comprehensive quantitative analysis and qualitative results show that AgentStore expands the capabilities of existing agent systems in both generalization and specialization. We believe that as basic AGI models continue to evolve, AgentStore will integrate more powerful agents, progressively advancing toward the vision of building the specialized generalist computer assistant.

6 Acknowledgements

This work was supported by the National Nature Science Foundation of China (No. 62192781, No. 62272374), the Natural Science Foundation of Shaanxi Province (2024JC-JCQN-62), the National Nature Science Foundation of China (No. 62202367, No. 62250009), the Key Research and Development Project in Shaanxi Province No. 2023GXLLH-024, Project of China Knowledge Center for Engineering Science and Technology, and Project of Chinese academy of engineering “The Online and Offline Mixed Educational Service System for ‘The Belt and Road’ Training in MOOC China”, and the K. C. Wong Education Foundation.

Ethics Statement

This research focuses on building a scalable platform to integrate heterogeneous agents dynamically. The data datasets or benchmarks we employed are properly cited. There are no discrimination, bias, or fairness issues that need to be declared in this paper. Further, the outputs are not expected to be potentially harmful. To ensure reproducibility, we provide experimental details in Section 4 and their corresponding appendices. All source code will be made public.

Limitations

While AgentStore has proven effective in multiple benchmarks, there still exist the following two directions for exploration: (1) Latency under high workloads: While task routing is efficient, the simultaneous deployment of multiple agents in high-demand scenarios can lead to increased response times. Further optimization is required to minimize delays in such environments. (2) Scaling with larger AgentPool: The current framework demonstrates promising results with moderately sized agent pools, but further research is needed to enable seamless integration and management of significantly larger AgentPools. It is essential to explore more robust mechanisms that can dynamically allocate resources and resolve conflicts among a large number of agents. Additionally, evaluating the system on more complex and diverse benchmarks involving extensive multi-agent collaboration will be critical for validating its scalability and adaptability.

References

Kirti Aggarwal and Harsh K Verma. 2015. Hash_rc6—variable length hash algorithm using rc6. In 2015 International Conference on Advances in Computer Engineering and Applications, pages 450–456. IEEE.

Ziwei Chai, Guoyin Wang, Jing Su, Tianjie Zhang, Xuanwen Huang, Xuwu Wang, Jingjing Xu, Jianbo Yuan, Hongxia Yang, Fei Wu, et al. 2024. An expert is worth one token: Synergizing multiple expert limbs as generalist via expert token routing. arXiv preprint arXiv:2403.16854.

Harrison Chase. 2022. LangChain.

Yibin Chen, Yifu Yuan, Zeyu Zhang, Yan Zheng, Jinyi Liu, Fei Ni, and Jianye Hao. 2024a. Sheetagent: A generalist agent for spreadsheet reasoning and manipulation via large language models. arXiv preprint arXiv:2403.03636.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024b. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821.

Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935.

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayyaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378.

Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2024. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. Advances in neural information processing systems, 36.

Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919.

Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyuan Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. 2023. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352.

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2024. Coagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290.

Guohao Li, Hasan Hammoud, Hani Itani, Dmitri Khizhbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991–52008.

Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and ZHAO-XIANG ZHANG. 2024. Sheetcopolit: Bringing software productivity to the next level through large language models. Advances in Neural Information Processing Systems, 36.

Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2023. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170.

Open-interpreter. 2024. open-interpreter.

OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.

Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22.

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, and et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403.05530.

Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. 2024. Ai models collapse when trained on recursively generated data. Nature, 631(8022):755–759.

Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. 2024. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint arXiv:2403.14734.

Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. 2023. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration. arXiv preprint arXiv:2310.00280.

Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, et al. 2024. Towards general computer control: A multimodal agent for red dead redemption ii as a case study. arXiv preprint arXiv:2403.03186.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291.

Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2024. Opendevin: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In The 61st Annual Meeting Of The Association For Computational Linguistics.

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155.

Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and
Lingpeng Kong. 2024. Os-copilot: Towards generalist computer agents with self-improvement. Preprint, arXiv:2402.07456.

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. Os-world: Benchmarking multimodal agents for open-ended tasks in real computer environments. Preprint, arXiv:2404.07972.

Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771.

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.

Kaiyan Zhang, Biquing Qi, and Bowen Zhou. 2024. Towards building specialized generalist ai with system 1 and system 2 fusion. arXiv preprint arXiv:2407.08642.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.
A AgentPool

The AgentPool is a collection of all available agents within AgentStore. To build the prototype of AgentStore, we organized 20 agents within AgentPool, each with distinct functionalities. As shown in Table 6, these agents range from unimodal to multimodal, from open-source to closed-source models, and from Command-Line Interfaces (CLI) to Graphical User Interfaces (GUI). The diverse capabilities of these agents cover common applications and tasks in both daily life and professional settings. In addition to the domain-specific agents we developed, we also integrated existing agents, such as Friday (Wu et al., 2024) and (He et al., 2024). This demonstrates the scalability of our approach, which allows third-party agents to be added to the platform.

Specifically, for closed-source model agents, we uniformly use GPT-4o as the base model. For open-source model agents, single-modality agents are based on Llama 3.1 (Touvron et al., 2023), while multi-modality agents are built on InternVL2 (Chen et al., 2024b). The last column of Table 6 indicates whether the agent has the capability to solve tasks outside its own domain.

Figure 6 illustrates the distribution of different types of agents, showing that the initial version of AgentStore maintains a consistent balance between GUI and CLI agents. Most models also support extensions to handle additional tasks. Due to the significant gap between open-source and close-commercial models, most agents in this version are currently based on close-commercial models.

![Pie charts showing the distribution of agents by GUI/CLI, single/multi-modal, open/close base model, and extension/no-extension](page_755_186_393_312.png)

Figure 6: The agent distribution across different types.

<table>
  <tr>
    <th>CLI or GUI?</th>
    <th>Single or Multi Modal?</th>
    <th>Open or Close Base Model?</th>
    <th>Domain for OSworld</th>
    <th>Support Extension?</th>
  </tr>
  <tr>
    <td>OSAgent</td>
    <td>GUI</td>
    <td>Multi</td>
    <td>Close</td>
    <td>OS</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Friday (Wu et al., 2024)</td>
    <td>CLI</td>
    <td>Single</td>
    <td>Close</td>
    <td>OS</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>SheetAgent</td>
    <td>CLI</td>
    <td>Single</td>
    <td>Close</td>
    <td>Calc</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>CalcAgent</td>
    <td>GUI</td>
    <td>Multi</td>
    <td>Close</td>
    <td>Calc</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>SlideAgent</td>
    <td>CLI</td>
    <td>Single</td>
    <td>Close</td>
    <td>Impress</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>ImPressAgent</td>
    <td>GUI</td>
    <td>Multi</td>
    <td>Close</td>
    <td>Impress</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>WordAgent</td>
    <td>CLI</td>
    <td>Single</td>
    <td>Close</td>
    <td>Writer</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>WriterAgent</td>
    <td>GUI</td>
    <td>Multi</td>
    <td>Close</td>
    <td>Writer</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>VLCAgent</td>
    <td>GUI</td>
    <td>Multi</td>
    <td>Close</td>
    <td>VLC</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>MailAgent</td>
    <td>GUI</td>
    <td>Multi</td>
    <td>Close</td>
    <td>TB</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>ChromeAgent</td>
    <td>GUI</td>
    <td>Multi</td>
    <td>Close</td>
    <td>Chrome</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>WebAgent (He et al., 2024)</td>
    <td>GUI</td>
    <td>Multi</td>
    <td>Close</td>
    <td>Chrome</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>VSAgent</td>
    <td>GUI</td>
    <td>Multi</td>
    <td>Open</td>
    <td>VSC</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>VGUAIAgent</td>
    <td>CLI</td>
    <td>Single</td>
    <td>Close</td>
    <td>VSC</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>GimpAgent</td>
    <td>GUI</td>
    <td>Multi</td>
    <td>Close</td>
    <td>GIMP</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>ImageAgent</td>
    <td>CLI</td>
    <td>Single</td>
    <td>Open</td>
    <td>GIMP</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Searcher</td>
    <td>CLI</td>
    <td>Single</td>
    <td>Close</td>
    <td>-</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>GoogleDrive</td>
    <td>CLI</td>
    <td>Single</td>
    <td>Close</td>
    <td>-</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>CoderAgent</td>
    <td>CLI</td>
    <td>Single</td>
    <td>Open</td>
    <td>-</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>VisionAgent</td>
    <td>CLI</td>
    <td>Multi</td>
    <td>Open</td>
    <td>-</td>
    <td>✗</td>
  </tr>
</table>

Table 6: The presentation of agents in the AgentPool.
Integration Protocol: When a developer creates a new OS agent and seeks to integrate it into AgentStore, it is essential to register the agent’s information in a standardized format. To ensure consistency in the integration process, we established an agent integration protocol. As shown in the template below, during enrollment, the developer completes a predefined form outlining the agent’s capabilities, limitations, the applications it interacts with, and demonstrations of its functionality.

In the actual enrollment process, we encourage developers to provide more demonstrations—the greater the number, the more comprehensive the document will be, which also facilitates agentToken training during the self-instruct process. In this paper, we provide 10 demonstrations for each agent, which is relatively lightweight but still effectively aids the Metaagent in learning and understanding the corresponding agent.

Template: AgentName

# Applications:
# List the applications or tools that the agent supports or interacts with.

# Capabilities
# Describe the main functions and abilities of the agent. Include details about the tasks it can perform and the libraries or technologies it utilizes.

# Limitations
# Outline the constraints and tasks the agent cannot perform. This helps set clear boundaries for the agent’s functionality.

# Demonstrations
# Demostation_1: <Description of the first demonstration task.> <path_to_demonstration_image_1>
# Demostation_2: <Description of the second demonstration task.> <path_to_demonstration_image_2>
# Demostation_3: <Description of the third demonstration task.> <path_to_demonstration_image_3>
# Demostation_4: <Description of the fourth demonstration task.> <path_to_demonstration_image_4>
......
End!

B  Details of Agents within AgentPool

Following the above template, we present six typical agent documents related to LibreOffice tasks to provide a clearer view of the agents in the AgentPool. Due to space limitations, further details on additional agents will be available when the entire project is open-sourced.

SlideAgent and ImPressAgent: These two agents serve distinct roles in slide processing tasks, with SlideAgent being CLI-based and ImPressAgent GUI-based. SlideAgent is derived through enhanced prompting of the FridayAgent (Wu et al., 2024), leveraging the Python-pptx library documentation. It effectively utilizes bash commands to locate file paths and generate Python code to perform operations on slides, such as adding titles or modifying text. In contrast, ImPressAgent specializes in handling tasks via GUI interactions with LibreOffice Impress, excelling in modifying software defaults, adjusting layout details, and performing other visual modifications that require direct interface manipulation.

WordAgent and WriterAgent: These two agents serve distinct roles in document processing tasks, with WriterAgent being CLI-based and WordAgent GUI-based. WriterAgent utilizes Python’s python-docx library to perform operations such as modifying paragraphs, adding or removing tables, and executing other programmatic edits on Word documents. In contrast, WordAgent specializes in handling GUI operations in LibreOffice Writer software, excelling in modifying software defaults, adjusting layout settings, and performing visually oriented tasks that require direct interaction with the user interface.

SheetAgent and CalcGUI: These two agents are designed for spreadsheet processing tasks, with SheetAgent being CLI-based and CalcGUI GUI-based. SheetAgent leverages Python’s openpyxl library to excel in tasks such as data entry, formula insertion, chart creation, and spreadsheet formatting, offering
programmatic efficiency for managing spreadsheets. On the other hand, CalcGUI specializes in handling GUI operations in LibreOffice Calc software, excelling in modifying software defaults, adjusting layout settings, and performing visually driven tasks that require direct interface manipulation.

AgentName: SlideAgent

# Applications:
Terminal, LibreOffice Impress

# Capabilities
Specializes in creating and modifying PowerPoint presentations using Python's python-pptx library. It can handle tasks involving slide creation, layout management, text and content insertion, and formatting adjustments. Also capable of detecting open PowerPoint presentations using Bash commands.

# Limitations
Cannot handle GUI operations, cannot perform tasks outside the capabilities of the python-pptx library such as directly interacting with embedded videos and complex animations. Additionally, cannot modify LibreOffice Impress software defaults or preferences.

# Demostations

Demostation_1: Can you add a new slide at the end of my presentation with the title 'Conclusion' and the text 'Thank you for your attention'?"

![Screenshot of a PowerPoint presentation showing a new slide being added](page_374_629_700_246.png)

Demostation_2: Can you add a footer with text 'Company Confidential' to all slides in the current PowerPoint presentation?

![Screenshot of a PowerPoint presentation showing a footer added to all slides](page_374_1012_700_246.png)

......
End!
AgentName: ImPressAgent

# Applications:
LibreOffice Impress

# Capabilities
Specializes in handling tasks using GUI operations and can modify LibreOffice Impress software defaults or preferences.

# Limitations
Cannot handle complex tasks such as creating and modifying PowerPoint presentations using Python's python-pptx library.

# Demonstrations

Demostation_1: Enable the "Grid" view to help with precise placement of objects.

![Screenshot showing the LibreOffice Impress grid view enabled](page_246_370_957_246.png)

Demostation_2: Change the default font for all text in the presentation to "Helvetica".

![Screenshot showing the LibreOffice Impress font change dialog](page_246_682_957_246.png)

......
End!
AgentName: WordAgent

# Applications:
Terminal, LibreOffice Writer

# Capabilities
Excels at identifying and manipulating Word documents using Python's python-docx library. Can manage tasks involving document modification, data insertion, and formatting adjustments. Capable of detecting open Word or other documents using Bash commands.

# Limitations
Cannot handle GUI operations, cannot perform tasks outside the capabilities of the python-docx library such as directly interacting with embedded media and scripts within the documents. Additionally, cannot modify LibreOffice Writer software defaults or preferences.

# Demonstrations

Demostation_1: Add the text 'Grand Opening' as a title at the beginning of the document."

![Screenshot showing the addition of a title 'Grand Opening' at the beginning of a Word document](page_328_495_792_180.png)

Demostation_2: Insert a horizontal line above the 'ABSTRACT' heading.

![Screenshot showing a horizontal line inserted above the 'ABSTRACT' heading in a Word document](page_328_764_792_180.png)

End!
AgentName: WriterAgent

# Applications:
LibreOffice Writer

# Capabilities
Specializes in handling GUI operations and can perform tasks outside the capabilities of the python-docx library such as directly interacting with embedded media and scripts within documents. Can modify LibreOffice Writer software defaults or preferences.

# Limitations
Cannot identify and manipulate Word documents using Python's python-docx library, and cannot manage tasks involving document modification, data insertion, and formatting adjustments. Additionally, cannot detect open Word or other documents using Bash commands.

# Demonstrations

Demostation_1: Enable the "Show Changes" feature to track document edits location.

![Screenshot showing the "Show Changes" feature enabled in LibreOffice Writer](page_324_563_798_246.png)

Demostation_2: Create a custom keyboard shortcut for "Print" set to Ctrl+P.

![Screenshot showing the creation of a custom keyboard shortcut for Print](page_324_823_798_246.png)

End!
AgentName: SheetAgent

# Applications:
Terminal, LibreOffice Calc

# Capabilities
Specializes in creating, analyzing, and modifying Excel spreadsheets using Python's openpyxl library. It can handle tasks involving data entry, formula insertion, chart creation, and spreadsheet formatting. Also capable of detecting open Excel files using Bash commands.

# Limitations
Cannot handle GUI operations, cannot perform tasks outside the capabilities of the openpyxl library such as directly interacting with complex macros. Additionally, cannot modify LibreOffice Calc software defaults or preferences.

# Demonstrations

Demostation_1: Highlight rows where the total sales exceed $1000.

![Screenshot of an Excel sheet with highlighted rows based on total sales exceeding $1000](page_324_563_800_246.png)

Demostation_2: Filter out players older than 35 and list their names and ages in a new sheet named "Veteran Players".

![Screenshot of an Excel sheet showing filtered players older than 35](page_324_900_800_246.png)

End!
AgentName: CalcGUI

# Applications:
LibreOffice Calc

# Capabilities
Specializes in handling tasks using GUI operations and can modify LibreOffice Calc software defaults or preferences.

# Limitations
Cannot handle complex tasks such as creating, analyzing, and modifying Excel spreadsheets using Python's openpyxl library.

# Demonstrations

Demostation_1: Set the row height to 18 pixels for better readability.

![Screenshot showing LibreOffice Calc window with row heights set to 18 pixels](page_246_370_957_246.png)

Demostation_2: Filter the data to show only rows where "Health: Mortality, under-5" is greater than 50.

![Screenshot showing LibreOffice Calc window with filtered data for "Health: Mortality, under-5" greater than 50](page_246_670_957_246.png)

End!
C  Cost analysis

While AgentStore introduces some computational overhead during deployment, its architecture demonstrates notable advantages in reducing task execution time and cost through the efficient assignment of specialized agents. The comparative analysis of AgentStore and MMAgent highlights these benefits, as summarized in Table 7. Due to potential variations in API response times influenced by network conditions, the reported running time and cost should be considered approximate estimates.

Table 7: The running times and costs between MMAgent and AgentStore.

<table>
  <tr>
    <th>Method</th>
    <th>Base-model</th>
    <th>running time</th>
    <th>Budget Cost</th>
  </tr>
  <tr>
    <td>MMAgent (Xie et al., 2024)</td>
    <td>GPT-4o</td>
    <td>\( \approx 12h \)</td>
    <td>\( \approx 250\$ \)</td>
  </tr>
  <tr>
    <td>AgentStore(AT)</td>
    <td>GPT-4o+Llama 3.1+InternVL2</td>
    <td>\( \approx 7h \)</td>
    <td>\( \approx 100\$ \)</td>
  </tr>
</table>

Such improvements are attributable to the following factors:
(1) Task-Specialized Assignment: By employing a diverse pool of agents, AgentStore allocates tasks to agents best equipped for specific domains, minimizing redundant or inefficient processing steps.
(2) Optimized Collaboration Framework: The MetaAgent’s AgentToken strategy enables effective coordination among agents, ensuring tasks are completed collaboratively where appropriate, without unnecessary computational duplication.
(3) Model Efficiency: The inclusion of open-source models, such as Llama 3.1 and InternVL2, reduces dependence on commercial models like GPT-4o for tasks.
These findings underscore the potential of AgentStore to serve as a cost-effective and time-efficient platform for automating diverse tasks in operating system environments.

D  Automated process with self-instruct

In this section, we provide more details about the Automated data generation process, including threshold selection and the greedy filtering algorithm.

Threshold Selection  To ensure the reliability of threshold selection, we first studied the distribution of thresholds in real-world tasks based on human-labeled standards. As shown in Figure 7, in tasks labeled by OSworld, the 95% threshold distribution of BertScore across different domains is primarily concentrated between 0.77 and 0.92. Therefore, to further strictly control the quality of generated data, we ultimately selected a threshold of 0.8 for \( \tau_1 \) and 0.9 for \( \tau_2 \) to filter the data.

This approach offers several advantages. By selecting thresholds of 0.8 for \( \tau_1 \) and 0.9 for \( \tau_2 \), we strike a balance between retaining high-quality data and ensuring the diversity necessary for robust training. The \( \tau_1 \) threshold helps in eliminating low-quality samples, while \( \tau_2 \) enforces stricter criteria for the final selection of data, ensuring that only the most relevant and high-quality data points are used. This dual-threshold filtering process not only improves the precision of the generated data but also enhances the overall performance of agent training, reducing the risk of overfitting to noise or irrelevant tasks.

Greedy Filtering Algorithm  Algorithm 1 presents a greedy algorithm for filtering a set of newly generated demonstrations, \( S'_i \), ensuring that each selected demonstration maintains a BERTScore similarity within the specified bounds \( \tau_1 \) and \( \tau_2 \) relative to both existing demonstrations \( S_i \) and previously selected new demonstrations \( S_i^{new} \). The key improvement lies in the prioritization of demonstrations that are optimally positioned between the two thresholds, thereby enhancing both relevance and diversity.

A prioritization mechanism selects demonstrations optimally positioned between the similarity thresholds. By calculating the minimum distance of each candidate’s BERTScore to the thresholds, the algorithm ensures that selected demonstrations are neither too similar nor too dissimilar to existing ones. This strategic ordering facilitates the inclusion of the most appropriate demonstrations first, thereby maximizing both the relevance and diversity of the refined set \( S_i^{new} \). Consequently, the quality of the training data for AgentToken is significantly improved, fostering more effective training outcomes.
![BertScore Distribution across Domains](page_184_120_1080_370.png)

Figure 7: BertScore distribution across different domains.

Algorithm 1 Greedy Filtering of Generated Demonstrations using BERTScore with Prioritized Selection

Require:   \( S'_i = \{y'_1, y'_2, \ldots, y'_m\} \): Set of newly generated demonstrations
    • \( S_i = \{y_1, y_2, \ldots, y_n\} \): Existing set of demonstrations
    • \( \tau_1 \): Lower bound for BERTScore similarity
    • \( \tau_2 \): Upper bound for BERTScore similarity
Ensure:   \( S^{new}_i \): Refined set of new demonstrations satisfying the similarity constraints
1: Initialize \( S^{new}_i \leftarrow \emptyset \)
2: For each \( y' \in S'_i \), compute the minimum distance to the thresholds:

\[
d(y') = \min(|\text{BERTScore}(y', y) - \tau_1|, |\text{BERTScore}(y', y) - \tau_2|) \quad \forall y \in S_i
\]

3: Sort \( S'_i \) in descending order based on \( d(y') \)
4: for each \( y' \in S'_i \) in sorted order do
5:     Initialize a flag \( valid \leftarrow \) True
6:     for each \( y \in S_i \cup S^{new}_i \) do
7:         Compute BERTScore(\( y', y \))
8:         if BERTScore(\( y', y \)) < \( \tau_1 \) or BERTScore(\( y', y \)) > \( \tau_2 \) then
9:             \( valid \leftarrow \) False
10:            break
11:        end if
12:    end for
13:    if \( valid \) then
14:        Add \( y' \) to \( S^{new}_i \)
15:    end if
16: end for
17: return \( S^{new}_i \)

E OSWorld

OSWorld (Xie et al., 2024) is a scalable, computer environment designed for multimodal agents. This platform provides a real-world environment for assessing open-ended computer tasks involving various applications. In this section, we provide a detailed introduction to OSworld, focusing on three key aspects: the open-ended and diverse nature of tasks, the reliability of evaluations in real-world environments, and the varied capability requirements for agents. This aims to help readers understand the rationale behind
Figure 9: OSWorld can serve as a unified environment for evaluating open-ended computer tasks in the real-world computer environment.

using OSworld as the primary evaluation platform in the main text.

E.1 OSWorld Tasks
OSWorld is a benchmark suite consisting of 369 real-world computer tasks, primarily based in an Ubuntu Linux environment, with a smaller portion covering Microsoft Windows. The tasks are sourced from the authors as well as various platforms like forums, tutorials, and guidelines. Each task is paired with a natural language instruction and a hand-crafted evaluation script for scoring. Figure 8 provides a detailed classification of tasks, showcasing their diversity and effectively reflecting the nature of open-ended tasks in real-world scenarios.

E.2 Real-world Computer Environment
As shown in Figure 9, OSworld provides an executable and controllable environment that supports task initialization, execution-based evaluation, and interactive agent learning in a range of real operating systems. It also provides easily accessible system screenshots, ally-tree information, and interfaces that facilitate agent output for mouse and keyboard operations. This rich system information, real-time execution, and comprehensive task evaluation offer an interactive environment that is not limited to specific applications or domains.

E.3 Representitive Examples
In Table 8, we present several representative examples from OSworld, which aim to illustrate the distinct operational logic involved in different tasks and the diverse capabilities required. These examples help readers better understand the broad range of generalization and specialized skills necessary in real-world computer environments, which are challenging for a single agent to fully encompass.

Table 8: Representative Examples from OSWorld to illustrate the distinct operational logic and the diverse capabilities involved in different tasks.

<table>
  <tr>
    <th>Related App(s)</th>
    <th>Instruction(s)</th>
    <th>Screenshot</th>
    <th>Abilities Needed</th>
  </tr>
  <tr>
    <td>OS</td>
    <td><i>I want to install Spotify on my current system. Could you please help me?</i></td>
    <td>![Screenshot of a jellyfish image](page_1047_1532_312_180.png)</td>
    <td>specialized knowledge of OS; Proficient GUI operations</td>
  </tr>
</table>

Continued on next page
<table>
  <tr>
    <th>Related App(s)</th>
    <th>Task Instruction</th>
    <th>Screenshot of Initial State</th>
    <th>Abilities Needed</th>
  </tr>
  <tr>
    <td>Calc</td>
    <td><i>I have a lookup table for the officers of each branch. Please, here is another table in which I need to fill with the officer names according the headoffice (i.e., the branch name). Help me to complete this.</i></td>
    <td>![Screenshot of Calc application showing a table and some data](page_420_186_370_180.png)</td>
    <td>Familiarity with the openpyxl library and command-line proficiency</td>
  </tr>
  <tr>
    <td>Impress</td>
    <td><i>I closed the slide pannel on the left and idk how to get it back please help</i></td>
    <td>![Screenshot of Impress presentation software showing a blank slide](page_420_420_370_180.png)</td>
    <td>specialized knowledge of Slide software; imagine about UI layouts; Proficient GUI operations</td>
  </tr>
  <tr>
    <td>Chrome</td>
    <td><i>Can you help me clean up my computer by getting rid of all the tracking things that Amazon might have saved? I want to make sure my browsing is private and those sites don’t remember me.</i></td>
    <td>![Screenshot of Chrome browser showing settings menu](page_420_654_370_180.png)</td>
    <td>specialized knowledge of Chrome browser, proficient GUI operations</td>
  </tr>
  <tr>
    <td>VLC</td>
    <td><i>Hey, could you turn this video the right way up for me? And once it’s flipped around, could you save it for me with the name ‘1984_Apple.mp4’ on the main screen where all my files are?</i></td>
    <td>![Screenshot of VLC media player showing a video window](page_420_888_370_180.png)</td>
    <td>software knowledge; spatial judgment ability</td>
  </tr>
  <tr>
    <td>Thunderbird</td>
    <td><i>Create a local folder called "Promotions" and create a filter to auto move the inbox emails whose subject contains “discount” to the new folder</i></td>
    <td>![Screenshot of Thunderbird email client showing folder creation and filter setup](page_420_1122_370_180.png)</td>
    <td>Knowledge of the Thunderbird mail system; GUI operations</td>
  </tr>
</table>

Continued on next page
<table>
  <tr>
    <th>Related App(s)</th>
    <th>Task Instruction</th>
    <th>Screenshot of Initial State</th>
    <th>Abilities Needed</th>
  </tr>
  <tr>
    <td>VS Code</td>
    <td><i>Please modify VS Code’s settings to disable error reporting for Python missing imports.</i></td>
    <td>![Screenshot of Visual Studio Code settings window](page_420_186_370_180.png)</td>
    <td>software knowledge to deal with settings; reasoning to understand the cause and solution of the error</td>
  </tr>
  <tr>
    <td>GIMP</td>
    <td><i>Could you tone down the brightness of my photo?</i></td>
    <td>![Screenshot of GIMP photo editing interface](page_420_420_370_180.png)</td>
    <td>Proficiency in using ImageMagick and CLI operations</td>
  </tr>
  <tr>
    <td>GIMP</td>
    <td><i>Help me choose the yellow triangle and position it at the center of my picture.</i></td>
    <td>![Screenshot of GIMP with a yellow triangle](page_420_654_370_180.png)</td>
    <td>spatial perception and reasoning, as well as precise control of actions</td>
  </tr>
  <tr>
    <td>Multiple (VLC+GIMP)</td>
    <td><i>Could you help me create an Animated GIF from a video file using VLC and GIMP from the source of video “src.mp4”, 5-second clip beginning at 00:03?</i></td>
    <td>![Screenshot of VLC media player playing a video](page_420_888_370_180.png)</td>
    <td>specialized software knowledge; ability to process multi-step procedure successfully</td>
  </tr>
  <tr>
    <td>Multiple (Chrome+Calc)</td>
    <td><i>Could you help me extract data in the table from a new invoice uploaded to my Google Drive, then export it to a Libreoffice calc .xlsx file in the desktop?</i></td>
    <td>![Screenshot of Google Drive and LibreOffice Calc](page_420_1122_370_180.png)</td>
    <td>specialized ability to do table data; ability to process multi-step procedure successfully</td>
  </tr>
</table>
F OSWorld-Multi Benchmark

Building on OSworld, we further developed a new benchmark, OSWorld-Multi, to evaluate MetaAgent’s ability to predict and coordinate multiple agents for collaborative task execution. OSWorld-Multi consists of 101 tasks, each requiring collaboration with paired agents from the AgentPool. In the following sections, we will introduce the construction process, task examples, and evaluation metrics.

Construction process   To maximize the reuse of tasks, system states, and evaluation functions from OSworld, we adopted a reverse synthesis approach. By mining paired examples in OSWorld, we generated tasks requiring agent collaboration. Specifically, we first traversed all pairwise combinations of subtasks, applying a two-step validation process: an initial filtering with a large language model (LLM), followed by manual review. This method allowed us to select meaningful collaborative tasks. Moreover, this approach enabled the synthesis of tasks requiring not only two-agent collaboration but also those involving three or more agents. In the following section, we will present some of the generated collaborative task results to demonstrate the outcomes of this synthesis process.

Task examples   In the table below, we present several synthesized examples to help readers understand the generation process. Another advantage of this reverse synthesis approach is the presence of natural ground truth, allowing us to evaluate not only execution accuracy but also the accuracy of agent predictions and task decomposition. This enables a comprehensive assessment of collaborative task execution. In the following sections, we will provide a detailed explanation of the corresponding evaluation metrics.

Synthesis task 1

# Agent:Subtask-1
VLCAgent: Snap a photo of the current video scene, save it as 'interstellar.png', and put it on the Desktop, please.

# Agent:Subtask-2
WriterAgent: Add page number for every page at the bottom left.

# Synthesis task
Capture a scene from a video in VLC and insert the image into a LibreOffice document with a page number.

# Required: VLCAgent + WriterAgent

Synthesis task 2

# Agent:Subtask-1
VLCAgent: Help me modify the folder used to store my recordings to Desktop.

# Agent:Subtask-2
Friday: Change the permission of all regular files under current directory tree to 644.

# Synthesis task
Modify VLC's recording folder to Desktop and set file permissions to 644 for all files in this directory.

# Required: VLCAgent + Friday
Synthesis task 3

# Agent:Subtask-1

VLCAgent: Can you start streaming the video from this link for me?
https://www.youtube.com/watch?v=pgBsyTKAWLw

# Agent:Subtask-2

ChromeGUI: Could you help me clear browsing history from Youtube?

# Synthesis task

Could you stream a video from a YouTube link in VLC and clear all YouTube browsing history in Chrome after to ensure a clean search experience?

# Required: VLCAgent + ChromeGUI

Synthesis task ……

…….

Evaluation metrics   We propose three metrics for evaluation: AgentMatch, SubtaskAcc, and ExecutionAcc, which respectively measure multi-agent prediction accuracy, subtask decomposition accuracy, and execution success rate.

AgentMatch is designed to assess the accuracy of the agent prediction process during collaborative task execution. It compares the predicted set of agents selected by the MetaAgent with the ground truth set of agents that are required for successful task completion. Essentially, AgentMatch measures how well the MetaAgent can correctly identify the appropriate agents from the AgentPool for a given task. The metric is computed by calculating the accuracy of the predicted agent set relative to the actual agents involved in the task. Specifically, it checks whether the predicted agents match the expected agents. A high AgentMatch score indicates that the MetaAgent is effectively coordinating and predicting the correct agents for task execution.

SubtaskAcc is an evaluation metric that measures the accuracy of task decomposition by comparing the predicted subtasks assigned to each agent with the ground truth subtasks. It evaluates how well the MetaAgent decomposes a given task and assigns the correct subtasks to the respective agents. To assess SubtaskAcc, we use a textual comparison between the predicted subtasks and the actual subtasks for the same agent. This comparison is based on textual similarity, using BERTScore as the evaluation metric. As per our analysis in D, if the BERTScore is below 0.77, the two subtasks are considered too dissimilar, and the decomposition is deemed unsuccessful. Conversely, if the BERTScore exceeds this threshold, the decomposition is considered accurate. This threshold ensures that only decompositions with sufficiently high textual similarity are counted as correct. SubtaskAcc thus reflects how effectively the MetaAgent can break down a complex task and allocate the correct components to individual agents. A high SubtaskAcc score indicates that the MetaAgent is accurately identifying the required subtasks for each agent, contributing to the overall success of the collaborative task execution.

ExecutionAcc is an evaluation metric that measures the success rate of task execution by reusing the assessment methods from OSworld. This metric focuses on determining whether the predicted subtasks are correctly executed by the agents, based on their final state in the environment.

To evaluate ExecutionAcc, we rely on OSworld’s system of getter and evaluator functions. The getter function extracts key components from the final state of the environment (e.g., a modified file or text contents displayed in a window element), while the evaluator function assesses success based on these extracted components. If a necessary function does not exist, it is constructed and added to the function library of the environment. Each task is evaluated by comparing its final execution state with the expected outcome, and the evaluation process is designed to be robust.
In the context of our system, ExecutionAcc provides a direct measure of how successfully the agents complete their assigned tasks, reflecting the practical performance of task execution in real-world scenarios. A high ExecutionAcc indicates that the agents are accurately following the predicted subtasks and completing them correctly in the environment.

G  Prompt Details

We provide examples of MetaAgent prompts in different modes to help readers understand the inference process. It is important to note that in manager mode, the prompt templates in Section G.3 for AgentToken and ICL are identical. The key difference is that AgentToken reduces the number of input documents, effectively shortening the context length, which in turn improves performance.

Additional prompts, including those related to each individual agent and self-instruct, will be provided when the project is open-sourced.

G.1  Prompt for router mode for AgentToken

Prompt: Router for AgentToken

Imagine you have a complex task that needs to be executed on an operating system.
This task can be decomposed into sub-tasks corresponding to the model's capabilities.
You have several agents with different specializations available.
Requirements:
The task is assigned to one agent, the model should return the one token of that agent.
Now your task is: {task_name}

G.2  Prompt for router mode for ICL

Prompt: Router for ICL

Imagine you have a complex task that needs to be executed on an operating system.
This task can be decomposed into sub-tasks corresponding to the model's capabilities.
You have several agents with different specializations available.
{agent_1_document},{agent_2_document},...{agent_n_document}
Requirements:
The task is assigned to one agent, the model should return the name of that agent.
like:
###CalcAgent###
Now your task is: {task_name}

G.3  Prompt for manager mode

Prompt: Manager Mode

Imagine you have a complex task that needs to be executed on an operating system.
This task can be decomposed into sub-tasks corresponding to the model's capabilities.
You have agents with different specializations available:
{agent_1_document},{agent_2_document},...{agent_n_document}
Requirements:
The task requires multiple agents, the model should specify which sub-tasks each agent should handle.
The model should ensure that the task assignment optimizes efficiency and effectiveness, considering the unique capabilities of each agent.
return like:
###AgentName1:compute the sum of data in a new sheet.###
###AgentName2:upload the computed file to the google Drive###
Be careful not to assign the same agent to perform tasks consecutively.
don't return like this:
###Agent1:compute the sum of data in a new sheet.###
###Agent1:rename this sheet.###
Now your task is: {task_name}

Reviewer paper summaries: The paper introduces AgentStore, a scalable platform designed to integrate multiple agents for completing complex tasks. The authors propose MetaAgent and AgentToken to facilitate task routing among specialized agents, enabling effective collaboration. Experiments demonstrate that AgentStore achieves significant performance improvements on OSWorld and APPWorld, outperforming previous baseline agent designs.

This paper introduces AgentStore, a novel framework for building autonomous computer assistants that can handle a wide range of tasks within an operating system. AgentStore uses a collection of specialized agents with different skills and a central "MetaAgent" to intelligently assign tasks to the most suitable agent. This approach allows the system to be both general and specialized.  The researchers developed a novel training method that requires less manual input and showed that AgentStore significantly outperforms existing systems in various operating system tasks, including complex scenarios requiring collaboration between agents. AgentStore also proved to be more efficient in terms of time and cost compared to previous solutions.
Summary of strengths: 1. The design of AgentStore, resembling an app store, is relatively novel and effectively supports the scaling of different agents, establishing itself as a robust platform.
2. The AgentToken mechanism for task routing enhances multi-agent collaboration, improving the handling of complex tasks while reducing resource costs.
3. The figures are well-designed, clearly illustrating the key concepts. The analysis is comprehensive, with case studies and quantitative evaluations effectively supporting the claims.

* Clear motivation: this paper addresses an important question of multi-agent collaboration in OS. The motivation is strong and clear.
* The paper is well written and easy to follow.
* Strong Performance: Demonstrates significant improvements over state-of-the-art methods across multiple challenging benchmarks. Comprehensive analytical experiments have been done to explore the cost/efficiency/data requirement.
Summary of weaknesses: 1. Regarding the training process, platforms like this are typically augmented iteratively to scale up. For the AgentToken design, it appears that when a new agent is added, the MetaAgent requires continuous fine-tuning to learn how to use the new token. Could this introduce a bias toward relying on later-incorporated tokens? How might the related issue be addressed in practical applications?
2. The existing agent designs in the agent pool are not well-detailed. They can only be inferred from Figures 1 and 2, which mention agents like sheet agent and web agent. Including a table that introduces the agents currently in the pool and their specific roles would improve clarity.
3. The related works section appears outdated. Claims such as agents “struggle with open-ended tasks” and “agents are fixed with pre-defined roles” are inaccurate. Systems like AutoGen and structured designs like Dylan already support flexible agent adaptations. This section lacks relevance to AgentStore’s design and fails to effectively motivate the paper.
4. The experimental results lack breadth and generalizability. The authors primarily test on OSWorld and APPWorld, which have a very limited number of tasks. Incorporating more diverse benchmarks, such as Mind2Web, could strengthen the claim that the design scales effectively. Additionally, the choice of base agent model, benchmarks, and baselines requires better justification.

* One major concern about this method is scalability. Though mentioned in Limitation, the method relies very much on the newly trained AgentTokens with SELF-INSTRUCT. When the total number of agents is large, or the problem (environment) is complex, data scarcity issue becomes significant. 
* Although the targeting issue is important, this paper provides more technical contributions. AgentPool and AgentToken, one of the most important ideas in this paper, are similar to ToolkenGPT (as cited in the paper). The author should provide more insights into how OS Agent is different from tools, etc.
Comments, suggestions and typos: As above

The margin below Figure 2 can be larger.
Overall assessment scores: 3.5

2.5