AIOS: LLM Agent Operating System
Authors: Kai Mei
Xi Zhu
Wujiang Xu
Mingyu Jin
Wenyue Hua
Zelong Li
Shuyuan Xu
Ruosong Ye
Yingqiang Ge
Yongfeng Zhang
Venue: COLM 2025 (2025)
Keywords: Large Language Model
LLM Agent
TL;DR: Design and implement a novel infrastructure for serving agents

Abstract: LLM-based intelligent agents face significant deployment challenges, particularly related to resource management. Allowing unrestricted access to LLM or tool resources can lead to inefficient or even potentially harmful resource allocation and utilization for agents. 
Furthermore, the absence of proper scheduling and resource management mechanisms in current agent designs hinders concurrent processing and limits overall system efficiency. To address these challenges, this paper proposes the architecture of AIOS (LLM-based AI Agent Operating System) under the context of managing LLM-based agents. It introduces a novel architecture for serving LLM-based agents by isolating resources and LLM-specific services from agent applications into an AIOS kernel. This AIOS kernel provides fundamental services (e.g., scheduling, context management, memory management, storage management, access control)  for runtime agents. To enhance usability, AIOS also includes an AIOS SDK, a comprehensive suite of APIs designed for utilizing functionalities provided by the AIOS kernel. Experimental results demonstrate that using AIOS can achieve up to $2.1\times$ faster execution for serving agents built by various agent frameworks. The source code is available at https://github.com/agiresearch/AIOS.

Published as a conference paper at COLM 2025
AIOS: LLM Agent Operating System
Kai Mei
Xi Zhu
Wujiang Xu
Mingyu Jin
Wenyue Hua
Zelong Li
Shuyuan Xu
Ruosong Ye
Yingqiang Ge
Yongfeng Zhang
Department of Computer Science, Rutgers University
Abstract
LLM-based intelligent agents face significant deployment challenges, par- ticularly related to resource management. Allowing unrestricted access to LLM or tool resources can lead to inefficient or even potentially harmful resource allocation and utilization for agents. Furthermore, the absence of proper scheduling and resource management mechanisms in current agent designs hinders concurrent processing and limits overall system efficiency. To address these challenges, this paper proposes the architecture of AIOS (LLM-based AI Agent Operating System) under the context of managing LLM-based agents. It introduces a novel architecture for serving L.MM- based agents by isolating resources and L.M-specific services from agent applications into an AIOS kernel. This AIOS kernel provides fundamental services (e.g,, scheduling, context management, memory management, stor- age management, access control) for runtime agents. To enhance usability, ATOS also includes an ATOS SDK, a comprehensive suite of APIs designed for utilizing functionalities provided by the AIOS kernel. Experimental results demonstrate that using AIOS can achieve up 2.1 x faster execution for serving agents built by various agent frameworks. The source code is available at https: htte:://ittti www..h..b...... cwm/ hamm//airreseacc/AIOs.
Introduction
In the field of autonomous agents, research efforts (Wooldridge & Jennings, 1995; Jennings et al., 1998; Bresciani et al., 2004) are made towards agents that can perceive environments, understand instructions, make decisions, take action and learn from feedbacks. The advent of large language models (LLMs) (Achiam et al., 2023; Touvron et al., 2023a; Team et al., 2023) has brought new possibilities to the agent development Ge et al. (2023a). Current LLMs have shown great power in understanding instructions (Ouyang et al., 2022; Chung et al., 2022; Touvron et al., 2023b; Geng et al., 2022), reasoning and solving problems (Kojima et al,, 2022; Nijkamp et al., 2022; Taylor et al., 2022; Hao et al., 2023; Kim et al., 2023), and interacting with human users (Ross et al., 2023) as well as external environments (Driess et al., 2023; Brohan et al., 2023). Built upon these powerful LLMs,, emergent LLM-baseed agents (Ge et al., 2023a; Yao et al., 2023; Shinn et al., 2023; Deng et al., 2023; Packer et al., 2023; Wu et al., 2024) can present strong task fulfillment abilities in diverse environments, ranging from virtual assistants to more sophisticated reasoning and problem-solving systems.
An illustrative example of LLM-based agent's real-world task execution is demon- strated in Figure 1, where a travel agent processes a trip organization request. The agent methodically decomposes this request into executable steps-bookkin booking flights, reserving accommodations, processing payments, and updating calendars according to user prefer- ences. Throughout execution, the agent exhibits reasoning and decision-making capabilities derived from its LLM foundation, distinguishing it from traditional applications constrained by predetermined functions or workflows. Implementing this travel scenario requires the agent to seamlessly integrate LLM-related services (preference retrieval, API selection, response generation) with conventional OS services (disk access, software execution).
"Author emails: {kai. mei, xi. zhu, wujiang. mingyu. jin, wenyue. hua, zelong. 1i, xu, ruosong. ye, yingqiang. ge,
ge, yongfeng. zhangg)rutgers..dd
1

---

Published as a conference paper at COLM 2025
User: I'm flying from San Francisco to New York for business next month, please help organize the trip.
Travel Agent: Understood. I'll plan and book itinerary according to your previous preferences.
Step
Step ?
Step S
Step
Step y
Step e
Step Reviews and sps
Proferance Retreval
Flight and hotel Paoommer
Photo ID Uplcad
Seat Selection
Payment Processing
Ado ticket Calendar
LLM Ciorage
Tool
Dos Storage
Cotvurn
Softwrre
Text Generzzon LLM
Figure 1: A motivating example of how an agent (i.e., travel agent) requires both LLM- related and Non-LLM-related (i.e., OS) services to complete a task, where color in red represents services related to LLM and color in blue represents services not related to LLM.
Current agent frameworks exhibit critical design limitations by granting agents direct access to system-level resources like LLMs and tools (Qin et al., 2024), compromising resource optimization and introducing potential vulnerabilities. Without proper scheduling, agents can monopolize resources such as flooding the LLM with requests while others wait. The absence of effective resource management mechanisms significantly impairs system efficiency. Under concurrent conditions, existing frameworks (e.g., Autogen, Langchain) employ an inefficient trial-and-error approach for LLM calls: prompts are converted to tensors and loaded into GPU memory until CUDA memory limits trigger exceptions, forcing tensor deallocation and requiring multiple retry attempts, which substantially degrades system throughput in scenarios where numerous agents compete for limited LLM resources.
To mitigate above-mentioned limitations, we introduce AIOS, an architecture designed to serve LLM-based agents more efficiently. Our contributions are as below.
New Agent-serving Architecture. We introduce AIOS, novel LLM-based agent serving ar- chitecture. AIOS divides agent applications and resources for agents such as LLMs and tools into distinct layers, i.e, the application layer and the kernel layer. This separation facilitates systematic resource management, efficiency optimization, and safety enhancement.
AIOS Kernel Design and Implementation. At the core of AIOS, we design and implement an AIOS kernel to encapsulate resource management abstractions. In this kernel, agent queries are decomposed into sub execution units (i.e., AIOS syscalls) to facilitate parallelism. We design an agent scheduler to orchestrate syscall execution across modules, while memory, storage, and tool managers and the LLM core are responsible for handling dispatched syscalls. We also design the context manager to handle context interruption and design the access manager to verify agent operations to ensure the reliability of the AIOS kernel.
AIOS SDK Development. We develop the AIOS SDK, which provides a higher level C
abstraction of kernel functionalities, allowing developers to focus on application logic and higher-level functionalities without being burdened by complicated kernel details.
Empirical Results. We conduct extensive evaluations of ATOS on agents developed using various agent frameworks. The results demonstrate that ATOS can maintain the perfor- mance of agents across standard agent benchmarks and can even enhance performance in benchmarks that involve tool calling. Furthermore, AIOS significantly improves execution efficiency, achieving up to 2.1x x increase in execution speed for serving agents across different frameworks. These experimental results underscore the effectiveness of AIOS in optimizing both agent performance and execution speed in serving agents.
2 The Architecture of AIOS
As depicted in Figure 2, the AIOS architecture is divided into three distinct layers: the application, kernel, and hardware layers. This layered design is intended to establish a clear separation of concerns within the system. Higher-level applications abstract the complexities of the underlying layers, interacting with them through well-defined interfaces such as software development kits (SDKs) and system calls.
Application Layer. The application layer leverages the AIOS SDK, providing interfaces for requesting system resources within AIOS. This design relieves agents from resource
2

---

Published as a conference paper at COLM 2025
Agent Application
Travel Agent
Agent
17 Coding Agani
Agent
Namative Agent
File Openates
Ressuiing rlanring
KighosslMont
Ragazd
Non-L Relased Queries
AIOS-Agen SDK
Querick
Code Generason
Ted Parking
Application Layer
Keree Layer
Kernel
AIOS Kemel
Process Scheduler
AIOS System Call
Memory Manager
LLM
Agent Scheduler
Context Manager
Memory Manager
Storage Manager
Tool Manager
Access Manager
Corefs)
Filesystem
Hardware Driver
OS System Call
Layer
Hardware
GPU
Memory
Peripheral Devices
Figure An overview the ATOS architecture of distinct layers. Application layer facilitates the design and development of agent applications. Kernel layer manages core functionalities and resources to serve agent applications. Hardware layer controls and manages physical computing resources and devices to support kernel layer functionalities.
management while enforcing isolation by preventing direct resource manipulation. The AIOS SDK supports both native agent development and non-native agents adapted from diverse frameworks including ReAct (Yao et al., 2023), Reflexion (Shinn et al., 2023), Autogen (Wu et al., 2023), Open-Interpreter (Lucas, 2024), and MetaGITT (Hong al., 2023). Non- native agents interact with AIOS kernel resources through adapter functions, while native development is streamlined via pre-defined APIs that invoke system calls. This abstraction allows developers to focus on agent logic rather than resource management details.
Kernel Layer The kernel layer integrates two components: the traditional OS kernel for non-LLM computing tasks and our innovative AIOS kernel. Within the ATOS kernel, specialized modules process agent requests through system calls. A scheduler dispatches these calls to appropriate modules using advanced scheduling strategies (detailed in Section 3.3). We design a unified interface encapsulating LLMs as cores, akin to CPU cores, enabling implement
integration of diverse LLM endpoints. To support LLM context switching, context manager with snapshot and restoration capabilities (Section 3.4). For efficient agent data handling, we develop memory manager for runtime operations (Section 3.5) and a storage manager for persistent storage (Section 3.6). Additionally, a tool manager loads tools and resolves call conflicts for AIOS SDK supported tools (Section 3.7), while an access manager implements access control and user intervention protocols (Section 3.8).
Hardware Layer. The hardware layer consists the physical components of the system, such as the CPU, GPU, memory, disk, and peripheral devices. The hardware layer is not the main focus of the work-A1OS kernel does not directly interact with the hardware but relies on OS system calls to access the physical resources in the hardware layer.
AIOS Kernel
In this section, we start with an overview of the AIOS kernel, highlighting how each module collaborates with other modules to support integrated functionalities. Following this, we provide an in-depth look into the design and implementation of each module, discussing their roles and contributions to the overall AIOS architecture.
3.1 Relationship and Connection between Modules
Within the AIOS kernel, agent queries are decomposed into categorized system calls (LLM processing, memory access, storage operations, tool usage) as shown in Figure 3, with comprehensive syscall catalog available in Appendix A.1. Each syscall is thread-bound
3

---

Published as a conference paper at COLM 2025
Agent Queries
Query
122
Query E3
Query F3
Ceetar: leed Tyur Chat
Convet poo)
Cenlat: lood] Type Tao Uhe
Consel: bood
Conne: Daxs Type Opection
Chenee (eex)
Type Sie Operatio
Tyne
Query Decompusition
AIOS System Call
B1-1
C2I LLM
LLM Syeal
Syvall
D2 11LM8pall
Strrnge Spscil
83 Specal
13.2 Tool Syszl!
B1.3 Mezecy Spvcil
233 Marwey Syaal
P3. Marazy Syacai
AIOS Kernel
Thread Binding
Context Manager
Scheduler
I.I.M Core(s)
LLM Quene
Memory Queue
A1
Memory Manager
Tool Manager
Tool Queue
Storage Queue
Storage Manager
In-processing system call
Exccution chain
Figure 3: How agent queries are decomposed into AIOS system calls and how AIOS system calls are dispatched and scheduled. We omit the access manager module here as the access- related system calls will not be dispatched by the scheduler.
and dispatched by the scheduler, which centralizes queue management across all modules. Syscalls are routed to appropriate module queues based on their attribute sets, with each module monitoring its designated queue for scheduled calls. Context manager will be triggered within LLM core for handling context interruptions instead of being scheduled.
3.2 LLM Core
Due to the various deployment options of LLMs, e.g., which LLM is used, whether the LLM hosted on cloud or on local device, what hardware conditions the LLM requires, or which inference framework is used, we encapsulate each LLM instance adopting different deployment options as a core, akin to CPU core in a traditional operating system. This design allows to treat each LLM instance modularity and extensibility within the AIOS architecture. To accommodate different LLM instances, introduce wrapper for each LLM instance and design unified system calls within this wrapper specifically for LLM inference. By abstracting an LLM instance as and im implementing standardized system calls, AIOS provides flexible way to integrate LLM instances under different deployment options, attributed to the modular design of the LLM core. Detailed information of LLM core is provided in Appendix A.2.
dedicated processing unit, enhancing the
3.3 Scheduler
We centralize all queues within the scheduler module rather than distributing them across processing modules, isolating request management responsibilities and allowing each module to focus solely on execution. This centralization simplifies cross-module task coordination and provides unified scheduling framework. For managing system calls, implement two classic algorithms: First- In-First-Out (FIFO), which processes calls in arrival order but may increase waiting times for later requests, and Round Robin (RR), which cycles through calls time-sliced fashion for more balanced resource distribution. The RR strategy supported by our context interrupt mechanism for LLM inference, detailed in Section A.3.
3.4 Context Manager
LLM inference time creates bottlenecks through long-running system calls that monopo- lize resources. Our context interrupt mechanism addresses this via task interruption and resumption through snapshot and restoration operations. The context manager designs two approaches: text-based (for closed-soures LLMs without logits access, saving decoded outputs) and logits-based (preserving intermediate search tree structure for finer-grained state restoration). The logits-based method is illustrated in Figure 4. Using beam search (common in LLMs (Touvron et al., 2023b; Jiang et al., 2023; Biderman et al., 2023)), with beam width 1 for simplicity, demonstrate the process: When processing Determine whether there
4

---

Published as a conference paper at COLM 2025
Agent raquest: Determine whether a min in the destination of fight UAC57.
Restora
LLM
LLM
0.8
0.6
Context Manager
Search
Rasrive
Search
Retreee
0.6
0.2
0,8
0.4 Rutrisse
Search
Snapehot
4
0.6
Suspended
Seerch
Weather
Search
Flelraaee
0
0.7
0-3
Weather
Plrts
London
0,0
0.2
Search weather Paris
Figure 4: Illustration of the oggit--based context snapshot and restoration process. We use beam search algorithm where beam width is set to as an example.
will rain in the destination of flight UA057, the LLM evaluates candidate tokens at each step. If suspended by the scheduler mid-gennr,tion, the context manager snapshots the interme- diate results. Upon resumption, it reloads this snapshot to continue from the suspension point, reaching the final answer: Search weather in Paris without restarting computation.
3.5 Memory Manager
Unlike traditional OS memory managers handling physical RAM, AIOS's memory manager addresses agent interaction histories during runtime (Lerman & Galstyan, 2003; Zhang et al., 2024), including conversation logs and tool-calling results. It manages memory structure, allocation, read/write /write operations, deletion, and updates. Agent memory resides in RAM by default, but when allocated space approaches capacity, the manager implements memory swapping between RAM and disk. When an agent's memory usage exceeds its block limit (e.g., 80% of allocation), the memory manager initiates a K-Least Recently Used (LRU--K eviction policy, transferring items from RAM to disk via the storage manager (detailed in Section 3.6). LRU-K prioritizes retaining items in RAM that have been accessed at least K times recently, while moving less frequently accessed items to disk. This balances memory efficiency by offloading infrequently accessed data while ensuring retrievability when needed. Detailed implementations of memory manager are in A.5.
3.6 Storage Manager
The storage manager handles persistent data storage for agents, including files or knowledge bases that agents depend on to run and the agent memories that need to be persistently stored. During an agent's runtime, when the agent's memory usage exceeds the allocated limit, the memory manager calls the storage manager to swap the data into the disk. Specifically, the storage manager reads and writes data based on the agent ID passed from the memory manager. In addition to the memory manager, the agent itself may also request to read and write data on disk during runtime, and these agent requests are also handled by the storage manager. Specifically, the agent calls the storage API in the SDK, which is further converted into storage-relate system calls and put into the storage queue by the scheduler. The storage manager then processes the requests in the queue to fulfill the agent requests. The storage manager is implemented using local files and vector database (e.g., chromadb). Implementation details of the storage manager are included in Appendix A.6.
3.7 Tool Manager
The tool manager in the AIOS kernel is responsible for managing a broad suite of API tools supported by the ATOS SDK.
5

---

Published as a conference paper at COLM 2025
Interaction History with Agent (Agent ID=2)
need to buy new phone. Looking for something with great camera, at least 128GB storage, and compatible. Budget is around $800.
User:
Agent: Understood, I'll call the google search of related items for you.
Memory Manager (Short-term)
Storage Manager (Long-term)
Original Interaction History
Agent
Memory Limit
Memory Note
Agent B
Agent ID:
Agent C
Agent ID:
LRI-K
Eviction
Agent ID:
Agent X
Figure 5: Illustration of memory and storage manager well as their relationship. An agent's memory item in its memory block will be evicted to storage if its memory usage exceeds the memory limit, which is set to 80% of the memory block size. This threshold is configurable through AIOS configuration.
Standardized Tool Loading. The manager employs standardized interface to handle diverse tools uniformly, while incorporating parameter validation before execution to prevent tool crashes. When invoked by name, the tool manager dynamically loads the tool instance including executables initialization and dependencies verification.
Resolution of Tool Call Conflicts. For tools with parallel access constraints, the system utilizes a hashmap to monitor real-time instance counts. Request processing involves hashmap verification against usage and parallel limits; upon detecting conflicts, the system advances subsequent queue requests until identifying conflict-free candidate. The implementation details are presented in Appendix A.7.
3.8 Access Manager
The access manager in the AIOS kernel provides the following two key functionalities.
The access manager regulates cross-agent data read/ mad/write write operations by implementing privilege-based access control mechanisms. It assigns each agent to specific privilege group and enforces permissions through hashmap architecture that maps agent IDs to their corresponding privilege groups. Access requests are validated against this permission structure before execution, ensuring agents only access resources of other agents within their shared privilege domain.
Access Control.
User Intervention. To mitigate risks of irreversible operations (deletion, overwrite, privilege modification), user intervention interface provided for user confirmation. This mandates explicit user verification before proceeding with potentially destructive operations on files or tools. Implementation details can be found in Appendix A.8.
3.9 ATOS SDK
We design the AIOS SDK to streamline the development and integration of agents on the AIOS architecture. This SDK not only empowers developers to build agents that interact with the core functions in the ATOS kernel but also abstracts complex system calls, allowing developers to focus on the agent's 's internal workflows.
Tool Integration. To support diverse agent functionalities, the AIOS SDK integrates wide range of tools sourced from various platforms and supports different cases of input-output modalities. Detailed information on these integrated tools is provided in Appendix B.3.
Interaction Interface with the AIOS Kernel. To facilitate the utilization of functions provided by AIOS system calls in the AIOS kernel, the SDK defines different API functions that agents use to invoke system calls and request resources.

---

Published as a conference paper at COLM 2025
Table 1: Evaluation of agent performance on benchmarks w/o and w/ AIOS, respectively. Success rate (SR%) is used as the metric for all the benchmarks. "-" represents methods that failed GAIA benchmark tasks due to lack of API support.
Method
HumanEval
MINT (Code)
GATA
SWE-Bench-Lite
ReAct w/oAIDs
48.8
29.4
5.5
3,9
ReAct wr AIO6
50.6
30.1
7.3
4.3
Reflexion w/oAIDS
50.6
32.4
6.7
4.7
Reflexion ABOS
51.8
33.8
7.8
5.1
Autogen w/o AIO
87.8
42.5
7.3
4.3
Autogen wy AIOE
87.8
42.5
9.7
4.3
Open-Interpreter
85.4
45.9
4.7
Open-Interpreter w/
86.0
48.7
5.1
MetaCPT A306
82.9
41.1
5.9
MetaGPT AOOS
82.9
41.8
5.9
Agent Framework Adapter. To support agents built with various agent creation frameworks, such as Autogen (Wu et al., 2023), Open-Interpreter (Lucas, 2024), and MetaGPT (Hong et al., 2023), the AIOS SDK provides adapters for these frameworks. These adapters locate the core functions in the aforementioned frameworks and redirect them to the functions in AIOS. This adaptation allows agents from different frameworks to operate within the AIOS environment without modification of the agent code. Further details on the core functions and specific adaptations for each agent framework provided in Appendix B.5.
Evaluation
In this section, we conduct experiments to answer the following research questions.
Can AIOS maintain or even enhance the performance of agents on standard bench- marks when running multiple agent instances simultaneously?
RQI:
RQ2: How effectively can AIOS optimize system execution throughput and reduce response latency when serving numerous agents built with different agent frameworks?
RQ3: How scalable is AIOS as the number of concurrently running agents increases?
4.1 Setup
Models. We use the GPT--o-mini (Achiam al., 2023) the closed-source API, and use two open-source LLMs, i.e, Llama-3.1-8b (Dubey et al., 2024) and Mistral-7b (Jiang et al., 2023), as the LLM core, respectively, during the experiments. The open-source models are both instruction-tuned versions and we use float16 precision.
Hardware. Our experiments are conducted on an Ubuntu 22.04 machine equipped with NVIDIA RIX A5000 GPUs (24GB), We run all experiments using single A5000 GPU.
Agent Frameworks. We conduct evaluation by running agents built from various popular agent frameworks: ReAct (Yao et al., 2023), Reflexion (Shinn et al., 2023), Autogen (Wu et al., 2023), Open-Interpreter (Lucas, 2024) and MetaGPT (Hong et al., 2023). Details of these agent frameworks are introduced in Appendix B.5.
Workloads. We evaluate on resource-consten conttrai- scenario in which agents run concur- rently with a single LLM deployed that can process only one prompt request at a time. To create these concurrent conditions, by default, i.e., at most 250 agents can run concurrently at the same time. The impact of increasing the number of agents will be analyzed in Section 4.4. By default, use RR as the scheduling strategy for AIOS to run agents. The impact of using other strategy (i.e., FIFO) is reported in Appendix D.
set the maximum number of working threads to 250
4.2 Agent Performance (RQ1)
To evaluate the impact of using AIOS on agent performance in standard benchmarks, adopt four agent benchmarks, iee, HumanEval (Chen et al., 2021a), MINT (the code subset) (Wang et al., 2023b), GAIA (Mialon et al., 2023) and SWE-Bench-Lite (Jimenez et al., 2024) to agents without and with AIOS, respectively. We use the success rate
7

---

Published as a conference paper at COLM 2025
With AIOS
Without A/OS
With AIOS
Without AIOS
Thoughtn foomattre)
1.
00-
ReAm
Refexcinn
MetaSPT
0.0
RoAcl
RaTuxieen
Autagen
Open-ntaaprear
MataPP
Agentaikgent Framowors
AgentsPgent
Frameworks
(a) Normalized throughput. Higher is better.
(b) Normalized latency. Lower is better.
Figure 6: Efficiency analysis on different agent frameworks evaluated on the Llama-3. 1-88bb model on the HumanEval benchmark.
With
Without
Thraphod Nomalzzd)
6.
0.5
0.5
Latency
0.0
Rellecon
A.llagen
Opennindepradd:
MeNnGPP
ReAd
Reiissoon
Autlogen
Open-indegrebee
MebxGPT
Agonts/kgentFammee Fomeootss
EgentaAgentFramman Frammerkoss
(a) Normalized throughput. Higher is better.
(b) Normalized latency. Lower is better.
Figure 7: Efficiency analysis on different agent frameworks evaluated on the Mistral-7b model on the HumanEval benchmark.
(SR%) the metric, consistent with the original benchmarks and use GPT-4o-mini the LLM core to run all the agents. We set the temperature as 1.0 for GPT-4o-mini in experiments. Detailed descriptions of the benchmark setups and configurations found in Appendix C. As shown in Table 1, incorporating AIOS consistently maintains agent performance across standard benchmarks. In some cases, AIOS can also contribute to agent performance improvements. For example, in code generation benchmarks such as MINT, HvaanEval, and SWE-Bench-Lite, AIOS boosts agent performance by prompt enhancement, which embeds the system prompts with more structural input and output within the LLM wrapper. These enhanced prompts provide the LLM with additional context and structural guidance for response generation. In tool calling benchmarks like GAIA, agent performance is even boosted by two following mechanisms: pre-execution parameter validation via structural regex to check the format of tool calls before execution, and (2) conflict resolution hashmaps to mitigate concurrent access issues.
be
4.3 Efficiency Analysis (RQ2)
In our efficiency experiments, evaluate system performance using two key metrics: throughput and latency. Throughput is measured by counting the number of AIÓS system calls executed per second, indicating the system's capacity handle multiple requests in parallel. Latency, on the other hand, is measured as the average waiting time experienced by agents, from the moment query submitted to the completion of the response, reflecting the system's responsiveness. To ensure controlled and consistent testing environment, we conduct these evaluations using the two open-source models, Llama-3.1-8 and Mistral-7b, both hosted locally. Hosting these models locally reduces potential variability in LLM API response times due to network-related latency issues. As shown in Figure 6a and Figure 7a, the results demonstrate that AIOS achieves significantly higher throughput across different agent frameworks, to 2.1 x increase throughput when using Reflexion-basee agents flama-3.1-2 This improvement is attributed to the scheduling employed in the AIOS kernel, which prevents unnecessary trial-and-error attempts by avoiding prompts that cannot be loaded onto the GPU for execution. In terms of latency, as illustrated in Figure 6b and Figure 7b, the average waiting time for agents is also substantially reduced. This reduction highlights the efficiency of AIOS in serving LLM-based agents.
4.4 Scalability Analysis (RQ3)
We evaluated AIOS scalability by incrementally increasing active agents from 250 to 2000, using Llama-3.1-80 and Mistral-7b models on the HumanEval benchmark. We duplicated HumanEval's 164 samples to match agent counts, enabling large-scale concurrent execution. As demonstrated in Figure 8, AIOS maintains approximately linear relationships between both overall execution time and average agent waiting time relative to agent count. This
8

---

Published as a conference paper at COLM 2025
Execution Time Agents
(b) Average Agent Waiting Time v.s. Agents
on Matne-bb AIOS
Matralwit AK2S
1000
on Wwwaal/wwhhoooeee AIOS
MetalliinoorMM A05
Limadd 5-lb with
Average Agent Waiting Time (seconds)
Lirma wiB
Overall Exccution Time (econds)
on Liana 3:1.5 wi willuut A0SS
Liiia wiROLLAISS
XC0
15.0
12.5
400
7.5
9.0
200
2.5
500
1003
1250
1760
250
2000
Agents
Agents
Figure 8: Overall execution time and average agent waiting time when agent number increases from 250 to 2000.
indicates that AIOS can efficiently handle workload even under increasing demand. In contrast, the gap of execution and waiting times between without AIOS and using AIOS widens as agent counts increase. This growing differential underscores AIOS's scalability under high concurrent workloads.
5 Related Work
The evolution of operating systems (OS) has progressed from rudimentary to sophisticated interactive systems. This evolution transitioned from basic batch processing (IBM, 2010) to advanced process management including time-sharing (Ritchie & Thompson, 1974) and multi-tasking (Hoare, 1974; Engler et al., 1995), enabling complex task handling. Devel- opment advanced toward modular architecture with specialized components for process scheduling (Liu & Layland, 1973; Dijkstra, 2002), memory management (Denning, 1968; Daley & Dennis, 1968), and filesystem operations (Rosenblum & Ousterhout, 1992; McKu- sick et al., 1984), improving system efficiency. The introduction of graphical user interfaces (GUIs) in Macintosh, Windows, and GNOME enhanced user interaction. Currently, AI models, particularly LLMs, are migrating from application to system layers, providing standardized services across applications.
Large Language Model Agents are used to solve complex planning and reasoning tasks (Xie et al., 2024; Ge et al., 2023a). Single agents engage with either digital environment or physical environment, which may invoke APIs (Ge et al., 2023a; Schick et al., 2023; Yao & Narasimhan, 2023; Parisi et al., 2022; Tang al., 2023; Xie al., 2024), browse websites (Nakano et al., 2022; Deng et al., 2023; Wu et al., 2024), or execute codes (Zhang et al., 2023; Yang et al.), while agents in the physical environment may manipulate objects (Brohan et al., 2023; Fan et al., 2022; Wang et al., 2023a), carry out lab experiments (Boiko et al., 2023; Bran et al., 2023), or make actionable decisions (Huang et al., 2022; Xiang et al., 2023). LLM-based multi-agent systems (MAS) leverage the interaction among multiple agents for problem solving. The relationship among the multiple agents could be cooperative (Wang et al., 2023c; Mandi et al., 2023), competitive (Chan et al., 2023; Du et al., 2023), of cooperation and competition (Ge et al., 2023b). In cooperative multi-agent systems, each agent takes and assesses the information provided by other agents, thereby working together to solve complex tasks, such as role playing (Li et al., 2023; Chen et al., 2023; Zhu et al., 2023), social simulation (Park al., 2023) and software development (Hong et al., 2023; Qian et al., 2023; Wu et al., 2023; Josifoski et al., 2023).
mixture
6 Conclusion and Future Work
This paper introduces AIOS, an architecture serving LLM-based agents through an in- novative kernel that isolates resources and LLM services from agent applications. The complementary AIOS SDK enables agent applications to leverage kernel functionalities efficiently. Experimental validation confirms AIOS maintains or enhances agent perfor- mance on standard benchmarks while significantly accelerating execution time, improving

---

Published as a conference paper at COLM 2025
system throughput, and demonstrating scalability with increasing concurrent agent loads. We envision this work catalyzing future innovations that refine and expand the architecture to address evolving requirements for developing and deploying LLM-based agents.
7 Acknowledgement
We thank Balaji Rama, Hang Gao, Shuhang Lin, Jian Zhang and Zhenting Wang for their valuable discussions and suggestions during the project.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Daniil A Boiko, Robert MacKnight, and Gabe Gomes. Emergent autonomous scientific research capabilities of large language models. ar preprint arXiv:2304.05332, 2023.
Andres M Bran, Sam Cox, Andrew D White, and Philippe Schwaller. Chemcrow: Aug- menting lange-la models with chemistry tools. arviv preprint arXii:2304.05376, 2023.
Paolo Bresciani, Anna Perini, Paolo Giorgini, Fausto Giunchiglia, and John Mylopoulos. Tropos: An agent-oriented software development methodology. Autonomous Agents and Multi-Agent Systems, 8:203-236, 2004.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as can, not as i say: Grounding language in robotic affordances. In Conference robol learning, pp. 287-318. PMLR, 2023.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agent debate. In The Tweefth International Conference on Learning Representations, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo- hammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Jaseph, Greg Brockman, et al. Evalu- ating large language models trained on code. ar Xiv preprint ar >xXi:21007003374 2021b.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent col- laboration and exploring emergent behaviors in agents. arXiv preprint 2rXiv:2308.10845, 2023.
10

---

Published as a conference paper at COLM 2025
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. arXiv preprint ar X1::22210.11446 2022.
Robert C Daley and Jack B Dennis. Virtual memory, processes, and sharing in multics. Communiiattoons of the ACM, 11(5):306-31, 1968.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2023.
Peter J Denning. The working set model for program behavior. Communications of the ACM, 11(5):323-333 1968.
Edsger W Dijkstra. Cooperating sequential processes. In The origin of concurrent programming: from semaphores to remote procedure calls, pp. 65-138. Springer, 2002.
Danny Driess, Fei Xia, Mehdi SM Sajaadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, al. Palm-e: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, pp. 8469 8488, 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14335, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint ar Xrv:24407 217783 2024.
Dawson R Engler, M Frans Kaashoek, and James O'Toole Jr. Exokernel: An operating system architecture for application-leve resource management. ACM SIGOPS Operating Systems Review, (955)2251-26666 1995.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Build- ing open-ended embodied agents with internet-scale knowledge. Advances Neural Information Processing Systems, 35:18343-18362, 2022.
Yingqiang Ge, Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, and Yongfeng Zhang, OpenAGl: Processing Systems, 36, 2023a.
When LLM Meets Domain Experts. Advances in Neural Information
Yingqiang Ge, Yujie Ren, Wenyue Hua, Shuyuan Xu, Juntao Tan, and Yongfeng Zhang LLM as OS, Agents Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystem. arXiv:2312.03815, 2023b.
Shijie Geng, Shuchang Liu, Zuohui Fu, Yinggiang Ge, and Yongfeng Zhang. Recommen- dation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference on Recommender Systems, pp.
299-315, 2022.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.
2023.
Charles Antony Richard Hoare. Monitors: An operating system structuring concept. Com- munications of the ACM, 17(10):549-55 557, 1974.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta program- ming for multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2023.
11

---

Published as a conference paper at COLM 2025
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. International Conference on Machine Learning, pp. 9118-9147. PMLR, 2022.
Corporation IBM. What is batch processing? z/OS Concepts, 2010.
Nicholas R Jennings, Katia Sycara, and Michael Wooldridge. roadmap of agent research and development. Autonomous agents and multi-agent systems, 1:7-38, 1998.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. aiv preprint ar Xi::23310006825, 2023.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
Martin Josifoski, Lars Klein, Maxime Peyrard, Yifei Li, Saibo Geng, Julian Paul Schnitzler, Yuxing Yao, Jiheng Wei, Debjit Paul, and Robert West. Flows: Building blocks of reasoning and collaborating ai. arv preprint Xiv:22330 0oeo:2888001.,,,, 2023.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Advances Neural Information Processing Systems, 36, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22211 2022.
Kristina Lerman and Aram Galstyan. Agent memory and adaptation in multi-agent systems. In Proceedings of the second international joint conference on Autonomous agents and multiagent systems, pp. 797-803, 2003.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36, 2023.
Chung Laung Liu and James W Layland. Scheduling algorithms for multiprogramming in hard-real-time environment. Journal of the ACM (JACM), 20(1):46-1,, (0())))) 1973.
Killian Lucas. Open interpreter. https:///httb..coommmmmttrttetes 2024.
Zhao Mandi, Shreeya Jain, and Shuran Song. Roco: Dialectic multi-robot collaboration with large language models. arxiv preprint urXio:2307.04738, 2023.
Marshall K McKusick, William N Joy, Samuel Leffler, and Robert Fabry. A fast file system for unix. ACM Transactions on Computer Systems (TOCS), 2(3):181-197, 1984.
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. arXiv preprint ar Xrie:2331..1293, 2023.
Reiichiro Nakano, Jacob Suchir
Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2022.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. Xiv preprint ar XrXit:2203.13474 2022.
12

---

Published as a conference paper at COLM 2025
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Ch Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-2777 -77744, 2022.
Charles Packer, Vivian Fang, Shishir G Patil, Kevin Lin, Sarah Wooders, and Joseph E Gonzalez. Memgpt: Towards Ilms operating systems. arXiv preprint arXto:2310.08550, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. Xiv preprint Xri::22055112555 2022.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-22, 2023.
Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXio preprint arXio:2307.07924, 2023.
Yujia Qin, Shihao Liang Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. ICLR, 2024.
Dennis M Ritchie and Ken Thompson. The unix time-sharing system. Communications of the ACM, 17(7):365-375, 1974.
Mendel Rosenblum and John K Ousterhout. The design and implementation of log- structured file system. ACM Transactions on Computer Systems (TOCS), 10(1):26-55 1992.
Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. The programmer's 's assistant: Conversational interaction with large language model for software development. In Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 491-514, 2023.
Timo Schick, Jane Dwivedi-Yos Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprini arXtv:2302.04761, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023.
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. ar preprint Xii:2300.00301,, 2023.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.0000 2022.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arv preprint ar arXiv:2312.11803 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anm Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXio preprint 2rXi::2302.13971,
2023a.
13

---

Published as a conference paper at COLM 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (xxi::2077.09288, 2023b.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. In Intrinsically-Mottiuuee and Open-Ended Learning Workshop@ Newr1PS2023, 2023a.
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating ilms in multi-turn interaction with tools and language feedback. Xiv preprint ar Xiv::309. 10691, 2023b.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Un- leashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint ar arXi::2307.00 Ki0:2307.05300, 1(2)3,, 2023c.
Michael Wooldridge and Nicholas R Jennings, Intelligent agents: Theory and practice. The knowledge engineering review, 10(2):115-152, 1995.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li liang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint ar Xiix:2308.08155, 2023.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. arviv preprint ar Xiv:24402.07456 2024.
Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Language models meet world models: Embodied experiences enhance language models. Advances in neural information processing systems, 36, 2023.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: benchmark for real-world planning with language agents. Xix preprint ar arXi::2402.0162 2024.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering.
Shunyu Yao and Karthik Narasimhan. Language agents in the digital world: Opportunities and risks. princeton-nlp-gheee github.., 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. International Conference on Learning Representations, 2023.
Kechi Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. Toolcoder: Teach code generation models to use apis with search tools. arviv preprint ar arXiv:235.004333 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. Xiv preprint ar arXiv:1904.09675, 2019.
Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents. Xrr preprint ar Xir::2440 13501, 2024.
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, al. Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. Xiv preprint ar Xiv:2333 17144, 2023.
14

---

Published as a conference paper at COLM 2025
APPENDIX
This appendix contains additional details for this paper. The appendix is organized as: Section ŠA provides AIOS Kernel Implementation Details. Section $B reports more about ATOS SDK. Section ŞC reports more Details of Agent Benchmarks. Section $D shows more Additional Experimental Results. Section ŞE analyzes Discussion.
A
AIOS Kernel Implementation Details
A.1
AIOS System Call
The modules in AIOS achieve their functionalities by invoking system calls. Table shows more comprehensive list of system calls correspondent to different modules and present the arguments for invoking these system calls.
Table 2: AIOS modules and their correspondent system calls.
Module
System Call
LLM Core
execute 1lm syscall, get model response, process process.model model response
Scheduler
execute -syscal, start, stop
Context Manager
load contexo,,xxt, clear_contex,
Memory Manager
execute memory syscall, add memory, remove memory update memory, retrieve memory
Storage Manager
execute storage syscall, sto create file, create directory sto mount, sto write, sto retrieve, sto rollback, sto share
Tool Manager
execute.tool.syscal load instanccce add privilee, check access, ask permission
Access Manager
Thread Binding, Each system call within AIOS is bound to a separate thread for execution, allowing for concurrent processing. The thread binding is implemented by inheriting the Thread class and overwrites its init and run methods.
class (rsCall(Thread))
def ...init_. agent_nane, request_data):
super()...inie
ageet_agent_nmee agent_nane
rolffrequest_data.e request_data
self.event = threading. Event
self.pid
self.status None
relf.nes
self.tiie_linit None
self.crreaed__ime Nane
self.stare start_tiiee None
self.end_tine None
def run ruels)))):
self.set see_ native_id) ( self. event. eeettt.aat(((
A.2 LLM Core
AIOS implements unified interface through the LLMAdapter class, which provides consistent function interface for integrating LLM instances from various backends. This architecture allows for seamless interaction with different LLM providers while maintaining a standardized API. Table 3 presents the supported LLM backends and their respective functionalities within the ATOS framework.
15

---

Published as a conference paper at COLM 2025
Table 3: Adapted LLM backends in AIOS and supported features.
Supported Features in ATOS
Backend
Structured Output
Function Calling
OpenAT (cloud)
Anthropic (cloud)
Google (cloud)
Groq (cloud)
7
Bedrock (cloud)
Huggingface (local) vllm (local)
Ollama (local)
class LLHAdapter:
The LLMAdapter class is an abstraction layer wraps LLN core instances from different LLN backends.
def _.init_._
self.
iin_configs: List[Dict[str, Anyll.
api_key: Optional[Union(strr, List[str]]]
log_mode: str "console", see_context_managee bool False,
strategy: optional[[outeessraatee]] RouterStrategy. suterrttratey.Ssqunntiall
):
Initielize the LLMAdapter.
def ketup_api_kessseel1f None:
# Set up API keys for different providers from config or environnent. pass
def initialize wln.s (self) None:
a Initialize LLM backends based configurations.
pass
def antiaalize_single 1lm(self, config: LLMConfig) >> None:
Initialize single LLN based configuration. pass
def handle_ handle_completteeeeeeeeeeeeeeeee cempletion_erroreeeeo (self, Exception) LLMResponse:
Handle errors that occur during LLN completion. pass
def execute_ execute_11n_syscall.
self,
1ls_syscall, temperature: float 0.0
>> LLNResponse:
Address pass
from the
def get_nodel_response(
self,
madel_name:
model: Union[str, #fLocalBackend, OpenAI], nessages: List(Dict],
tools: Optional[List).
llm_syscall,
api_base: Optional[str] None,
ressage_return_tty Optional[str] "text", response_format: (Dtttsr, Dict]l None, temperature: float = 1.00
nax_tokens:
>> Any:
response
def process_response(
completed_respone str List, # either response message of a string or list of self.
finished: bool,
tools: Optional[List] None,
model: Union[str. OpenAI, #flocalBackend] = None, ressage_return_tyee Optional[str] None
LLNResponse:
a Process the model's s response into the appropriate fornat.
pass
16

---

Published as a conference paper at COLM 2025
A.3 Scheduler
ATOS implements a flexible scheduler architecture through the BaseScheduler class, which serves as the foundation for various scheduling strategies. By designing an extensible inheritance model, specialized schedulers (implementing algorithms such FTFO, Round Robin, and priority-based scheduling) can be derived from this base class. This modular design ensures that new scheduling algorithms can be introduced without modifying existing implementations, thus maintaining strong isolation between different scheduling strategies and enhancing the overall system flexibility.
class BaseScheduler:
Task scheduler implementation.
def _..init__(
self, LLMAdapter,
nemory_nanager: MemoryWanager,
noragee StorageNanager,
tool_nanager: ToolManager,
log_mode: str.
get_lln_syscall LuBequesstQuuueGettssssg...
get_memory_syscall Nenorryteqqessqquuuddeettttsss
get_storage_syscall staragetrousstuuuuetttsessssss
get_tool__yscall: Toolequustquendeettsssgg
)
Initielize the Scheduler.
pass
def syscall
self, syscall: Any,
executor: Any.
syscall_type- str
>>
(Dictssr Anyll:
Execute system call with proper status tracking and handling. pass
def process_ process_lln_reeeeeeee.s. 1em_ (sel) >> None: Process LLM requests from the queue.
def hetrs_nenorry__eueeseee
# Process Hemory requests from the queue. pass
def Drcesssstrrae_rrquuttcccc
None:
Process Storage requests fron the pass
def process_ process_tool_reeeee rew..o >> None: requests
queue.
pass
def start(self) None: # Start all request processing threads. pass
def stel()
Stop all request processing threads. pass
A.4 Context Manager
The SimpleContettManager implements an efficient mechanism for managing LLM genera- tion contexts with time-aware interruption capabilities. This component enables preemptive multitasking by preserving LIMM states across generation phases. When an L.LM successfully begins token generation within its allocated time slice (evidenced by at least one decoded token), the context manager captures and preserves the intermediate generation state. This ensures that subsequent resumption can continue from the exact point of interruption, eliminating redundant computation. This architecture supports various LLM backends (OpenAI, HuggingFace, and others) through unified interface while enforcing strict time boundaries on generation tasks. By managing both streaming responses and enforcing time limits, the context manager helps improve fairness among different queries to L.Mss.
17

---

Published as a conference paper at COLM 2025
class (eagrttttttttaeerBaaBassstttttte
...
simple context manager for handling LLN context saving and loading.
This class provides functionality to the current state of LLM generation, load previously saved states, and manage context for different processes.
def
(self):
Initialize SimpleContexthanager pass
eapty context dictionary.
def ge..steaiingccooecciioeeeesssss self, model_or_client: Union[str, OpenAI], model_name: str.
nessages: List strll,
tools: optional[List(Dictttt:: Any]l], temperature:
float.
nax_tokens:
response_format Optional[Dict(strr, Any]] None, bool
) >> Any:
Get completion response pass
litella OpenAI client.
def rroess.collciioo.tttnnnn..rrropp
self,
response:
time_linit: str.
initial_content.
Tuple[str, bool]:
Process streaming response with time limit enforcement.
def is www.iuggace.....cccccccoo.oooeee sareece_nddeleeeeeee (self, model) bool: a Check if the model is HuggingFace model instance. pass
def gemerate_with_tine_liie self, model,
nessages: nax_tokens: int,
List[Dict(str, str]].
tomperature: float,
pid: int.
time_limit float Tuple[str, bool, Dict]:
# Generate text with HuggingFace model with time limit enforcement. pass
generate_response.c wwt.epossssiiiiiteerutttttaanaaa
nodel_name:
model: Union[str, OpenAI, Anyl,
nessages: List[Dict[str, strll,
tools: Optional[List(Diettss nessage_return_tyye str, temperature: float.
Any]]].
nax_tokens: int,
pid: Union[int, strl.
time_linit: float,
response_formae Optional(Dict(str,, Any]] >> Tuple[Any, bool]:
handles different types models (string-based, OpenAI client, or HuggingFace) and diffarent response types
context
generation.
(text, JSON, or tool calls), manages streaming responses and enforces time limits.
pass
def cad__coneettseef, pid): previously saved pass
process.
def clear_context (self, pid): # Clear the saved context for process. pass
18

---

Published as a conference paper at COLM 2025
A.5 Memory Manager
The memory manager provides RAM-based memory operations for the AIOS system, handling transient session-specific data that clears when agent sessions end. Built the MaseMeemoryMaangeer class, it offers comprehensive memory access through atomic operations, automatic metadata synchronization, and thread-safe access patterns. The system efficiently processes memory through a complete suite of CRUD operations and advanced retrieval mechanisms, functioning similarly to pointer management in low-level programming while ensuring data integrity during concurrent operations.
class BaseMemoryKanager:
provides core functionality memory operations
system.
including adding, removing, updating, and retrieving memories. It acts wrapper for nemory access, similar to working with pointers in low-level languages.
def iniit (self, log_mode):
# Initialize the BaseNenoryhanager. pass
def www.ouuurryttt.errrr...
query: MemoryQuery) >> 'MemoryNote'; Convert HemoryQuery to HemoryNote object.
def exeutte_ennrry.sccalllee nemory.syscall):
a Route memory syscall to the appropriate nethod. pass
def add_menory(
self,
nemory_note
# Add menory note to the storage.
pass
def remove_nemory(
self,
nemory_id
Remove nenory note from storage. pass
def update_memory
self, nemory_note
Update an existing nemory note. pass
def get get_menory( self,
nemory_id: str "MemoryNote's
Retrieve menory nate by I0. pass
def _rotrieve.menory_raa
self, nemory_query: MenoryQuery
# Retrieve memories sinilar to the query content. pass
def retrieve_nenory(
self, nemory_query: MemoryQuery
# Retrieve memories sinilar to the query content.
pass
A.6 Storage Manager
The Storage Manager orchestrates persistent data operations in AIOS, combining traditional file storage with vector database capabilities. It implements versioned file management, thread-safe access through file-specific locks, and semantic search functionality. The storage
19

---

def sto_rollback( Published as a conference paper at COLM 2025
manager provides file operations including file management, semantic file retrieval, version control with rollback capabilities, and file sharing. Function interfaces are shown as below.
class StorageHanager:
nanager provides versioning. locking, and database integration ... efficient file operations and retrieval.
def -.init_. (self, root_dir, sse_vector_db=rru pax_versions=20) Initiilize with specified root directory configuration.
def dedel (self):
Destructor stops systen pass
is deleted.
def hr-ffil_hashhselt file_path: str)
# Generate SHA-256 hash for file path.
pass
get_file filefile_loce (sllf path str) threading. Lock: # Get create thread lock for specific file path. pass
def handle_ file_lllllleeeeeeeeee caee_chaaeeeeee (self, file_path; str, change_type: str):
Handle file changes with proper lock managenent. pass
def hr__file_histee history (self, file_path: str, listt: int None) >> list: Retrieve version history for a file from Redis cache.
def restore_ version file_path; str, version_index: int) >> bool: previous
pass
def execute_ soouute_strage__.. wtt.:sstteeeccacaleeeeeeeeeeeeee (sef,, storage_sysee storage_syscall):
Process and route storage syscalls to appropriate file system operations.
pass
foo_creeat_filesseff file_name: str, file_path: str, collection_nane None) bool:
Create new empty file in the file systen.
def coo_crrate_ddrecctrry.es dir_nane: str, dir_path: str, collection_nane. None)
-> bool:
Create directory
system.
pass
def sto_mount collection_name: root_dir: str)
# Hount directory far an agent and build the vector database.
pass
setrieeveeee( self.
collection_nane: str.
query_text:
k: str "3", keywords: None
) -> list: Retrieve documents from vector database using semantic search.
self,
file_path,
n=1,
time=None
) bool:
Roll back file to previous version index tinestamp. pass
def shaatte_sshare__ikkkse file_path: str) >> str: publicly accessible pass
def sto_share(self file_path: collection_nane:
None) >> dict:
a Share file by generating a public access link with proper lock management. pass
20

---

Published as a conference paper at COLM 2025
A.7 Tool Manager
The tool manager module is responsible for loading tools executing tools with tool conflict prevention mechanisms. Implementation details are shown below.
class ToolManager:
def _._ini___(
self
log_mode: str "console",
):
pass
def exccute_tool_sycccl(eee tool_syscall) >> ToolResponse:
pass
def instteecsssttteeeeseeer.
pass
A.8 Access Manager
The access manager provides two key functions: First is to check access when agents attempt access other agents' resources. Second is request user permission before agents execute irreversible actions such as deletion of files. Function interfaces are shown below.
class AccessNanager:
def (self): pass
def ped_privilege sid, tid):
# Assigns agent into another agent's privilege group pass
def aacckssccesssec (self, sid, tid):
checks if pass
is in target agent's priviledge group
def fsk_peraissionneelff operation):
a Prompts the user for confirmation before an irreversible operation
A.9 Module Hooks
To effectively separate the interface of calling the AIOS kernel modules from the implemen- tation details, employ hook mechanism to initialize modules and export the necessary call interfaces. Here are the hooks we use for initializing modules.
évalidate (LLMParans)
UseLLM(parans: LLMParans)
Initialize and return Language Learning Model (LLM) instance.
parans (LLMParams): Paraneters required for LLN initialization.
Returns: LLN: instance initialized LLM.
return LLM(**parans nodel_dump())
evalidate (MenorsyaaaaggeeParrss
def useHenoryyMaagger (parans:: MemoryKanagerParans) MemoryManager: Initialize and return a menory instance.
Args:
parans :MenoryParans)): Paraneters required Nemory Hanager Initialization.
Returns:
Hemory Manager: An instance of the initislized Hemory Manager.
MenoryManager (**parans. mordel_dunmdeelddurr nodel_dunp())
21

---

Published as a conference paper at COLM 2025
@validate (Storaggthaaggerraann) usestorageHanagerr (parans: StorageHanggeParaans) StorageManager: Initialize and return storage instance.
Args:
parans (StorageHanagerParang Paraneters required for Menory Manager Initialization.
Returns:
Storage Manager:
initialized Storage Manager.
return StorageManager (**parans, model_dunp())
@validate (ialidattetoollaaaaeePaaaa def veeToolHanagerrpprraa TooiHanaggeParrass) >> ToolNanager:
... Initialize return instance.
Args:
(ToolManagerParrans): Paraneters required for Tool Nanager Initialization.
Returns:
Tool Nanager: An instance of the initialized Tool Manager.
...
return ToolManager (*=paan (odel_
B AIOS SDK
The AIOS SDK provides structured interface between user device applications and the ATOS kernel through different functional modules. All queries from these modules are ultimately channeled through central send_ request)) function in the SDK, which then communicates with the AIOS kernel via HTTP requests (either to localhost or remote URL). From the agent developer's perspective, their agents are essentially composed of code snippets which could include agent logic and resource-related commands for LLM, memory, storage and tool that interact with their respective modules through the SDK apis. This creates clean separation between the application logic and request of kernel resources.
B.1 Query and Response
The AIOS SDK defines a robust query-response architecture that enables seamless commu- nication between agent applications and the AIOS kernel. This framework is built on two foundational data structures: Query and Response, which facilitate structured data exchange across the system.
The Query class serves the abstract base for all input requests, estab- lishing a consistent interface for agent interactions with the kernel. It branches into four specialized implementations:
Query Structure.
Facilitates natural language interactions with configurable parameters for temperature, token limits, and various action types including chat, JSON output, tool calls, and file operations.
LLMQuery:
MemoryQuery: Handles transient data operations including adding, retrieving, updating, and removing memories with specialized support for agentic memory management.
StorageQuery: Manages persistent storage operations through parameterized requests for file and directory manipulation.
ToolQuery: Enables access to external capabilities through structured tool calls, extending the system's is functionality.
The Response class provides complementary structure for stan- dardized outputs from the AIOS kernel. Each response type corresponds to its query counterpart:
Response Structure.
LLMResponse: Returns generated text, tool call outcomes, completion status, and error information from language model operations.
Delivers memory content, metadata, search results, and operation status for memory management functions.
MemoryResponse:
22

---

Published as a conference paper at COLM 2025
Agents
AS
Autogen
Agent A
Open-Interpecter
Agent
G
MetaGPT
Agent
Agent B
Agent
Agent F
AlOS-Agent (De(Cebenu)))
LLM Code Snippet
Code Snippet
Memory Code Snippet
Storage Code Snippet
weyg plansing
saup, planieg) pigescos skes
planseg))
planing A preasaces dog
perpocrts days
of gogeos aeeeooetti
ape
moeenryy api
scones see
pooperate
meile
maah
send requesi Querry)
HTTP (localhost/remote url)
AIOS Kernel
Figure 9; Illustration of how agent applications leverage the AIOS SDK to send queries to the AIOS Kernel. For simplicity, queries sent directly the OS kernel are omitted.
StorageResponse: Provides operation outcomes, completion status, and error details for storage-related activities.
Returns tool execution results, completion status, and error information from external tool operations.
ToooResp:ns::
class Query (BaseHodel):
class all query types in the system.
serves foundation specialized query classes LLMQuery, MenoryQuery, Storagequery, and Toolquery. It defines the mininum structure required for a valid query within the AIOS ecosysten.
Attributes:
query_class: Identifier
query type, ["1ls", "menory", "storage", "tool"]
***
class: Literal["lln', "nenory", "storage", "tool"]
class Response (BaseNodel):
Base class for all response types in the AIOS This class serves as the foundation for specialized response classes like LLNResponse, MemoryRespanse, StorageResponse,
ToolRespanse. It defines the minimum structure required for a valid response within the AIOS ecosysten.
...
response_class: Literal["1ln", "memory", "storage", "tool"]
class LLMQuery
query_class: "llm"
11ms: Optional[List(Dieetttttt Anyll] Field
messages: List(Dict[str, Union[str, Anyll]
tools: SprionallLisstDDeettttttt Anyl]] Fieil(deaaltt
type:: citeral phaa__wihh_json_otpuu call_tool,,
temperature: float = Field(default=1.00)
mox_new_tokens:
Fieed((efaultt1688)
ressage_return_tvv tepettrr__tye:::: Literal[ "json"] Field(defaultt=text))
response_format. Optional Anyl] Field Field(defullt=None)
class Config:
arbitrary.type. allowwd
23

---

Published as a conference paper at COLM 2025
class (Raespppnneeeeepnnse
...
Response class for LLM operations.
This class represents the output structure after performing LLM actions.
sisponse_class: str "lln"
response_ sesposeeeeessge:: [ptional[str]
tool_calls: optional[List(Dictttttt. Anyll] None finished: False
error: Optional[str]] = None status_code: int 200
class Config: arbitrary.types. alowwd True
(uuory)uery(Querr))
Query class for nemory operations.
query_class: str "menory"
operation_type: Literal["add_ "get_nemory", "update_nenorr", retrieve_nenory" parans: Dict[str, Any] faattttff
"renove_netory",
class Config:
arbitrary_typea. allowed allowdd
True
class NenoryResponse (Response):
Response memory operations.
response_clan
"nemory*
menory_id: Optional[str] None
content: Optional [str] None
metadata: (pitonal
search_results: OptionallList(Dccc(ttrr Any]]] None
success:
errar: [ptional[str] = None status_code: int 200
class Config: arbitrary_types_cc allowed
Storagequeryy(ueryy
Query storage operations.
query_class: str "storage"
params: Dict[str, Union[str, Any]] operation_type: str Field(default="tece) text")
class Config:
Eetttrary_types.allooe True
class StorageResponse (Response):
*en
Response class for storage operations.
...
response_c class: str "storage"
:eoonse Optional[str] None finished: bool False
errar: Optional[str] = None status_code:
24

---

Published as a conference paper at COLM 2025
class (oolquery(Query).
...
Query class for tool operations.
..*
query_class: str "tool"
tool_calls: List(Dict(str,, Union[str, Any]l]
class Config: arbitrary.types. True
class ToolResponse (Responsee)
Response class for taol operations.
response_class.
"tool"
m Optional[str] = None finished: bool False
[ptienal[str] None
status_code: int 208
B.2 AIOS SDK APIs
AIOS SDK also provides multiple API functions that are constructed using the Query and Response data structure and renu_st functions. Avaiable APIs are shown in Table
Table 4: AIOS SDK APIs
Module
APIs
LLM Core
Ilm llm,chat, 1lm Llm_chht.vv wwwatwwithhpppaeoooo withhjjson_outpuie llm mom_calltool,, llm_operate file
Memory
create create_ memory, get memory, delete memory, delete_memory, search memorris
update memory
Storage
mount, retrieve freate_file, create_dir, write_file rollback_file share file
Tool
call
B.3 Supported Tools.
As is illustrated in Table 5, AIOS SDK integrates diverse computational tools to address wide spectrum of information processing tasks. The SDK incorporates 17 native tools spanning multiple modalities and functionalities, enabling sophisticated interaction patterns across text, image, and audio domains. These tools can be categorized into three primary sources: established technology providers (Google, Bing, WolframAlpha), specialized API hubs (Rapid API Hub), and advanced Al model providers (Huggingface).
The toolkit's architecture demonstrates particular strength in text-based operations, with 12 tools supporting text input output modalities. This includes fundamental information retrieval services (Arxiv, BingSearch, Wikipedia), specialized analytical tools (Currency- Converter, (oonnhaaeSearch), and domain-specific applications (ImdbRank, TripAdvisor). Furthermore, the SDK exhibits robust cross-modal capabilities through tools like Visu- alQuestionAnswering (image-text integration), TextToAudio (text-to-speech synthesis), and Volce-AttivtyReeonnition (speech-to-text conversion).
B.4
Agent Examples
Here we provide examples of agents developed by leveraging AIOS SDK.
Travel Agent: The travel agent is designed for trip planning, including searching for flights, accommodations, and local activities.
Rec Agent: The recommendation agent is designed for suggesting movies and TV series.
Math Agent: This agent is designed to solve equations, perform calculations, and provide step-by-step explanations for different math problems.
25

---

Published as a conference paper at COLM 2025
Table 5: Tools supported by AIOS SDK, ordered by names in alphabet.
Tool Name
Source
Type
Modality (Input - Output)
Arxiv
Arxiv
API
Text + Text
BingSearch
Bing
Text +
CurrencyConverter
Rapid API Hub
API
Text Text
GooglePlace
Google
API
Image/ TTxt Text
GoogleSearch
Google
API
Text Image
ImageCaption
Huggingface Local Model/APl
Text + Text
ImdbRank
Rapid API Hub
API
Text + Text
MoonPhaseSearch
Rapid API Hub
Text + Text
Shazam
Rapid API Hub
Text + Text/ Audio
TextToAudio
Huggingface
Local Model/APl
Text Audio
TextTolmage
Huggingface
Local Model/APl
Text - Image
TripAdvisor
Rapid API Hub
API
Text - Text
TisualQuestionAAnswee
Huggingface
Local Model/APl
Image & Text ++ Text
Voice Recoonitionnin
Huggingface
Local Model/AR API
Audio + Text
Wikipedia
Wikipedia
Text + Text
WolframAlpha
WolframAAlpha
Text +
WordsAPI
Rapid API Hub
API
Text Text
The creation agent is tailored for content generation tasks, such as writing, graphic design, even video editing.
Creation Agent:
Academic Agent:
The academic agent is designed to support research and learning, such as literature reviews and provide explanations on complex academic topics.
Agent Profile
Description:
You are an expert in planning and managing travel itineraries.
Workflow:
1. Identify the destination and search for hotel locations using the hotel location search tool.
2. Based on the hotel locations, find suitable hotels using the hotel search tool, and select the best one.
3. Get detailed information about the selected hotel using the get.hotel tool.
4. Search for the nearest airport the origin using the airport search tool.
5. Search for the nearest airport to the destination using the airport searchhserrhhhcceeeee tool. search tool
6. Find available flights the destination airport using the flight flight-search using the correct date.
7. Search for restaurant locations near destination using the restaurant www.looonceaaaatteoeooeo
tool.
8. Based on the restaurant locations, find suitable restaurants using the restaurant search tool.
9. Get detailed information about the selected restaurants using the ret_restara -etails tool.
10. Gather additional relevant information about the destination the user is visiting using the wikipedia tool.
11. Integrate the information gathered from the previous steps to provide a comprehensive travel plan.
Available tools:
1. Tripodviso
2. Wikipedia
Example of task inputs: want to take a trip to Paris, France from July 4th to July 10th, 2024, and I am traveling from New York City. Help me plan this trip.
26

---

Published as a conference paper at COLM 2025
RecAgent Profile
Description:
You are expert who is good at recommending TV series and movies.
Workflow:
1. Identify the tool that you need to call to obtain information.
2. Based on the information, give recommendations for the user based on the constrains.
Available tools:
1. TripAdvisor
2. Wikipedia
Example of task inputs: Recommend three action movies from the past five years ranked between 1 and 20 with ratings above 8.0.
CreationAgent Profile
Description:
You expert who is good at content creation.
Workflow:
1. Convert the vague description of the content requirements into concrete objects and fill in more details.
2. Identify the tool to call the tool to create content based the filled details. Available tools:
1. SDXL-Turbo
Example of task inputs: Create an image of a sleek, high-tech futuristic city with vibrant nightlife atmosphere.
MathAgent Profile
Description:
You are an expert who is good at solving mathematical problems.
Workflow:
1. Identify the tool to call some pre-calculation.
2. Perform mathematical operations using the pre-calculated result, which could involve addition, subtraction, nultiplication, or division with other numeric values to solve the problem. Available tools:
1. Currency Converter
2. WolframAlpha
Example of task inputs: Convert 15000 MXN to Canadian Dollars and find out how much would be in USD 1 CAD equals 0.79 USD.
AcademicAgent Profile
Description:
You expert who is good at looking up and obtaining information from academic articles.
Workflow:
1. Identify the tool to call based on the academic requirements and call the tool.
2. Gather the information obtained from the tool to write an outline or summarization.
Available tools:
1. Arxiv API
Example of task inputs: Summarize recent studies on the role of artificial
intelligence in drug discovery from 2018 to 2023.
27

---

Published as a conference paper at COLM 2025
B.5 Support of Agent Frameworks
The core idea to adapt agents built by existing agent frameworks for AIOS is to identify the core functions that will interact with system resources and change that with functions in our native adapters. In this section, we illustrate the crucial adaptation function that needs to be changed run agents built by other agent frameworks on AIOS.
ReAct (Yao et al., 2023). The ReAct framework integrates reasoning and action steps in language models, allowing them to generate intermediate reasoning traces alongside actionable steps for complex task completion. This dual approach helps models not only plan and track their thought process but also interact with external tools, improving performance on tasks like question answering, game environments, and decision-making problems that require multi-step reasoning and adaptability. By alternating between reasoning and action, ReAct reduces errors from solely predictive responses and enables more accurate, contextually aware task completion.
Reflexion (Shinn et al., 2023). The Reflexion framework enhances language agents with feedback-driven mechanism, allowing them to learn from mistakes and adapt behavior through self-reflective feedback loops. By leveraging verbal reinforcement learning, agents assess and adjust their actions, which improves performance on complex tasks through iterative learning. This approach makes language agents more resilient and adaptive, enabling them to handle tasks with evolving requirements and uncertainty.
Autogen (Wu et al., 2023). AutoGen introduces framework that leverages multiple lan- guage model agents with distinct roles (such as Planner, Executor, and Reflector) collabo- ratively solve complex tasks through structured, goal-oriented conversations. By enabling agents to communicate and share intermediate results, AutoGen coordinates multi-step processes like data analysis, decision-making, and iterative problem-solving, significantly enhancing efficiency and accuracy beyond single model's capabilities. This approach empowers wext-generation applications, allowing LLMs to tackle dynamic workflows, adapt to task-specific nuances, and achieve higher performance the code of adapting Autogen for ATOS. Due to ongoing refactoring work by the Autogen team, only Autogen-0.2 (the latest stable version) is supported.
real-world scenarios. Below is
eadd_ fww.errrrn_ adapter ("AutoGen '8.2*)
def [usparr__uttoen___220 ... Replace OpenAIWrapper and ConversableAgent nothods with aios's implementation.
This function is used to adapt autogen's API to aios's API, and it is used internally by aios.
Replace OpenAIMrapper nethod
wtt._rrrappr...init... adapter_.utogen_clliee
openATWrapper.crren crraaaapppprrrrcteeeeee = daatte_ccliet_croatt
tetpatt_textttrrcoopeeee object BennAWwapper..extaace
Replace agent nethod
Canversablelgenn..ppitt...r
ConversableAgent. www.eeeettntt..ggattteeeeeeeeeee _eg.erateecoi_rroll__frrr.
ConversableAgent. wtw.://eeeeegggegateeeeeeee senerate_tool_calss..eea
gnnversableteenteexxuue execute_function
seaapter_executeffue
ConversableAgent. sooeetttttotttttttttooooo
ConversableAgent.
wpwpbbleeggen.uuute_e_eeee sppdate_ttol_signe
adapter_.utoggn.ggentttt siaitt__tool_signtuuee
conversableAgent.
Open-Interpreter (Lucas, 2024). Open Interpreter is an open-source framework that enables users interact with LLMs through ChaiGPT-like interface interpret and execute complex instructions across programming languages directly in the terminal. It supports both locally-hosted and cloud-based LLMs, allowing for streamlined code execution and debugging in natural language. By translating natural language instructions into executable code, Open Interpreter offers an intuitive environment that not only simplifies development workflows but also facilitates learning by providing detailed explanations and interactive support for various coding challenges, making suitable for developers at skill levels. Below is the core function to be adapted for Open-Interpreter.
28

---

Published as a conference paper at COLM 2025
dttprrkeeaaattttoon-nnnaee prepare_interpreterd.
Prepare the interpreter far running LLN in aios.
Set the completion function in the interpreter interpreter co.coonesssons adppter_eios_comlletiinn
def www.aiioooooeeiiionncccppp
completions replace fixed_litella, completions interpreter Run completion
attempts first_error None
for attempt (attentt:):::)):
try:
send_request get_request_funcl
response send_request(
query=LLMQuery(
aassagespprranss"eessss kools=(parans("tools= if "tools" in parans else None)
[" ['"eepposs"")
format similar to completion in interpreter
conletion ['choices" ([ :[('delt'': (}}}
conletion ("coiicee
response rssonse["rssponse_eessss
if response tool_calls is not None:
"eheiionn"choice "elaa""
return [conletion] If the completion is successful, exit the function
except KeyboardInterrupt: print ("xittn"xitingg...) sys. ex)
Exception if attempt a= first_error
first error
if first_error None: raise first_error
MetaGPT (Hong et al., 2023). MetaGPT proposes meta-programming approach that optimizes I.L.-driven multi-agent systems by integrating task-oriented programming paradigms for complex, collaborative problem-solving. MetaGPT encodes Standardized Op- erating Procedures (SOPs) directly into structured prompt sequences, creating streamlined workflows that empower agents with human-like domain expertise to systematically verify intermediate outputs and proactively mitigate errors. Along this line, MetaGPT addresses the limitations of existing LLM-based frameworks, such as hallucination and cascading errors during agent chaining. This framework facilitates the decomposition of complex tasks into manageable, interdependent subtasks, improving overall system robustness, especially in high-stakes, iterative processes where reliability across agent interactions is crucial. Below is the core function to be adapted for MetaGPT.
feddd_fraewwok_eee adapter "etaGPT")
def prepare_netaggt ..* Prepare the metagpt module aios.
prepare_.etaget_configg
BeseLLN. aask adapter_aask async def adapter_aask
self,
Union[str. list[diet[str, str]]].
[list[ssr]]] systen_nsgs: [[ston]]][ist(t:]]]]] None,
format_asgs: Optional [list(diet(str, strlll
inages: Optional [Union[str, list[str]]] = None, Limeout-USE_CONFICCTTEEEE stream=True,
str: await cdopter.ccopletionn.tttteesees strean=strean, timeout=self g timeout)) rsp rsp else ."
29

---

Published as a conference paper at COLM 2025
C Details of Agent Benchmarks
C.1
HumanEval
The authors (Chen et al., 2021b) 1 introduced HumanExal, a benchmark dataset comprising 164 handwritten programming problems for evaluating functional correctness of code gen- eration models. Each problem consists of a function signature, docstring, implementation body, and comprehensive test suite, with an average of 7.7 test cases per problem. The hand-written nature of these problems is crucial, given that modern language models are typically trained on large portions of GitHub code containing existing solutions program- ming challenges and contest problems. HumanEval is designed to assess multiple aspects of code generation capability: natural language comprehension, logical reasoning, algorith- mic thinking, and mathematical operations. Through this publicly available benchmark, researchers can conduct rigorous and standardized evaluations of code generation models.
C.2 MINT
MINT (Wang et al., 2023b)² introduced a benchmark to evaluate LLMs' ability to solve challenge tasks through multi-turn interactions. The benchmark focuses on code generation, decision making, and reasoning tasks that require LLMs to utilize tools and incorporate natural language feedback. MINT was constructed by curating multiple single-turn datasets, reducing an original collection of 29,30 99,307 instances to 586 carefully selected examples. The benchmark uses success rate (SR) its primary evaluation metric, measuring the percentage of successfully completed tasks. For a given interaction limit k ranging from 1 to 5, each LLM is allowed up to k turns of interaction, with performance measured as SR₃- In our experiments, we set k = 5 and focus exclusively on MINT's code generation subset.
C.3 GAIA
General AT Assistant (GATA) (Mialon et al., 2023)3 is a benchmark designed to represent a significant milestone in Al research by evaluating fundamental capabilities essential for general intelligence. Unlike traditional benchmarks that focus specialized professional knowledge, GAIA emphasizes everyday tasks that require core abilities including logical reasoning, multi-modal processing, web navigation, and effective tool utilization. GAIA comprises 466 questions that evaluate Al assistants across multiple capabilities including reasoning, multi-modal understanding, coding, and tool usage (particularly web browsing), with tasks involving various data formats like PDFs, spreadsheets, images, videos, and audio. The benchmark organizes questions into three difficulty levels based on the number of required steps and tools: Level 1 requires minimal tool usage (5 5 steps), Level 2 demands multiple tools and 5-10 steps, while Level 3 tests advanced general assistance capabilities through complex, multi-step sequences requiring diverse tool combinations. Additionally, while web browsing is central to GAIA, the benchmark deliberately excludes complex web interactions like file uploads or posting comments, leaving such evaluations for future research.
C.4 SWEBench-Lite
SWE-bench (Jimenez et al., 2024)* is a software engineering benchmark constructed through rigorous three-stage pipeline that processes GitHub pull requests (PRs) from 12 popular Python repositories. The pipeline filters approximately 90,000 PRs based on attributes (issue resolution and test contribution) and execution criteria (successful installation and fail-to-pass test transitions), resulting in 2,294 high-quality task instances. Each task requires models to generate patch files that resolve software issues, with success determined by com- prehensive test coverage. The benchmark distinguishes itself through real-world challenges, extensive input context (averaging 195 words per issue), cross-context editing requirements (typically spanning 1.7 files and 32.8 lines per solution), and robust test-based evaluation. Notably, SWE-bench's 's automated collection process enables continuous updates with new task instances from GitHub repositories, ensuring benchmark relevance over time.
'Tee dataset can be found at https://ww/ www.ggithe. gi www.hb.cco/oopeeeeeeee www.opppnnanaaaeeeaaaaeeee Phttps: //xwang dew/mintt-bench/ 9nttps://hugginggf co/ wwggaaabbenchaar httt::::://wwwwswwbbee
30

---

Published as a conference paper at COLM 2025
Table 6: Impact of using different scheduling strategies, where NONE represents without
using AIOS, FIFO and RR represent using AIOS with the two different scheduling strategies. All metrics are reported in seconds.
Agent waiting time
Strategy
Overall execution time
Avg.
p90
None
152.1
9.8
11.0
FIFO
74.2
3.0
5,0
RR
77.3
3.2
4.2
AICS
Witouu AJOS
Wihh
Withoul A/O8
Thrughoad 0.5
2.5
RoAet
McboPT
2.0
Rohat
Refioiion
MctoCFT
Agonts/ggem Fanooooss
Agentssigent Dorts//gentFaarrroooo Frorowoko
(a) Normalized throughput. Higher is better.
(b) Normalized latency. Lower is better.
Figure 10: Efficiency analysis on different agent frameworks evaluated on the Llama-3.1-80 model on the MINT benchmark.
D Additional Experimental Results
D.1 Efficiency analysis.
We also report the throughput and latency of running agents on other three benchmarks on Llama-3.1-8b and Mistral-7b compared between using AIOS and without using AIOS. The results are shown in Figure 10 and Figure 11, Figure 12 and Figure 13, Figure 14 and Figure 15, respectively.
Impact of Different Scheduling Strategies. To further analyze the impact of different scheduling strategies on system efficiency, we conduct an ablation study using agents built with ReAct on the HumanEval benchmark with the Llama-3.1 model. We test three strategies: without AIOS, FIFO, and Round Robin (RR), and measure the overall execution time and agent waiting time (average and p90).
As shown in Table 6, the FIFO strategy achieves the shortest overall execution time com- pared to the other strategies. RR comes second in terms of overall execution and average agent waiting time, its context switching introduces additional overhead. However, RR performs better on the p90 metric (i.e., the value below which 90% of waiting times fall) due to its fairer scheduling approach, which reduces the likelihood of later tasks having longer waiting time, which can typically occur in FIFO.
D.2 Correctness of context switch.
To assess the correctness of the context switch supported by the context manager, we employ the BLEU score (Papineni et al., 2002) and BERT score (Zhang et al., 2019) to measure text similarity. The similarity is calculated against the final outputs generated for the same agent under the same conditions, only varying with context switch enabled and disabled. As demonstrated Table 7, both BLEU and BERT scores achieve a value of 1.0. The suggests that the context switch does not introduce discrepancies in output quality, suggesting the reliability of the AIOS.
E Discussion
E.1 Ethical Consideration
In this section, discuss both potential positive and negative societal impacts of the work.
31

---

Published as a conference paper at COLM 2025
With AIOS
Withou AIOS
WIhAIOS
Without AVOS
Thruughod (Neerwbree))
Normatted
0.0
0.5
Aurogen Interp
Agonts/ggom: Fanceoos
Agents/Age 11 Frarowooo
(a) Normalized throughput. Higher is better.
(b) Normalized latency. Lower is better.
Figure 11: Efficiency analysis on different agent frameworks evaluated on the Mistral-7b model on the MINT benchmark.
WiOh
Without AVOS
WItAIOS
WVOSoutAOOS
1.0
Laweno (oomrloo
1.0
0.5
Open-Irtep
Agents/Age:t Open-mmennter
Agenttlggem: Frameeoots
Normalized throughput. Higher better.
(b) Normalized latency. Lower is better.
Figure 12: Efficiency analysis on different agent frameworks evaluated on the Llama-3.1-8b model on the GAIA benchmark.
The potential positive societal impacts include: 1) Enhanced efficiency and productivity: AIOS can automate routine tasks, achieve more efficient operations, optimize resource allocation, and reduce bottlenecks, leading to better service and improved efficiency for agent developers; 2) Improved user experience: with better context, memory, and storage management, AIOS can offer more personalized and responsive interactions, enhancing user satisfaction across various applications; 3) Innovation ecosystem: the creation AIOS could foster vibrant ecosystem of agent developers and researchers, driving innovation in AI technologies and applications.
The potential negative societal impacts include: 1) Privacy concerns: the integration of LLMs into operating systems may raise privacy concerns, as Al models such as LLMs may require access to personal data to provide effective services; 2) Security risks: as AI systems become more integral critical infrastructure, they could become targets for cyberattacks, potentially compromising sensitive data and operations; 3) System failures: the failure of integrated systems could have widespread consequences, affecting multiple sectors simultaneously and causing disruptions.
Balancing the impacts: To maximize the positive impacts and mitigate the negative ones, is crucial to adopt a balanced approach to the development and deployment of AIOS, such as Rules and standards: Implementing responsible development rules and standards to ensure data privacy, security, and ethical use of A1; 2) Robust design: implementing robust system design, regular maintenance, comprehensive testing, continuous monitoring, backup
Table 7: Correctness of context switch (text-based and logits-based), which checks the similarity between the generated final outputs with context switch enabled and disabled.
LLM Core
Method
BLEU Score
BERT Score
Mistral-7B
Text-based
1.0
1.0
Logits-based
1.0
1.0
Llama-3-8B
Text-based
1.0
1.0
Logits-based
1.0
1.0
32

---

Published as a conference paper at COLM 2025
With AIOS
Withou AIOS
WIhAIOS
Without AVOS
Thruughod (Neerwbtee))
Normatted
0.0
0.5
Aurogen Intorp Agonts//gom Fanceoos
Aulogo
Agents/Age Frarowoko
(a) Normalized throughput. Higher is better.
(b) Normalized latency. Lower is better.
Figure 13: Efficiency analysis on different agent frameworks evaluated on the Mistral-7b model on the GAIA benchmark.
With AIDS
Withou AJOS
Aits
Without A/OS
Laknoy mortoozo
Thoughoe 0.5
2.5
RaAd
Taflecon
Opendtrieprale
MetaGPT
0.0
Faeilee
Opern-ineenntee
Meta(PP
Agarta/Agen: Framseorts
AgurtalAgent 1i Frarrwweds
(a) Normalized throughput. Higher is better.
(b) Normalized latency. Lower is better.
Figure 14: Efficiency analysis on different agent frameworks evaluated on the Llama-3.1-8b model on the SWE-Bench-Lit benchmark.
and recovery plans, developer training, careful documentation, clear communication, and leveraging AI for predictive maintenance and automated recovery; 3) Public engagement: engaging with the public to raise awareness about the benefits and challenges of AI, ensuring that societal concerns are addressed the development process.
By addressing these considerations, society can harness the potential of AIOS while mitigat- ing its risks, leading to more equitable and prosperous future.
E,2 Future Directions
With AIOS as foundation, there are many directions for future research to pursue. This section outlines potential areas of study that expand upon AIOS.
Semantic Scheduling Algorithms. The scheduling function of AIOS lays the groundwork for the development more advanced algorithms. Future research could focus on algo- rithms that perform dependency analysis among agent requests, optimizing the allocation of computational resources. Additionally, some of the tool resources are locally deployed models, which can also be incorporated into the scheduling paradigm. This includes the management of tool status and snapshots, suggesting a move towards unified scheduling framework that encompasses both agents and their tools.
Efficiency of Context Management. More efficient mechanisms can be devised to assist context management. For example, the pursuit time-efficient context management techniques could significantly augment user experience by expediting the processes of context snapshotting and restoration. Also, context compression techniques can also be leveraged prior to snapshotting, which can yield a more space- -ffficeeff solution.
Optimization of Memory and Storage Architecture. In the context of agent collaboration and communication, the future design of memory and storage systems can adopt a shared approach, enabling the sharing of memory and storage between agents. Such an architecture would enable agents to access à communal pool memory and storage, thereby improving the agents' decision-making ability since one agent can benefit from other agents' memory or storage. Moreover, future work can explore hierarchical storage solutions, designed to optimize data retrieval and storage efficiency. This could involve prioritizing quicker access and reduced storage allocation for frequently accessed data, and vice versa for less frequently accessed information.
33

---

Published as a conference paper at COLM 2025
Thruughod (Neerwbree))
With AIOS
Withou AIOS
WIhAIOS
Without AVOS
0.0
0.5
Aurogen Interp
Agonts/ggom Fanceoos
Agents/Age Frarowoko
(a) Normalized throughput. Higher is better.
(b) Normalized latency. Lower is better.
Figure 15: Efficiency analysis on different agent frameworks evaluated on the Mistral-7b model on the SWE-Bench-Lite benchmark.
Safety and Privacy Enhancements. The aspect of safety in AIOS necessitates protective measures against various attacks, ensuring the system's 's resilience against malicious attacks, such jailbreaking of LLM unauthorized access of other agents' memory. In the realm of privacy, the exploration advanced encryption techniques is vital for safeguarding data transmission within AIOS, thus maintaining the confidentiality of agent communications. Furthermore, the implementation of watermarking techniques could serve to protect the intellectual property of agent developers by embedding unique identifiers in outputs, facilitating the tracing of data lineage.
In a nutshell, AIOS stands research opportunities. Each outlined direction not only can build upon the foundational elements of AIOS but also can contribute to the advancement of the field large.
motivating body of work that brings a broad spectrum of
34


Reviews:
[{"review_id": "SsoKuRvUv9", "reviewer": "Reviewer_UuGH", "summary": "- The paper targets the lack of formal scheduling and resource-isolation support in today’s LLM-agent frameworks, where agents can overload the GPU or external tools.\n- It proposes AIOS, a three-layer architecture. (1) A core layer that handles scheduling (deciding which agent runs when). (2) Tools that manage memory, storage, and access to resources. (3) An interface that lets AI agents use these tools easily.\n- Experiments on five agent frameworks show equal or better benchmark success rates and up to 2.1× higher throughput plus lower average latency on a single RTX-A5000.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work addresses a clear, real-world bottleneck: many concurrent agents fighting for one LLM instance or tool chain.\n- System design is well documented. The appendix includes syscall catalogs, adapter APIs, and pseudocode.\n- Evaluation covers closed-source (API) and open-source (local) models,  and 4 benchmarks (HumanEval, MINT, GAIA, SWE-Bench-Lite).", "reasons_to_reject": "- The baseline “w/o AIOS” is vaguely defined. Please clarify whether it uses (i) the same prompts, (ii) the same scheduling method, and (iii) the same backend for LM inference.\n- The paper only includes ablation of the scheduler. I wonder how other part in the system influence the latency and throughput.\n- In Figure 8, I understand that with poor scheduling, the agent waiting time can increase roughly linearly as the number of agents increases. However, once GPU utilization reaches 100%, I would expect the total execution time to become less sensitive to the scheduling strategy. Could you clarify why the results in the figure still show differences in execution time under 2K agent?\n- All tests use a single GPU. It would be helpful to discuss how the AIOS scales with multiple GPUs or distributed LLM.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CV2u9dz2MB", "reviewer": "Reviewer_FGa7", "summary": "This paper proposes the architecture of AIOS in the context of managing LLM-based agents.\n\nThe primary contribution of this paper is the design and implementation of AIOS, a novel operating system-like architecture specifically tailored for managing LLM-based agents.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work is original. While concepts like treating LLMs as operating systems have been explored, AIOS is the first (to my knowledge) to formalize and implement a comprehensive OS-like system with a dedicated kernel and SDK for concurrent LLM-based agent management.\n\nThe experimental results are competitive. Benchmarks across multiple frameworks and tasks show that AIOS maintains or improves performance while enhancing throughput (up to 2.1×) and reducing latency.", "reasons_to_reject": "The implementation details in the appendix are extensive and might be better summarized in the main text.\n\nReal-world deployment scenarios (e.g., user studies or production cases) are not discussed.", "questions_to_authors": "You mention treating each LLM instance as a “core.” How does the system handle load balancing and dynamic allocation among these cores?\n\nThe benchmarks used represent structured tasks. Do they reflect real-world scenarios of intensive, continuous multi-agent use? What limitations do these benchmarks impose on evaluating scalability?", "extra_scores": {}}, {"review_id": "O8FYgQUGe0", "reviewer": "Reviewer_4vNH", "summary": "This paper introduces the AIOS framework, designed to enhance the parallel efficiency of multiple LLM-based agents. By isolating the resources of LLM services within a dedicated AIOS kernel, the framework achieves up to a 2.1× improvement in efficiency. Additionally, it provides an SDK to improve usability and ease of integration.\n\nThe paper clearly outlines the architecture of AIOS and presents experimental results across various agent frameworks. The proposed system significantly improves concurrency and overall task efficiency for agent-based applications.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The framework demonstrates superior efficiency in high-concurrency settings, achieving higher throughput and lower latency.\n\n2. Validated across various agent tasks, the framework maintains comparable task performance to traditional systems while offering faster execution speed.\n\n3. An SDK is provided to streamline integration and improve ease of use.", "reasons_to_reject": "1. Lack of experimental comparison with classical frameworks (e.g., Autogen, Langchain).", "questions_to_authors": "1. It looks like each component in the AIOS architecture has a special design. I wonder if there is an ablation study that shows that the efficiency improvement is credited more to certain designs\n2. What is the experimental setup for the legend without AIOS on the bar charts? How is the LLM service call scheduled in this baseline?\n3. How does Context Manager ensure efficient snapshot and restore\noperations? These operations seem to be taking up a lot of cache and time in the case of multiple concurrency.", "extra_scores": {}}, {"review_id": "kZQU47Xa6L", "reviewer": "Reviewer_ZN9n", "summary": "The blueprint for a native operating system for artificial intelligence is visionary but incomplete, and will require more stringent technical requirements and broader validation to realise its potential.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an interesting direction for resource management for LLM-based agents.\n\nExperimental results suggest that AIOS can improve execution speed for serving agents.", "reasons_to_reject": "1. The paper mentions FIFO and RR scheduling algorithms. These are standard operating system scheduling algorithms. It is unclear if and how these algorithms accommodate the specific nuances of LLM agent requests (e.g., different computational costs of LLM calls, dependencies between agent tasks, and heterogeneous computational requirements such as dynamic batching, priority preemption).\n\n2. There is no in-depth discussion of practical feasibility, overheads, and compatibility issues (e.g., storage overheads, recovery accuracy) between different LLM architectures (especially closed-source architectures that do not have direct access to logit).\n\n3. The memory manager handles agent interaction history and uses the LRU-K eviction policy. While managing dialogue history is important, it is not entirely clear how this differs significantly from the complex caching and memory management techniques already employed at the application or middleware layers.\n\n4. No mention of robustness issues like security mechanisms, whether debugable, fault tolerance, and the overhead, except for ‘access control’, which is risky to put into a production environment.", "questions_to_authors": "How AIOS' concepts of “resource isolation” and “scheduling” are irreplaceable with the core mechanisms of traditional distributed systems (e.g. Kubernetes) or LLM service frameworks (e.g. Ray)?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces the AIOS framework, designed to enhance the parallel efficiency of multiple LLM-based agents. By isolating the resources of LLM services within a dedicated AIOS kernel, the framework achieves up to a 2.1× improvement in efficiency. Additionally, it provides an SDK to improve usability and ease of integration. The paper clearly outlines the architecture of AIOS and presents experimental results across various agent frameworks. The proposed system significantly improves concurrency and overall task efficiency for agent-based applications.

Reasons To Accept:
* The work addresses a clear, real-world bottleneck: many concurrent agents fighting for one LLM instance or tool chain.
* System design is well documented. The appendix includes syscall catalogs, adapter APIs, and pseudocode.
* Evaluation covers closed-source (API) and open-source (local) models, and 4 benchmarks (HumanEval, MINT, GAIA, SWE-Bench-Lite).

Reasons To Reject:
* The baseline “w/o AIOS” is vaguely defined. Please clarify whether it uses (i) the same prompts, (ii) the same scheduling method, and (iii) the same backend for LM inference.
* The paper mentions FIFO and RR scheduling algorithms. These are standard operating system scheduling algorithms. It is unclear if and how these algorithms accommodate the specific nuances of LLM agent requests (e.g., different computational costs of LLM calls, dependencies between agent tasks, and heterogeneous computational requirements such as dynamic batching, priority preemption).
* The paper only includes ablation of the scheduler. I wonder how other part in the system influence the latency and throughput.
Decision: Accept