Stop Wasting Your Tokens: Towards Efficient Runtime Multi-Agent Systems

## Abstract

Abstract While Multi-Agent Systems (MAS) excel at complex tasks, their growing autonomy with operational complexity often leads to critical inefficiencies, such as excessive token consumption and failures arising from misinformation.
Existing methods primarily focus on post-hoc failure attribution, lacking proactive, real-time interventions to enhance robustness and efficiency.
To this end, we introduce SupervisorAgent , a lightweight and modular framework for runtime, adaptive supervision that operates without altering the base agent’s architecture.
Triggered by an LLM-free adaptive filter, SupervisorAgent intervenes at critical junctures to proactively correct errors, guide inefficient behaviors, and purify observations.
On the challenging GAIA benchmark, SupervisorAgent reduces the token consumption of the Smolagent framework by an average of 29.45% without compromising its success rate.
Extensive experiments across five additional benchmarks (math reasoning, code generation, and question answering) and various SoTA foundation models validate the broad applicability and robustness of our approach.
The code is available at https://github.com/LINs-lab/SupervisorAgent .

## 1 Introduction

The advent of powerful Large Language Models (LLMs) has catalyzed significant advancements in Multi-Agent Systems (MAS) (Liu et al., 2025a; ang Gao et al., 2025), enabling them to achieve remarkable performance across diverse and challenging domains such as mathematical reasoning (Shang et al., 2025), code generation (Lu et al., 2025), and complex question answering (Luo et al., 2025).
This progress has spurred research into sophisticated agent architectures, including self-evolving systems that learn from feedback and experience (Shi et al., 2025b; Liu et al., 2025b), and dynamic topologies that adapt to task complexity (Li et al., 2025a; b).
However, a critical paradox has emerged: as these systems grow more capable and complex, they often become less robust and economically viable (Wu et al., 2025a; Huang et al., 2025).
Systemic inefficiencies incur prohibitive computational costs, while intricate interactions introduce vectors for unpredictable failures (Zhang et al., 2025e).

This lack of robustness stems from the operational complexity of modern MAS, which introduces a significant reliability challenge (Tian et al., 2025).
The long chain of interactions inherent in these systems creates fertile ground for error propagation (Dong et al., 2025; Shen et al., 2025).
For instance, a single piece of misinformation generated by an agent, a common risk with today’s powerful yet occasionally hallucinatory foundation models (Kalai et al., 2025; Farquhar et al., 2024), can be committed to memory and subsequently poison the reasoning of all downstream agents (as explained in Figure [1](https://arxiv.org/html/2510.26585v1#S1.F1)a).
These vulnerabilities mean that even a state-of-the-art MAS can fail on tasks well within its theoretical capabilities, simply due to a lack of operational robustness (Chen et al., 2024).

Furthermore, the issue of economic inefficiency is a major barrier to the real-world deployment of MAS (Wang et al., 2025a).
We identify two primary sources of this inefficiency.
First, agents often struggle with excessively long observations, such as verbose web pages or tool outputs, which flood their context windows.
This not only inflates token costs but can also obscure critical information, causing the agent to lose focus and derail its task execution (Hosseini et al., 2025).
Second, agents may adopt sub-optimal strategies, entering into repetitive action loops or choosing unnecessarily complex paths to a solution (Cemri et al., 2025), further wasting computational resources (see Figure [1](https://arxiv.org/html/2510.26585v1#S1.F1)a).
Our own analysis reveals that cutting-edge architectures can incur staggering token costs, with some GAIA benchmark (Mialon et al., 2023) tasks consuming upwards of two million tokens.

To address these intertwined challenges, we propose SupervisorAgent, a lightweight and modular framework that enhances the robustness and efficiency of Multi-Agent Systems (MAS) through real-time supervision (see Figure [1](https://arxiv.org/html/2510.26585v1#S1.F1)c).
Incorporating an adaptive filter, SupervisorAgent enables proactive process control, exemplified by its GAIA Level 2 performance in Figure [1](https://arxiv.org/html/2510.26585v1#S1.F1)e.
It adaptively intervenes at critical junctures to mitigate key operational risks: it conducts proactive error diagnosis, provides pragmatic guidance for inefficient behaviors, and performs adaptive observation purification to reduce contextual noise from long observations.

Figure: Figure 1: The SupervisorAgent Framework: Concept and Impact. (a) Illustrative examples of common failure modes in MAS, including error propagation and inefficient loops, and the corresponding intervention by our SupervisorAgent. (b) An overview of a conventional MAS, highlighting the high-risk interaction loci (agent-agent, agent-tool, agent-memory) where such failures occur. (c) The core workflow of our SupervisorAgent, which monitors these interactions to provide real-time intervention. (d) The resulting Supervised MAS (SMAS), which integrates the SupervisorAgent to enhance robustness and efficiency. (e) Performance on GAIA (Level 2), where SMAS (blue) reduces token cost by 35% and variance by 63% versus the baseline (red).
Refer to caption: x1.png

In summary, our main contributions are:

- 1.
We propose and implement SupervisorAgent, a novel, lightweight, and non-intrusive meta-agent framework for real-time MAS supervision.
It improves agent robustness and efficiency through proactive error correction, inefficiency guidance, and adaptive observation purification, without altering the base agents’ architecture.
- 2.
We conduct extensive experiments on the challenging GAIA benchmark and demonstrate a significant Pareto improvement.
When applied to the Smolagent framework (Roucher et al., 2025), SupervisorAgent reduces token consumption by an average of 29.68% while maintaining competitive task success rates.
- 3.
We validate the general applicability of our approach across five additional benchmarks spanning mathematical reasoning (GSM8k-Hard, AIME), code generation (HumanEval, MBPP), and question answering (DROP).
Our method consistently delivers substantial efficiency gains, highlighted by a 23.74% token reduction on HumanEval alongside an accuracy improvement.
The framework’s effectiveness is further confirmed across various foundation models, including the GPT-4.1, Gemini-2.5-pro, and Qwen3 series.

## 2 Related Work

##### The increasing complexity of Multi-Agent Systems (MAS).

Recent advancements in Large Language Models have spurred the development of increasingly sophisticated Multi-Agent Systems (MAS) capable of tackling complex, multi-step tasks (Tran et al., 2025; He et al., 2025).
Frameworks like Tongyi DeepResearch (Team, 2025c), AgentOrchestra (Zhang et al., 2025f), and Aime (Shi et al., 2025a) exemplify this trend, introducing complex features such as hierarchical structures (Zhu et al., 2025; Cheng et al., 2025), dynamic agent management (Wu et al., 2025b; Zhang et al., 2025g), and end-to-end training (Li et al., 2025b; Ye et al., 2025).
However, this escalating architectural complexity invariably introduces significant challenges in maintaining operational robustness and computational efficiency, which we address in this work.

##### Failure attribution and robustness.

A significant body of work has emerged to address the challenge of MAS robustness, primarily focusing on post-hoc *failure attribution* (Zhang et al., 2025e).
Systems like Aegis (Song et al., 2025) and SHIELDA (Zhou et al., 2025) propose taxonomies for failure analysis, while AgenTracer (Zhang et al., 2025b) and A2P (West et al., 2025) introduce methods to better trace the root causes of task failures.
While valuable, these methods are fundamentally reactive, analyzing failures after they have occurred.
In contrast, our SupervisorAgent is designed for *proactive, real-time intervention*, aiming to detect and mitigate high-risk steps *before* they lead to systemic failure.

##### Efficient Multi-Agent Systems.

Another stream of research targets the efficiency of MAS, a critical factor largely driven by token consumption. Most approaches focus on *design-time optimization*. Some prune the system’s architecture by eliminating agents with AgentDropout (Wang et al., 2025b) or communication links with SafeSieve (Zhang et al., 2025d). Others generatively construct efficient prompts (Han et al., 2025) or agent topologies from the outset, as seen in MetaAgent (Zhang et al., 2025h), MaAS (Zhang et al., 2025a), and HiVA (Tang et al., 2025). A second direction, *context compression*, aims to reduce token count by summarizing or distilling observations (Chen et al., 2025; Mou et al., 2025). Our work is orthogonal to these methods. Instead of focusing on static design or message content, we introduce runtime process control. SupervisorAgent addresses dynamic inefficiencies *during* execution, a complementary approach that can enhance existing systems.

## 3 Preliminary

In this section, we first establish a formalism for our proposed Supervised Multi-Agent System (SMAS).
We then detail the core components of our framework: the SupervisorAgent’s action space and the contextual information it leverages for decision-making.

### 3.1 A Formalism for Supervised Multi-Agent Systems

Our work is predicated on the idea that the complex, often chaotic, interactions within a Multi-Agent System (MAS; see Figure [1](https://arxiv.org/html/2510.26585v1#S1.F1)b) can be actively managed to improve both robustness and efficiency.
To formalize this, we introduce the concept of a Supervised Multi-Agent System (SMAS; see Figure [1](https://arxiv.org/html/2510.26585v1#S1.F1)d).

###### Definition 1 (Supervised Multi-Agent System (SMAS)) .

A SMAS is a Multi-Agent System augmented with a meta-level control agent, henceforth referred to as the Supervisor. The Supervisor’s objective is to monitor agent interactions in real-time, proactively detecting and mitigating operational risks without altering the core logic of the agents it oversees. In this work, we implement this conceptual Supervisor as a concrete agent named SupervisorAgent.

The fundamental unit of supervision is the interaction, which occurs when an agent engages with other system components. We categorize interactions into three primary types:

- 1.
Agent-Agent Interactions: Communication or delegation between agents.
In architectures like ReAct (Yao et al., 2023), where an agent’s output becomes another’s input, this channel is highly susceptible to the propagation of hallucinated or erroneous information (Shen et al., 2025);
- 2.
Agent-Tool Interactions:
The invocation of external tools or APIs.
This interaction is a primary source of external information, but it is also fraught with risks, including factually incorrect, irrelevant, or outdated data that can corrupt the agent’s context (Qian et al., 2025);
- 3.
Agent-Memory Interactions:
The retrieval of information from short- or long-term memory stores.
While crucial for self-evolving systems, memory introduces the hazard of acting upon stale or flawed information from past experiences (Xiong et al., 2025).

###### Definition 1 (Supervised Multi-Agent System (SMAS)) .

### 3.2 The SupervisorAgent ’s Context Window

To make informed decisions, the SupervisorAgent is provided with a rich, real-time snapshot of the MAS’s state, which we formalize as the *context window*.

###### Definition 2 (Context Window) .

The standard context window, $\mathcal{W}$, is a tuple of five key elements:

$$ $\mathcal{W}=(N,Q_{g},Q_{l},T_{l},S)\,,$ $$

where $N$ is the name of the agent under review, $Q_{g}$ and $Q_{l}$ are the global and local tasks, $T_{l}$ is the local trace of agent $N$’s recent actions and observation summaries, and $S$ is a summary of the agent’s latest interaction step.
For diagnosing system-wide inefficiencies, we augment this to an extended context window $\mathcal{W}_{\text{ext}}=\mathcal{W}\cup\{T_{g}\}$, where $T_{g}$ is the global trace of all agent interactions.

###### Definition 2 (Context Window) .

### 3.3 The SupervisorAgent ’s Action Space

The role of the SupervisorAgent is to diagnose high-risk interactions and execute a targeted intervention (Figure [2](https://arxiv.org/html/2510.26585v1#S4.F2)c). We define three primary intervention contexts, $c\in\mathcal{C}=\{c_{\text{error}},c_{\text{inefficient}},c_{\text{excessive}}\}$, which activate one of three core supervision strategies:

- •
Proactive Error Correction: Triggered by $c_{\text{error}}$, this strategy aims to diagnose the root cause of an explicit error and provide a direct fix or a verification task to resolve it.
- •
Guidance for Inefficiency: Triggered by $c_{\text{inefficient}}$, this strategy provides pragmatic, course-correcting hints for sub-optimal behaviors, while also critically permitting productive, albeit repetitive, processes to continue via an approve action.
- •
Adaptive Observation Purification: Triggered by $c_{\text{excessive}}$, this strategy refines excessively long or noisy observations to improve the signal-to-noise ratio for the agent.

These strategies are implemented by selecting an action $a$ from the global action space $\mathcal{A}$. The specific subset of permissible actions, $\mathcal{A}(c)$, is formally defined by the intervention context as follows:

$$ $\displaystyle\mathcal{A}(c)=\begin{cases}\{\textit{correct_observation},\textit{provide_guidance},\textit{run_verification}\}&\text{if }c=c_{\text{error}}\\ \{\textit{approve},\textit{provide_guidance}\}&\text{if }c=c_{\text{inefficient}}\\ \{\textit{correct_observation}\}&\text{if }c=c_{\text{excessive}}\end{cases}$ $$

The implementation of each action is detailed in Section [4.3](https://arxiv.org/html/2510.26585v1#S4.SS3).

## 4 Methodology

Figure: Figure 2: The architecture and workflow of SupervisorAgent. (a) The LLM-free adaptive filter for identifying high-risk interactions. (b) The context window, aggregating goals and traces for situational awareness. (c) The spectrum of intervention actions, from simple approval to intensive verification. (d, e) Case study on a GAIA task, comparing the baseline MAS (d) with our SMAS (e), which cuts steps by 43% and token cost by over 70%. (f) The supervise workflow for an interaction, from filtering to a final supervision action.
Refer to caption: x2.png

Building upon the formalism of a Supervised Multi-Agent System (SMAS) introduced in Section [3](https://arxiv.org/html/2510.26585v1#S3), we now detail the architecture and operational workflow of our SupervisorAgent (illustrated in Figure [2](https://arxiv.org/html/2510.26585v1#S4.F2)). Our methodology is structured around three fundamental questions: What to supervise, When to supervise, and How to supervise. We defer the specific implementation details, including all hyperparameters and prompts, to Appendix [A.3](https://arxiv.org/html/2510.26585v1#A1.SS3) and [A.5](https://arxiv.org/html/2510.26585v1#A1.SS5).

### 4.1 What to Supervise: High-Risk Interaction Points

The primary targets for our supervision are the three high-risk interaction points defined in our preliminary formalism (Section [3.1](https://arxiv.org/html/2510.26585v1#S3.SS1), see also Figure [2](https://arxiv.org/html/2510.26585v1#S4.F2)a): Agent-Agent, Agent-Tool, and Agent-Memory interactions. These points are the primary channels through which errors and inefficiencies are introduced and propagated throughout the system. Our goal is to monitor these specific channels to maintain the operational integrity of the MAS.

### 4.2 When to Supervise: The Adaptive Filter

While a naive approach might monitor every interaction, the associated computational cost is prohibitive and would undermine our goal of improving efficiency.
Therefore, the cornerstone of our framework is a lightweight, LLM-free adaptive filter (see in Figure[2](https://arxiv.org/html/2510.26585v1#S4.F2)a) designed to trigger supervision only at critical junctures (see case studies in Figures [2](https://arxiv.org/html/2510.26585v1#S4.F2)d and [2](https://arxiv.org/html/2510.26585v1#S4.F2)e).
This approach ensures that the SupervisorAgent’s resources are deployed judiciously, maximizing impact while minimizing overhead.
The filter is designed to be fast and heuristic-based, monitoring the MAS for three pre-defined, high-risk scenarios:

- •
Error occurrence:
The manifestation of an explicit error (e.g., in tool use or code execution) is a critical trigger.
Unlike current MAS that often pass the full error log into a cluttered context for a subsequent agent to debug, our filter immediately flags these events for a focused, real-time intervention.
- •
Inefficient behavior:
An agent may enter a loop of sub-optimal or repetitive actions that, while not explicit errors, lead to high token consumption and latency.
Our filter is designed to detect such patterns, such as an agent repeatedly using the page_down action instead of a more direct search strategy.
- •
Excessive observation length:
Interactions with tools can return excessively long and noisy observations (e.g., raw HTML) that inflate costs and distract the agent.
Our filter identifies such cases for immediate information purification.

### 4.3 How to Supervise: Memory-Augmented, Multi-Level Intervention

Once a high-risk interaction is flagged, SupervisorAgent leverages a rich context window and a spectrum of intervention strategies to deliver a nuanced, effective response.

##### Memory-augmented context window.

To make an effective decision, a supervisor must possess a more comprehensive understanding of the system’s state than any single agent.
This is why SupervisorAgent is conceptualized with its own memory module, not a simple monitor.
As illustrated in Figure [2](https://arxiv.org/html/2510.26585v1#S4.F2)b, this is achieved through a dynamic context window $\mathcal{W}$, which aggregates the global task $Q_{g}$, the agent’s local task $Q_{l}$, interaction summary $S$, and its recent local action trace $T_{l}$.
Crucially, for diagnosing complex inefficiencies, SupervisorAgent also accesses the global trace $T_{g}$, granting it a holistic perspective that transcends the limited view of any individual agent.
This elevated viewpoint is what enables it to provide genuinely strategic guidance.

##### A spectrum of intervention actions.

With this rich context, SupervisorAgent selects an action from a multi-level action space $\mathcal{A}$, adapting intervention intensity tailored to issue severity (Figure [2](https://arxiv.org/html/2510.26585v1#S4.F2)c).
These actions range from a minimal nudge to a comprehensive correction:

- •
*approve*: A minimal intervention that permits a productive, albeit repetitive, agent behavior to continue. Primarily used in the *inefficient* context, its purpose is to avoid disrupting a process that is pragmatically the best path forward from its current state.
- •
*provide_guidance*: A semi-intrusive action that steers an agent away from a sub-optimal strategy or logical flaw. This action appends a concise, directive hint to the existing observation, correcting the agent’s reasoning path without altering the core context data.
- •
*correct_observation*: A direct and forceful intervention that refines the agent’s sensory input. It is the sole action for *excessive observations*, where it purifies the content, and is also used in *error* contexts to fix factually incorrect data.
This action replaces the original raw observation entirely with a cleaned and corrected version.
- •
*run_verification*: The deepest intervention, used in complex *error* contexts when internal information is insufficient.
It invokes a verification sub-agent for external fact-checking or advanced debugging, returning a definitive, verified result.

## 5 Experiments

### 5.1 Experimental Setup

We empirically validate the effectiveness of SupervisorAgent through a series of extensive experiments.
We begin by outlining our evaluation metrics, datasets, and baselines.
For a more detailed description of the experimental settings, please refer to Appendix [A.2](https://arxiv.org/html/2510.26585v1#A1.SS2).

##### Datasets.

We evaluate our method on a diverse suite of six benchmarks spanning three domains.
Our primary benchmark is the challenging GAIA validation set (Mialon et al., 2023), which provides a comprehensive test of an MAS’s general problem-solving capabilities.
To demonstrate broader applicability, we use five additional benchmarks: for mathematical reasoning, we use AIME 2024 (HuggingFaceH4, 2024) and a random subset of 600 samples from GSM8k-Hard (Gao et al., 2022); for code generation, we use the full HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) datasets; and for question answering, we use a subset of 800 samples from the DROP (Dua et al., 2019) dataset, following the sampling strategy of prior work (Zhang et al., 2025c).

##### Baselines.

On several benchmarks, we compare SupervisorAgent against a comprehensive set of agentic systems equipped with web-browsing and code execution capabilities.
These baselines fall into two categories: (1) single agent execution methods: including vanilla LLM, Self Consistency CoT (3 answers) (Wang et al., 2023), and CodeAgent (Roucher et al., 2025); and (2) multi-agent systems, including Smolagent (Roucher et al., 2025), OAgents (Zhu et al., 2025), MetaAgent (Zhang et al., 2025h), OWL (role playing) (Hu et al., 2025), and AWorld (Xie et al., 2025; Yu et al., 2025).

##### Implementation details.

To assess model-agnosticism, we test SupervisorAgent with multiple foundation models.
For the demanding GAIA benchmark, we primarily use GPT-4.1 as the base model for all agents (see Table [1](https://arxiv.org/html/2510.26585v1#S5.T1)), and evaluate SupervisorAgent when powered by GPT-4.1 (OpenAI, 2025), Gemini-2.5-pro-0605 (Team, 2025a), and Qwen3-235B-2507 (Team, 2025b); see Figure [4(b)](https://arxiv.org/html/2510.26585v1#S5.F4.sf2).
For all other benchmarks (see Table [2](https://arxiv.org/html/2510.26585v1#S5.T2)), we employ the efficient and powerful Qwen3-32B (Team, 2025b) for both the base agents and the SupervisorAgent to assess performance in a more resource-constrained setting.

##### Testbed Selection.

We selected Smolagent as our primary experimental testbed, which provides a flexible framework upon which we build our agentic systems (SMAS). Critically, Smolagent’s capabilities stem primarily from its internal agentic interactions rather than powerful external tools(e.g. web APIs or solvers). This provides an ideal, controlled environment to isolate and evaluate the direct impact of our SupervisorAgent on an agent’s core reasoning and communication processes.

##### Metrics.

For GAIA and the code generation benchmarks, we report the standard pass@k metric.
For our main baseline, Smolagent, we report pass@1, 2, and 3.
For math reasoning, we report the final solve rate (%).
For question answering, we report the F1 score for DROP.
In all experiments, we meticulously track and report the total token consumption as a primary measure of efficiency.

**Table 1: Overall performance on the GAIA validation set (based on GPT-4.1). Our SMAS consistently reduces the average token cost comparing to Smolagent baseline while achieving competitive success rates.**
| Method | Avg. Acc. | Avg. Tokens (K) | L1 Acc. | L1 Tokens (K) | L2 Acc. | L2 Tokens (K) | L3 Acc. | L3 Tokens (K) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CodeAgent | 40.00 | 120.40 | 56.60 | 92.84 | 34.88 | 131.90 | 23.08 | 138.54 |
| OWL | 45.40 | 111.07 | 56.56 | 67.72 | 43.02 | 110.36 | 29.16 | 209.34 |
| OAgents | 49.09 | 340.50 | 66.04 | 260.27 | 47.67 | 358.63 | 19.23 | 444.11 |
| Smolagent | 50.91 | 527.76 | 62.26 | 298.51 | 53.49 | 619.59 | 19.23 | 691.33 |
| AWorld | 60.00 | 128.27 | 67.92 | 69.61 | 62.79 | 164.08 | 34.62 | 133.65 |
| pass@1 |  |  |  |  |  |  |  |  |
| Smolagent | 50.91 | 527.76 | 62.26 | 298.51 | 53.49 | 619.59 | 19.23 | 691.33 |
| + SMAS (ours) | 50.91 | 371.12 ↓29.68% | 62.26 | 258.28 ↓13.48% | 51.16 | 404.96 ↓34.64% | 26.92 ↑7.69% | 489.22 ↓29.23% |
| pass@2 |  |  |  |  |  |  |  |  |
| Smolagent | 58.18 | 467.19 | 69.81 | 275.85 | 59.30 | 548.02 | 30.77 | 589.92 |
| + SMAS (ours) | 58.79 ↑0.61% | 389.54 ↓16.62% | 73.58 ↑3.77% | 270.07 ↓2.10% | 56.98 | 420.97 ↓23.18% | 34.62 ↑3.85% | 529.20 ↓10.29% |
| pass@3 |  |  |  |  |  |  |  |  |
| Smolagent | 61.82 | 502.40 | 71.70 | 282.14 | 63.95 | 605.05 | 34.62 | 611.87 |
| + SMAS (ours) | 63.03 ↑1.21% | 369.52 ↓26.45% | 75.47 ↑3.77% | 276.84 ↓1.88% | 62.79 | 409.05 ↓32.39% | 38.46 ↑3.84% | 427.72 ↓30.10% |

### 5.2 Results and Analysis

##### Significant efficiency gains with competitive accuracy.

The main experimental results, presented in Table [1](https://arxiv.org/html/2510.26585v1#S5.T1), confirm the substantial benefits of SupervisorAgent.
On the GAIA validation set, when integrated with the Smolagent framework, SupervisorAgent achieves an average token reduction of 29.68% at pass@1, while maintaining a statistically equivalent success rate.
Notably, the efficiency gains are even more pronounced on more difficult tasks, with token savings reaching 32.39% on Level 2 and 30.10% on Level 3 tasks at pass@3.

*Across the other five benchmarks, SupervisorAgent generally achieves a Pareto improvement* (see Table [2](https://arxiv.org/html/2510.26585v1#S5.T2)).
In mathematical reasoning, it raises the AIME solve rate by 6.67% while cutting token costs by 18.92%.
In code generation, it maintains competitive accuracy on HumanEval and further reduces token use by 23.74%, likely due to its ability to streamline repetitive debugging cycles.
Occasionally, SupervisorAgent may overcompress long contexts during purification, causing minor accuracy or F1 drops on certain benchmarks.
These results underscore SupervisorAgent’s ability to act as a universal efficiency enhancer across diverse problem domains.

**Table 2: Generalization across diverse benchmarks (based on Qwen3-32B). SupervisorAgent consistently reduces token costs while maintaining or improving accuracy on tasks spanning mathematical reasoning, code generation, and question answering. All reported gains are relative to the Smolagent baseline.**
| Method | Metrics | GSM-hard | AIME | HumanEval | MBPP | DROP |
| --- | --- | --- | --- | --- | --- | --- |
| Vanilla | Acc / F1 (%) | 67.17 | 26.67 | 76.82 | 80.09 | 76.36 |
| Avg. Tokens (K) | 0.37 | 2.01 | 0.28 | 0.27 | 0.46 |  |
| CoT SC (3-shot) | Acc / F1 (%) | 69.01 | 30.00 | 77.78 | 81.26 | 77.72 |
| Avg. Tokens (K) | 2.62 | 14.26 | 1.42 | 1.29 | 2.73 |  |
| OWL | Acc / F1 (%) | 72.48 | 33.33 | 90.74 | 79.08 | 79.85 |
| Avg. Tokens (K) | 15.67 | 56.11 | 31.87 | 54.80 | 11.47 |  |
| MetaAgent | Acc / F1 (%) | 72.14 | 26.67 | 74.08 | 79.86 | 78.16 |
| Avg. Tokens (K) | 4.35 | 6.24 | 2.59 | 6.39 | 1.43 |  |
| Smolagent | Acc / F1 (%) | 74.33 | 30.00 | 92.07 | 85.68 | 81.08 |
| Avg. Tokens (K) | 11.59 | 59.14 | 40.91 | 111.07 | 12.01 |  |
| + SMAS (ours) | Acc / F1 (%) | 75.50 | 36.67 | 92.68 | 84.43 | 79.80 |
| Avg. Tokens (K) | 10.55 ↓8.92% | 47.95 ↓18.92% | 31.19 ↓23.74% | 103.71 ↓6.62% | 11.34 ↓5.60% |  |

Figure: Figure 3: SupervisorAgent enhances performance consistency on the GAIA benchmark. (a) Violin plots of token cost distributions, revealing the more compact and predictable performance of our Supervised MAS (SMAS). (b) A direct comparison quantifying the substantial reduction in token cost variance achieved by our SMAS across all difficulty levels.
Refer to caption: resources/figures/violin_and_variance.png

##### Model-Agnostic generalization.

To demonstrate that the benefits of SupervisorAgent are architectural rather than model-specific, we evaluated it with three different powerful LLMs as its inference engine on GAIA.
As shown in Figure [4(b)](https://arxiv.org/html/2510.26585v1#S5.F4.sf2), *SupervisorAgent consistently yields significant token savings and maintains robust performance across all models, including GPT-4.1, Gemini-2.5-pro, and Qwen3-235B.*
This validates that our supervision framework is a model-agnostic component that can enhance a wide variety of LLM-powered agent systems.

##### Improving robustness and performance consistency.

Beyond average performance, we define robustness as the consistency of an agent’s performance.
As illustrated by the violin plots in Figure [3](https://arxiv.org/html/2510.26585v1#S5.F3), *SupervisorAgent significantly reduces the variance in token consumption per task.*
The distributions for the SMAS are visibly shorter and wider, indicating a more concentrated and predictable performance profile.
The bar chart on the right further quantifies this, showing a marked decrease in token cost variance, especially for the more complex Level 2 and 3 tasks.
This demonstrates that *our method not only makes the MAS more efficient on average but also more reliable and less prone to extreme resource consumption outliers.*

##### Ablation study.

Our ablation study, conducted on the subset of GAIA benchmark (top-10 most token-intensive tasks per GAIA level), dissects the contribution of SupervisorAgent’s three core strategies (Table [3](https://arxiv.org/html/2510.26585v1#S5.T3), Figure [4(a)](https://arxiv.org/html/2510.26585v1#S5.F4.sf1)). We compare the full SupervisorAgent against three ablated versions: w/o Correction (disabling Proactive Error Correction), w/o Guidance (disabling Guidance for Inefficiency), and w/o Purification (disabling Adaptive Observation Purification). The results reveal that Purification is the primary driver of token reduction. Concurrently, removing the Correction and Guidance modules leads to a significant drop in accuracy, confirming their critical role in ensuring task success and robustness. This highlights a clear trade-off: while Purification is key to efficiency, the other two strategies are crucial for maximizing performance, justifying their marginal token cost by preventing costly failures.

**Table 3: Ablation study of SupervisorAgent’s components (based on GPT-4.1) on the subset of GAIA benchmark (top-10 most token-intensive tasks per GAIA level), dissecting the contributions of each module.**
| Method | Avg. Acc. | Avg. Token | Level 1 Avg. Token | Level 2 Avg. Token | Level 3 Avg. Token |
| --- | --- | --- | --- | --- | --- |
| Smolagent | 40.00 | 1,446,526 | 933,013 | 2,037,437 | 1,369,131 |
| + SMAS (w/o Correction) | 40.00 | 719,075 ↓50.28% | 426,786 ↓54.26% | 755,543 ↓62.91% | 974,895 ↓28.79% |
| + SMAS (w/o Guidance) | 40.00 | 706,831 ↓51.14% | 453,623 ↓51.38% | 913,109 ↓55.18% | 753,761 ↓44.95% |
| + SMAS (w/o Purification) | 46.67↑6.67% | 851,747 ↓41.11% | 585,411 ↓37.26% | 990,769 ↓51.37% | 979,061 ↓28.49% |
| + SMAS | 46.67 ↑6.67% | 721,332 ↓50.13% | 522,364 ↓44.01% | 960,694 ↓52.85% | 680,939 ↓50.26% |

Figure: (a) Ablation study on GAIA.
Refer to caption: x3.png

##### MAS-Agnostic generalization.

To verify MAS-agnostic feature of SupervisorAgent, we integrate SupervisorAgent into two distinct multi-agent system frameworks: AWorld (Xie et al., 2025) and OAgents (Zhu et al., 2025), and evaluate their performance on the subset of GAIA benchmark(the same as in ablation study).
The results, presented in Table [4](https://arxiv.org/html/2510.26585v1#S5.T4), indicate that SupervisorAgent consistently enhances the performance of both frameworks, underscoring its versatility and effectiveness across different MAS architectures.

Specifically, integrated with AWorld (Xie et al., 2025), our SMAS(AWorld) achieved superior average accuracy over both the original AWorld (without Guard) and the Guard-enabled version. Furthermore, SMAS(AWorld) demonstrated substantial token efficiency, saving 36.54% on average versus AWorld (with Guard). With savings reaching 48.38% on Level 3 tasks, it confirms SupervisorAgent’s ability to enhance sophisticated MAS.

Applying SupervisorAgent to OAgents (Zhu et al., 2025) further validated its general applicability, reducing average token consumption by 39.36% while maintaining competitive accuracy. Interestingly, the largest token reduction (50.19%) occurred on Level 1 tasks, exceeding the savings on Level 3 (40.63%). While this might reflect OAgents’ inherent proficiency on harder tasks, the significant overall savings underscore SupervisorAgent’s broad utility.

**Table 4: Cross-framework performance of SupervisorAgent on Smolagent, AWorld, and OAgents (based on GPT-4.1). Evaluated on the same GAIA subset (top-10 token-intensive tasks per level) as in Table [3](https://arxiv.org/html/2510.26585v1#S5.T3).**
| Method | Avg. Acc. | Avg. Token | Level 1 Avg. Token | Level 2 Avg. Token | Level 3 Avg. Token |
| --- | --- | --- | --- | --- | --- |
| Smolagent | 40.00 | 1,446,526 | 933,013 | 2,037,437 | 1,369,131 |
| + SMAS | 46.67 ↑6.67% | 721,332 ↓50.13% | 522,364 ↓44.01% | 960,694 ↓52.85% | 680,939 ↓50.26% |
| AWorld (without Guard) | 23.33 | 155,239 | 50,851 | 217,332 | 166,500 |
| AWorld (with Guard) | 30.00 | 353,738 | 135,413 | 463,083 | 376,878 |
| AWorld (with SMAS) | 36.67 ↑6.67% | 224,480 ↓36.54% | 90,569 ↓33.12% | 355,051 ↓23.33% | 194,561 ↓48.38% |
| OAgents | 46.67 | 530,939 | 430,852 | 359,511 | 802,454 |
| + SMAS | 46.67 | 321,957 ↓39.36% | 214,604 ↓50.19% | 274,875 ↓23.54% | 476,393 ↓40.63% |

## 6 Discussion

##### Supervisor as a foundational MAS component.

Our work positions SupervisorAgent as a foundational component for future Multi-Agent Systems, akin to established modules like memory banks and tool-usage frameworks. By providing real-time, adaptive supervision, SupervisorAgent alleviates critical challenges of robustness and efficiency that are pervasive across diverse MAS architectures. Its modular design allows for seamless integration with existing systems, enhancing their performance without necessitating fundamental changes to their core logic. This underscores the potential of supervisory agents as universal enhancers of MASs, capable of elevating both reliability and cost-effectiveness across a wide range of applications.

##### Comparison with related supervisory agents.

A related concept is the Guard agent in AWorld (Xie et al., 2025), which is invoked at key steps primarily for factual verification to enhance task accuracy. While valuable, its scope differs significantly from our method. SupervisorAgent adopts a broader objective of improving overall system efficiency and robustness through continuous (albeit adaptively filtered) monitoring and a wider range of interventions, including error correction, inefficiency guidance, and observation purification, complementing the Guard’s focus on accuracy.

##### Broader insights.

Our work also yields critical insights for the broader field. First, we discovered that seemingly “noisy” information, such as HTML structure and truncation cues, serves as a vital signal for ReAct-style agents. The overly aggressive purification can paradoxically harm performance. This highlights a fundamental trade-off between information density and the preservation of environmental texture.
Second, our focus on token cost underscores the need for a more holistic efficiency evaluation for MAS. A comprehensive analysis must also account for the frequency and complexity of external tool API calls, which offload significant burdens from the MAS.
This very trade-off informed our choice of Smolagent as a primary testbed - its reliance on internal agentic reasoning, rather than powerful external tools, provided a controlled environment to isolate and evaluate our SupervisorAgent’s impact on the interaction process itself.

##### Future directions.

These insights inform several promising avenues for future work. While our method proved efficient and robust, developing a self-evolving, memory-augmented version remains an exciting opportunity. Other key directions include creating more sophisticated purification techniques that address the “noise-as-signal” problem and developing a universal resource consumption metric for MAS. Ultimately, we posit that incorporating such real-time, meta-level supervision is a foundational component for building the next generation of truly scalable and reliable MAS.

## 7 Conclusion

In this work, we introduced SupervisorAgent, a lightweight and non-intrusive meta-agent framework that enhances the robustness and efficiency of Multi-Agent Systems. Through real-time, adaptive supervision, SupervisorAgent mitigates common failure modes and reduces computational overhead using three core strategies: proactive error correction, pragmatic inefficiency guidance, and adaptive observation purification. Our extensive experiments demonstrate a significant Pareto improvement. On the challenging GAIA benchmark, SupervisorAgent reduces token consumption by an average of 29.68% while maintaining competitive task success rates, a crucial step towards building more practical and scalable agentic systems.

## References

- ang Gao et al. (2025)
Huan ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, Hongru Wang, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, Qihan Ren, Cheng Qian, Zhenghailong Wang, Minda Hu, Huazheng Wang, Qingyun Wu, Heng Ji, and Mengdi Wang.
A survey of self-evolving agents: On path to artificial super intelligence, 2025.
URL [https://arxiv.org/abs/2507.21046](https://arxiv.org/abs/2507.21046).
- Austin et al. (2021)
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton.
Program synthesis with large language models, 2021.
URL [https://arxiv.org/abs/2108.07732](https://arxiv.org/abs/2108.07732).
- Cemri et al. (2025)
Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica.
Why do multi-agent llm systems fail?, 2025.
URL [https://arxiv.org/abs/2503.13657](https://arxiv.org/abs/2503.13657).
- Chen et al. (2025)
Junzhi Chen, Juhao Liang, and Benyou Wang.
Smurfs: Multi-agent system using context-efficient dfsdt for tool planning.
In *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp. 3281–3298, 2025.
- Chen et al. (2021)
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.
Evaluating large language models trained on code, 2021.
URL [https://arxiv.org/abs/2107.03374](https://arxiv.org/abs/2107.03374).
- Chen et al. (2024)
Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li.
Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases, 2024.
URL [https://arxiv.org/abs/2407.12784](https://arxiv.org/abs/2407.12784).
- Cheng et al. (2025)
Yuyang Cheng, Yumiao Xu, Chaojia Yu, and Yong Zhao.
Hawk: A hierarchical workflow framework for multi-agent collaboration, 2025.
URL [https://arxiv.org/abs/2507.04067](https://arxiv.org/abs/2507.04067).
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021.
- Dong et al. (2025)
Shen Dong, Shaochen Xu, Pengfei He, Yige Li, Jiliang Tang, Tianming Liu, Hui Liu, and Zhen Xiang.
A practical memory injection attack against llm agents, 2025.
URL [https://arxiv.org/abs/2503.03704](https://arxiv.org/abs/2503.03704).
- Dua et al. (2019)
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.
DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.
In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pp. 2368–2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
doi: 10.18653/v1/N19-1246.
URL [https://aclanthology.org/N19-1246/](https://aclanthology.org/N19-1246/).
- Farquhar et al. (2024)
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal.
Detecting hallucinations in large language models using semantic entropy.
*Nature*, 630(8017):625–630, jun 2024.
ISSN 1476-4687.
doi: 10.1038/s41586-024-07421-0.
URL [https://www.nature.com/articles/s41586-024-07421-0](https://www.nature.com/articles/s41586-024-07421-0).
- Gao et al. (2022)
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.
Pal: Program-aided language models.
*arXiv preprint arXiv:2211.10435*, 2022.
- Han et al. (2025)
Yichen Han, Bojun Liu, Zhengpeng zhou, Guanyu Liu, Zeng Zhang, Yang Yang, Wenli Wang, Isaac N Shi, Yunyan, Lewei He, and Tianyu Shi.
Mapgd: Multi-agent prompt gradient descent for collaborative prompt optimization, 2025.
URL [https://arxiv.org/abs/2509.11361](https://arxiv.org/abs/2509.11361).
- He et al. (2025)
Junda He, Christoph Treude, and David Lo.
Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead.
*ACM Transactions on Software Engineering and Methodology*, 34(5):1–30, 2025.
- Hosseini et al. (2025)
Peyman Hosseini, Ignacio Castro, Iacopo Ghinassi, and Matthew Purver.
Efficient solutions for an intriguing failure of LLMs: Long context window does not mean LLMs can analyze long sequences flawlessly.
In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), *Proceedings of the 31st International Conference on Computational Linguistics*, pp. 1880–1891, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics.
URL [https://aclanthology.org/2025.coling-main.128/](https://aclanthology.org/2025.coling-main.128/).
- Hu et al. (2025)
Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Qiguang Chen, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Bernard Ghanem, Ping Luo, and Guohao Li.
Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025.
URL [https://arxiv.org/abs/2505.23885](https://arxiv.org/abs/2505.23885).
- Huang et al. (2025)
Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael R. Lyu.
Competing large language models in multi-agent gaming environments.
In *Proceedings of the Thirteenth International Conference on Learning Representations (ICLR)*, 2025.
- HuggingFaceH4 (2024)
HuggingFaceH4.
Aime 2024 dataset.
[https://huggingface.co/datasets/HuggingFaceH4/aime_2024](https://huggingface.co/datasets/HuggingFaceH4/aime_2024), 2024.
- Kalai et al. (2025)
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang.
Why language models hallucinate, 2025.
URL [https://arxiv.org/abs/2509.04664](https://arxiv.org/abs/2509.04664).
- Li et al. (2025a)
Shiyuan Li, Yixin Liu, Qingsong Wen, Chengqi Zhang, and Shirui Pan.
Assemble your crew: Automatic multi-agent communication topology design via autoregressive graph generation, 2025a.
URL [https://arxiv.org/abs/2507.18224](https://arxiv.org/abs/2507.18224).
- Li et al. (2025b)
Weizhen Li, Jianbo Lin, Zhuosong Jiang, Jingyi Cao, Xinpeng Liu, Jiayu Zhang, Zhenqiang Huang, Qianben Chen, Weichen Sun, Qiexiang Wang, Hongxuan Lu, Tianrui Qin, Chenghao Zhu, Yi Yao, Shuying Fan, Xiaowan Li, Tiannan Wang, Pai Liu, King Zhu, He Zhu, Dingfeng Shi, Piaohong Wang, Yeyi Guan, Xiangru Tang, Minghao Liu, Yuchen Eleanor Jiang, Jian Yang, Jiaheng Liu, Ge Zhang, and Wangchunshu Zhou.
Chain-of-agents: End-to-end agent foundation models via multi-agent distillation and agentic rl, 2025b.
URL [https://arxiv.org/abs/2508.13167](https://arxiv.org/abs/2508.13167).
- Liu et al. (2025a)
Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, Yuheng Cheng, Suyuchen Wang, Xiaoqiang Wang, Yuyu Luo, Haibo Jin, Peiyan Zhang, Ollie Liu, Jiaqi Chen, Huan Zhang, Zhaoyang Yu, Haochen Shi, Boyan Li, Dekun Wu, Fengwei Teng, Xiaojun Jia, Jiawei Xu, Jinyu Xiang, Yizhang Lin, Tianming Liu, Tongliang Liu, Yu Su, Huan Sun, Glen Berseth, Jianyun Nie, Ian Foster, Logan Ward, Qingyun Wu, Yu Gu, Mingchen Zhuge, Xinbing Liang, Xiangru Tang, Haohan Wang, Jiaxuan You, Chi Wang, Jian Pei, Qiang Yang, Xiaoliang Qi, and Chenglin Wu.
Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems, 2025a.
URL [https://arxiv.org/abs/2504.01990](https://arxiv.org/abs/2504.01990).
- Liu et al. (2025b)
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu.
Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners, 2025b.
URL [https://arxiv.org/abs/2504.14239](https://arxiv.org/abs/2504.14239).
- Lu et al. (2025)
Xu Lu, Weisong Sun, Yiran Zhang, Ming Hu, Cong Tian, Zhi Jin, and Yang Liu.
Requirements development and formalization for reliable code generation: A multi-agent vision, 2025.
URL [https://arxiv.org/abs/2508.18675](https://arxiv.org/abs/2508.18675).
- Luo et al. (2025)
Yajie Luo, Yihong Wu, Muzhi Li, Fengran Mo, Jia Ao Sun, Xinyu Wang, Liheng Ma, Yingxue Zhang, and Jian-Yun Nie.
An entity linking agent for question answering, 2025.
URL [https://arxiv.org/abs/2508.03865](https://arxiv.org/abs/2508.03865).
- Mialon et al. (2023)
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom.
Gaia: a benchmark for general ai assistants, 2023.
URL [https://arxiv.org/abs/2311.12983](https://arxiv.org/abs/2311.12983).
- Mou et al. (2025)
Xinyi Mou, Chen Qian, Wei Liu, Xuanjing Huang, and Zhongyu Wei.
Ecolang: Efficient and effective agent communication language induction for social simulation, 2025.
URL [https://arxiv.org/abs/2505.06904](https://arxiv.org/abs/2505.06904).
- OpenAI (2025)
OpenAI.
Introducing gpt-4.1 in the api.
OpenAI blog post, Apr 2025.
URL [https://openai.com/index/gpt-4-1/](https://openai.com/index/gpt-4-1/).
- Qian et al. (2025)
Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji.
Smart: Self-aware agent for tool overuse mitigation, 2025.
URL [https://arxiv.org/abs/2502.11435](https://arxiv.org/abs/2502.11435).
- Roucher et al. (2025)
Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki.
‘smolagents‘: a smol library to build great agentic systems.
[https://github.com/huggingface/smolagents](https://github.com/huggingface/smolagents), 2025.
- Shang et al. (2025)
Ning Shang, Yifei Liu, Yi Zhu, Li Lyna Zhang, Weijiang Xu, Xinyu Guan, Buze Zhang, Bingcheng Dong, Xudong Zhou, Bowen Zhang, Ying Xin, Ziming Miao, Scarlett Li, Fan Yang, and Mao Yang.
rstar2-agent: Agentic reasoning technical report, 2025.
URL [https://arxiv.org/abs/2508.20722](https://arxiv.org/abs/2508.20722).
- Shen et al. (2025)
Xu Shen, Yixin Liu, Yiwei Dai, Yili Wang, Rui Miao, Yue Tan, Shirui Pan, and Xin Wang.
Understanding the information propagation effects of communication topologies in llm-based multi-agent systems, 2025.
URL [https://arxiv.org/abs/2505.23352](https://arxiv.org/abs/2505.23352).
- Shi et al. (2025a)
Yexuan Shi, Mingyu Wang, Yunxiang Cao, Hongjie Lai, Junjian Lan, Xin Han, Yu Wang, Jie Geng, Zhenan Li, Zihao Xia, Xiang Chen, Chen Li, Jian Xu, Wenbo Duan, and Yuanshuo Zhu.
Aime: Towards fully-autonomous multi-agent framework, 2025a.
URL [https://arxiv.org/abs/2507.11988](https://arxiv.org/abs/2507.11988).
- Shi et al. (2025b)
Yucheng Shi, Wenhao Yu, Zaitang Li, Yonglin Wang, Hongming Zhang, Ninghao Liu, Haitao Mi, and Dong Yu.
Mobilegui-rl: Advancing mobile gui agent through reinforcement learning in online environment, 2025b.
URL [https://arxiv.org/abs/2507.05720](https://arxiv.org/abs/2507.05720).
- Song et al. (2025)
Kevin Song, Anand Jayarajan, Yaoyao Ding, Qidong Su, Zhanda Zhu, Sihang Liu, and Gennady Pekhimenko.
Aegis: Taxonomy and optimizations for overcoming agent-environment failures in llm agents, 2025.
URL [https://arxiv.org/abs/2508.19504](https://arxiv.org/abs/2508.19504).
- Tang et al. (2025)
Jinzhou Tang, Jusheng Zhang, Qinhan Lv, Sidi Liu, Jing Yang, Chengpei Tang, and Keze Wang.
Hiva: Self-organized hierarchical variable agent via goal-driven semantic-topological evolution, 2025.
URL [https://arxiv.org/abs/2509.00189](https://arxiv.org/abs/2509.00189).
- Team (2025a)
Gemini Team.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025a.
URL [https://arxiv.org/abs/2507.06261](https://arxiv.org/abs/2507.06261).
- Team (2025b)
Qwen Team.
Qwen3 technical report, 2025b.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
- Team (2025c)
Tongyi DeepResearch Team.
Tongyi-deepresearch.
[https://github.com/Alibaba-NLP/DeepResearch](https://github.com/Alibaba-NLP/DeepResearch), 2025c.
- Tian et al. (2025)
Fangqiao Tian, An Luo, Jin Du, Xun Xian, Robert Specht, Ganghua Wang, Xuan Bi, Jiawei Zhou, Ashish Kundu, Jayanth Srinivasa, Charles Fleming, Rui Zhang, Zirui Liu, Mingyi Hong, and Jie Ding.
An outlook on the opportunities and challenges of multi-agent ai systems, 2025.
URL [https://arxiv.org/abs/2505.18397](https://arxiv.org/abs/2505.18397).
- Tran et al. (2025)
Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D. Nguyen.
Multi-agent collaboration mechanisms: A survey of llms, 2025.
URL [https://arxiv.org/abs/2501.06322](https://arxiv.org/abs/2501.06322).
- Wang et al. (2025a)
Ningning Wang, Xavier Hu, Pai Liu, He Zhu, Yue Hou, Heyuan Huang, Shengyu Zhang, Jian Yang, Jiaheng Liu, Ge Zhang, Changwang Zhang, Jun Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou.
Efficient agents: Building effective agents while reducing cost, 2025a.
URL [https://arxiv.org/abs/2508.02694](https://arxiv.org/abs/2508.02694).
- Wang et al. (2023)
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.
Self-consistency improves chain of thought reasoning in language models, 2023.
URL [https://arxiv.org/abs/2203.11171](https://arxiv.org/abs/2203.11171).
- Wang et al. (2025b)
Zhexuan Wang, Yutong Wang, Xuebo Liu, Liang Ding, Miao Zhang, Jie Liu, and Min Zhang.
Agentdropout: Dynamic agent elimination for token-efficient and high-performance llm-based multi-agent collaboration, 2025b.
URL [https://arxiv.org/abs/2503.18891](https://arxiv.org/abs/2503.18891).
- West et al. (2025)
Alva West, Yixuan Weng, Minjun Zhu, Zhen Lin, and Yue Zhang.
Abduct, act, predict: Scaffolding causal inference for automated failure attribution in multi-agent systems, 2025.
URL [https://arxiv.org/abs/2509.10401](https://arxiv.org/abs/2509.10401).
- Wu et al. (2025a)
Chen Henry Wu, Rishi Shah, Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried, and Aditi Raghunathan.
Dissecting adversarial robustness of multimodal lm agents, 2025a.
URL [https://arxiv.org/abs/2406.12814](https://arxiv.org/abs/2406.12814).
- Wu et al. (2025b)
Feijie Wu, Zitao Li, Fei Wei, Yaliang Li, Bolin Ding, and Jing Gao.
Talk to right specialists: Routing and planning in multi-agent system for question answering, 2025b.
URL [https://arxiv.org/abs/2501.07813](https://arxiv.org/abs/2501.07813).
- Xie et al. (2025)
Zhitian Xie, Qintong Wu, Chengyue Yu, Chenyi Zhuang, and Jinjie Gu.
Profile-aware maneuvering: A dynamic multi-agent system for robust gaia problem solving by aworld, 2025.
URL [https://arxiv.org/abs/2508.09889](https://arxiv.org/abs/2508.09889).
- Xiong et al. (2025)
Zidi Xiong, Yuping Lin, Wenya Xie, Pengfei He, Jiliang Tang, Himabindu Lakkaraju, and Zhen Xiang.
How memory management impacts llm agents: An empirical study of experience-following behavior, 2025.
URL [https://arxiv.org/abs/2505.16067](https://arxiv.org/abs/2505.16067).
- Yao et al. (2023)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models, 2023.
URL [https://arxiv.org/abs/2210.03629](https://arxiv.org/abs/2210.03629).
- Ye et al. (2025)
Rui Ye, Shuo Tang, Rui Ge, Yaxin Du, Zhenfei Yin, Siheng Chen, and Jing Shao.
Mas-gpt: Training llms to build llm-based multi-agent systems, 2025.
URL [https://arxiv.org/abs/2503.03686](https://arxiv.org/abs/2503.03686).
- Yu et al. (2025)
Chengyue Yu, Siyuan Lu, Chenyi Zhuang, Dong Wang, Qintong Wu, Zongyue Li, Runsheng Gan, Chunfeng Wang, Siqi Hou, Gaochi Huang, Wenlong Yan, Lifeng Hong, Aohui Xue, Yanfeng Wang, Jinjie Gu, David Tsai, and Tao Lin.
Aworld: Orchestrating the training recipe for agentic ai, 2025.
URL [https://arxiv.org/abs/2508.20404](https://arxiv.org/abs/2508.20404).
- Zhang et al. (2025a)
Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang.
Multi-agent architecture search via agentic supernet.
*arXiv preprint arXiv:2502.04180*, 2025a.
- Zhang et al. (2025b)
Guibin Zhang, Junhao Wang, Junjie Chen, Wangchunshu Zhou, Kun Wang, and Shuicheng Yan.
Agentracer: Who is inducing failure in the llm agentic systems?, 2025b.
URL [https://arxiv.org/abs/2509.03312](https://arxiv.org/abs/2509.03312).
- Zhang et al. (2025c)
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu.
Aflow: Automating agentic workflow generation, 2025c.
URL [https://arxiv.org/abs/2410.10762](https://arxiv.org/abs/2410.10762).
- Zhang et al. (2025d)
Ruijia Zhang, Xinyan Zhao, Ruixiang Wang, Sigen Chen, Guibin Zhang, An Zhang, Kun Wang, and Qingsong Wen.
Safesieve: From heuristics to experience in progressive pruning for llm-based multi-agent communication, 2025d.
URL [https://arxiv.org/abs/2508.11733](https://arxiv.org/abs/2508.11733).
- Zhang et al. (2025e)
Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu.
Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems, 2025e.
URL [https://arxiv.org/abs/2505.00212](https://arxiv.org/abs/2505.00212).
- Zhang et al. (2025f)
Wentao Zhang, Liang Zeng, Yuzhen Xiao, Yongcong Li, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An.
Agentorchestra: A hierarchical multi-agent framework for general-purpose task solving, 2025f.
URL [https://arxiv.org/abs/2506.12508](https://arxiv.org/abs/2506.12508).
- Zhang et al. (2025g)
Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, and Volker Tresp.
Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pp. 23378–23386, 2025g.
- Zhang et al. (2025h)
Yaolun Zhang, Xiaogeng Liu, and Chaowei Xiao.
Metaagent: Automatically constructing multi-agent systems based on finite state machines, 2025h.
URL [https://arxiv.org/abs/2507.22606](https://arxiv.org/abs/2507.22606).
- Zhou et al. (2025)
Jingwen Zhou, Jieshan Chen, Qinghua Lu, Dehai Zhao, and Liming Zhu.
Shielda: Structured handling of exceptions in llm-driven agentic workflows, 2025.
URL [https://arxiv.org/abs/2508.07935](https://arxiv.org/abs/2508.07935).
- Zhu et al. (2025)
He Zhu, Tianrui Qin, King Zhu, Heyuan Huang, Yeyi Guan, Jinxiang Xia, Yi Yao, Hanhao Li, Ningning Wang, Pai Liu, Tianhao Peng, Xin Gui, Xiaowan Li, Yuhui Liu, Yuchen Eleanor Jiang, Jun Wang, Changwang Zhang, Xiangru Tang, Ge Zhang, Jian Yang, Minghao Liu, Xitong Gao, Jiaheng Liu, and Wangchunshu Zhou.
Oagents: An empirical study of building effective agents, 2025.
URL [https://arxiv.org/abs/2506.15741](https://arxiv.org/abs/2506.15741).

## Appendix A Appendix

### A.1 LLM usage

The large language model (LLM) was utilized as a writing assistant during the preparation of this manuscript. Its application was strictly limited to improving the clarity and grammatical accuracy of the text. Specific uses included rephrasing sentences for better flow and translating initial concepts and drafts from Chinese to English. All core scientific contributions, including the conceptualization of our SupervisorAgent framework, the design of the methodology and experiments, and the analysis and interpretation of the results, are solely the work of the authors. The authors take full responsibility for all claims and the final content of this paper.

### A.2 Experimental Setup

#### A.2.1 Datasets

Here, we provide a detailed introduction to the datasets used in this paper:

- •
GAIA (Mialon et al., 2023) serves as a benchmark designed to evaluate next-generation LLMs that possess enhanced capabilities through the incorporation of tools, efficient prompting strategies, and access to external search resources. This benchmark comprises over 450 challenging questions, each with a clear and unequivocal answer, necessitating varying degrees of tooling and autonomy for resolution. Accordingly, the questions are categorized into three distinct levels: Level 1 is expected to be solvable by proficient LLMs, while Level 3 signifies a substantial increase in the model’s capabilities. Each level includes a fully public development set for validation purposes, as well as a test set containing private answers and associated metadata. In our experiments, we utilize the test set, which encompasses 164 tasks.
- •
GSM-hard (Gao et al., 2022) is an advanced version of the GSM8K mathematics reasoning dataset (Cobbe et al., 2021). This enhanced dataset presents models with increased challenges, featuring larger numerical values and more complex relationships within the problems.
- •
AIME-2024 (HuggingFaceH4, 2024) is a dataset comprising problems derived from the American Invitational Mathematics Examination (AIME) 2024. AIME is a prestigious mathematics competition for high school students, recognized for its challenging problems that span various mathematical domains. This benchmark serves multiple purposes: it evaluates the mathematical reasoning capabilities of LLMs, assesses their problem-solving abilities on complex mathematical challenges, and investigates AI performance on structured mathematical tasks.
- •
HumanEval (Chen et al., 2021) is a dataset released by OpenAI that includes 164 programming problems, each containing a function signature, a docstring, a body, and several associated unit tests. These problems were handwritten to ensure that they were not included in the training dataset for code-generation models. This benchmark is crucial for evaluating code-generation models, providing a structured set of challenges in Python that facilitates the assessment of both the quality and correctness of code produced by language models.
- •
MBPP(Mostly Basic Python Problems Dataset) (Austin et al., 2021) comprises approximately 1,000 crowd-sourced Python programming problems that are specifically designed to be solvable by entry-level programmers. The dataset covers essential programming fundamentals and standard library functionalities. Each problem includes a task description, a corresponding code solution, and three automated test cases.
- •
DROP(Data Retrieval Open Answering) Dua et al. (2019) is a reading comprehension benchmark that requires discrete reasoning over paragraphs. This dataset consists of 96,000 questions developed through crowd sourcing and adversarial methods. It challenges systems to resolve references within the questions, which may point to multiple input positions. The tasks entail performing discrete operations, such as addition, counting, and sorting, necessitating a substantially more comprehensive understanding of paragraph content than that demanded by prior datasets. In our experiment, we sampled 800 tasks for evaluation.

#### A.2.2 Baselines

- •
Vanilla is the original Large Language Model (LLM) that processes input using only the question and a basic prompt, without any prompt engineering or external tool integration. This straightforward approach emphasizes the model’s inherent capabilities in handling natural language tasks. By operating in this simplistic manner, Vanilla LLM serves as a critical baseline for evaluating the performance of more advanced techniques that incorporate sophisticated prompt strategies or additional tools, thereby providing valuable insights into the effectiveness of various methodologies in natural language processing.
- •
CoT-SC(Chain-of-Thought Self-Consistency) (Wang et al., 2023) serves as a baseline for enhancing the reasoning capabilities of language models. This approach generates multiple reasoning chains, which are then aggregated to produce a coherent summary. By leveraging self-consistency, CoT-SC improves the reliability of the model’s outputs, allowing for better performance in complex reasoning tasks. This structured process facilitates deeper analysis of the model’s thought processes, providing a foundation for comparing more advanced reasoning strategies and understanding their impact on overall performance.
- •
MetaAgent (Zhang et al., 2025h) is a groundbreaking framework designed to automatically construct multi-agent systems by specifying the objectives of a given task. A distinctive feature of MetaAgent is its ability to generate these multi-agent systems without relying on external training data. This capability allows the produced multi-agent systems to effectively address all scenarios within the specified task domain. The underlying architecture of the Multi-Agent System is based on Finite State Machines(FSM), which facilitates structured decision-making and state transitions, thereby enhancing the system’s operational efficiency and adaptability.
- •
OWL(Open Web Language) (Hu et al., 2025) serves as a foundational framework for knowledge representation in multi-agent systems. By enabling agents to process and reason over complex data in a machine-readable format, OWL is crucial for facilitating interoperability among diverse agents. It allows for the creation of ontologies that define intricate relationships and constraints within the environment, thereby enhancing collaborative behaviors among agents. The expressive power of OWL supports advanced inference capabilities, empowering agents to share knowledge effectively and make informed decisions. This framework establishes a robust baseline for evaluating and enhancing the performance of multi-agent systems in various applications.
- •
Smolagent (Roucher et al., 2025) is a lightweight library designed to facilitate the development and implementation of AI agents that can think and operate using code. It emphasizes simplicity and efficiency, enabling users to create multi-agent systems with minimal code. Smolagent’s architecture allows for smart threading, dependency management, and context sharing, making it ideal for orchestrating complex tasks. By providing a streamlined framework, Smolagent serves as a foundational model for evaluating the performance and capabilities of more advanced agent-based systems in various applications.
- •
OAgents (Zhu et al., 2025) is a modular multi-agent framework that conducts a thorough empirical study of key agent components (planning, memory, tool use, test-time scaling) on benchmarks such as GAIA and BrowseComp. It delivers great performance among open-source agent frameworks. Importantly, OAgents builds on the lightweight agentience model provided by Smolagent (which emphasises code-based agent orchestration and minimal overhead) and extends it with fine-grained task decomposition, dynamic workflow adaptation, multi-source web browsing and more extensive tool and memory modules.
- •
AWorld (Yu et al., 2025) is an open-source framework for large-scale agent–environment interaction, designed to operationalize the “learning from practice” paradigm in agentic AI. It features a hierarchical multi-agent architecture composed of specialized agents such as the *Execution Agent*, which performs primary reasoning and tool-use operations, and the *Guard Agent*, which intervenes at critical steps to verify and refine intermediate outcomes. AWorld adopts a modular design supporting dynamic supervision, context tracking, and distributed orchestration, enabling efficient coordination across diverse tasks and environments. By treating agents and tools as interchangeable components within a unified orchestration layer, it facilitates flexible composition, concurrent execution, and fine-grained control over reasoning workflows, illustrating a scalable and extensible paradigm for constructing adaptive multi-agent systems.

### A.3 Implementation details

In this section, we provide a detailed description of how the conceptual framework of SupervisorAgent is implemented in our codebase. Our implementation is centered around the supervise_and_correct function, which serves as the primary entry point for all supervisory actions. We structure our explanation following the same *What, When, and How* logic presented in our main methodology.

#### A.3.1 What to Supervise: The ActionStep Object

Our supervision targets the discrete interaction steps performed by each agent within the MAS. In our framework, every such interaction is encapsulated in a data structure we refer to as an ActionStep object. This object contains all relevant information for a single step, including the agent’s thought process (model_output), the executed tool_calls, the resulting observations, and an error attribute which is populated if an exception occurs. Our SupervisorAgent is implemented as a callback function that intercepts every ActionStep object generated by any agent in the system.

#### A.3.2 When to Supervise: The Prioritized Adaptive Filter

To avoid the prohibitive cost of constant intervention, we employ a lightweight, LLM-free adaptive filter. This filter is implemented as a prioritized conditional chain at the beginning of the supervise_and_correct function. It evaluates each ActionStep to determine if supervision is warranted. The conditions are checked in the following order of precedence:

- 1.
Sub-Agent Completion: The highest priority is to check if the observation contains a final report from a sub-agent (identified by the presence of a "<summary_of_work>" string). If so, it triggers the specialized Adaptive Observation Purification strategy to distill the findings for the manager agent.
- 2.
Error Occurrence: If the step.error attribute is not None, the Proactive Error Correction strategy is triggered. Our implementation includes a defensive check to ensure this does not fire for known, non-critical tool failures that the base agent can handle.
- 3.
Inefficient Behavior: If no error is present, we then check for inefficiency using our heuristic-based _check_for_inefficiency function. This function detects patterns such as hard loops (identical actions and observations) and excessive step counts for a given sub-task, triggering the Guidance for Inefficiency strategy.
- 4.
Excessive Observation Length: Finally, if none of the above conditions are met, the filter checks if the length of the step.observations string exceeds a pre-defined threshold (3,000 characters in our implementation). If it does, the general type of Adaptive Observation Purification strategy is activated.

If none of these conditions are met, the step is approved by default, and no costly LLM-based supervision is invoked.

#### A.3.3 How to Supervise: The Intervention Pipeline

Once the adaptive filter flags an interaction, the supervise_and_correct function executes a three-stage intervention pipeline:

##### 1. Context Aggregation

Before making a decision, the Supervisor aggregates a context window ($\mathcal{W}$). This process involves retrieving the global task ($G$) and the agent’s local task ($L$), formatting the agent’s recent local action history ($T_{l}$) via the _format_local_trace_for_prompt function, and generating a summary of the current step ($S$) using the _summarize_interaction function. For inefficient behavior, the full global trace ($T_{g}$) is also included.

##### 2. LLM-based Decision Making

The aggregated context is then compiled into a specialized prompt tailored to the triggered supervision type (e.g., Proactive Error Correction). This prompt instructs our main model (e.g., GPT-4.1) to analyze the situation and return a structured JSON object containing its analysis, a chosen action (from the set {approve, correct_observation, provide_guidance, run_verification}), and the necessary parameters to execute that action.

##### 3. Action Execution

The returned JSON is parsed, and the chosen action is executed.

- •
correct_observation: The original step.observations is entirely replaced with the new_observation provided in the parameters. A “[Supervisor’s Note: …]” is prepended to inform the agent of the modification.
- •
provide_guidance: The guidance string from the parameters is appended to the end of the existing step.observations, leaving the original sensory data intact while providing a corrective hint.
- •
run_verification: The task parameter is passed to a dedicated, fully-equipped verification agent, and its conclusive findings are appended to the step.observations.

### A.4 Case Study

### A.5 Prompts

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Do the reported token costs for SMAS in all figures/tables take into account all supervisor token costs?
2. Can the authors describe the overhead introduced by SupervisorAgent, not only in terms of the token costs, but also in terms of wall-clock times, or number of additional LLM calls invoked (especially relevant since once a SupervisorAgent intervention will be invoked, the agent system's execution will need to be halted).
3. Can the authors' comment on the interaction of SupervisorAgent when implemented in other MAS frameworks, which could include significant interaction with external environments (for example, through MCP)?
soundness: 4
strengths: 1. The paper introduces a novel, lightweight, non-intrusive monitoring of MAS. This could potentially become a major MAS development pattern going ahead.
2. The paper is well-to-read, and educational from the perspective of understanding MAS design, issues with current MAS, and nicely introduces the proposed intervention.
3. Evaluation: The authors validate the intervention across 6 benchmarks, and across 3 leading LLMs, including open and proprietary models.
summary: The paper addresses two critical challenges in MASs, 1) Operational Inefficiency (measured through token usage here), 2) Robustness Failures (error propagation, misinformation, etc.). Unlike prior methods that focus on post-hoc error attribution and diagnosis, the paper introduces SupervisorAgent, a meta-agent, that monitors the interactions in an MAS (agent-agent, agent-tool, agent-memory), uses an LLM-free "adaptive-filter" to determine high-risk interactions, and if invoked, the SupervisorAgent intervenes through actions like proactively correcting errors, guiding agents away from inefficient behaviors (like loops), and purifying excessively long or noisy observations to reduce token load.

The authors implement the SupervisorAgent over the SmolAgents framework, and compare against strong baselines across several benchmarks, finding that the intervention reduces token consumption by average as much as 29.45% without losing accuracy. The paper concludes with an ablation study of the various SupervisorAgent interventions, finding that purification leads to efficiency gains, whereas correction and guidance are necessary for performance.
weaknesses: 1. The overhead introduced due to SupervisorAgent is not described in detail. 
2. The details about the working of adaptive filter are not described. Since one of the core features of the SupervisorAgent is "lightweight" monitoring, the authors should clearly describe how the adaptive filter works without LLMs, especially to identify "inefficient behavior" which seems to be a highly subjective criteria unlike the other 2 high-risk interactions identified.
3. Many MAS proposed in the past have included an additional agent that acts as a verifier or quality-assurance agent (for example, ChatDev includes a "Tester" agent). However, one issue with such interventions is that often the verifier can itself fail, introduce misinformation, or propagate errors. The authors do not discuss the failure modes or failure rate of complex SupervisorAgent actions like "correct_observation".

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Do the token savings reported in Table 1 and 2 (e.g., the 29.68% on GAIA)  account for the tokens consumed by the SUPERVISORAGENT itself? If not, what is the net token savings for the entire system (base agents + supervisor)?

2. The ablation study (Table 3) shows that "Purification" is the main source of token savings, while "Correction" and "Guidance" are key for accuracy . This suggests that for tasks where the baseline is already robust (high accuracy), a "Purification-Only" supervisor might offer the best cost-benefit. Have you considered this variant?

3. The "Excessive Observation Length" trigger is a hardcoded "3,000 characters". How sensitive is the framework's performance (both token savings and accuracy) to this hyperparameter?
soundness: 4
strengths: The paper's framing of MAS inefficiency as a runtime process control problem is a fresh and valuable perspective.


The hybrid "LLM-free filter + LLM supervisor"  is a very strong and practical design choice that balances cost and capability.


The 29.45% token reduction on GAIA with no accuracy loss is a headline-worthy result, strongly supported by the data.



The ablation in Table 3 is a highlight, perfectly justifying the three-part design of the supervisor's intervention strategies (Purification for efficiency, Correction/Guidance for robustness) .


The method is shown to be model-agnostic (Figure 4b) and benchmark-agnostic (Table 2), proving it is a general framework and not a one-off trick.
summary: This paper addresses two critical and intertwined problems in modern Multi-Agent Systems (MAS): operational inefficiency (excessive token consumption) and lack of robustness (failures from error propagation). The authors argue that existing methods are either post-hoc (analyzing failures after they occur) or design-time (optimizing the agent architecture itself), with a lack of tools for proactive, real-time intervention .




To fill this gap, the authors propose SUPERVISORAGENT, a lightweight, modular meta-agent framework designed for runtime supervision. The framework's core is a lightweight, LLM-free adaptive filter that monitors agent interactions. This filter triggers interventions only at critical junctures, which it identifies as: 1) explicit error occurrences, 2) inefficient behaviors (like loops), or 3) excessive observation lengths.





When triggered, the SUPERVISORAGENT (which is an LLM) analyzes the system's state using a rich context window and performs one of four actions: approve, provide_guidance, correct_observation, or run_verification .


The authors demonstrate the system's effectiveness by applying it to the Smolagent framework on the GAIA benchmark, achieving a 29.45% reduction in token consumption without any loss in task success rate. The paper further validates this approach's generalizability across five other benchmarks (for math, code, and QA) and multiple state-of-the-art foundation models .
weaknesses: 1. The paper's main claim is "Stop Wasting Your Tokens", and it reports a ~30% token reduction. However, it appears this 30% saving applies only to the base agents (Smolagent). The paper never states the **token cost of the SUPERVISORAGENT itself**. The supervisor is an LLM (e.g., GPT-4.1)  and is called every time the filter is triggered. To make a true claim of efficiency, the paper must report the net token savings (i.e., Baseline_Tokens - (SMAS_Agent_Tokens + Supervisor_Agent_Tokens)). Without this, the primary claim of token reduction is incomplete.

2. The paper focuses exclusively on token efficiency. However, the proposed framework introduces an extra LLM call into the execution loop every time an intervention occurs. This will "stop the world" and add wall-clock latency. It's possible the system saves tokens but becomes significantly slower to run. A comprehensive "efficiency" analysis should include latency, especially when the interventions are designed to cut down on inefficient loops (which also consume time).

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Why was SUPERVISORAGENT not applied to SOTA baselines like AWorld? If integrated, could it further reduce AWorld's token costs (e.g., from 128K to below 100K) while maintaining its 60% accuracy? If incompatible, please explicitly discuss this limitation in the paper.

How does SUPERVISORAGENT's own overhead (e.g., from context aggregation or LLM calls in interventions) factor into total token costs?
soundness: 3
strengths: The paper effectively highlights underexplored pain points in MAS, such as runtime inefficiencies from error propagation and excessive observations, which lead to high token costs (for example, up to 2 million tokens on GAIA tasks). This framing is fresh and relevant, building on recent MAS advancements while addressing a critical paradox: increased autonomy often reduces robustness and economic viability.

SUPERVISORAGENT is a lightweight, non-intrusive framework that integrates seamlessly with existing MAS without architectural changes. Its LLM-free adaptive filter for triggering interventions (e.g., at agent-agent, agent-tool, or agent-memory points) is efficient and proactive, focusing on runtime process control rather than static optimizations. The action space (e.g., approve, provide_guidance, correct_observation, run_verification) is well-formalized, and the memory-augmented context window enables holistic oversight. This design is modular and broadly applicable, as validated across diverse benchmarks and models (e.g., GPT-4.1, Gemini-2.5-pro, Qwen3).
summary: This paper addresses critical challenges in MAS, such as excessive token consumption and error propagation, by introducing SUPERVISORAGENT, a lightweight, non-intrusive meta-agent framework for runtime supervision. The approach is innovative in its use of an LLM-free adaptive filter to trigger targeted interventions, enabling proactive error correction, inefficiency guidance, and observation purification without modifying the base agents' architecture. The experiments are comprehensive, spanning multiple benchmarks and models, and demonstrate efficiency gains on a specific baseline. However, the paper's claims of a "significant Pareto improvement" are undermined by unfavorable comparisons to SOTA baselines presented in its own results, raising concerns about the method's broader applicability and impact.
weaknesses: The paper claims a "significant Pareto improvement" (reduced tokens without compromising success rates), but this holds only against the weak Smolagent baseline (50.91% accuracy, 527.76K tokens reduced to 371.12K). In contrast, the paper's own Table 1 shows SOTA baseline AWorld achieving higher accuracy (60.00%) at much lower cost (128.27K tokens). The supervised system (Smolagent + SMAS) is thus both less accurate and more expensive than AWorld, contradicting the Pareto claim. While the authors justify Smolagent for isolating "agentic interactions" over external tools, this does not excuse avoiding direct SOTA integration or comparison. It makes the method's impact limited.

The LLM-free filter relies on hardcoded heuristics (e.g., detecting repetitive actions or observations exceeding 3,000 characters). These thresholds lack justification, sensitivity analysis, or ablation, raising concerns about brittleness and potential overfitting to Smolagent or GAIA. Without robustness tests (e.g., varying thresholds across datasets), the filter's generalizability is questionable.

Token reductions vary inexplicably between main results (29.68% on full GAIA) and ablation on 30 high-token tasks (50.13%). This suggests benefits are skewed toward long-tail cases, potentially masking uneven performance. The abstract and conclusions claim "average" savings (e.g., 29.45%) without clarifying this distribution, which could mislead readers and requires more transparent analysis.

---

confidence: 5
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. What fraction of ActionSteps trigger each condition on GAIA (by level) and on each external benchmark? Please report a table of invocations per action and supervisor token cost so readers can reason about net savings.

2. Why was the ablation study limited to the 30 most token-intensive tasks from the GAIA validation set? Do the individual contributions of the Correction, Guidance, and Purification modules remain consistent when evaluated across the entire dataset, including less token-intensive tasks?

3. The current filter is rule-based. Have you considered a learning-based approach for the supervisor? For example, could a policy be trained to decide when and how to intervene, potentially leading to a more adaptive and less heuristic-driven system?

4. Have you tested a tool-rich agent configuration (e.g., retrieval-augmented browsing with multi-source aggregation)? If the majority of cost/latency comes from tools rather than LLM tokens, does the supervisor still help?
soundness: 3
strengths: 1. The paper addresses a critical and highly relevant challenge in the field of agentic AI. The prohibitive token cost and lack of operational robustness are major barriers to the real-world deployment of complex Multi-Agent Systems. The work's focus on runtime efficiency is a valuable contribution.

2. The empirical validation is extensive and rigorous. The authors test their method on six different benchmarks spanning multiple domains, which strongly supports the claim of general applicability.

3. The paper highlights reduced variance and a more compact per-problem token distribution after supervision—an under-discussed but practically important property for cost predictability. 

4. The prioritized LLM-free filter (sub-agent summary → error → inefficiency → length) is a neat engineering idea to avoid constant, costly oversight while still catching the most harmful failure modes. The concrete heuristics (e.g., length threshold ≈3,000 chars; loop/inefficiency checks) are spelled out sufficiently to reproduce.
summary: The paper introduces SUPERVISORAGENT, a lightweight, runtime “supervisor” that sits on top of a multi-agent system (MAS). A non-LLM prioritized adaptive filter detects four trigger conditions (sub-agent completion, error occurrence, inefficient behavior, excessive observation length) and, when activated, a three-stage intervention pipeline decides among four actions: approve, provide_guidance, correct_observation, run_verification. The goal is to reduce token usage and improve robustness without modifying underlying agents. Experiments on GAIA (Smolagent) and several code/math/QA benchmarks report sizable token savings with roughly maintained—or sometimes improved—accuracy, and an ablation suggests observation “purification” is the main driver of efficiency, while guidance/correction help accuracy.
weaknesses: 1. The core idea—a meta-controller that detects errors/loops/long observations and either prunes context or issues guidance—resembles prior runtime oversight, routing, or budgeted-reasoning controllers. The paper would benefit from a sharper contrast to existing runtime monitors/filters (as opposed to offline compression or architectural agent redesign). As written, the conceptual delta may feel incremental.

2. The authors justify Smolagent because it relies less on “powerful external tools,” isolating the MAS. But in many real MAS workloads, tool calls dominate; it is unclear whether the same gains hold with tool-heavy agents, where tool output is the main token pressure and mis-tooling is the dominant failure mode. The paper acknowledges tokens meticulously but not end-to-end cost/latency including tool latencies and external compute.

3. The authors justify using Smolagent as the base framework because its capabilities stem from "internal agentic interactions rather than powerful external tools". While this provides a controlled environment, it also means the baseline might be inherently less efficient and robust. Applying SUPERVISORAGENT to a weaker baseline could inflate its perceived benefits.

4. The prioritized conditions (e.g., 3,000-char cutoff; loop detectors) appear hand-tuned. How sensitive are results to these thresholds across tasks and models? The paper claims model-agnosticism but does not deeply analyze failure cases where the supervisor fires too often/too rarely on different backbones.


Decision: Accept (Poster)