Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL

## Abstract

Abstract Recent advances in large language models (LLMs) and multi-agent systems have demonstrated remarkable capabilities in complex problem-solving tasks such as deep research, vibe coding, and mathematical reasoning. However, most existing multi-agent systems are built upon manual prompt/workflow engineering with sophisticated agent frameworks, making them computationally inefficient, less capable, and can not benefit from data-centric learning. In this work, we introduce Chain-of-Agents (CoA), a novel paradigm of LLM reasoning that enables native end-to-end complex problem-solving in the same way as a multi-agent system (i.e., multi-turn problem solving with multiple tools and multiple agents) within one model. In chain-of-agents problem-solving, the model dynamically activates different tool agents and role-playing agents to simulate multi-agent collaboration in an end-to-end fashion. To elicit end-to-end chain-of-agents problem-solving abilities in LLMs, we introduce a multi-agent distillation framework to distill state-of-the-art multi-agent systems into chain-of-agents trajectories for agentic supervised fine-tuning. We then use agentic reinforcement learning on verifiable agentic tasks to further improve the models’ capabilities on chain-of-agents problem solving. We call the resulting models Agent Foundation Models (AFMs).
Our empirical studies demonstrate that AFM establishes new state-of-the-art performance across diverse benchmarks in both web agent and code agent settings. We make the entire research, including the model weights, code for training and evaluation, and the training data, fully open-sourced, which offers a solid starting point for future research on agent models and agentic RL.

## 1 Introduction

Recent advances in multi-agent systems (MAS) [6, 2, 80, 81, 82, 83, 44, 46, 53] have demonstrated remarkable capabilities in complex problem-solving tasks such as deep research and vibe coding. These multi-agent frameworks enable complex problem-solving via collaboration between multiple agents with diverse roles and tool sets.
Despite their impressive performance, current multi-agent systems suffer from several crucial limitations: (1) high computational overhead due to redundant communication between agents and sophiscated workflow design, (2) challenges in generalizing to new domains and tasks without substantial reconfiguration, i.e., prompt engineering and workflow engineering [74, 72], (3) inability to perform data-centric learning so that the performance of multi-agent systems can improve by training on agentic tasks, and (4) the backbone large language models(LLMs) used in multi-agent systems are generally not trained to support multi-turn, multi-agent, and multi-tool workflows and are prompt engineered to do so.

Tool-Integrated Reasoning (TIR) models, a recent line of work, explicitly incorporates tool usage into the reasoning process [21, 28, 29, 65, 52, 73, 79, 66]. Specifically, recent work such as Search-R1 [21] and WebThinker [29] improved end-to-end information seeking with large language models (LLMs) by training the models to call <search> at appropriate reasoning steps. The TIR framework makes LLMs support the “think-action-observation” pipeline, which corresponds to the ReAct [68] framework, in an end-to-end fashion. Recent empirical studies [65, 27, 54] demonstrated that TIR training significantly improves the performance of ReAct agents compared to those built with general LLMs via prompt engineering. On the other hand, recent empirical studies on multi-agent frameworks [26, 46, 82, 15] clearly show the advantage of multi-agent systems on complex problem-solving tasks by supporting more diverse tool sets and collaboration between multiple role-playing agents, demonstrated by significant performance improvements across various tasks and benchmarks. However, the current TIR paradigm cannot train LLMs to support multi-agent systems in an end-to-end fashion.

To bridge this gap, we introduce Chain-of-Agents (CoA), a novel paradigm of LLM reasoning that enables native end-to-end complex problem-solving in the same way as a multi-agent system. In contrast to conventional TIR methods that only support a ReAct-like trajectory (i.e., the “think-action-observation” pattern), the CoA paradigm supports almost any multi-agent system by flexibly defining multiple agents corresponding to different tools and roles (defined in the system prompt for CoA) and dynamically activating them to simulate multi-agent collaboration in an end-to-end fashion within a single model. Compared to conventional MAS, CoA eliminates the need for sophisticated prompt engineering and workflow engineering, reducing the computational overhead for inter-agent communication, and supports end-to-end training. These features make the CoA paradigm more efficient and (potentially) more capable compared to conventional multi-agent systems. We refer to our models that support native Chain-of-Agents problem-solving as “Agent Foundation Models” (AFMs).

To elicit end-to-end Chain-of-Agents problem-solving abilities in LLMs, we introduce a Chain-of-Agents tuning framework. Our framework starts with agentic task generation and filtering following the procedure describe in Shi et al. [49]. Then we propose a novel multi-agent distillation framework to distil the capabilities of state-of-the-art multi-agent frameworks such as OAgents [82] into LLMs. Specifically, we use a multi-agent system to solve these agentic tasks and convert the successful trajectories into CoA-compatible ones. We then fine-tune the LLM with the generated CoA trajectories to distill the designs and capabilities of any state-of-the-art multi-agent frameworks and enable end-to-end CoA complex problem solving. We then use agentic reinforcement learning on verifiable agentic tasks to further improve the models’ capabilities for Chain-of-Agents problem solving.

To demonstrate the effectiveness of the Chain-of-Agents paradigm and the Chain-of-Agents tuning framework, we conduct empirical studies on various agentic tasks and benchmarks, including both web agent and code/mathematical reasoning agents. Our experimental results demonstrate that AFM establishes new state-of-the-art performance across nearly 20 diverse agent benchmarks. Specifically, with a 32B model size, AFM achieves new state-of-the-art Pass@1 success rates on various challenging web agent benchmarks: 55.3% on GAIA [37], 11.1% on BrowseComp [63], and 18.0% HLE [41]. With code interpreter as the main tool, AFMs achieve 47.9% on LiveCodeBench v5 [20] and 32.7% on CodeContests [31], significantly outperforming existing TIR methods. In mathematical reasoning, our model achieves a 59.8% solve rate on the challenging AIME2025 benchmark, leading to an absolute improvement of over 10.5% compared to previous best-performing TIR methods, including ReTool [7] and SimpleTIR [66]. Furthermore, our analysis reveals that AFM reduces the inference cost (in terms of token consumption) by 84.6% compared to traditional multi-agent systems while achieving competitive performance.

In summary, our key contributions include:

- •
We introduce Chain-of-Agents, a novel paradigm for LLM-based problem-solving that integrates multi-agent collaboration capabilities within a single model.
- •
We propose multi-agent distillation, a novel framework to distill the capabilities of state-of-the-art multi-agent systems into end-to-end agent models, and an agentic RL framework to optimize agent models with RL.
- •
We train AFMs with the proposed methods and show that AFMs establish new state-of-the-art across multi-hop question answering, web search, code generation, and mathematical reasoning tasks, demonstrating superior performance compared to TIR approaches.
- •
We make the entire research, including the model weights, code for training and evaluation, and the training data, fully open-sourced, which offers a solid starting point for future research on agent models and agentic RL.

## 2 Background

Complex task reasoning often requires structured decomposition, specialized capabilities, and external tool integration. We review three prominent paradigms that motivate our framework:

- •
ReAct: This framework augments LLMs with structured reasoning by interleaving thought steps $\tau_{t}\in\mathcal{T}$ for planning, action steps $a_{t}\in\mathcal{A}$ for tool use, and observation steps $o_{t}\in\mathcal{O}$ for outcome processing. The reasoning trajectory follows:
$(\tau_{1},a_{1},o_{1},\tau_{2},a_{2},o_{2},...,\tau_{T})$
(1)
where each thought $\tau_{t}$ conditions on the history $h_{t}=[\tau_{1:t-1},a_{1:t-1},o_{1:t-1}]$ to determine the next action.
- •
Multi-Agent Systems: A Multi-Agent System comprises specialized agents $\mathcal{A}=\{a_{1},a_{2},...,a_{N}\}$, where each agent $a_{i}$ maintains an internal state $s_{i}^{t}\in\mathcal{S}_{i}$ and executes a policy $\pi_{a_{i}}:\mathcal{S}_{i}\rightarrow\Delta(\mathcal{A}_{i})$ over its action space $\mathcal{A}_{i}$. Agents communicate via messages $m_{i\rightarrow j}^{t}\in\mathcal{M}$ to share states and outputs, with state transitions governed by:
$s_{j}^{t}=f_{j}(s_{j}^{t-1},\{m_{i\rightarrow j}^{t-1}\}_{a_{i}\in\mathcal{A}})$
(2)
Here, $s_{j}^{t}$ represents agent $j$’s state at time $t$, updated based on previous state $s_{j}^{t-1}$ and incoming messages $m_{i\rightarrow j}^{t-1}$ from other agents.
- •
Tool-Integrated Reasoning: TIR enables a single agent to leverage external tools $\mathcal{T}=\{t_{1},t_{2},...,t_{M}\}$ by maintaining a global state $S_{t}$ and selecting tools via policy $\pi(t_{k}\mid S_{t})$. After executing tool $t_{k}$, the agent observes outcome $o_{t}\sim\mathcal{O}(t_{k},S_{t})$ and updates its state:
$S_{t}=f(S_{t-1},t_{k},o_{t-1})$
(3)
where $S_{t}$ denotes the reasoning state, $t_{k}$ represents the selected tool, and $o_{t}$ captures tool execution outcomes.

## 3 Method

### 3.1 Chain-of-Agents Paradigm

Figure: Figure 2: Illustration of TIR and CoA paradigms. TIR uses a static “Think-Action-Observation” workflow whereas CoA supports any workflow that can be modeled by a multi-agent system, supporting more diverse role-playing agents and tool agents.
Refer to caption: /html/2508.13167/assets/x2.png

We propose the CoA framework to tackle complex queries via dynamic module orchestration within a unified model $\mathcal{M}_{\theta}$. Given a query $\mathbf{q}$ such as "Summarize recent AI breakthroughs", CoA consists of two core components:

Role-playing Agents: High-level reasoning and coordination agents:

- •
Thinking Agent: Orchestrates the reasoning pipeline by activating specialized agents and maintaining solution state coherence
- •
Plan Agent: Decomposes $\mathbf{q}$ into structured task sequences $\langle\phi_{\text{search}},\phi_{\text{crawl}},\ldots\rangle$
- •
Reflection Agent: Conducts self-critique through knowledge fusion and inconsistency resolution
- •
Verification Agent: Validates reasoning integrity against formal correctness criteria

Tool Agents: Domain-specific execution agents including:

- •
Search Agent: Formulates optimized queries (e.g., "2024 AI breakthroughs") with source prioritization
- •
Crawl Agent: Performs parallel content extraction and technical detail parsing
- •
Code Generate Agent: Generates and executes code snippets within sandbox environments

As illustrated in [Figure˜2](#S3.F2), unlike Tool-Integrated Reasoning, the CoA paradigm orchestrates multi-agent collaboration within a single decoding (inference) process: the Thinking Agent dynamically coordinates this ecosystem through state transitions:

$$ $\mathcal{S}_{t}=f_{\theta}(\mathcal{S}_{t-1},\phi_{t-1},\mathbf{o}_{t-1}),\quad\phi_{t}\sim P(\phi\mid\mathcal{S}_{t})$ (4) $$

where $\mathcal{S}_{t}$ maintains persistent reasoning state, and $\phi_{t}\in\{\phi_{\text{think}},\phi_{\text{plan}},\phi_{\text{search}},\ldots\}$ denotes activated roles.

**Table 1: Comparative analysis of agent paradigms.**
| Paradigm | Tool Integration | End-to-end Execution | Multi-agent Collaboration | Data-centric Optimization |
| --- | --- | --- | --- | --- |
| ReAct | ✓ | ✗ | ✗ | ✗ |
| Multi-Agent System | ✓ | ✗ | ✓ | ✗ |
| Tool-Integrated Reasoning | ✓ | ✓ | ✗ | ✓ |
| Chain-of-Agents | ✓ | ✓ | ✓ | ✓ |

compared to contentional TIR’s rigid pipelines, CoA achieves adaptive, dynamic multi-agent collaborative problem-solving via dynamic agent orchestration within a unified model, enabling efficient and coherent problem-solving. This corresponds to the advantages of multi-agent systems upon ReAcT agents, which is demonstrated in various previous empirical studies [82].

On the other hand, when compared with popular multi-agent systems built with agent frameworks and workflow/prompt engineering, our Chain-of-Agents paradigm maintains contextual continuity within multi-agent orchestration and collaboration. It is also more computationally efficient because much token consumption for intra-agents communication in traditional multi-agent systems is alleviated. By modeling multi-agent collaboration within a single model, CoA can be directly optimized by training the model with both supervised training and reinforcement learning, whereas the LLM backbones in agent frameworks are static and cannot be directly optimized for agentic use in most cases.
[Table˜1](#S3.T1) presents a comparison between AFM and other agentic problem-solving paradigms.

### 3.2 Agentic Supervised Fine-tuning

Figure: Figure 3: Illustration of the proposed multi-agent distillation framework, which synthesizes Chain-of-Agents trajectories with state-of-the-art multi-agent systems such as OAgents [82]
Refer to caption: /html/2508.13167/assets/x3.png

#### 3.2.1 Training Data Generation

##### Multi-Agent Knowledge Distillation.

Our approach leverages agent-level knowledge distillation to transfer capabilities from state-of-the-art multi-agent systems into chain-of-agents trajectories. This method extends sequence-level knowledge distillation principles [24] to the multi-agent domain, where we distill the sequential decision-making patterns of expert multi-agent systems rather than word-level distributions.

In sequence-level knowledge distillation, a teacher model’s entire sequence distribution is transferred to a student model. Similarly, our agent-level distillation captures the complete execution trajectory of a multi-agent system, preserving the sequential reasoning patterns and agent activation sequences that lead to successful task completion.

Given a multi-agent system, we extract chain-of-agents trajectories by monitoring its execution process. Each agent interaction is recorded as a step in the trajectory, formalized as:

$$ $\tau=\{(\mathcal{S}_{t},\phi_{t},\mathbf{o}_{t})\}_{t=1}^{T}$ (5) $$

where $\mathcal{S}_{t}\in\mathcal{S}$ represents the reasoning state, $\phi_{t}\sim P(\phi|\mathcal{S}_{t})$ denotes the activated agent, and $\mathbf{o}_{t}$ captures the agent’s observation.

In this work, we use OAgents [82], the state-of-the-art open-source multiagent system to extract trajectories by recording each agent’s activation, reasoning state, and output in sequence. When OAgents executes a task, we monitor the agent selection process, capture the reasoning state before each agent acts, and record the agent’s output. This transforms OAgents’ mutli-agent collaboration procedure into a CoA-like trajectory suitable for agentic supervised fine-tuning.

The trajectory construction follows an iterative refinement cycle:

$$ $\mathcal{S}_{t}=\Gamma(\mathcal{S}_{t-1},\mathbf{o}_{t-1}),\quad a_{t}\sim\pi(\cdot|\mathcal{S}_{t}),\quad\mathbf{o}_{t}=\Phi(a_{t})$ (6) $$

where $\Gamma$ denotes the state transition function, $\pi$ the action policy, and $\Phi$ the agent execution environment. This formulation captures the distillation of multi-agent system interactions into a structured sequence of agents. The resulting trajectories, consisting of these agent sequences, are constructed into CoA distillation datasets specifically designed for AFM training (see [Appendix˜C](#A3) for detailed examples of CoA distillation trajectories).

##### Progressive Quality Filtering.

Given the variability in trajectory quality across different data sources, we implement a progressive progressive filtering mechanism to ensure that only high-quality, non-trivial samples are used for SFT. We formulate four-stage filtering processes:

(1) Complexity filtering: Trajectories with $\textless 5$ total agent-tool interactions are excluded to eliminate overly simplistic tasks.

(2) Quality filtering: "Dirty data" is removed, including instances with incorrect answers, redundant tool inputs, or failure to strictly follow instructions (validated via prompting, even for otherwise correct answers).
The correctness of QA and search tasks is evaluated using large language models, as documented in [76]. In contrast, the validity of code-related tasks is determined by whether the generated code successfully passes all test cases. For mathematical reasoning tasks, correctness is assessed through a direct comparison between the generated answers and the predefined golden answers.

(3) Reflection enrichment: Trajectories lacking reflection mechanisms (e.g., self-reflection, self-refinement) are downsampled to prioritize instances modeling self-critical reasoning. Note that for math or code tasks, we drop trajectories without reflection mechanisms.

(4) Error-correction trajectory upsampling: For search or QA tasks, trajectories where the <double_check> agent initially yields low credibility scores (assessed via GRM [33] credibility metrics) but ultimately achieves correct answers through iterative re-reasoning are upsampled. This prioritizes samples that demonstrate the ability to identify and rectify initial errors, enhancing the dataset’s focus on robust error-correction capabilities.

The resulting corpus is distinguished by three key traits:

(1) All trajectories necessitate multi-tool collaborative coordination, embodying complex functional interdependencies that demand advanced planning and execution capabilities;

(2) Reasoning chains span 5–20 hops, significantly surpassing the 2–3 hop range typical of standard benchmarks [14, 58, 42];

(3) It is enriched with high-quality reflective trajectories—particularly those featuring iterative error correction.

Based on the aforementioned filtering criteria, we formulate the SFT training trajectories into the following format:

<think> $\mathcal{C}_{\text{cot}}$ </think><tools> $\alpha_{m}(\alpha_{p})$ </tools><observation> $\mathcal{O}_{t}$ </observation><reflection> $\mathcal{F}_{t}$ </reflection>...<answer> $\mathcal{A}_{t}$ </answer>

where $\mathcal{C}_{\text{cot}}$ denotes chain-of-thought rationales, $\alpha_{m}$ the tool action, $\mathcal{F}_{t}$ denotes the reflection or reasoning over the observation for subsequent decision-making, and $\mathcal{O}_{t}$ tool observations. The training objective minimizes:

$$ $\mathcal{L}_{\text{SFT}}=-\tsum\slimits@_{t\notin\mathcal{O}}\log\pi_{\theta}(\tau_{t}|\tau_{<t},\mathbf{q})$ (7) $$

with observation masking ($\mathcal{O}$) to prevent environmental noise propagation. This establishes robust cold start for downstream RL. The overall training framework is illustrated in [Figure˜4](#S3.F4).

Figure: Figure 4: Overview of the training framework. (I) The SFT stage utilizes reformatted ReAct data with both short and long chains of thought for cold start. (II) The RL stage performs tool-aware rollouts on unused QA pairs and optimizes the policy.
Refer to caption: /html/2508.13167/assets/x4.png

### 3.3 Agentic Reinforcement Learning

#### 3.3.1 Data Sampling for RL Training

Given the heterogeneous quality distribution across our integrated diverse data sources, we implement a multi-stage filtering protocol to ensure query quality. This curation strategy addresses data variance through quality filter and strategic sampling for web agent and only quality filter for code agent.

##### Quality filter for Web Agent.

We employ Qwen-2.5-72B-Instruct [45] to evaluate question solvability without tool assistance. For each query $\mathbf{q}$ in the QA dataset:

$$ $r_{q}=\frac{1}{N}\tsum\slimits@_{i=1}^{N}\mathbb{I}\left[\text{EM}(a_{i},y_{\text{gt}})=1\right]$ (8) $$

where $N\!=\!32$ is the number of model predictions, $a_{i}$ denotes the $i$-th prediction, $y_{\text{gt}}$ represents the ground truth, and $\text{EM}(\cdot)$ computes the exact match score between two inputs. This pass rate $r_{q}$ quantifies parametric knowledge contamination risk.
Queries with $r_{q}>0.3$ are excluded as they either represent: 1) Trivially solvable cases requiring no tool usage, or 2) Highly contaminated samples vulnerable to parametric recall. This threshold ensures genuine tool engagement.

##### Strategic sampling.

We adopt a randomly selecting strategy to sample queries from the remaining challenging ones (with $r_{q}\leq 0.3$), which are ultimately used for RL training:

$$ $\mathcal{Q}_{\text{RL}}=\{q_{j}\mid r_{q_{j}}\leq 0.3\}_{j=1}$ (9) $$

The sampled subset, which is excluded from the SFT dataset, forms the final RL dataset. This composition focuses on queries where tool-based reasoning offers substantial value. By design, the strategic sampling ensures that the RL training emphasizes challenging cases in which effective tool coordination is critical, while reducing the influence of trivial or potentially unuseful samples.

##### Quality filter for Code Agent.

For the Code Agent, we likewise perform quality filtering to eliminate overly simplistic queries and thereby accelerate training efficiency. Concretely, a fine-tuned 7B AFM model is used to sample each query 8 times. Any query that is solved correctly in all 8 trials is deemed insufficiently challenging and consequently discarded.

#### 3.3.2 Reward Function Design

The reinforcement learning stage refines the agent’s policy for multi-tool orchestration using outcome-driven rewards. Building upon SFT initialization, we optimize for long-term task success through environment feedback, enhancing strategic tool usage and adaptive reasoning in dynamic interactions.

##### Web Agent Reward Function.

Reward signals are critical for shaping RL dynamics in open-ended web agent tasks. Our framework adopts a streamlined design, built on two key considerations:
Format consistency is inherently ensured through high-quality supervised fine-tuning and effective cold-start, obviating the need for explicit format validation rewards (e.g., prior $score_{format}$). For evaluating answer correctness, traditional rule-based metrics (F1, EM) fail to capture the nuance of diverse valid outputs in open-ended tasks. Instead, we use LLM-as-Judge[76], where judge model $M_{j}$ provides binary assessments. Our reward function is:

$$ $\mathcal{R_{\text{web}}}(\tau)={score}_{\text{answer}}$ (10) $$

where ${score}_{\text{answer}}\in\{0,1\}$ is 1 if $M_{j}$ judges the final prediction correct.
This design prioritizes core correctness, avoids instability from fragmented rewards, mitigates reward hacking via binary signals, and enables flexible evaluation of diverse outputs through LLM judgment.

##### Code Agent Reward Function.

For programming and mathematical reasoning, we employ a reward function that reflects both answer correctness and the format correctness.
The reward is defined as:

$$ $\mathcal{R_{\text{code}}}(\tau)={score}_{\text{answer}}\cdot{score}_{\text{format}}$ (11) $$

where ${score}_{\text{answer}}\in\{0,1\}$ reflects answer correctness. For code generation tasks, the solutions are executed in a secure sandbox and must pass all test cases. For mathematical tasks, the answers are evaluated with Math-Verify(^1^11[https://github.com/huggingface/Math-Verify](https://github.com/huggingface/Math-Verify)).
And ${score}_{\text{format}}\in\{0,1\}$ denotes whether each call of Code agent is in the format of <code>\n‘‘‘py\n...‘‘‘\n</code>
Only outputs that both comply with format expectations and pass semantic verification receive full reward (${score}=1$).

## 4 Experiments

### 4.1 Web Agent Experiments

#### 4.1.1 Experimental Setup

##### Training Dataset.

For the sake of comparing different baselines, we train and evaluate different models by constructing two types of datasets, which take into account variations in task types and difficulty levels.

(1) MHQA Dataset Construction: In the SFT stage of the MHQA task, we sample a set of question-answer pairs from the NQ [25] and HotpotQA [67] datasets. From [Section˜3.2.1](#S3.SS2.SSS1), we generate about 8.8k training data by applying the trajectory synthesis and quality-filtering pipeline. For the RL stage, we adopt the same dataset setting as Search-R1 [21], using the full set of NQ [25] and HotpotQA [67] datasets.

(2) Web Agent Dataset Construction: We construct a comprehensive Web Agent dataset through systematic integration of synthetic and filtered real-world sources. The dataset draws from two primary sources, both refined through rigorous filtering processes to ensure complexity and quality:

- •
Generated Agentic Datasets.The first part of agentic search data used for training AFM for web agent tasks is generated with an autonomous agentic task generation pipeline as described in Shi et al. [49]. Specifically, starting with unlabeled corpora (e.g., PDF documents, HTML pages) aligned with tool input requirements, we first generate atomic tasks: for each input text segment, we extract an initial task and derive corresponding textual content via tool execution. To enhance task complexity, we apply two extension strategies: Depth-based extension: Constructing multi-step tasks requiring sequential tool executions, where each step’s output directly informs the next; Width-based extension: Generating tasks that must be decomposed into parallel subtasks, each requiring independent tool usage to solve the original problem collectively.
- •
Filtered Real-World QA Datasets. The second source consists of filtered single-hop and multi-hop QA data from established benchmarks, including NQ [25], TQ [23], and HotpotQA [67], among others. These datasets are processed to align with the demands of complex web agent scenarios.

**Table 2: MHQA Dataset Composition.**
| Source | NQ | HotpotQA | Total |
| --- | --- | --- | --- |
| SFT Phase |  |  |  |
| Filtered Size | 1717 | 7109 | 8826 |
| Avg. Hops | 3.43 | 4.57 | 4.35 |
| RL Phase |  |  |  |
| Filtered Size | 79168 | 90447 | 169615 |

We have curated a total of 16433 high-quality trajectories for supervised fine-tuning (SFT), comprising 8826 trajectories from the MHQA Dataset and 7607 trajectories from the Web Agent Dataset. A key characteristic of these trajectories is their extended reasoning chains, spanning 5–20 hops (with each agent interaction counted as one hop)—a significant advancement over the shorter 2–3 hop ranges typical in prior benchmarks.

Additionally, the reinforcement learning (RL) data for our experiments is sourced from open channels: the MHQA Dataset uses 169615 instances, following the setup in [21], while the Web Agent Dataset incorporates 10427 instances.

The following tables present detailed statistics on the dataset composition, trajectory characteristics, and quality metrics. These detailed statistics demonstrate our dataset’s comprehensive coverage of complex tool-use behaviors, enabling effective distillation of sophisticated reasoning patterns into foundation models.

##### Benchmarks.

We evaluate our approach on both single-hop QA and multi-hop QA datasets, following prior works [79, 73]. The single-hop evaluation comprises 22328 examples from NQ, TQ and HotpotQA, while the multi-hop evaluation uses 29385 examples from TriviaQA, 2Wiki, MuSiQue [58], Bamboogle [42], and PopQA [36], which exhibit diverse question formulations and information distributions. To assess performance on complex information retrieval tasks, we further evaluate on three specialized benchmarks: GAIA [37] (103 text-only examples for fair comparison with [29, 65]), BrowseComp [63], and HLE [41]. These benchmarks collectively enable systematic assessment across diverse task typologies and complexity levels.

**Table 4: General retrieval dataset specifications.**
| Dataset | Category | Domain Focus | Size |
| --- | --- | --- | --- |
| NQ [25] | Single-Hop QA | Open-domain | 3610 |
| TQ [23] | History/Culture | 11313 |  |
| HotpotQA [67] | Multi-domain | 7405 |  |
| PopQA [36] | Multi-Hop QA | Popular culture | 14267 |
| 2Wiki [14] | Wikipedia-based | 12576 |  |
| Musique [58] | Compositional QA | 2417 |  |
| Bamboogle [42] | Counterfactual | 125 |  |

- •
Single-Hop QA: The single-hop benchmark consists of 11,015 examples: 3,610 from NQ, 11313 from TQ and 7405 from HotpotQA.
- •
Multi-Hop QA: The out-of-domain set comprises 29385 examples: 14267 from PopQA, 12576 from 2Wiki, 2417 from Musique, and 125 from Bamboogle.

**Table 5: Complex task dataset specifications.**
| Dataset | Task Focus | Evaluation Set |
| --- | --- | --- |
| GAIA [37] | Multi-step reasoning & tool orchestration | 103 |
| BrowseComp [63] | Advanced web navigation & information extraction | 1,266 |
| HLE [41] | Frontier academic problem-solving | 500 |

- •
GAIA [37] is a benchmark for General AI Assistants that evaluates multi-step reasoning and tool-use proficiency through real-world questions. While conceptually simple for humans (92% solve rate), these questions are challenging for AI systems. We use its text-only subset (103 validation samples) to ensure fair comparison with prior work [29, 65], requiring fundamental abilities including web browsing and tool orchestration.
- •
BrowseComp [63] assesses advanced web navigation capabilities through deliberately obscure yet verifiable questions. It requires persistent, creative search strategies to locate hard-to-find information that cannot be discovered via simple queries or brute-force methods, with verification through short, factual answers. We evaluate on the full benchmark (1,266 examples).
- •
HLE [41] is a frontier academic benchmark at the limits of human knowledge, featuring 2,500 multi-modal questions across mathematics, humanities, and natural sciences. These questions require expert-level reasoning and cannot be resolved through simple internet retrieval. For methodological consistency, we evaluate exclusively on its text-only subset (500 samples), which exposes significant capability gaps in state-of-the-art systems.

**Table 6: Main results on 7 Multi-hop Question Answering (MHQA) benchmarks, with Qwen-2.5 family models serving as the default backbone unless otherwise noted.**
| Method | Backbone | Single-Hop QA | Multi-Hop QA | Avg. |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NQ | TriviaQA | PopQA | HotpotQA | 2Wiki | MuSiQue | Bamboogle |  |  |  |
| Model Inference |  |  |  |  |  |  |  |  |  |
| Qwen2.5-3B-Instruct | - | 10.6 | 14.9 | 28.8 | 10.8 | 24.4 | 2 | 2.4 | 13.4 |
| Qwen2.5-7B-Instruct | - | 11.6 | 35.6 | 1.2 | 16.4 | 22.2 | 4.8 | 14.4 | 15.2 |
| Tool-integrated Methods |  |  |  |  |  |  |  |  |  |
| Search-R1 | Qwen2.5-3B-base | 40.6 | 58.7 | 43.5 | 28.4 | 27.3 | 4.9 | 8.8 | 30.3 |
| ZeroSearch | 43.0 | 61.6 | 41.4 | 33.8 | 34.6 | 13.0 | 13.9 | 34.5 |  |
| StepSearch | - | - | - | 32.9 | 33.9 | 18.1 | 32.8 | - |  |
| AFM-SFT | 37.5 | 57.6 | 40.4 | 42.4 | 41.0 | 18.7 | 40.0 | 39.7 |  |
| AFM-RL | 39.3 | 58.2 | 42.4 | 41.1 | 43.4 | 19.0 | 45.6 | 41.3 |  |
| Search-R1 | Qwen2.5-3B-instruct | 34.1 | 54.5 | 37.8 | 32.4 | 31.9 | 10.3 | 26.4 | 32.5 |
| ZeroSearch | 41.4 | 57.4 | 44.8 | 27.4 | 30.0 | 9.8 | 11.1 | 31.7 |  |
| O^2-Searcher | 44.4 | 59.7 | 38.8 | 42.9 | 37.4 | 16.0 | 34.4 | 39.1 |  |
| StepSearch | - | - | - | 34.5 | 32.0 | 17.4 | 34.4 | - |  |
| AFM-SFT | 36.0 | 56.4 | 39.7 | 42.0 | 41.1 | 19.0 | 44.8 | 39.9 |  |
| AFM-RL | 41.9 | 57.7 | 38.0 | 41.9 | 43.9 | 18.9 | 43.2 | 40.8 |  |
| Search-R1 | Qwen2.5-7B-base | 48.0 | 63.8 | 45.7 | 43.3 | 38.2 | 19.6 | 43.2 | 43.1 |
| ZeroSearch | 42.4 | 63.5 | 51.7 | 32.0 | 34.0 | 18.0 | 33.3 | 41.0 |  |
| ReSearch | - | - | - | 40.6 | 44.7 | 21.7 | 43.2 | - |  |
| StepSearch | - | - | - | 38.0 | 38.5 | 21.6 | 46.7 | - |  |
| AFM-SFT | 38.8 | 59.7 | 39.5 | 45.0 | 47.9 | 21.5 | 48.8 | 43.0 |  |
| AFM-RL | 45.7 | 64.3 | 45.9 | 45.6 | 45.9 | 20.2 | 48.8 | 45.2 |  |
| Search-o1 | Qwen2.5-7B-instruct | 19.4 | 40.6 | 11.4 | 17.0 | 27.0 | 8.6 | 30.4 | 22.1 |
| Search-R1 | 39.3 | 61.0 | 39.7 | 37.0 | 41.4 | 14.6 | 36.8 | 38.5 |  |
| ZeroSearch | 43.6 | 61.8 | 51.5 | 34.6 | 35.2 | 18.4 | 27.8 | 39.1 |  |
| ReSearch | - | - | - | 43.5 | 47.6 | 22.3 | 42.4 | - |  |
| StepSearch | - | - | - | 38.6 | 36.6 | 22.6 | 40.0 | - |  |
| ReasonRAG | - | - | 41.5 | 38.4 | 43.6 | 12.8 | 36.0 | - |  |
| AFM-SFT | 39.8 | 59.6 | 39.3 | 38.8 | 50.7 | 19.5 | 44.4 | 41.7 |  |
| AFM-RL | 43.9 | 63.3 | 46.5 | 43.9 | 49.2 | 22.3 | 49.6 | 45.5 |  |

##### Metrics.

Model performance is evaluated using the LLM-as-Judge method, with Qwen-2.5-72B serving as the judge [76, 52, 65]. The judge provides binary correctness assessments for each prediction, yielding accuracy scores per dataset. The standardized judging prompt is detailed in [Section˜D.4](#A4.SS4).

##### Implementation Details.

Our experimental framework is implemented using the Qwen-2.5 model family as the backbone architecture. Specifically, we evaluate the Qwen2.5-3B-Instruct, 7B-Instruct, and 32B-Instruct variants [45] to analyze performance across different model scales. All models are configured with a maximum sequence length of 32768 tokens to support complex reasoning chains and the integration of lengthy retrieved content. During inference, we set the generation temperature to 1.0, the top-p sampling threshold to 0.9, and the top-k sampling parameter to 20.

For SFT, we use a batch size of 256 for 2.5 epochs with a learning rate of 1.4e-5 and AdamW optimizer with cosine decay. The fine-tuning procedure is implemented using the LLaMA-Factory framework [77]. Following established practice in prior work [21, 52], we mask external tool call outputs during fine-tuning to preserve the integrity of the learning process by excluding extraneous external knowledge.
RL stage employs Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) [69] with the following protocol: Each training iteration processes 64 prompts, generating 8 rollouts per prompt through environment interaction. Each rollout permits up to 24 steps and 32k tokens followed by final answer generation. We use the VeRL framework [48] for DAPO training.

**Table 7: Results on agentic benchmarks including GAIA, WebWalker, BrowseComp and HLE. We port Pass@1 metric for all tasks. Gray-highlighted values represent our reproduced results.**
| Method | Backbone | GAIA | WebWalker | BrowseComp | HLE |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Level 1 | Level 2 | Level 3 | Avg. | Avg. | Avg. | Avg. |  |  |
| Model Inference |  |  |  |  |  |  |  |  |
| Qwen2.5-32B-Instruct | - | 12.8 | 3.8 | 0.0 | 6.8 | 3.1 | 0.6 | 5.4 |
| QwQ-32B | - | 30.8 | 15.4 | 25.0 | 22.3 | 4.3 | 0.5 | 9.6 |
| Deepseek-R1-671B | - | 43.6 | 26.9 | 8.3 | 31.1 | 10.0 | 2.0 | 8.6 |
| Agent Frameworks |  |  |  |  |  |  |  |  |
| OWL | GPT-4.1 | 71.0 | 50.0 | 28.6 | 53.6 | 10.2 | - | 6.4 |
| OAgents | 66.7 | 57.7 | 33.3 | 58.3 | - | 13.7 | 20.2 |  |
| DeepResearch | - | 74.3 | 69.1 | 47.6 | 67.4 | - | 51.5 | 26.6 |
| Tool-integrated Methods |  |  |  |  |  |  |  |  |
| R1-Searcher | Qwen-2.5-7B-Instruct | 28.2 | 19.2 | 8.3 | 20.4 | - | - | - |
| WebDancer | 41.0 | 30.7 | 0 | 31.0 | 36.0 | - | - |  |
| WebSailor | - | - | - | 37.9 | - | 6.7 | - |  |
| AFM-SFT | 36.5 | 33.3 | 16.7 | 34.0 | - | - | - |  |
| AFM-RL | 53.8 | 32.7 | 33.3 | 40.8 | 55.6 | 5.8 | 15.6 |  |
| Search-o1 | QwQ-32B | 53.8 | 34.6 | 16.7 | 39.8 | 34.1 | - | 10.8 |
| WebThinker-Base | 53.8 | 44.2 | 16.7 | 44.7 | 41.9 | - | 13.0 |  |
| WebThinker-RL | 56.4 | 50.0 | 16.7 | 48.5 | 46.5 | 2.8 | 15.8 |  |
| SimpleDeepSearcher | - | - | - | 50.5 | - | - | - |  |
| WebDancer | 61.5 | 50.0 | 25.0 | 51.5 | 47.9 | 3.8 | 7.2 |  |
| WebShaper | 69.2 | 50.0 | 16.6 | 53.3 | 49.7 | - | - |  |
| Search-o1 | Qwen-2.5-32B-Instruct | 33.3 | 25.0 | 0.0 | 28.2 | - | - | - |
| WebDancer | 46.1 | 44.2 | 8.3 | 40.7 | 38.4 | - | - |  |
| SimpleDeepSearcher | - | - | - | 40.8 | - | - | - |  |
| WebShaper | 61.5 | 53.8 | 16.6 | 52.4 | 51.4 | - | - |  |
| WebSailor | - | - | - | 53.2 | - | 10.5 | - |  |
| AFM-SFT | 56.4 | 51.9 | 25.0 | 50.5 | 61.5 | 10.0 | 16.3 |  |
| AFM-RL | 69.2 | 50.0 | 33.3 | 55.3 | 63.0 | 11.1 | 18.0 |  |

##### Baselines.

We conduct comprehensive comparisons against state-of-the-art methods to evaluate our approach across two evaluation datasets.

Multi-hop Question Answering Baselines.
We compare against two categories of methods on MHQA benchmarks:

- •
Direct Inference: We evaluate against baseline LLMs that rely on their internal knowledge for question answering, including Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct [45]. These models serve as the backbone model for AFM and have demonstrated excellent performance across various established benchmarks.
- •
Tool-integrated Framework: We systematically evaluate a suite of tool-augmented methods, including Search-o1 [28], Search-R1 [21], ZeroSearch [51], ReSearch [3], StepSearch [60], and ReasonRAG [75].

Complex Web Tasks Baselines.
For GAIA, WebWalker, BrowseComp, and HLE benchmarks, we compare against:

- •
Direct Inference: For complex web tasks, we evaluate against more advanced baseline LLMs, including Qwen2.5-32B-Instruct [45], QwQ-32B [55], and Deepseek-R1-671B [9].
- •
Agent Framework: We additionally compare against two SOTA agent frameworks: OAgents [82] and OWL [15], which are widely recognized for their strong performance in web agent tasks.
- •
Tool-integrated Frameworks: We compare against specialized web agents including: Search-o1 [21], R1-Searcher [50], WebThinker [29], SimpleDeepSearcher [52], WebDancer [65], WebSailor [27], and WebShaper [54].

All baselines utilize publicly available implementations with performance reported for their optimal configurations. To ensure fair comparison while isolating architectural contributions, we use the same backbone models (Qwen-2.5-7B/32B-Instruct or QwQ-32B) across all methods where applicable.

#### 4.1.2 Experimental Results

##### MHQA Performance.

From [Table˜6](#S4.T6), empirical results demonstrate that AFM achieves strong performance across both single-hop and multi-hop test sets of the MHQA benchmark against comparably-sized models, with consistent effectiveness observed across models of varying sizes compared to other approaches. Specifically, our AFM-SFT demonstrates exceptional performance, having surpassed the previous state-of-the-art methods. This validates the effectiveness of multi-agent distillation in transferring collaborative intelligence. Notably, our AFM-RL, after strategy optimization, has established a state-of-the-art in average performance across 7 datasets. When evaluated on models of the same size and type, our framework achieves 41.3% (Qwen-2.5-3B-base), 40.8% (Qwen-2.5-3B-instruct), 45.2% (Qwen-2.5-7B-base), and 45.5% (Qwen-2.5-7B-instruct), respectively. Compared to the previous best methods, these represent improvements of 6.8%, 1.7%, 2.1%, and 6.4%, respectively. A key finding is the exceptional generalization capability of our framework. Despite being trained solely on NQ and HotpotQA, our models achieve even more significant performance gains on the unseen validation and test sets of other multi-hop QA datasets. This outperformance is a direct result of our framework’s core strengths: advanced task decomposition and effective tool utilization, which are critical for solving complex, multi-step reasoning problems.

##### Complex Web Tasks Performance.

From [Table˜7](#S4.T7), empirical results demonstrate that AFM establishes a new state-of-the-art on knowledge-intensive complex tasks. With the Qwen-2.5-32B-Instruct backbone, it achieves a new state-of-the-art (among the same model size) average success rate of 55.3% on the GAIA benchmark. This represents a significant 2.1% improvement over WebSailor, with even greater gains of 3.8% relative to the RL-enhanced WebDancer model (which scores 51.5 on GAIA with the QwQ-32B backbone, which is stronger than the backbone for AFMs). AFM also achieves a new state-of-the-art among 32B models on BrowseComp with a success rate of 11.1%. On the WebWalker benchmark, AFM’s 63.0% average accuracy substantially exceeds WebThinker-RL (46.5%), WebDancer (47.9%), and WebShaper (51.4%) demonstrating its problem-solving capabilities in dynamic web environments. AFM also achieves 18.0% on the challenging HLE benchmark, outperforming strong baselines including WebThinker-RL (15.8%) and WebDancer (7.2%).

Figure: Table 8: SFT performance comparison of 32B models (all using Qwen2.5-32B-Instruct as backbone) across GAIA, WebWalker, and BrowseComp.

Moreover, our method enables models across different scales to achieve impressive performance, even comparing favorably with GPT-4.1-based Multi-Agent Systems and state-of-the-art reasoning models. With the Qwen2.5-32B-Instruct backbone, AFM attains a GAIA score of 55.3%, approaching the performance of GPT-4.1 based systems like OWL (55.8%) and OAgents (58.3%). Besides, in HLE, it reaches 18.0%—surpassing not only OWL’s GPT-4.1 based result of 12.6% but also outperforming leading reasoning models such as DeepSeek-R1 (8.6%) and QwQ-32B (9.6%). Even with the smaller Qwen-2.5-7B-Instruct backbone, AFM maintains exceptional performance, achieving a HLE score of 15.6%—a result that is only 0.2% lower than the 15.8% attained by WebThinker-RL with the QwQ-32B backbone. Notably, this 7B model’s performance even surpasses that of other 32B tool-integrated methods across different benchmarks, further validating the effectiveness of the Chain-of-Agents paradigm for agentic problem-solving and our multi-agent distillation framework in transferring collaborative intelligence robustly across model scales.

Specifically, [Table˜8](#S4.T8) presents a direct comparison of SFT performance across 32B models, highlighting the advantages of our approach. With the Qwen2.5-32B-Instruct backbone, AFM-SFT achieves a GAIA score of 50.5%—outperforming WebSailor-SFT-32B (46.6%), WebDancer-SFT-32B (35.0%), and WebShape-SFT-32B (44.6%) by notable margins.
The superiority of AFM-SFT extends beyond GAIA: on WebWalker, it reaches 61.5%—a substantial lead over WebShape-SFT-32B (44.6%), the only other model with reported results on this benchmark. On BrowseComp, AFM-SFT scores 10.0%, surpassing WebSailor-SFT-32B (7.2%) to claim the top position among compared methods.
These consistent performance gains across benchmarks directly validate the effectiveness of our proposed multi-agent distillation framework, which enhances SFT outcomes by distilling high-quality collaborative reasoning patterns into the model.

These findings demonstrate that AFM effectively resolves the Tool Coordination Dilemma by enabling bidirectional interaction between search and code tools, as reflected in its superior performance across multi-level GAIA tasks. Furthermore, the framework overcomes the Multi-Agent Transfer Dilemma by distilling distributed collaboration patterns into a single foundation model, as indicated by its leading results in knowledge-intensive scenarios.

### 4.2 Code Agent Experiments

#### 4.2.1 Experimental Setup

##### Training Dataset.

Our code agent training dataset is assembled by unifying several publicly-available datasets spanning both code generation and mathematical reasoning problems. Specifically, we draw from

- •
Pure code tasks: LiveCodeBench v1–v3 [20] and CodeForces [40].
- •
Pure math tasks: Retool-SFT [7] and DAPO-Math [69].
- •
Mixed code & math tasks: Skywork-OR1-RL-Data [13].

These sources are selected because they are verifiable: every code problem carries an average of $\geq 50$ test cases, and proof-based math problems are discarded. The dataset is also diverse and challenging, spanning common to contest-level programming problems, high-school to Olympiad mathematics problems.

For SFT stage, we take the full splits of LiveCodeBench v1–v3, Retool-SFT, and Skywork-OR1-RL-Data, and the verifiable-prompts split of CodeForces. After applying the trajectory synthesis and quality-filtering pipeline ([Section˜3.2.1](#S3.SS2.SSS1)), we retain about 47 k reasoning traces whose final answers pass all unit tests or numerical verifications.

For RL stage, we use LiveCodeBench v1–v3, Skywork-OR1-RL-Data, and DAPO-Math[69]. Skywork-OR1-RL-Data contains more than 100 k math problems—far exceeding the size of the code generation dataset—so we discard questions that even a DeepSeek-distill-Qwen-7B model fails on all 16 samples (as tagged in the original release). The remaining Skywork-OR1-RL-Data math dataset contains 35 k math problems.
Then we apply the quality filter as mentioned in [Section˜3.3.1](#S3.SS3.SSS1) eliminate overly simplistic queries. Notebly, we intentionally do not deduplicate prompts between the SFT and RL sets; instead we rely on the DAPO algorithm’s implicit difficulty-based filtering during RL to prevent over-training on already-mastered tasks.

The statistical results of the final SFT and RL datasets are presented in [Table˜9](#S4.T9). Here, "Avg. Hops" denotes the average number of agent invocations per sample in the SFT dataset.

**Table 9: Code Agent Dataset Composition**
| Source | LCB v1-v3 | CodeForces | Sky-Code | ReTool | Sky-Math | DAPO-MATH | Total |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SFT Phase |  |  |  |  |  |  |  |
| Filtered Size | 443 | 2695 | 10339 | 1112 | 45350 | - | 59929 |
| Avg. Hops | 9.1 | 9.4 | 8.3 | 8.0 | 6.5 | - | 7.0 |
| RL Phase |  |  |  |  |  |  |  |
| Filtered Size | 392 | - | 10033 | - | 23766 | 13369 | 47560 |

##### Benchmarks.

As shown in [Table˜10](#S4.T10), we evaluate the code agent on two types of benchmarks: mathematical reasoning benchmarks and code generation benchmarks. The former includes two benchmarks at the level of competition programming problems: CodeContests[31] and LiveCodeBench v4-v5 [20]; the latter adopt five mathematical competition benchmarks that cover difficulty levels from intermediate-level math competition problems to Olympiad-level ones: AIME24 [38], AIME25 [39], MATH500 [32], OlympiadBench [12], and AMC23. Detailed information about these evaluation benchmarks are as folows:

- •
AIME24 [38] and AIME25 [38]: Curated from the 2024 and 2025 American Invitational Mathematics Examination, these datasets encapsulate 30 authentic, competition-grade problems whose depth surpasses that of mainstream high-school contests, thereby furnishing a rigorous test-bed for advanced mathematical reasoning.
- •
MATH500 [32]: A stratified sample of 500 problems drawn from OpenAI’s PRM800K corpus. The selection spans a broad spectrum of mathematical domains and difficulty strata, ensuring comprehensive coverage of typical challenge archetypes.
- •
AMC23: Comprising problems released in the 2023 American Mathematics Competitions, this benchmark interrogates models across algebra, geometry, combinatorics and number theory. The tasks are moderately difficult yet non-routine, demanding multi-step symbolic manipulation and creative insight rather than mechanical computation. Consequently, AMC’23 serves as an effective gauge of a system’s end-to-end reasoning capacity within the scope of standard high-school competitions.
- •
OlympiadBench [11]: A rigorously curated collection of problems transcribed from premier high-school Olympiads in mathematics and physics (e.g., IMO Shortlist, AIME, PUPC, and national contests). For our evaluation we retain the English, text-only mathematics subset—674 problems in total—thereby preserving Olympiad-level rigor while obviating multimodal dependencies.
- •
LiveCodeBench (v4–v5) [20] is a dynamic and contamination-free benchmark dataset specifically designed to assess the coding capabilities LLMs. Its primary goal is to offer a realistic programming evaluation setting while mitigating issues such as data leakage and overfitting that commonly affect static benchmarks. In our evaluation, we adopt versions v4 and v5, which include problems released between August 2024 and January 2025.
- •
CodeContests [31]: The CodeContests dataset is constructed by Google DeepMind for the development of the AlphaCode model. The corpus is sourced from public competitions hosted on several major online programming platforms. The selected evaluation subset comprises 165 problems, each accompanied by multiple sets of test cases. The difficulty of the problems ranges from beginner level to advanced competition grade.

**Table 10: Mathematical reasoning and code generation benchmarks**
| Dataset | Task Types | Evaluation Set Size |
| --- | --- | --- |
| LiveCodeBench (v4–v5) [20] | Contest-level Programming | 268 |
| CodeContests [31] | Contest-level Programming | 165 |
| AIME24 [38] | Advanced Mathematics | 30 |
| AIME25 [39] | Advanced Mathematics | 30 |
| MATH500 [32] | General Mathematics | 500 |
| AMC23 | High-school Mathematics | 40 |
| OlympiadBench [11] | Olympiad Mathematics | 674 |

##### Metrics.

For code generation tasks, the test sets provide predefined test cases. The final generated code is executed in a sandbox environment, and a task is considered successfully solved only if all test cases are passed. Model performance is evaluated based on the pass@1 rate on each test set.
In mathematical reasoning tasks where each question is associated with a ground-truth answer, Math-Verify is utilized for robust answer extraction and correctness assessment. Owing to the limited sample sizes of AMC23, AIME24, and AIME25, pass@1 estimates exhibit high variance. To attenuate this stochasticity, we report the mean pass@1 over 16 independently drawn samples (avg@16) as the primary metric for these datasets, whereas standard pass@1 is retained for all remaining benchmarks.

##### Implementation Details.

Our code agent utilizes Qwen2.5-Coder-7B-Instruct and Qwen2.5-Coder-32B-Instruct models as backbones [17].
SFT is performed for 2 epochs, with a batch size of 32 employed for the 7B model and a batch size of 64 utilized for the 32B model.
Optimization is driven by the AdamW optimizer [34]; the initial learning rate is 3e-5 for the 7B model and 1.4e-5 for the 32B model, both scheduled with a linear warm-up of 10% steps followed by cosine decay.
In accordance with prior work [7], we mask the loss of external tool–call outputs, thereby preventing gradient updates from sandbox feedback.
The RL stage leverages the VeRL [48] framework.
We adopt the DAPO algorithm [69], configured with 8 rollouts per prompt and an overlong buffer set to 1/8 of the maximum response length.
To improve sample efficiency, we constrain each rollout to at most 8 tool calls before a final answer is emitted.
The train-batch size is 256 while the mini-batch size is 32.
The 7B model is trained with a context window of 32k tokens throughout the training process. For the 32B model, a 16k token context window is initially adopted to expedite early-stage training; subsequently, this window is expanded to 32k tokens at the 40th global training step. The total number of global training steps conducted for the 7B model amounted to 150, whereas the 32B model underwent 120 global training steps. The number of warm up steps is set to 10.

For inference of AFM-7B models, we set the temperature at 0.8, which is 0.6 for AFM-32B models. Both configurations operate in a context window of 32k tokens, with a top-p of 1.0 and a maximum of 12 tool calls.

##### Baselines.

We compare our method against two representative families of competitive approaches.

- •
Text-only Reasoning Models: These systems solve tasks exclusively through textual reasoning, without recourse to external tools. The set comprises (i) the Qwen2.5-Coder-Instruct family [45], which serves as the backbone of our Code Agent, and (ii) high-performance reasoning models such as QwQ-32B-Preview [55], and models based on qwen, include SimpRL-Zoo [71], and Eurus-2-PRIME [4].
- •
Tool-Integrated Reasoning Models: These models improve reasoning capabilities by incorporating the ability to generate and execute code, based on the Qwen-2.5-7B/32B parameter families, including ToRL [30], SimpleTIR [66], ReTool [7], AutoTIR [64], ZTRL [35], Effective TIR [1], Reveal [22], and Verl-Tool [56]. Notably, Reveal is the only model trained on code generation tasks during the RL stage, whereas other models are trained using mathematical and other datasets.

#### 4.2.2 Experimental Results

**Table 11: Results comparison of mathematical benchmarks. For each column, best results are shown in bold and second-best results are underlined. We sample 16 responses and report the avg@16 metric for AIME24, AIME25, AMC23. For MATH500 and OlympiadBench, we report Pass@1 metric.**
| Method | AIME24 | AIME25 | MATH500 | AMC23 | OlympiadBench | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| Model Inference |  |  |  |  |  |  |
| Large-Scale Models |  |  |  |  |  |  |
| OpenAI o1-mini | 56.7 | - | - | - | 65.3 | - |
| DeepSeek-V3 | 39.2 | 36.6 | 90.2 | - | - | - |
| 7B Models |  |  |  |  |  |  |
| Qwen-2.5-Coder-7B | 7.5 | 2.9 | 68.6 | 16.4 | 11.9 | 21.5 |
| SimpleRL-Zoo-7B | 24.0 | - | 80.2 | 70.0 | 39.0 | - |
| Eurus-2-7B-PRIME | 26.7 | 13.3 | 79.2 | 57.4 | 42.1 | 43.7 |
| 32B Models |  |  |  |  |  |  |
| Qwen-2.5-Coder-32B | 13.3 | 10.0 | 73.0 | 50.6 | 35.9 | 36.6 |
| SimpleRL-Zoo-32B | 27.2 | - | 82.4 | 67.5 | 46.4 | - |
| QwQ-32B-Preview | 50.0 | 40.0 | 90.6 | 80.0 | - | - |
| Tool-integrated Methods (7B) |  |  |  |  |  |  |
| TIR Methods From Qwen-2.5-7B Family Models |  |  |  |  |  |  |
| ToRL | 43.3 | 30.0 | 82.2 | 75.0 | 49.9 | 56.1 |
| Effective TIR | 42.3 | 29.2 | 86.4 | 74.2 | - | - |
| ZTRL | 50.0 | 26.7 | 80.2 | - | - | - |
| Verl-Tool | 40.0 | - | 83.4 | 65.0 | 50.2 | - |
| SimpleTIR-Multi | 50.5 | 30.9 | 88.4 | 79.1 | 54.8 | 60.7 |
| AutoTIR | 33.3 | 16.7 | 62.6 | - | - | - |
| AFM-SFT | 27.5 | 15.4 | 74.0 | 60.3 | 40.3 | 43.5 |
| AFM-RL | 51.9 | 37.8 | 89.4 | 81.6 | 60.6 | 64.3 |
| Tool-integrated Methods (32B) |  |  |  |  |  |  |
| TIR Methods From Qwen-2.5-32B Family Models |  |  |  |  |  |  |
| ZTRL-32B | 56.7 | 33.3 | 87.8 | - | - | - |
| ReTool-32B | 67.0 | 49.3 | 93.2 | 96.1 | 66.4 | 74.4 |
| SimpleTIR-32B-Multi | 59.9 | 49.2 | 92.9 | 91.6 | 63.7 | 71.5 |
| AFM-SFT | 41.0 | 31.0 | 82.8 | 78.9 | 51.1 | 60.0 |
| AFM-RL | 66.7 | 59.8 | 94.6 | 96.6 | 72.1 | 78.0 |

**Table 12: Code generation benchmarks results comparison. We report Pass@1 metric.**
| Method | LiveCodeBench v4 | LiveCodeBench v5 | CodeContests | Avg. |
| --- | --- | --- | --- | --- |
| Model Inference |  |  |  |  |
| Qwen-2.5-Coder-7B | 15.8 | 18.0 | 0.0 | 6.8 |
| Qwen-2.5-Coder-32B | 28.7 | 28.1 | 0.6 | 11.5 |
| Tool-integrated Methods |  |  |  |  |
| TIR Methods From Qwen-2.5-32B Family Models |  |  |  |  |
| Retool | 19.0 | 23.4 | 10.3 | 10.5 |
| ReVeal | - | 42.4 | - | - |
| AFM-SFT | 28.0 | 26.3 | 13.3 | 13.5 |
| AFM-RL | 29.0 | 28.7 | 18.8 | 15.3 |
| AFM-SFT | 37.0 | 43.1 | 22.4 | 20.5 |
| AFM-RL | 43.0 | 47.9 | 32.7 | 24.7 |

In this section, we analyze the performance of Code Agent from two perspectives: mathematical reasoning tasks and code generation tasks. Additionally, we incorporate curves from the reinforcement learning process of the 32B model within [Appendix˜B](#A2), including the changes in AIME25 performance, response length, and score with respect to global training steps.

##### Mathematical problem-solving Capabilities.

The empirical results presented in [Table˜11](#S4.T11) demonstrate that AFM significantly outperforms existing baseline models across both 7B and 32B parameter scales, validating the effectiveness of our proposed method in mathematical reasoning tasks. Specifically, at the 7B scale, AFM-RL-7B achieved the best performance across all five benchmarks, with an average accuracy of 64.3%—representing a 3.6% improvement over the second-best performer, SimpleTIR-7B-Multi. At the 32B scale, AFM-RL-32B attained an average accuracy of 78.0%, which is 3.6% higher than the current state-of-the-art ReTool-32B. Notably, on the datasets AIME25 and OlympiadBench, AFM-RL-32B achieved absolute improvements of 10.5% and 5.7% respectively, indicating that AFM possesses stronger generalization and problem-solving capabilities in complex mathematical reasoning scenarios.

To quantify the contribution of our training methodology, we analyze performance improvements across different training stages compared to the base model. For the 7B model, SFT contributed an average accuracy gain of 22.0%, with RL providing an additional 20.8% improvement over the SFT baseline. For the 32B model, the corresponding gains are 23.4% from SFT and 18% from RL. Taken together, the results reveal that the SFT phase endows the model with Chain-of-Agents reasoning capabilities, such as planning, reflection, and tool calling, through multi-agent distillation process. The RL phase further consolidates and strengthens these capabilities.

##### Code generation Capabilities.

[Table˜12](#S4.T12) summarizes performance comparisons of our models against backbone models and other TIR methods across three competitive programming tasks, revealing substantial improvements.
Compared to our backbones, RL-enhanced AFM achieves average accuracy gains of 8.5% (7B) and 13.2% (32B), confirming our method’s efficacy in boosting code generation capabilities. Among baseline TIR models utilizing code interpreters—ReTool-32B (trained exclusively on mathematical datasets) and Reveal-32B (specialized for code generation)—our approach outperforms both. Notably, even our SFT-only model surpasses these baselines on LiveCodeBenchV5, validating that our multi-agent distillation framework and data filtering strategies enhance complex programming performance. Further, RL-refined models gain an additional 1.8% (7B) and 3.2% (32B) over their SFT counterparts, confirming that agentic reinforcement learning further strengthens programming capabilities.

We present two illustrative cases in [Appendix˜C](#A3) to demonstrate how our Code Agent addresses mathematical reasoning and code generation tasks.

## 5 Analysis

### 5.1 Computational Efficiency

We conduct a comparative analysis of tool calls and token consumption across 3 state-of-the-art frameworks, using 10 randomly sampled instances from the GAIA dataset as the evaluation set: OAgents [82], WebThinker [29], and AFM. As depicted in [Figure˜5](#S5.F5), AFM demonstrates superior efficiency across two critical dimensions: (a) tool efficiency, measured as tool calling numbers per successful task completion and (b) token efficiency, measured as prompt engineering cost per successful task completion.
We can see that AFM not only achieves the lowest token consumption (both in overall tokens and tool call tokens) but also uses the fewest number of tool calls among all compared methods. Furthermore, compared to methods other than the agentic framework OAgents, AFM demonstrates notable latency improvements in inference time.
This efficiency gain stems from our data construction mechanism, which mitigates redundant token accumulation through targeted filtering of irrelevant and non-useful content.

Figure: Figure 5: Performance efficiency comparison of AFM with MAS and TIR methods.
Refer to caption: /html/2508.13167/assets/x5.png

### 5.2 Generalization on Unseen Agents

Under the Chain-of-Agents framework, we evaluated the model’s zero-shot agent generalization ability. During training, the Code Agent model was exposed exclusively to code and math tasks executed by a Python interpreter agent and had never encountered tool agents such as Web search or Visual inspector. At inference, the complete tool descriptions and invocation schemas were explicitly inserted into the prompt, with the GAIA test set serving as the task suite. Results show that the code-agent model strictly adheres to the prompt-specified formats and correctly orchestrates the unseen tools on the first attempt. In the honey-density task, it first queries Web search for the value and then computes the result via the Python executor ([Case Study](#A3)).

For counter-validation, we instructed the web-agent model, whose training data consists solely of search tasks with tools limited to Web Search and Crawl Page, to invoke the Python executor and Visual inspector under the same prompt. Although the web-agent model issues the appropriate calls at the right moment, the Python executor requires code blocks to be wrapped in triple backticks, and Visual inspector demands a JSON string with complete fields and no extraneous spaces. In the vast majority of test cases, the web-agent model fails to generate invocations that satisfy these fine-grained format constraints, resulting in parser errors and task abortion.

We further analyze and find that, for conventional tasks such as report generation, all models exhibit strong generalization. However, when tool usage requires character-level precision, the performance of the web agent model degrades significantly. In contrast, the code agent model, which was trained under strict code formatting constraints, remains robust and consistently achieves correct tool invocation and successful task completion.

### 5.3 Agentic Test-Time Scaling

Figure: Figure 6: Performance of AFM with test-time scaling on GAIA, BrowseComp, WebWalker, and HLE.
Refer to caption: /html/2508.13167/assets/x6.png

We conduct an in-depth analysis of test-time scaling (TTS) performance following the training of the AFM model, which establishes itself as the top-performing baseline post-training. We further validate and assess TTS across multiple agentic benchmarks: GAIA, WebWalker, BrowseComp, and HLE. As presented in [Table˜13](#S5.T13), our methods—AFM, AFM-Bo3, and AFM-Pass@3—are evaluated against a set of competitive agentic models employing diverse backbones across different model families. Specifically, AFM-Bo3 denotes a strategy that selects the optimal trajectory from N=3 candidate answers, with judgments facilitated by the Qwen2.5-72B-Instruct model; AFM-Pass@3 employs the principle of "three attempts with one correct answer" to enhance performance.

**Table 13: TTS results on agentic benchmarks including GAIA, WebWalker, BrowseComp and HLE.**
| Method | Backbone | GAIA | WebWalker | BrowseComp | HLE |
| --- | --- | --- | --- | --- | --- |
| Agent Framework |  |  |  |  |  |
| SmolAgents | Claude-3-7 | 52.1 | - | - | - |
| SmolAgents-Pass@3 | 63.6 | - | - | - |  |
| OAgent | 66.7 | - | - | - |  |
| OAgent-Pass@3 | 73.9 | - | - | - |  |
| Open Agentic Models |  |  |  |  |  |
| WebDancer | QwQ-32B | 51.5 | - | 3.8 | - |
| WebDancer-Pass@3 | 62.1 | - | 7.9 | - |  |
| WebSailor | Qwen-2.5-32B-Instruct | 53.2 | - | 10.5 | - |
| WebSailor-Pass@3 | 63.1 | - | 15.0 | - |  |
| Our Methods |  |  |  |  |  |
| AFM | Qwen-2.5-32B-Instruct | 55.3 | 63.0 | 11.0 | 18.0 |
| AFM-Bo3 | 57.3 | 64.7 | 11.3 | 23.0 |  |
| AFM-Pass@3 | 69.9 | 78.7 | 19.2 | 33.2 |  |

Notably, AFM-Bo3 outperforms the base AFM model across key benchmarks. On GAIA, AFM achieves an average score of 55.3, while AFM-Bo3 climbs to 57.3, marking a clear improvement. The gains are even more pronounced on HLE, where AFM-Bo3 reaches 23.0 compared to AFM’s 18.0, underscoring the effectiveness of the best-of-3 selection strategy in refining results.

AFM-Pass@3 delivers an even more substantial performance boost. On GAIA, it surges to 69.9—representing a remarkable 14.6-point increase over the base AFM (55.3)—a gain that far outpaces the improvements seen in other models’ Pass@3 variants. For context, WebDancer-Pass@3 improves by 10.6 points (from 51.5 to 62.1) on GAIA, and WebSailor-Pass@3 gains 9.9 points (from 53.2 to 63.1), highlighting the superior TTS capability of our framework. This trend holds across other benchmarks: on WebWalker, AFM-Pass@3 hits 78.7, leaping from AFM’s 63.0 and AFM-Bo3’s 64.7; on HLE, it reaches 33.2, more than 10 points above AFM-Bo3’s 23.0.

Furthermore, our 32B end-to-end model demonstrates a compelling performance trajectory against SmolAgents (Claude-3-7 backbone) on GAIA: at Pass@1, AFM scores 55.3 compared to SmolAgents’ 66.7, while with the Pass@3 strategy, our AFM-Pass@3 reaches 69.9—substantially narrowing the gap relative to SmolAgents-Pass@3 (73.9). This marked convergence underscores the effectiveness of our test-time scaling in bridging performance differences between open-source and proprietary backbones. It also significantly surpasses other end-to-end models: on GAIA, AFM outperforms WebDancer (51.5) and matches WebSailor (53.2), while AFM-Pass@3 extends this lead to 16.8 points over WebDancer-Pass@3 and 6.8 points over WebSailor-Pass@3. These results collectively demonstrate that end-to-end agent models integrated with our Chain-of-Agents framework derive greater benefits from test-time scaling strategies compared to traditional multi-agent systems.

## 6 Related Work

### 6.1 Multi-Agent Systems

Recent research has decisively shifted from static retrieval to dynamic multi-agent frameworks. Orchestrator-worker architectures enable strategic query planning with parallel retrievals for search tasks [10]. In code generation, while systems like MapCoder simulate development cycles [18], they suffer from error propagation due to late-stage testing – a limitation addressed through simulation-based plan verification (CodeSim [19]) or decoupled test generation for bias mitigation (AgentCoder [16]). Crucially, comprehensive analyses like OAgents [82] provide empirical module-level insights into coordination mechanisms for complex open-ended tasks.

Multi-Agent Systems attempt to resolve scalability limitations through specialized agents $\{a_{i}\}_{i=1}^{N}$ with dedicated policies $\pi_{a_{i}}$, enabling distributed expertise for complex tool orchestration. However, this architectural specialization introduces prohibitive coordination overhead that scales with pairwise agent interactions, alongside state fragmentation across agents. The coordination cost grows unbounded as the number of agents increases, fundamentally constraining real-world deployment, while the lack of a global state representation inhibits cross-agent tool synergies. These limitations collectively manifest as suboptimal utility when handling queries requiring adaptive coordination across extended reasoning chains. Collectively, while specialized agent coordination surpasses monolithic models, prevailing pipelined architectures constrain emergent synergies compared to end-to-end trainable systems.

### 6.2 Tool-Integrated Reasoning

Chain-of-Thought (CoT) reasoning [62] established a foundational paradigm for complex problem solving through explicit decomposition into stepwise traces, where each rationale incrementally builds upon previous reasoning steps. This framework demonstrates exceptional efficacy in closed-world tasks such as mathematical reasoning [47] where solution paths reside within the model’s parametric knowledge. However, CoT exhibits fundamental limitations when handling queries requiring external information, as the likelihood of relying on out-of-scope knowledge increases with reasoning complexity – a constraint manifesting as practical deficiencies in real-time information processing and domain-specific operations beyond the model’s training distribution.

Building on CoT, Tool-Integrated Reasoning architectures enhance reasoning through explicit external tool integration. Prompting-based methods (IRCoT [57], ReAct [68]) initiated tool-reasoning loops via static templates, with ReAct establishing a foundational paradigm through iterative Thought-Action-Observation cycles. Formally, ReAct generates trajectories by sequentially sampling reasoning thoughts, tool invocations, and environmental responses. While effective for small-scale tool integration, this monolithic policy architecture encounters significant scalability challenges in open-domain environments: static retrieval mechanisms fail to adapt to dynamic tool ecosystems, and the combinatorial explosion in action space renders optimal tool selection computationally intractable for large knowledge bases. Subsequent advances extended these frameworks to dynamic prompting (OpenResearcher [78]) and memory-augmented control (AirRAG [8]). SFT-optimized approaches (CoRAG [61], Auto-RAG [70]) learned tool-use from demonstrations but remained constrained by static supervision. RL-driven frameworks (Search-R1 [21], WebThinker [29], WebDancer [65]) optimized policies through reward maximization yet face critical limitations: they lack mechanisms for synergistic multi-tool coordination, suffer from reward sparsity in multi-step sequences. These gaps fundamentally restrict current systems’ ability to establish bidirectional tool dependencies required for dynamic information-seeking scenarios.

### 6.3 Reinforcement Learning for Reasoning

The integration of external tools into language models has shown promise in enhancing reasoning capabilities, particularly for structured tasks such as mathematical computation and code generation [30, 43]. Early approaches relied on supervised fine-tuning with manually curated tool-use data, limiting their adaptability to new domains [55]. More recent frameworks leverage RL to learn adaptive tool invocation strategies, enabling real-time execution within reasoning processes [30, 43]. However, these methods often focus on single-tool settings and overlook the efficiency cost of repeated tool calls. To address this, OTC [59] introduces a reward formulation that jointly optimizes correctness and tool efficiency, while Tool-Star [5] explores multi-tool collaboration through hierarchical reward design and scalable data synthesis. Despite these advances, most existing systems remain constrained by isolated reasoning-execution loops or reliance on costly annotations. AFM builds toward a unified framework that enables efficient and coordinated multi-tool reasoning without heavy dependence on manual engineering or explicit supervision.

## 7 Conclusion

This work introduces Chain-of-Agents, a new paradigm for building native agent models that supports end-to-end multi-agent problem-solving. Compared with the recent tool-integrated-reasoning paradigm, which corresponds to ReAcT-like agents, our Chain-of-Agents paradigm supports any multi-agent system that demonstrated superior performance compared to ReAcT agents. We propose a multi-agent distillation method to generate supervised training data and an agentic reinforcement learning method to optimize the model. We train a series of agent foundation models with the proposed methods. Our experimental results show that our approach significantly outperforms existing tool-integrated-reasoning methods across various domains, including RAG-based agents, web agents, and code agents. All code and data used for training and evaluation are open-sourced to facilitate future research on agent models and agentic RL.

## 8 Contributions

Core Contributors

- 0
Weizhen Li
\__color_backend_reset:0
’
\__color_backend_reset:       0
Jianbo Lin
\__color_backend_reset:0
’
\__color_backend_reset:
0
Zhuosong Jiang
\__color_backend_reset:0
’
\__color_backend_reset:       0
Jingyi Cao
\__color_backend_reset:0
’
\__color_backend_reset:
0
Xinpeng Liu
\__color_backend_reset:0
’
\__color_backend_reset:       0
Jiayu Zhang
\__color_backend_reset:0
’
\__color_backend_reset:
0
Zhenqiang Huang
\__color_backend_reset:0
’
\__color_backend_reset:       0
Qianben Chen
\__color_backend_reset:0
’
\__color_backend_reset:
0
Weichen Sun
\__color_backend_reset:0
’
\__color_backend_reset:       0
Qiexiang Wang
\__color_backend_reset:0
’
\__color_backend_reset:

Contributors

- 0
Hongxuan Lu
\__color_backend_reset:0
’
\__color_backend_reset:       0
Tianrui Qin
\__color_backend_reset:0
’
\__color_backend_reset:
0
Chenghao Zhu
\__color_backend_reset:0
’
\__color_backend_reset:       0
Yi Yao
\__color_backend_reset:0
’
\__color_backend_reset:
0
Shuying Fan
\__color_backend_reset:0
’
\__color_backend_reset:       0
Xiaowan Li
\__color_backend_reset:0
’
\__color_backend_reset:
0
Tiannan Wang
\__color_backend_reset:0
’
\__color_backend_reset:       0
Pai Liu
\__color_backend_reset:0
’
\__color_backend_reset:
0
King Zhu
\__color_backend_reset:0
’
\__color_backend_reset:       0
He Zhu
\__color_backend_reset:0
’
\__color_backend_reset:
0
Dingfeng Shi
\__color_backend_reset:0
’
\__color_backend_reset:       0
Piaohong Wang
\__color_backend_reset:0
’
\__color_backend_reset:
0
Yeyi Guan
\__color_backend_reset:0
’
\__color_backend_reset:       0
Xiangru Tang
\__color_backend_reset:0
’
\__color_backend_reset:
0
Minghao Liu
\__color_backend_reset:0
’
\__color_backend_reset:       0
Yuchen Eleanor Jiang
\__color_backend_reset:0
’
\__color_backend_reset:
0
Jian Yang
\__color_backend_reset:0
’
\__color_backend_reset:       0
Jiaheng Liu
\__color_backend_reset:0
’
\__color_backend_reset:
0
Ge Zhang
\__color_backend_reset:0
’
\__color_backend_reset:

Corresponding Authors

- 0
Wangchunshu Zhou
\__color_backend_reset:0
’
\__color_backend_reset:

Project Responsibilities

- •
Web Agent: Weizhen Li (SFT, RL), Jianbo Lin (RL), Jingyi Cao (SFT), Qianben Chen (SFT), Chenghao Zhu (RL), Hongxuan Lu (Eval), Weichen Sun (Lead).
- •
Code Agent: Zhuosong Jiang (RL), Xinpeng Liu (SFT), Zhenqiang Huang (Eval), Qiexiang Wang (Lead).
- •
Data: Jingyi Cao (Agent Paradigm), Jiayu Zhang (Data Generation), Qianben Chen (Lead).
- •
Paper Writing: Qianben Chen, Qiexiang Wang, Weichen Sun, Tianrui Qin, Shuying Fan.

## References

- Bai et al. [2025]
Fei Bai, Yingqian Min, Beichen Zhang, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen.
Towards effective code-integrated reasoning.
*arXiv preprint arXiv:2505.24480*, 2025.
- Canese et al. [2021]
Lorenzo Canese, Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanò.
Multi-agent reinforcement learning: A review of challenges and applications.
*Applied Sciences*, 11(11):4948, 2021.
- Chen et al. [2025]
Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, Fan Yang, et al.
Learning to reason with search for llms via reinforcement learning.
*arXiv preprint arXiv:2503.19470*, 2025.
- Cui et al. [2025]
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al.
Process reinforcement through implicit rewards.
*arXiv preprint arXiv:2502.01456*, 2025.
- Dong et al. [2025]
Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen.
Tool-star: Empowering llm-brained multi-tool reasoner via reinforcement learning.
*arXiv preprint arXiv:2505.16410*, 2025.
- Dorri et al. [2018]
Ali Dorri, Salil S Kanhere, and Raja Jurdak.
Multi-agent systems: A survey.
*Ieee Access*, 6:28573–28593, 2018.
- Feng et al. [2025a]
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong.
Retool: Reinforcement learning for strategic tool use in llms.
*arXiv preprint arXiv:2504.11536*, 2025a.
- Feng et al. [2025b]
Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Jingyi Song, and Hao Wang.
Airrag: Activating intrinsic reasoning for retrieval augmented generation via tree-based search.
*arXiv preprint arXiv:2501.10053*, 2025b.
- Guo et al. [2025]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Hadfield et al. [2025]
Jeremy Hadfield, Barry Zhang, Kenneth Lien, Florian Scholz, Jeremy Fox, and Daniel Ford.
How we built our multi-agent research system.
*https://www.anthropic.com/engineering/built-multi-agent-research-system*, 2025.
- He et al. [2024a]
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al.
Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.
*arXiv preprint arXiv:2402.14008*, 2024a.
- He et al. [2024b]
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al.
Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.
*arXiv preprint arXiv:2402.14008*, 2024b.
- He et al. [2025]
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou.
Skywork open reasoner 1 technical report.
*arXiv preprint arXiv:2505.22312*, 2025.
- Ho et al. [2020]
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa.
Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps.
*arXiv preprint arXiv:2011.01060*, 2020.
- Hu et al. [2025]
Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Qiguang Chen, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Bernard Ghanem, Ping Luo, and Guohao Li.
Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025.
URL [https://arxiv.org/abs/2505.23885](https://arxiv.org/abs/2505.23885).
- Huang et al. [2023]
Dong Huang, Jie M.Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui.
Agentcoder: Multi-agent-based code generation with iterative testing and optimisation.
*arXiv preprint arXiv:2312.13010*, 2023.
- Hui et al. [2024]
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al.
Qwen2. 5-coder technical report.
*arXiv preprint arXiv:2409.12186*, 2024.
- Islam et al. [2024]
Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez.
Mapcoder: Multi-agent code generation for competitive problem solving.
*Association for Computational Linguistics*, 2024.
- Islam et al. [2025]
Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez.
Codesim: Multi-agent code generation and problem solving through simulation-driven planning and debugging.
*Association for Computational Linguistics*, 2025.
- Jain et al. [2024]
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica.
Livecodebench: Holistic and contamination free evaluation of large language models for code.
*arXiv preprint arXiv:2403.07974*, 2024.
- Jin et al. [2025a]
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han.
Search-r1: Training llms to reason and leverage search engines with reinforcement learning.
*arXiv preprint arXiv:2503.09516*, 2025a.
- Jin et al. [2025b]
Yiyang Jin, Kunzhao Xu, Hang Li, Xueting Han, Yanmin Zhou, Cheng Li, and Jing Bai.
Reveal: Self-evolving code agents via iterative generation-verification, 2025b.
URL [https://arxiv.org/abs/2506.11442](https://arxiv.org/abs/2506.11442).
- Joshi et al. [2017]
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer.
Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.
*arXiv preprint arXiv:1705.03551*, 2017.
- Kim and Rush [2016]
Yoon Kim and Alexander M Rush.
Sequence-level knowledge distillation.
In *Proceedings of the 2016 conference on empirical methods in natural language processing*, pages 1317–1327, 2016.
- Kwiatkowski et al. [2019]
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al.
Natural questions: a benchmark for question answering research.
*Transactions of the Association for Computational Linguistics*, 7:453–466, 2019.
- Li et al. [2023]
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.
Camel: Communicative agents for "mind" exploration of large language model society.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
- Li et al. [2025a]
Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang, Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, and Jingren Zhou.
Websailor: Navigating super-human reasoning for web agent, 2025a.
URL [https://arxiv.org/abs/2507.02592](https://arxiv.org/abs/2507.02592).
- Li et al. [2025b]
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou.
Search-o1: Agentic search-enhanced large reasoning models.
*arXiv preprint arXiv:2501.05366*, 2025b.
- Li et al. [2025c]
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou.
Webthinker: Empowering large reasoning models with deep research capability.
*arXiv preprint arXiv:2504.21776*, 2025c.
- Li et al. [2025d]
Xuefeng Li, Haoyang Zou, and Pengfei Liu.
Torl: Scaling tool-integrated rl.
*arXiv preprint arXiv:2503.23383*, 2025d.
- Li et al. [2022]
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al.
Competition-level code generation with alphacode.
*Science*, 378(6624):1092–1097, 2022.
- Lightman et al. [2023]
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step.
*arXiv preprint arXiv:2305.20050*, 2023.
- Liu et al. [2025]
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu.
Inference-time scaling for generalist reward modeling.
*arXiv preprint arXiv:2504.02495*, 2025.
- Loshchilov and Hutter [2017]
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
*arXiv preprint arXiv:1711.05101*, 2017.
- Mai et al. [2025]
Xinji Mai, Haotian Xu, Weinong Wang, Yingying Zhang, Wenqiang Zhang, et al.
Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving.
*arXiv preprint arXiv:2505.07773*, 2025.
- Mallen et al. [2022]
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi.
When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.
*arXiv preprint arXiv:2212.10511*, 2022.
- Mialon et al. [2023]
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom.
Gaia: a benchmark for general ai assistants.
In *The Twelfth International Conference on Learning Representations*, 2023.
- of America [MAA]
Mathematical Association of America (MAA).
American invitational mathematics examination (aime) 2024.
Competitive mathematics examination, 2024.
- of America [MAA]
Mathematical Association of America (MAA).
American invitational mathematics examination (aime) 2025.
Competitive mathematics examination, 2025.
- Penedo et al. [2025]
Guilherme Penedo, Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Edward Beeching, Agustín Piqueres Lajarín, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra.
Codeforces.
*Hugging Face*, 2025.
- Phan et al. [2025]
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al.
Humanity’s last exam.
*arXiv preprint arXiv:2501.14249*, 2025.
- Press et al. [2022]
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis.
Measuring and narrowing the compositionality gap in language models.
*arXiv preprint arXiv:2210.03350*, 2022.
- Qian et al. [2025]
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji.
Toolrl: Reward is all tool learning needs.
*arXiv preprint arXiv:2504.13958*, 2025.
- Qiu et al. [2025]
Jiahao Qiu, Xuan Qi, Tongcheng Zhang, Xinzhe Juan, Jiacheng Guo, Yifu Lu, Yimin Wang, Zixin Yao, Qihan Ren, Xun Jiang, et al.
Alita: Generalist agent enabling scalable agentic reasoning with minimal predefinition and maximal self-evolution.
*arXiv preprint arXiv:2505.20286*, 2025.
- Qwen et al. [2025]
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report, 2025.
URL [https://arxiv.org/abs/2412.15115](https://arxiv.org/abs/2412.15115).
- Roucher et al. [2025]
Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki.
‘smolagents‘: a smol library to build great agentic systems.
[https://github.com/huggingface/smolagents](https://github.com/huggingface/smolagents), 2025.
- Shao et al. [2024]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Sheng et al. [2024]
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
*arXiv preprint arXiv: 2409.19256*, 2024.
- Shi et al. [2025]
Dingfeng Shi, Jingyi Cao, Qianben Chen, Weichen Sun, Weizhen Li, Hongxuan Lu, Fangchen Dong, Tianrui Qin, King Zhu, Minghao Yang, et al.
Taskcraft: Automated generation of agentic tasks.
*arXiv preprint arXiv:2506.10055*, 2025.
- Song et al. [2025]
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen.
R1-searcher: Incentivizing the search capability in llms via reinforcement learning.
*arXiv preprint arXiv:2503.05592*, 2025.
- Sun et al. [2025a]
Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou.
Zerosearch: Incentivize the search capability of llms without searching.
*arXiv preprint arXiv:2505.04588*, 2025a.
- Sun et al. [2025b]
Shuang Sun, Huatong Song, Yuhao Wang, Ruiyang Ren, Jinhao Jiang, Junjie Zhang, Fei Bai, Jia Deng, Wayne Xin Zhao, Zheng Liu, et al.
Simpledeepsearcher: Deep information seeking via web-powered reasoning trajectory synthesis.
*arXiv preprint arXiv:2505.16834*, 2025b.
- Tang et al. [2025]
Xiangru Tang, Tianrui Qin, Tianhao Peng, Ziyang Zhou, Daniel Shao, Tingting Du, Xinming Wei, Peng Xia, Fang Wu, He Zhu, Ge Zhang, Jiaheng Liu, Xingyao Wang, Sirui Hong, Chenglin Wu, Hao Cheng, Chi Wang, and Wangchunshu Zhou.
Agent kb: Leveraging cross-domain experience for agentic problem solving.
In *ICML 2025 Workshop on Collaborative and Federated Agentic Workflows*, 2025.
- Tao et al. [2025]
Zhengwei Tao, Jialong Wu, Wenbiao Yin, Junkai Zhang, Baixuan Li, Haiyang Shen, Kuan Li, Liwen Zhang, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou.
Webshaper: Agentically data synthesizing via information-seeking formalization, 2025.
URL [https://arxiv.org/abs/2507.15061](https://arxiv.org/abs/2507.15061).
- Team [2024]
Qwen Team.
Qwq: Reflect deeply on the boundaries of the unknown, November 2024.
URL [https://qwenlm.github.io/blog/qwq-32b-preview/](https://qwenlm.github.io/blog/qwq-32b-preview/).
- TIGER-AI-Lab [2025]
TIGER-AI-Lab.
Verl-tool: A version of verl to support tool use, 2025.
URL [https://github.com/TIGER-AI-Lab/verl-tool](https://github.com/TIGER-AI-Lab/verl-tool).
Accessed: 2025-08-04.
- Trivedi et al. [2022a]
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal.
Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.
*arXiv preprint arXiv:2212.10509*, 2022a.
- Trivedi et al. [2022b]
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal.
Musique: Multihop questions via single-hop question composition.
*Transactions of the Association for Computational Linguistics*, 10:539–554, 2022b.
- Wang et al. [2025a]
Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji.
Otc: Optimal tool calls via reinforcement learning.
*arXiv e-prints*, pages arXiv–2504, 2025a.
- Wang et al. [2025b]
Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu.
Stepsearch: Igniting llms search ability via step-wise proximal policy optimization.
*arXiv preprint arXiv:2505.15107*, 2025b.
- Wang et al. [2024]
Ziting Wang, Haitao Yuan, Wei Dong, Gao Cong, and Feifei Li.
Corag: A cost-constrained retrieval optimization system for retrieval-augmented generation.
*arXiv preprint arXiv:2411.00744*, 2024.
- Wei et al. [2022]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
*Advances in neural information processing systems*, 35:24824–24837, 2022.
- Wei et al. [2025a]
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese.
Browsecomp: A simple yet challenging benchmark for browsing agents, 2025a.
URL [https://arxiv.org/abs/2504.12516](https://arxiv.org/abs/2504.12516).
- Wei et al. [2025b]
Yifan Wei, Xiaoyan Yu, Yixuan Weng, Tengfei Pan, Angsheng Li, and Li Du.
Autotir: Autonomous tools integrated reasoning via reinforcement learning, 2025b.
URL [https://arxiv.org/abs/2507.21836](https://arxiv.org/abs/2507.21836).
- Wu et al. [2025]
Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Yong Jiang, Pengjun Xie, et al.
Webdancer: Towards autonomous information seeking agency.
*arXiv preprint arXiv:2505.22648*, 2025.
- Xue et al. [2025]
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Zejun Ma, and Bo An.
Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning.
[https://simpletir.notion.site/report](https://simpletir.notion.site/report), 2025.
Notion Blog.
- Yang et al. [2018]
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning.
Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
*arXiv preprint arXiv:1809.09600*, 2018.
- Yao et al. [2023]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
In *International Conference on Learning Representations (ICLR)*, 2023.
- Yu et al. [2025]
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al.
Dapo: An open-source llm reinforcement learning system at scale.
*arXiv preprint arXiv:2503.14476*, 2025.
- Yu et al. [2024]
Tian Yu, Shaolei Zhang, and Yang Feng.
Auto-rag: Autonomous retrieval-augmented generation for large language models.
*arXiv preprint arXiv:2411.19443*, 2024.
- Zeng et al. [2025]
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He.
Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.
*arXiv preprint arXiv:2503.18892*, 2025.
- Zeng et al. [2023]
Zhen Zeng, William Watson, Nicole Cho, Saba Rahimi, Shayleen Reynolds, Tucker Balch, and Manuela Veloso.
Flowmind: automatic workflow generation with llms.
In *Proceedings of the Fourth ACM International Conference on AI in Finance*, pages 73–81, 2023.
- Zhang et al. [2025a]
Dingchu Zhang, Yida Zhao, Jialong Wu, Baixuan Li, Wenbiao Yin, Liwen Zhang, Yong Jiang, Yufeng Li, Kewei Tu, Pengjun Xie, et al.
Evolvesearch: An iterative self-evolving search agent.
*arXiv preprint arXiv:2505.22501*, 2025a.
- Zhang et al. [2024]
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al.
Aflow: Automating agentic workflow generation.
*arXiv preprint arXiv:2410.10762*, 2024.
- Zhang et al. [2025b]
Wenlin Zhang, Xiangyang Li, Kuicai Dong, Yichao Wang, Pengyue Jia, Xiaopeng Li, Yingyi Zhang, Derong Xu, Zhaocheng Du, Huifeng Guo, et al.
Process vs. outcome reward: Which is better for agentic rag reinforcement learning.
*arXiv preprint arXiv:2505.14069*, 2025b.
- Zheng et al. [2023]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.
Judging llm-as-a-judge with mt-bench and chatbot arena.
*Advances in Neural Information Processing Systems*, 36:46595–46623, 2023.
- Zheng et al. [2024a]
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma.
Llamafactory: Unified efficient fine-tuning of 100+ language models.
*arXiv preprint arXiv:2403.13372*, 2024a.
- Zheng et al. [2024b]
Yuxiang Zheng, Shichao Sun, Lin Qiu, Dongyu Ru, Cheng Jiayang, Xuefeng Li, Jifan Lin, Binjie Wang, Yun Luo, Renjie Pan, et al.
Openresearcher: Unleashing ai for accelerated scientific research.
*arXiv preprint arXiv:2408.06941*, 2024b.
- Zheng et al. [2025]
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu.
Deepresearcher: Scaling deep research via reinforcement learning in real-world environments.
*arXiv preprint arXiv:2504.03160*, 2025.
- Zhou et al. [2023]
Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, et al.
Agents: An open-source framework for autonomous language agents.
*arXiv preprint arXiv:2309.07870*, 2023.
- Zhou et al. [2024]
Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, et al.
Symbolic learning enables self-evolving agents.
*arXiv preprint arXiv:2406.18532*, 2024.
- Zhu et al. [2025a]
He Zhu, Tianrui Qin, King Zhu, Heyuan Huang, Yeyi Guan, Jinxiang Xia, Yi Yao, Hanhao Li, Ningning Wang, Pai Liu, Tianhao Peng, Xin Gui, Xiaowan Li, Yuhui Liu, Yuchen Eleanor Jiang, Jun Wang, Changwang Zhang, Xiangru Tang, Ge Zhang, Jian Yang, Minghao Liu, Xitong Gao, Wangchunshu Zhou, and Jiaheng Liu.
Oagents: An empirical study of building effective agents, 2025a.
URL [https://arxiv.org/abs/2506.15741](https://arxiv.org/abs/2506.15741).
- Zhu et al. [2025b]
King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, Changwang Zhang, Chenghua Lin, Jun Wang, Ge Zhang, and Wangchunshu Zhou.
Scaling test-time compute for llm agents, 2025b.
URL [https://arxiv.org/abs/2506.12928](https://arxiv.org/abs/2506.12928).

## Appendix A Tool Agents

### A.1 Web Agent

Our web agent utilizes two types of tool agents: web search and crawl page:

- •
Web search tool agent. We employ a mechanism to access the Google search engine for information retrieval. Specifically, Serpapi(^2^22[https://google.serper.dev/search](https://google.serper.dev/search)) is utilized to execute web search operations. The core parameters configured for Serpapi include the search query string and the specified number of results to be returned. In practice, searches are conducted using queries generated by the model, with the system set to retrieve the top 10 results for each query. Each result contains a title, a snippet, and the corresponding URL. This setup furnishes substantial support for subsequent analytical processes and decision-making actions.
- •
Crawl page tool agent. We implement a tool agent for web page crawling and content summarization. The core configuration parameters for the crawl page tool agent include URLs, web search query, and thinking content. URLs to be crawled are generated by the model, and each URL is crawled for information using Jina(^3^33[https://jina.ai/](https://jina.ai/)). Subsequently, the Qwen2.5-72B-instruct model is employed to produce summaries for each crawled page. The summary prompt is shown in [section˜D.1](#A4.SS1). These page summaries are then concatenated to form the result returned by the tool agent. Notably, we incorporate a requirement in the summary prompt template to retain relevant URLs, which enables the continuous use of the crawl page tool agent for in-depth web searching functionality.

### A.2 Code Agent

To ensure convenience and security, the code sandbox is implemented using nsjail(^4^44[https://github.com/google/nsjail](https://github.com/google/nsjail)), a lightweight tool for creating isolated execution environments for Python code. Nsjail enhances filesystem security via namespace isolation, mitigating unauthorized access to host system resources. A key advantage of this tool is its compatibility with containerized environments (e.g., Docker), enabling seamless migration across diverse training and testing setups. Furthermore, nsjail supports fine-grained resource constraints. For the AFM model, we enforce specific limits: a 5-second CPU time cap and 5 GB memory restriction to ensure controlled execution.

## Appendix B Dynamics Analysis during RL Training of Code Agent

Figure: (a) Training Reward
Refer to caption: /html/2508.13167/assets/x7.png

To analyze the dynamic performance of Code Agent during RL training, we performed a systematic analysis, as shown in [figure˜7](#A2.F7). To ensure stability, we expanded the model’s context length from 16k to 32k at step 40. The experimental results in [figure˜7(c)](#A2.F7.sf3) show a steady improvement in AIME25 accuracy, while the following two key metrics provide further insights:

- •
Training Reward: As shown in [figure˜7(a)](#A2.F7.sf1), the mean critic rewards consistently trend upward during RL training. This indicates a progressive improvement in model performance and demonstrates the stability and effectiveness of the DAPO training strategy.
- •
Response Length: [figure˜7(b)](#A2.F7.sf2) depicts the average response length, which consistently increases across training phases. This trend suggests that the model learns more complex coding strategies and engages in more tool interactions during the RL phase.

## Appendix C Case Study

## Appendix D Prompts

### D.1 Summary Prompt

### D.2 Mathematical problem-solving Prompt

### D.3 Code generation Prompt

### D.4 LLM-as-Judge Prompt

### D.5 Prompt for Agent Generalization Experiments

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: - The term "agent foundation model" is perhaps overstated, as the proposed agents are still task-specific. Can the authors elaborate on this terminology choice? How are agent foundation models different from foundation models that process text, more generically? 
- Note that most of the citations in the paper are in-text, but many of them should actually be parenthetical.
soundness: 3
strengths: - The empirical results of this work are strong, with state-of-the-art performance across a variety of benchmarks across various applications, including deep search, math, and MHQA tasks. 
- The token reduction compared to multi-agent systems is significant, reducing inference cost (as measured in tokens) by an impressive 84%. 
- The ablations suggest that both the SFT and RL components of the training pipeline contribute meaningfully, which is a useful ablation for understanding which components of the proposed technique matter.
summary: This paper proposes an agentic pipeline in which a single model can act as multiple agents to simulate a "multi-agent" workflow. The proposed procedure involves distilling multi-agent traces into a single model, which the authors refer to as an "agent foundation model" (AFM). The proposed chain of agents approach achieves state of the art performance when compared to single-agent tool integrated reasoning (TIR) pipelines.
weaknesses: - The framing around AFMs seems to be undermined, to some degree, by the fact that the proposed AFMs are domain-specific, requiring specialized AFMs (specialization is at odds with the colloquial meaning of the term "foundation model"). Section 5.3 described one example in which cross-domain generalization was observed, but this does not seem like strong enough evidence to claim that the proposed models are general-purpose agentic foundation models. 
- I may have missed something, but it remains unclear to me whether performance gains stem from the chain of agents technique itself or primarily from higher-quality training data obtained via multi-agent distillation and filtering. A prompting-only baseline applying chain of agents format to existing models would help to disentangle these potential contributions.
- The method relies heavily on LLM as a judge, which might raise questions about the scalability of the proposed method -- is the performance ceiling capped by the best available frontier model, for a given type of task?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Is there any intuition on why test-time adaptation performs better with the proposed approach than with the baselines as claimed?  
2. Can the authors clarify where the novelty stands relative to Search-R1?
3. Could the system be made more flexible to accommodate new tools or agent roles? This does not require experiments, but rather a discussion.
soundness: 3
strengths: 1. The main strength of this paper lies in its experimental section. The results are compelling, with strong evaluations against numerous baselines and models at different scales, including 32B ones. These elements collectively support the paper’s claims effectively.
2. The analysis in Section 5 highlights important contributing factors. It is particularly interesting to observe that test-time computation significantly benefits the trained network, although further clarifications would be helpful (see questions below).
3. The manuscript is well organized and clearly written.
summary: The paper presents a method for training a single LLM that integrates tool usage, enabled by data generated by multiple agents. The key idea is to create training data using a multi-agent setup, where each agent has access to specific tools. The interactions of these agents can be represented in the training data through special tokens. This data is then used to perform supervised fine-tuning or RL-based post-training on a single distilled LLM, with an LLM-as-a-judge component providing the reward signal for RL training. The effectiveness of the approach is evaluated across multiple backbones of varying sizes.
weaknesses: 1. The novelty of the work does not appear to be particularly strong; it seems more like a well-executed engineering contribution. My main concern arises from the comparison with Search-R1. The proposed framework resembles an N-agent extension of Search-R1, where multiple agents with different tools replace the single agent equipped with a search tool. While effective in practice, to me this looks to be quite incremental.
2. Another limitation is that distilling all capabilities into a single LLM restricts the integration of new tools, a process that is much more straightforward in traditional roleplay-based agentic setups. Is there any possible way to mitigate this limitation?

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: - Some critical details (training data composition, tool configurations, trajectory sources) are relegated to the appendix; I feel it would be better to include these in main text directly.

- Why use Qwen 2.5 instead of Qwen 3, which was available before submission? Do you expect similar improvements if retrained on the newer backbone?

- For baselines whose results are borrowed rather than rerun, how do you control for changes in web search outcomes and stochastic LLM-as-Judge evaluations?
soundness: 2
strengths: - The paper presents a well-structured design combining multi-role reasoning and domain-specific tool agents under a single model.

- Extensive benchmarks across search, reasoning, math, and coding show consistent improvements.

- The paper provides substantial implementation details and appendices explaining the tool setup and data processing.
summary: The paper proposes Chain-of-Agents (CoA), a framework for integrating multi-agent reasoning and tool use within a single model and execute end-to-end. Instead of running and orchestrating multiple interacting agents, CoA chain agents together and distills successful trajectories from expert multi-agent systems into a unified model.

The resulting system coordinates “role-playing agents” (planner, thinker, reflector, verifier) and “tool agents” (web search, wiki search, crawler, code executor), and finetune a base LLM using SFT/RL with synthesized agent trajectories.

Empirical results on a wide set of benchmarks (GAIA, BrowseComp, HLE, AIME, MATH500, etc.) show strong performance and efficiency gains compared to Tool-Integrated Reasoning (TIR) and multi-agent baselines.
weaknesses: - The main technical contribution is the Chain-of-Agents architecture. The data synthesis pipeline and training (SFT + RL on distilled trajectories) mostly follows well-established practices.

- The paper demonstrates performance gains but offers limited insight into why it outperforms explicit multi-agent systems or where the improvement come from, as well as why sometimes the performance difference is uneven, why in some dataset improvement is significant while in some cases it is marginal. Or the gap between SFT and RL being different across benchmarks.

- The model is trained on Qwen 2.5, released in September 2024, even though Qwen 3 was public several months before submission (April 2025). This raises questions about the timeliness and potential performance ceiling.

- Some reported numbers are taken directly from prior work rather than rerun. For baselines that depend on web search results or use LLM-as-Judge scoring, direct reuse could compromise strict comparability as they can change over time (search results drift, judgment variance),


Decision: Reject