In-the-Flow Agentic System Optimization for Effective Planning and Tool Use
Authors: Zhuofeng Li
Haoxiang Zhang
Seungju Han
Sheng Liu
Jianwen Xie
Yu Zhang
Yejin Choi
James Zou
Pan Lu
Venue: ICLR 2026 Oral (2026)
Keywords: Reinforcement Learning
Large Language Models
Agentic Systems
Tool Use
Planning
On-policy Optimization
Sparse Rewards
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce AgentFlow, a trainable agentic system, and Flow-GRPO, an on-policy RL algorithm that optimizes the planner "in-the-flow" by broadcasting a final outcome reward to all steps, enabling effective long-horizon planning and tool use.

Abstract: Outcome-driven reinforcement learning has advanced reasoning in large language models (LLMs), but prevailing tool-augmented approaches train a single, monolithic policy that interleaves thoughts and tool calls under full context; this scales poorly with long horizons and diverse tools and generalizes weakly to new scenarios. Agentic systems offer a promising alternative by decomposing work across specialized modules, yet most remain training-free or rely on offline training decoupled from the live dynamics of multi-turn interaction. We introduce AgentFlow, a trainable, *in-the-flow* agentic framework that coordinates four modules (planner, executor, verifier, generator) through an evolving memory and directly optimizes its planner inside the multi-turn loop. To train on-policy in live environments, we propose *Flow-based Group Refined Policy Optimization* (Flow-GRPO), which tackles long-horizon, sparse-reward credit assignment by converting multi-turn optimization into a sequence of tractable single-turn policy updates. It broadcasts a single, verifiable trajectory-level outcome to every turn to align local planner decisions with global success and stabilizes learning with group-normalized advantages. Across ten benchmarks, AgentFlow with a 7B-scale backbone outperforms top-performing baselines with average accuracy gains of 14.9% on search, 14.0% on agentic, 14.5% on mathematical, and 4.1% on scientific tasks, even surpassing larger proprietary models like GPT-4o. Further analyses confirm the benefits of in-the-flow optimization, showing improved planning, enhanced tool-calling reliability, and positive scaling with model size and reasoning turns.

Published as a conference paper at ICLR 2026
IH-THE-FLOW AGENTIC SYSTEM OPTIMIZATION FOR EFFECTIVE PLANNING AND TOOL USE
Zhuofeng Li Haoxiang Zhang*1.3, Seungju Han', Sheng Liu', Jianwen Xie
Yu Zhang?. Yejin Choi', James Zou'l, Pan Lu
1 Stanford University, "Texas A&M University, 'UC San Diego, *Lambda
Website: bstps://agentfleee stanford.
Code
Model
Demo
Visualize
ABSTRACT
Outcome-driven reinforcement learning has advanced reasoning in large language models (LLMs), but prevailing tool-augmented approaches train a single, mono- lithic policy that interleaves thoughts and tool calls under full context; this scales poorly with long horizons and diverse tools and generalizes weakly to new scenar- ios. Agentic systems offer promising alternative by decomposing work across specialized modules, yet most remain training-free or rely on offline training de- coupled from the live dynamics of multi-turn interaction. We introduce AGENT- FLOW. a trainable, in-the-flow agentic framework that coordinates four modules (planner, executor, verifier, generator) through an evolving memory and directly optimizes its planner inside the multi-turn loop. To train on-policy in live environ- ments, we propose Flow-based Group Refined Policy Optimization which tackles long-horizon, sparse-reward credit assignment by converting multi- turn optimization into sequence of tractable single-turn policy updates. broad- casts single, verifiable trajectory-level outcome to every turn to align local plan- ner decisions with global success and stabilizes learning with group- group-normalized advantages. Across ten benchmarks, AGENTFL.OW with 7B-scale backbone outperforms top-performing baselines with average accuracy gains of 14.9% on search, 14.0% agentic, 14.5% mathematical, and 4.1 % scientific tasks, even surpassing larger proprietary models like GPT-4o. Further analyses confirm the benefits of in-the-flow optimization, showing improved planning, enhanced tool-calling reliability. and positive scaling with model size and reasoning turns.
AgantFlow (wlo Flaw-GRPO)
AgontFlow
2Wiki (Search)
HotpotQA (Search)
GAIA (Agentic)
Bamboogle
+7.0%
MedOA
50.0
80.8
2Wiki
Accuraay (N6)
47.6
+10.1% Search
6
20
Solence
76.0
58.4
23.0
GPQA
37.0
51.3
Qwen-255-75
GFI-4e (-000)
Search-ff (71)
ReeSearch (71)
19.2
TIR(7))
ToRi,
AutoGen
AgentFlow (78)
63.0
31.0
16.7
26.3
AIME24 (Math)
Game0124 (Math)
GPQA (Sclence)
Gamedk24
47.4
Musique
Assuracy (96)
52
51.0
$2.5
33.1
+19.8%
40.0
GAIA
+15.9%
67
20.0
23
33.0 52.0
33
20
Agenic
AIMEZ4
Figure Left: Performance of AGENTFLOW with a 7B-scale backbone before and after Flow- GRPO tuning across ten diverse reasoning benchmarks. Flow-GRPO substantially improves per- formance by enhancing planning quality and tool-calling reliability. Right: AGENTFLOw achieves consistent gains top baselines, including base LLMs, tool-integrated RL models, and training- free agentic systems. All 7B results use Qwen2.5-775 7 as the backbone and tools.
*Equal contribution. CCo-senior authors. Work was partially done while and HZ were visiting Stanford.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent advances in large language models (LLMs) have unlocked remarkable reasoning capabilities, largely driven by reinforcement learning (RL) from antcome-based feedback. By fine-tuning models to maximize verifiable rewards, L.LMs like DeepSeek-RI (Guo et al., 2025) and SimpleRI. (Zeng et al., 2025b) have demonstrated sophisticated behaviors in self-correction and multi-step deduction.
complementary line of work augments LLMs with external tools (e.g., web search, code exe- cution) for knowledge retrieval and precise computation. Tool-integrated reasoning (TIR) extends reinforcement learning with verifiable rewards to learn when and how to call tools by interleav- ing reasoning (e.g., <think>) with tool invocations (e.g.. <tool call>) under full context (Jin et al., 2025; Song et al., 2025; Chen et al., 2025; Feng et al., 2025). Early systems supported only single tool type, whereas recent work enables multi-tool settings by encoding tool metadata into prompts (Dong et al., 2025; Qian et al., 2025a; Zhang et al., 2025). However, these methods still train a single, monolithic policy under multi-turn full-context reasoning. which introduces scaling challenges: (i) training becomes increasingly unstable as horizons lengthen, tool diversity grows, and environments shift with tool feedback (Wang et al., 2025c; Mai et al., 2025; Moonshot AI, 2025; Xue et al., 2025); and (ii) inference-tmme generalization remains brittle to unseen tasks or tools (Dong al., 2025; Hu al., 2025b).
Agentic systems (Wu et al.. 2024; Hong et al., 2024; Hu et al., 2025b) offer promising alter- native to monolithic tool-integrated reasoning models. They consist of multiple modules-often distinct LLMs with prescribed roles (e.g., planner, critic) or specialized components with dedicated tools and capabilities (e.g., executor, coder) coordinate via shared memory and moter- communication. By decomposing problems into sub-goals and iterating over multiple turns, these systems can tackle tasks that demand diverse tools, long horizons, or multi-stage reasoning. How- ever, achieving robust coordination in such systems ultimately requires training, since handcrafted logic or static prompting cannot reliably capture when and how modules should collaborate, adapt to evolving tool outputs, recover from early mistakes. At the same time, they introduce new training challenges: modules coordinate sequentially, outcome feedback propagates through long reasoning chains, and state distributions shift with evolving tool outputs. As a result, most systems remain training-free, relying on handcrafted logic or prompting heuristics. While some employ supervised fine-tuning or preference optimization for key modules (Motwani et al., 2024; Park et al., 2025), these off- tfflcolcyy approaches are decoupled from live dynamics and learn poorly from downstream successes or failures. Thus, agentic systems struggle with sparse rewards, brittle adaptation, and inefficient orchestration in dynamic environments.
To address the central challenge of learning long-horizon reasoning with sparse rewards in tool- integrated agentic systems, we introduce AGENTFLOW, trainable framework for effective plan- ning and tool use (Figure 2). AGENTFLOW comprises four specialized modules-plannee executor, verifier, and generator-that interact iteratively over multiple turns via shared evolving memory and a toolset. The system operates in the flow, with each turn cycling through planning, execu- tion, and verification. Unlike prior agentic systems, AGENTFLOW directly optimizes its planner on-policy, inside the live multi-turn loop, allowing it to dynamically adapt to trajectories shaped by tool calls, verifier signals, and memory updates. This evolving memory serves as a deterministic, structured record of the reasoning process, enabling transparent state tracking, controllable behavior, and bounded context growth.
To train the planner on-policy within this agentic system, we need to overcome the long-horizon credit assignment problem inherent to sparse, trajectory-level rewards. We introduce Flow-based Group Refined Policy Optimization (Flow-GRPO, Figure 4), an on-policy algorithm designed for this setting. GRPO operates on in-the-flow rollouts, which capture the full trajectory of states, actions, and tool events induced by the live system. Instead of attempting to assign credit with brit- tle, intermediate heuristics, we assign a single, verifiable final-outcome reward to the entire trajec- tory and broadcast it every turn. This design effectively transforms the multi-turn reinforcement learning challenge into a series of single-tum updates: at each turn, the planner has access to the full memory context and receives consistent reward signal aligned with global success. This approach, coupled with group- normalzomel advantages to stabilize training, enables robust credit assignment and allows the planner learn effective long-horizar strategies from sparse feedback.
We evaluate AGENTFLOW on ten benchmarks across diverse reasoning domains, as results high- lighted in Figure 1. AGENTFL.OW substantially outperforms top-performing specialized tool-
2

---

Published as a conference paper at ICLR 2026
Query
Toolkit Set
Planner
Executoo
Verifier
Planner io
Executor
Verifier
Memory
7um
M
Planner
Executor
Verifier
Tepat:
Tepent
Eggtt
(Guery drahsiu]
[carrun (oaa))
Deneratss Command)
Tinn
Planner
Exceutor
Verifier
Generator
[Peqaret Siela]]
Canhort)
[Teel Mettodaro
Outgut:
Trained
(Gureet) bub Geel) [selected Teal)
Outpen:
(Menery Aeahaiss
(Goneratid Commund))
[Venf ca isn Stanus]
Memory
Answer
Uee]
Frozen
(a) AgentFlow:
In-the-Flow Agentic System
(b) In-the-Flow Rollout at Turn
Figure 2: (a) Overview of AGENTFLOW, a trainable agentic system for in-the-flow planning and tool use. Four modules (planner, executor, verifier, generator) coordinate via a shared evolving memory M and toolset K, given a query q. The planner policy is optimized on-policy inside the system's multi-turn loop enable adaptive, long-horizon reasoning. (b) A single state transition, showing the action a", execution result e", and verifier signal v that update the memory from M' to M
integrated reasoning models and agentic systems, achieving average accuracy by 14.9% on snowledge-intensive search, 14.0% on broader agentic tasks, 14.5% mathematical reasoning, and 4.1% on scientific reasoning ($4.2). Notably, our 7B-backbone system even surpasses the ~200B parameter GPT-4o (Hurst et al., 2024) across all domains. Further analyses confirm that our in- the-flow optimization with Flow-GRPO is crucial, far surpassing offline supervised tuning ($4.3). The trained planner learns to optimize planning, enhance tool-calling reliability, and discover ef- fective solution pathways ($4.5). Moreover, our training approach proves highly efficient, leading to increased rewards and condensed responses compared to traditional tool-integrated RL methods ($4.6). Finally, we demonstrate that these benefits generalize, with consistent gains from scaling backbone size and turn budget ($4.4).
Our work makes three key contributions: (1) We present AGENTFLOW, a trainable in-the-flow agen- tic system that directly optimizes its planner inside the multi-turn loop. By coordinating specialized modules through an evolving memory, it enables adaptive Long-horizon planning and robust tool orchestration. (2) We introduce Flow-GRPO, an on-policy, outcome -riienn algorithm that hat con- verts multi-curn RL into sequence of tractable single-turn policy updates by broudcasting a sin- gle, verifiable final-outcome reward to every turn. (3) Through comprehensive experiments on ten benchmarks, we show that AGENTFLOW with a 7B backbone outperforms specialized baselines and even larger proprictary models. Further analyses reveal improved planning, enhanced tool-calling reliability, and positive scaling with model size and turn budgets.
PRELIMINARY
Reinforcement learning for reasoning LLMs. Recent progress reasoning LLMs has been sig- nificantly driven by reinforcement learning from outcome feedback, using a verifiable reward sig- nal (Shao et al., 2024: Yu et al., 2025). This paradigm fine-tunes a language model to maximize an outeome-based reward while remaining close a reference policy. Formally, the objective is to optimize a policy LLM Ty to generate a response for a given query from dataset D:
max Eyg:D, owwea(-lg) [R(g, o)] Tix ( ()) mer(o g)); TT
where Ri(g o) is the outcome-based reward, Tef is reference model to prevent policy collapse, and controls KL regularization. Algorithms like Group Relative Policy Optimization (GRPO) (Shao et al., 2024) implement this by sampling groups of responses, normalizing advantages by their re- wards, and updating the policy with a clipped objective to encourage high-reward outputs.
Tool inol-integrated reasoning models (LLM agents). LLMs can be augmented with external tools to access knowledge and perform precise computation under reinforcement learning with outcome- based reward. As shown in Figure 3(a), the LLM interleaves reasoning and tool calls, produc- ing a chain of thought within <think></thine The resulting trajectory is sequence of model generations and tool observations: -= {ss .c'....,s" "".a" where s' denotes the context, a' the generated action (thought tool call), and et the tool's execution result. The policy model лo is then trained to maximize a final outcome reward. Prior work has explored single- and multi-tool settings for search and code execution (Jin et al., 2025; Chen al., 2025; Feng al., 2025; Qian et al., 2025a).
tokens followed by tool invocations (c.g..
3

---

Published as a conference paper at ICLR 2026
Query
Query
Query
Query
LLM
<oools> </tools>
LLM
Module
Module
Module
Truined
-thinko </thin
thiinoo </think
Module
Module 3
Fr Frozen
ftol__call Laclllllllco.o. /atool_coll
-tooL_calb </tlol_caall
token
<onswer> </onswer>
</onswer>
Module
Module N
Answer
Answer
0
Answer
Answer
(a) Tool-Integrated Reasoning Models (LLM Agents) (b) Training-Free Agentic Systems
Figure 3: Comparison of two paradigms of LLMs with tool use. (a) Monolithic cool-integrated reasoning models train single policy to interleave reasoning (egg,, <think>) and tool calls (e.g., <tool_call>) within single, full-context trajectory. (b) Agentic systems decompose tasks across multiple specialized modules (e.g., planner, coder) that collaborate. These systems are typically traening-free, orchestrated by handcrafted logic prompting.
Agentic systems with tool usage. An alternative approach is the use of agentic systems (Wu et al., 2024; Hong et al., 2024; Lu et al., 2025). As shown in Figure 3(b), these frameworks deploy mul- tiple specialized modules-off a collaborative workflow. By decomposing tasks and assigning subproblems to modules with dedi- cated tools and capabilities (e.g.. planner, coder, critic), they can address complex problems such as web browsing, document processing, and multi-stage programming that exceed the scope of a sin- gle model. central limitation, however, is that these systems are typically training-free modules remain frozen pre-trained models orchestrated by handcrafted logic or prompting heuristics.
distinet L.MMs with carefully designed prompts and -itthn
IN-THE-FLOW AGENTIC SYSTEM OPTIMIZATION
We aim to bridge the gap between trainable but monolithic reasoning models and flexible yet static agentic systems. We present AGENTFLOW, flexible and trainable agentic system that integrates four specialized modules with an evolving memory ($3.1). Unlike prior agentic systems, AGENT- FLOW directly optimizes the planner within the multi-turn loop of an agentic system ($3.2).
3.1 AGENTFLOW: AN IN-THE- FLOW AGENTIC SYSTEM
We propose AGENTFLOW, a general general-purpose purpose tool-integrated agentic framework for solving com- plex reasoning tasks through fine-grained planning and effective tool within multi-turn archi- tecture. As shown in Figure 2, the framework comprises four specialized coaduules-Action Planner P. Tool Executor E, Execution Verifier V. and Solution Generator G co-odnniiatted by shared evolving memory M and a toolset K. These modules interact sequentially and iteratively to per- form action planning, tool execution, context verification, and solution generation, thereby enabling tool-integrated reasoning across multiple turns.
We formalize AGENTFLOW's problem-solving process as multi-turn Markov Decision Process (MDP). Given a query and toolset K, the system proceeds for a variable number of turns. Let M denote the memory state before turn (with M initialized from q). At turn t, the planner P (a trainable policy Ta) formulates a sub-goal, selects an appropriate tool k E K, and retrieves relevant context from memory, producing an action: a' To(a" q, M*).
The executor È invokes the chosen tool with context, yielding execution observation el E(e" a', K). The verifier then evaluates whether et is valid and whether the accumulated memory sufficient to solve the query, producing binary verification signal of V q. M*) If of 0, the memory is updated deterministically to incorporate new evidence: M = fmen( at, e", v²), where fmem (.) denotes the memory-updas function, which records agent-process information in concise, structured form along with contextual details such as time, turn index, and error signals.
The process repeats until of (termination) predefined maximum turn budget is reached. Upon termination at turn T, the solution generator & produces the final solution o, conditioned on the query and the accumulated memory: G(o q, MT).
4

---

Published as a conference paper at ICLR 2026
Flow-GRPO
Policy Model
g
M
XG
Reward
Model
Multi- car Group Computation
Referemoc Model
K
Trained
Models
Medels
OC
Multi-tum Agentic System
Figure 4: Optimization for our proposed agentic system AGENTFLOW. Given a query q, an evolving memory M, and toolset K, the policy model generates actions that target sub-goals and select tools. It is trained via Flow-based Group Refined Policy Optimization (Flow-GRPO), which enables multi-turn reinforcement learning and stable optimization under collaborative dynamics.
This formulation decomposes multi-ture turn, tool-integrated reasoning into structured, observable tran- sitions. After I' turns, the trajectory {(aa, e', )²))== records the history of planning, execution, and verification. The joint generative process can be written as
po (fa' e', E' o qg K)
II me(a" 1 K, K,) E(e' I K) V(v' g M²) G C M.M,
(2)
where {a', o' thoughts behind trajectories, our memory M is an explicit and deterministic record of the reasoning process, ensuring transparency and controllability of multi-turn decisions.
Jt=1 are explicit realizations of the latent reasoning chain. Importantly, unlike latent
3.2 IN-THE- THE-TH-FOOT FLOW REINFORCEMENT LEARNING OPTIMIZATION
We target tool-integrated agentic systems operating under long-horizon tasks with sparse rewards. In this setting, the Action Planner (the trainable policy AGENTFLOW) selects a sequence of interdependent actions while the state (q, K, M*) evolves with tool results and verifier feedback. Conventional offline training-ee e.g., supervised fine-tuning or preference fine-tuning on curated optcee-e the planner outside the active loop (Motwani et al., 2024; Park al., 2025). This decoupling prevents real-time coordination with the executor, verifier, and solution generator, induces distribution shift between training and deployment, and provides limited guidance about which intermediate decisions truly matter. As a result, planners often adapt poorly to multi-turn dynamics; early errors cascade, and post-hoc fixes are brittle.
In-the-flow learning. To address these issues, we optimize the planner in the flow of execution. We roll the full AGENTFLOW system under the current policy, collect the actual trajectory of states, actions, and tool events it induces, and update the policy within the agentic system using verifiable final-outcome signal. This exposes the multi-turn credit-assignment problem directly and trains the planner on the exact states it will face at inference. Our objective, Flow-GRPO, is designed to stabilize learning under sparse, trajectory-Ievel rewards over multiple turns.
As established in $3.1, rollouts in AGENTFLOW define fnite-horizon MDP with variable horizon T. At turn t, the planner observes the state (q. K, M"). selects an action a', the executor and verifier return (e', v²) and the memory updates deterministically to M
The planner policy me is trained to maximize the expected return over on-policy rollouts. Let R(T) be the reward for complete trajectory т. The objective is:
Policy optimization objective.
J(0) E R(T)],
#* arg max [()).
where a rollout is the sequence of decisions (T generated on-policy by øg.
Final-outcome reward. Assigning credit to intermediate actions is challenging because each a' influences the final solution only indirectly, and their value may only emerge after several turns (c.g., error or improvement accumulation). To avoid brittle local feedback, we adopt a final-outcome- based reward: every action within rollout receives the same global reward signal, based on the correctness of the final solution with respect to query and ground truth y":
R(a') R(o,g,v")) =t= 1,.....,77
5

---

Published as a conference paper at ICLR 2026
{ is assigned by LLM-as-judge rubric for semantic, numeric, and option-level equivalence (see $E.3). This propagates a trajectory-leve. success signal back through the reasoning chain, aligning every decision a' with global correctness.
where R(o. y²)
Objective fuunction. We formalize Flow-based Group Refined Policy Optimization for the plan- ner. The goal is to optimize the policy тe by maximizing the expected return over group of parallel rollouts. For each query-label pair from training corpus (g, y
'D, we sample a group of G on- policy trajectories {T by running the current behavior policy Tew inside AGENTFLOW, where -!.,,...., a. tion (a token sequence of length (a;D. D. and og the final response. This structure is key to addressing the long-horizon credit assignment challenge: by broadcasting single trajectory-level reward to all turns, effectively decompose the mulli-turn RL problem into set of independent, single-turn policy updates; we provide formal proof of this equivalence and analyze convergence properties in $B. Each update for an action af is conditioned on the full historical context encapsulated in the state sy and receives the same global success signal, simplifying optimization. The objective is
......,,'.ox, ox). Let sf = (g, K, M2) be the state at turn ( of rollout i, at the planner's ac-
Jown (0) = I (' JwD, (rd) a0
T5
[
Σ
min pisf clip(oks, 1- (6) DKi( )eत)
(5)
where T3 is the (variable) number of turns in rollout i, and
Te(ai,j s ((,3-1))))) Pl.y = (eo 8// 11.1
(6)
is the token-level importance ratio for the j-th token of a;, e >>>>> is the PPO clipping parameter, and controls the KL penalty fixed reference policy Tel..
Group-normalized advantages. Because the reward in Eq. 4 is single trajectory-level signal, the per-turn advantage A is constant over t within a rollout i. We reduce variance and sharpen credit assignment across the group by using group-normalizel advantage:
A R(oi q v,) - mean ((R(OK, 4, 4, (1
std ((R(ok: 9: y² )=
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
In our main experiments, all modules-Action Planner, Tool Executor, Executive Verifier, and Solu- tion Generator- instantiated with the Qwen2.5-7B IIIntrutt model (Yang et al., 2024a). Among these, only the Action Planner is trainable. The system operates with five interactive tools: Base Generator is an instance of Qwen2.5-7B-Inucee planner decides not to use an external tool; Python Coder generates and executes Python code given a query and returns the execution result; Google Search searches the web and returns summariza- tion of Top-K search results; Wikipedia Search searches articles matching given query and returns a summarization; and Web Search returns summarized information from given web page. Dur- ing the RL fine-tuning phase, we mix data from Search-R1 (Jin et al., 2025) and DeepMath (He al.. 2025) as training data, which provides paired question-anss answer examples across search and mathematical domains. We use a batch size of 32 with 8 rollouts per sample.
that acts as the default reasoning engine if the
To comprehensively evaluate tool-use capabilities of AGENTFLOOW we conduct experiments on four types of reasoning tasks: (1) bhowleddg-intenssee
search including Bamboogle (Press et al., 2023), 2Wiki (Ho et al., 2020), HotpoiQA (Yang et al., 2018), and Musique (Trivedi et al., 2022); (2) Agen- tic reasoning such as GAIA (Mialon et al., 2023) (where we adopt the textual split); (3) Logic-dense mathematical reasoning including AIME2024 (Art of Problem Solving, 2025), AMC23 (MAA, 2023), and GameOf24 (Lightman et al., 2023); and (4) Scientific reasoning including GPQA (Rein et al., 2024) and MedQA (Yang et al., 2024c). To mitigate randomness, we report the average accu- racy across three trials for all experiments. More experimental details in SC.

---

Published as a conference paper at ICLR 2026
Search Intensive
Agentic
Model
Size
Bamboogle
2Wiki
HotpotQA
Musique
Avg.
A
GAIA
A
Qwen-2.5-7B-Inuse
7B-Inst
12.0
23.0
21.0
6.0
15.5
41.8
3.2
29.9
Dwen-2.5-14PB 1.st-.ie----
14B-Inst
21.6
26.7
20.0
8.0
19.1
38.2
5.5
27.6
Owen--2.-328-Inssttu
32B-Inst
24.0
26.7
27.0
6.0
20.9
36.4
9.5
[ 233.6
Llama-3.3-70B-Isttae
70B-Inst
18.4
22.7
52.0
16.0
27.3
30.0
3.2
29.9
GPT-44--mini (Hurst al., 2024)
~8B
40.8
35.6
41.0
15.0
33.1
24.2
7.1
26.0
GPT-4o (Hurst et al., 2024)
200B
68.8
49.5
54.0
24.0
49.1
8.2
17.3
1 15.8
Supervised TinenT (SFT)
7B-Inst
12.0
25.9
22.0
6.6
16.6
40.7
3.2
29.9
Ite-ReiGen (Shao et al., 2023)
7B-Inst
36.8
33.6
37.4
17.8
31.4
1 25.9
3.9
29.2
Search-RI R1 (Jin
al., 2025)
7B-Inst
43.2
38.2
37.0
14.6
33.3
24.0
19.1
14.0
ZeroSearch (Sun et al., 2025)
7B-Base
27.8
35.2
34.6
18.0
28.9
28.4
16.5
16.6
ReSeurch (Chen al., 2025)
7B-Basc
42.4
47.6
43.5
22.3
39.0
18.3
17.3
15.8
StepSearch (Wang et al., 2025d)
7B-Base
40.0
36.6
38.6
22.6
34.5
22.8
VerlTool (Jiang al., 2025)
7B
45.3
19.3
39.0
18.3
11.2
AutoGen (Wa ul., 2024)
7B-Inst
59.6
44.0
50.0
15.9
42.4
14.9
6.3
26.8
AGENTFLOOW
In-Inst
58.4
60.0
51.3
19.2
47.2
12.1
17.2
15.9
AGENTFLOW (w/ Flow-GRPP)
7B-Inst
69.6
77.2
$7.0
25.3
57.3
33.1
Table I: Accuracy comparison search-intensive and agentic tasks. 7B-Base refers Qwen-
2.5-7B-Bas and 7B-Inst refers to Qwen-2.5-7B-Inntruue. AutoGen and our AGENTFLOOW method agentic systems, which use Qwen-2.5-7BB-nnttuu for the L.L.-powered agents and tools for fair comparison. We visualize the gains of AGENTFLOW the each baseline the A columns
Math Reasoning
Scientific Reasoning
Model
Size
AIME24 AMC23
GameO124
Avg.
A
| GPQA
MedQA
Avg.
A
Owen-2.5-7B-Innstt
7B-Inst
6.7
47.5
29.1
22.5
34.0
66.0
50.0
13.5
14B-Inst
6.7
60.0
25.0
30.6
21.0
31.0
75.0
53.0
10.5
Jlama-3.3-708-Instrucc 1Jamn-3. 1-0--.11-405--mstreee
70B-Inst
6.7
47.5
31.0
28.4
23.1
35.0
67.0
51.0
12:55
405B-Inst
26.7
47.5
23.0
32.4
19.1
30.0
62.0
46.0
17.5
GPT-4o-mini (Hurst et al., 2024)
8B
13.3
57.5
16.0
28.9
22.6
27.0
66.0
46.5
17.0
GPT-4o (Hurst et al., 2024)
200B
13.3
60.0
32.0
35.1
16.4
31.0
60.0
45.5
18.0
Supervised Fine- Tuninn (SFT)
BB-Ins
6.7
47.5
33.0
29.1
22:5
34.0
66.0
50.0
13.5
SimpleRL-reason iZeng et al., 2025b)
7B-Basc
16.7
60.0
33.0
36.6
15:0
45.0
65.0
50.0
13.5
Seat-RRessneee-Zeee (Hu et al., 2025a)
7B Base
16.7
54.9
32.0
34.5
17.0
34,0
54.0
44,0
19.5
General Reasoner
al., 2025)
BB-Base
13.3
33.8
17.7
35.5
61.0
48.3
15.2
Lufly (Yan et al., 2025)
7B BBst
30.7
44.8
33.0
36.2
15.3
34.0
77.0
55.5
8.0
TIR (Yang et al., 2024h)
In-
10.0
50.0
33.0
31.0
20.5
42.0
76.8
59.4
4.1
ToRL. (I.i et al., 2025b)
7B Inst
20.0
60.0
31.0
37.0
14.5
35.0
76.5
55.8
7.7
AutoGen (Wu et al., 2024)
7B Inst
13.3
57.5
24.0
31.6
19.9
42.0
72.0
57.0
1 6.5
AGENTFLOOW
BB-Inst
16.7
47.4
31.0
31.7
19.8
37.0
76.0
56.5
7.0
AGENTFLOW (w/ Flow GRPO)
7B Inst
40.0
61.5
53,0
51.5
47.9
80.0
63.5
Table 2: Accuracy comparison of mathematical and scientific reasoning tasks.
4.2 MAIN RESULTS
Baselines. As presented in Tables and 2, we include five categories of baselines: (1) Open- source LLMs: Qwen2.5 (Yang et al., 2024a). Llama-3.1. and Llama-3.3 (Dubey et al., 2024); (2) Proprietary L.LMs: GPT-4o-mini and GPT-4o: (3) Reasoning I.LMs: supervised fine-tuning (Yang et al., 2024b), SimpleRl.-reason Open- Open-RRaasonr-7Zerr, General-Reasonee and LUFFY,; (4) Tool- integrated reasoning LLMs: both search-enhanced, including RettRn, Search-R1, ZeroSearch, ReSearch, StepSearch, and VerlTool, and code-enhanced, including TIR and ToRL; (5) Training- agentic system: AutoGen. More details on baseline implementations are in SC.3.
Key insights. AGENTFLOW consistently outperforms all baseline models by large margins. Com- pared to the best-performing 7B models without tool integration, AGENTFLOW achieves absolute gains of 70.7% on search (SFT), 29.9% on agentic reasoning (SFT), 15.0% on math (SimpleRL- reason), and 8.0% on scientific tasks (Luffyy. Against specialized tool-integrated systems, AGENT- FLOW surpasses the top models by 14.9% in search (AutoGen), 14.0% in agentic reasoning (Search- RI), 14.5% in math (ToRL), and 4.1% in science (TIR). Notably, our 7B- baccbbooon AGENTFLOW even outperforms the 22000--araamter GPT-4o across all domains, with gains ranging from 8.2% to 18.0%. A detailed analysis is provided in I.
7

---

Published as a conference paper at ICLR 2026
Quertion: check digt Troprcos the Order Ground answer:
Selected tool: Wikppdia Search [Truncoted...) Result: found.
x
Selected tool: Google Scorch (Truncated...) Result: Tropicos 100370510.
Selected tool: Wilipcddaa Search
Sub goal: Recrivve Tropicos ID of Order Helottales from Wikipedo
Selected took: Python Coder
Tool call: querys" Tropicas to for the Order Helotiales"
goal: and exccute Python script to calculate the check digtt for tropicos_
Result: No results found for query: Trapicos for Order Helotales
se 6SN-10 number
Tool quarye" Calcolate the check dight for the LSBN-10 number frapicos_id if
Result: Error in execute, commond nd: name tropicos_ id not defined.
Selected tool: Geogle Search
Sub-gor flstriew the
the Order.
call: quary="Tropicoe TD
Helatiales"
Selected tool: Python Coder
Result: Tropicos for Order Helodales 100970510.
Sub goal: Write and excoute Python scrpp to calculate the check dight for
"tropicos, if iftt were an EBN 10 number.
Tool call: Compuut the check dight for 100370510 1SBN--1 number" Result (printed output).
Selected tool: Python Coder
Sub gool: Calculate the check figgtfor the Tropicos ID 100370510
dightiion
it were an ISNN -10 number
ilbn, sight (inikag)) for digit in isbn-91]]
* Tool call: query= Calculate check dight for 158N-10 10 number 100370610"
foldl Jum sumpposition * -dgit for paùi fon, dig in enumerelediin.diity startv1]) check_igh toto Jum % 11
Result: Isbel not delined.
if check digit 10:
neturn
Action stops 3 9 have simtler sub goots, commondss and results: the agent to stuck enorr loops and unable fox the naminn issue.
else:
noult str(chsck_dggh
digit 100570510")
print(" the check dig is (neud))))
Solution: The Python tool failed repestedly with the same 'shni val abble emor). The agent faiked to calculate the clack digi.
Solution: The check digit is 3, resuling in the number 1003705103.
AgentFlow (before Flow-GRPO Fine-tuning)
AgentFlow (after Flow-GRPO Fine-tuning)
One case study example. Initially failed with repetitive errors (left), AGENTFLOW, trained with Flow-GRPO, explores new solution pathway at turn 4 after two failed attempts (right).
Figure 5:
4.3 TRAINING STRATEGIES ON THE PLANNER
We conduct an ablation study to analyze the impact of different training strategies for the Action Planner module in AGENTFLOW, with results reported in Table 3. The executor, verifier, and gen- erator modules remain fixed as Swvn2.5-7B-Innstuue consistent with our main setup ($4.1).
Planner Model
Training
Bamboogle
2Wiki
GAIA
AIME24
AMC23
GameOr24 Avg.
Qwen-2.5-7B
Frozen
58.4
60.0
17.2
16.7
47.4
31.0
38.5
GPT-4o
Frozen
65.0 6.6
70.0 10.0
23.6
16.7 0.0
48.7 1.
42.0 r 11.0
44.3
Qwen-2.5-7B
SFT
30.4 28.0
32.7
27.3
6.3
10.9
3.3
13.4
37.5
7.0 24.0
19.5 ; 19.0
Qwen-2.5-7B
Flow-GRPO
19.6,1122 11.2
77.2 17.2
33.1 1 15.9
40.0 23.3
14.1
53.0 q 22.0
55.7 f 17.22
Table 3: Performance comparison of AGENTFL.OW across different training methods.
A more capable planner is beneficial, but has limits. Replacing the frozen Breen..5-7B-nstrree baseline with a stronger proprietary model, GPT-4,, yields only a modest 5.8% average gain. This indicates key bottleneck that, while more powerful model improves planning, its static nature prevents co-adaptation with the live dynamics of AGENTFLOW.
Offline SFT leads to performance collapse, while in-the-flow RL is crucial. The limitations of a static planner are further exposed when distilling GPT-4o's behavior via offline supervised fine- tuning (SFT) on its trajectories as Action Planner in AGENTFLOW This results in a catastrophic performance collapse, with an average accuracy drop of 19.0% compared to the frozen baseline. This failure arises from the token-level imitation objective of SFT, which misaligns with trajectory- level task success and prevents the planner from adapting to dynamic tool feedback or recovering from compounding errors. In contrast, training the planner with our on-policy Flow-GRPO method proves highly effective: by optimizing for the final outcome, the planner learns to handle long- horizon workflows, achieving a 17.2% average gain over the frozen baseline.
4.4 SCALING TRENDS IN AGENTFLOW
Training scaling in backbone size. We study how backbone LLM scale affects AGENTFLOW's performance and the efficacy of Flow-GRPO. We build two versions of the system: one using Qwen2.5-3B-Instruc and another using Qwen2.5-7B-lnstrucl for all four modules (planner, execu- tor, verifier, and generator) and tools. In both, only the planner is fine-tuned with Flow-GRPO. As shown in Figure 6, Flow-GRPO fine-tuning consistently improves performance across tasks for both backbones. This demonstrates that our in-the-flow optimization is effective across model capacities, enhancing AGENTFLOW regardless of LLM size.
8

---

Published as conference paper at ICLR 2026
AgentFlow 2et-22.5----sssuuuee
AgentFlow Soven-2.5-78-instruue
80
tuning After tuning
80
tuning After wning
Accuracy (16)
60
53.6
68.8
60
58.4
60.0
Accuracy (%)
60
Cammor24
60-
GNA
+16.7%)
20
13.3
20.0
20
17.2
16.7
6.63%
29
10
Bambooggee 2Wiki
GAIA
AJM224
Bamboogle 2Wiki
GAAA
AIME24
Max Allowed Turns
Figure 6: Flow-GRPO fine-tuning offers consistent gains on Figure 7:
Average accuracy
AGENTELOW as the backbone model size scales from 3B 7B.
with increased T max-
Inference scaling in turn budgets. We investi- gate how the maximum allowed turns (7 affect
Turns (Tmax)
3
5
10
reasoning depth and final performance of AGENT- FLOW during test-time inference with the Qwen2.5- 7B-Instrcct backbone. As shown in Figure 7, in- creasing Timx from 3 to 10 consistently improves outcomes across all tasks, accompanied by a rise in average turns consumed. On knowledge-intensive benchmarks such as 2Wiki and GAIA, a larger turn budget enables AGENTFLOW for deeper in- formation retrieval. On mathematical benchmarks like GameOf24 and AIME24, it supports de- composed sub-goals, alternative strategies, and refinement of eπrrrs. Final performance peaks at Twax 10 for all tasks, confirming that longer reasoning horizon benefits the system without causing degenerate loops. This validates that AGENTFL.OW adapts its turn allocation to problem complexity to achieve better solutions through iterative refinement.
2Wiki
2.22
3.18
3.81
4.44
GameOf24
1.63
2.12
2.36
2.67
AIME24
1.63
1.63
1.86
1.90
GAIA
2.43
3.46
4.28
5.42
Table 4: Average turns with increased Tmax-
4.5 IN-DEPTH ANALYSIS OF OPTIMIZED PLANNING
Flow-GRPO optimizes tool usage. We compare tool usage distribu- tions before and after in-the-flow RL training. Figure 8 shows results on two knowledge-intensive tasks, 2Wiki and MedQA, which exhibit distinct optimization patterns along- side improved task accuracy. For 2Wiki, which requires broad factual knowledge, Flow-GRPO optimizes the planner to increase Google Search usage by 42.0%. In contrast, for the specialized MedQA benchmark, which requires deep, domain-spectio information retrieval, fine- tuning shifts the planner away from general tools, reducing Google Search calls (66.2-110999) in favor of in-document Web Search (0-519.5% and specialized Wikipedia Search (0-+598%% This demonstrates that the planner learns to select aps-ooppre tools.
Base Generator
Google Search
Web Search
Wilipedio Search
Acc:60.0%
Acc: .2% (-17.2%)
78.0%)
80, 0%
70
After Fine-tuning
h0.8 +42.0
72
After buning
+598
Cal
30
20-
.22.4
20
+19.5
13.6
-24.8
10-
4..9
Step
Step
Step
(a) 2Wiki
Training Steps
(b) MedQA
Training
Figure 8: Tool call ratio change by Flow-GRPO fine-tuning.
Flow-GRPP incentivizes autonomous discovery of new solutions. We further examine qualita- tive examples in Figure and additional cases $F. These cases show that AGENTFLOW, trained with Flow-GRPO, develops enhanced capabilities for task planning and tool use. The planner ex- hibits adaptive efficiency, stronger self-correction, and spontaneous new integration tools through- out step-by-step problem-solving, autonomously discovering effective solution pathways.
4.6 TRAINING EFFICIENCY ANALYSIS
Optimized planning with increased rewards and condensed responses. We analyze the training dynamics of the AGENTFLOW planner by tracking its average reward and response length on the train set (Figure 9a). Training rewards steadily increase, indicating effective policy improvement via Flow-GRPO. Meanwhile, response length, after an initial exploratory rise, progressively shortens and stabilizes. This shows the planner learns to balance conciseness and informativeness, avoiding unnecessarily long outputs.

---

Published as a conference paper at ICLR 2026
Flow-GRPO efficiency over tool-
integrated reasoning RL. We
Training Reward (Acc.)
0.8
Training Reward Trend
Response Length Irend
-230
Reisporse Langth (tokeng)
Validation Reward (Acc.)
0.4
Flow-CiPPO (ours)
TdAL
compare AGENTFLOW (trained with Flow-GRPO) against a mono-
220
0.7-
0.3
lithic tool-integrated reasoning
0.6-
200
0.2
baseline (ToRL) on AIME24. As shown in Figure 9b, AGENTFLOW achieves sustained performance gains, with validation accuracy growing steadily. In contrast, ToRL's performance quickly stagnates and trends downwards, highlighting the superior efficiency of our agentic training approach, which uses decomposition and stable credit assignment to avoid the instability.
190
0.3
180
0.1
(a)
10
20 30 40
50
60
(b)
Training Steps
Training Steps
Figure 9: Training dynamics and efficiency of Flow-GRPO.
5 RELATED WORK
Reinforcement learning (RL) from outcome-based rewards has become a dominant paradigm for training LLMs to external tools. Much of this work trains single, monolithic policy in- terleave reasoning with tool calls. This strategy has proven effective in specialized, single-tool settings (Mai et al., 2025; Xue et al., 2025; Feng et al., 2025; Li et al., 2025b) and web search for knowledge-intensive questions (Chen et al., 2025; Jin et al., 2025; Song et al., 2025; Li et al., 2025a; Sun et al., 2025). Recent efforts have extended this monolithic framework to multi-tool en- vironments by focusing on data synthesis (Dong et al., 2025), unified training infrastructure (Jiang et al., 2025), and principled reward design (Qian al., 2025a; Zhang et al., 2025). However, these approach scales poorly as task complexity and planning horizons grow. The central challenge is long-horizer hong-horizon credit assignment; attributing a final outcome to specific intermediate tool calls re- mains difficult, even with fine-gra ane-grained, turn-level rewards (Zeng et al., 2025a; Wang al., 2025d). This difficulty leads to training instability and brittle inference-time generalization, manifesting as strategic deficiencies like tool overuse "cognitive offloading" (Wang al., 2025b; Qian et al., 2025b), suboptimal personalization (Cheng et al., 2025), and poor alignment with user preferences for tool invocation (Huang al., 2025).
Agentic systems with tool use. Agentic systems offer an alternative to monolithic models by de- composing tasks across specialized modules. Many such systems are training-free, orchestrating pre-trained LLMs with handcrafted logic and prompting, as seen in frameworks like AutoGen (Wu al., 2024), MetaGPT (Hong et al., 2024), and OctoTools (Lu al., 2025). This static approach, however, limits their ability to learn and adapt collaborative strategies from experience. Recognizing this, recent work explores training these systems to improve coordination (Deng et al., 2025; Liao et al., 2025). However, most training paradigms are effline, relying supervised fine-tuning or preference optimization on static datascts (Motwani et al., 2024; Park et al., 2025). These methods decoupled from the live, multi-turn dynamics of the system, preventing modules from learning to adapt to evolving tool outputs or recover from early mistakes. Training directly in the flow with on-policy RL is difficult due to sparse rewards and long-horizon credit assignment, where feedback is delayed across long reasoning chains and shifting state distributions (Wang et al., 2025c). Con- sequently, these systems often suffer from brittle adaptation and require complex reward shaping to learn effectively (Wang et al., 2025a).
6 CONCLUSION
We presented AGENTFLOW, a trainable, in-the-flow agentic system that coordinates four specialized modules via an evolving memory and optimizes planner directly inside the multi-turn loop. To enable stable on-policy learning under long-horizon, sparse-reward settings, we introduced Flow- GRPO, which converts multi-turn RL into a sequence of tractable single-turn policy updates by broadcasting single, verifiable trajectory-level outcome every turn and stabilizing credit as- signment with group- nooopnoomal advantages. Comprehensive experiments show that AGENTFLOw achieves strong cross-domain performance, surpassing specialized baselines and even larger propri- etary models. In-depth analyses confirm improved planning and tool-calling reliability, along with positive scaling trends in model size and allowed turn budgets.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
We would like to thank Yihe Deng, Xuehang Guo, and Kunlun Zhu for their valuable input during the early stages of this work. We are grateful Lambda for providing GPU resources. This work was partially supported by the Hoffman- Yee Research Grants program at Stanford HAL, the Al for Math Fund by Renaissance Philanthropy, ONR MURI N00014-24-1-2748 and the Al Research Hub Project through KAIST.
REFERENCES
Art of Problem Solving.
Aime problems and solutions. 2025. / //artofproblensclvinng com/ htt///iikkiiineeee pap/AIME Pwwblems_ssseeeeeee _and__ Solutions. 6, 21
URL https:
Mingyang Chen, Tianpeng Li. Haoze Sun, Yijic Zhou, Chenzheng Zhu, Haofen Wang. Jeff Z Pan, Wen Zhang, Huajun Chen, Fan Yang, et al. ReSearch: Learning to reason with search for Ilms via reinforcement learning. arXiv preprint arXiv:2503. 19470, 2025. 2, 3, 7, 10, 20
Zihao Cheng, Hongru Wang, Zeming Liu, Yuhang Guo, Yuanfang Guo, Yunhong Wang, and Haifeng Wang. ToolSpectrum: Towards personalized tool utilization for large language mod- els. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 20679-20699, 2025. 10
Yingfan Deng, Anhao Zhou, Yuan Yuan, Xian Zhang, Yifei Zou, and Dongxiao Yu. Pe-ma: Parameter-efficient co-evolution of multi-agent systems. arXiv preprint arXiv:2506.11803 2025.
10
Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen. Tool-star: Empowering Ilm-brained multi-tool reasoner via reinforcement learning. arXiv preprint arXiv:2505.16-110 2025. 2, 10
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:240..21783, 2024. 7, 19
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.1 1536, 2025. 2, 3,
Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025. 2
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k: large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.1144 11456, 2025. 6
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th Interna- tional Conference on Computational Linguistics (COLING), pp. 6609-6625, 2020. 21
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, al. MetaGPT: Meta programming for multi-agent collaborative framework. In International Conference on Learning Representations (ICLR), 2024. 2, 4, 10
Jingcheng Hu. Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zert zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint 2503:25003.2499 2025a. 7, 20

---

Published as a conference paper at ICLR 2026
Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Qiguang Chen, et al. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, arXiv preprint 25X1::2505..1885, 2025b. 2
Chengnui Huang, Shen Gao, Zhengliang Shi, Dongsheng Wang, and Shuo Shang. TTPA: Token- level tool-use preference alignment training framework with fine-grained evaluation. arXiv preprint wrXiv:2505.20016 2025. 10
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o system card. arXiv preprint arXiv:2410.21276 2024. 3. 19
Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, et al. VerITool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint wrXiv:2509.0105: 2025. 7, 10, 20
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training Ilms reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 2025. 2, 6, 7, 10, 19, 20
Di Jin, Fileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What dis- ease does this patient have? lcrge-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021. 21
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366 2025a. 10
Xuefeng Li, Haoyang Zou, and Pengfei Liu. ToRL: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383, 2025b. 7, 10, 20
Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang, Marft: Multi-agent reinforcement fine- tuning. arXiv preprint arXiv:2504.1 16129, 2025. 10
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations (ICLR), 2023. 6
Nathan Lile. Math twenty four (24s game) dataset. https:://hugginggaccc.. htt.:/aaacc..cooaaaaaeeeeeeeeeee
nl Allile// 24-ee/24..m 2024.7 21
Pan Lu, Bowen Chen, Sheng Liu. Rahul Thapa, Joseph Boen, and James Zou. OctoTools: An agentic framework with extensible tools for complex reasoning, arXiv preprint arXiv:2502.11271 2025. 2025.
4, 10
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General--easoner:: Advancing IIm reasoning across all domains. arXiv preprint arXiv:2505. 14652, 2025. 7, 20
MAA. American mathematics competitions. In American Mathematics Competitions, 2023. 6, 21
Xinji Mai, Haotian Xu, Xing W. Weinong Wang, Yingying Zhang, and Wenqiang Zhang. Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving. arXiv preprint 2rXiv:2505.0773. 2025. 2, 10
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Seialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations (ICLR), 2023. 6, 21
Moonshot AI. Kiei-Researcher: End-to-End RL Training for Emerging Agentic Capabilities.
https wwwoonssooaa..githee 1o/ weio/oii//eeeteeeeeeeeeeeeeeeeee wwi.iieeeeechee////e June 2025. 2
Sumcet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip HS Torr, Fabio Pizzati. Ronald Clark, and Christian Schroeder de Witt. Malt: Improv- ing reasoning with multi-agent Iim training. arXiv preprint arXiv:24/2.01928, 2024. 2,5, 10
12

---

Published as a conference paper at ICLR 2026
Chanwoo Park, Seungju Han, Xingzhi Guo, A. Ozdaglar, Kaiqing Zhang, and Joo- Joo-Kyung Kim. MA- PoRL: Multi-agent post-co-training for collaborative large language models with reinforcement learning. In Annual Meeting of the Association for Computational Linguistics (ACL, 2025. URL https:/ / /app..... semant ccscoo ar. wwt.oorgg/oooooooo Corpussd: :TT.:T::226600000 2, 5, 10
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp- 5687-5711, 2023. 6, 21
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani- -ür, Gokhan Tur. and Heng Ji. ToolRL: Reward all tool learning needs. arXiv preprint 1XXx:2554.13956 2025a. 2, 3, 10
Cheng Qian. Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Türr, Gokhan Tur, and Heng Ji. SMART: Self-aware agent for tool overuse mitigation. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 4604-4624 2025b. 10
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. First Conference on Language Modeling, 2024. 6, 21
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML), pp. 1889-1897. PMLR, 2015. 18
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retrieval-augmented large language models with iterative retrieval-genee -eaerratnnnn synergy. In Find- ings of the Association for Computational Linguistics: EMNLP 2023, pp. 9248-92788 2023. 7, 20
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint 2xXv:22022.33000, 2024. 3
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprini urXiv:2503.0559: 2025. 2, 10
Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of Ilms without searching. arXiv preprint S51i:::5005045888 2025. 7, 10, 20
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics (TACL), 10:539-554, 2022. 21
Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. SPA-RL: Reinforcing I1 agents via stepwise progress attribution. arXiv preprint arXiv:2505.20732, 2025a. 10
Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Acting less is reasoning more! teaching model to act efficiently. arXiv preprint ar arxiv:2504.1 250.77404444 2025b. URL https: pettps://arxivee /ttpxaaxxvv/.///////eee/eaaae..a org/ / 2504. 1 14870. 10
Zihan Wang. Kangrui Wang, Qineng Wang. Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. RAGEN: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning, arXiv preprint urXiv:2504.20073. 2025c. 2, 10
Ziliang Wang. Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. Stepsearch: Igniting iims search ability via stse- proximal policy optimization. arXiv preprint arXiv:2505. 15107 2025d. 7, 10, 20
13

---

Published as a conference paper at ICLR 2026
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang. Jiale Liu, et al. Autogen: Enabling next-gen Ilm applications via multi- agent conversations. In First Conference on Language Modeling (COLM), 2024. 2, 4. 7, 10, 20
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An. Sim- pletir: End-to-end reinforcement learning for multi-turn sol-innterrated reasoning, arXiv preprint arXiv:2509.02479, 2025. 2, 10
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. arXiv preprint urXiv:2504. 14945, 2025. 7,20
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15/ 151 2024a. 6, 7, 19
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5 5-math technical report: Toward mathematical expert model via self- celf-improvement. arXiv preprint arXiv:2409. 12122, 2024b. 7, 20
Hang Yang, Hao Chen, Hui Guo, Yineng Chen, Ching-Sheng Lin, Shu Hu, Jinrong Hu, Xi Wu, and Xin Wang. Llm-medqa: Enhancing medical question answering through case studies in large language models. arXiv preprint arXiv:2501 055644, 2024c.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 2369-2380, 2018. 21
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprini arXiv:2503. 14476, 2025.
Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong. Re- inforcing multi-turn reasoning in Ilm agents via turn-level credit assignment. arXiv preprint arXiv:2505. 11821, 2025a. 10
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 20o: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar Arxiv:2503. 10..003..88888 18892, 2025b. 2, 7, 19, 20
Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. Nemmoron-reseaacch-oooolll Tool-using language models with rein- forced reasoning. arXiv preprint arXiv:2505.000 00024, 2025. 2, 10
14

---

Published as a conference paper at ICLR 2026
TABLE OF CONTENTS
A Training Algorithm of AGENTFILOW
16
Theoretical Analysis of Flow-GRPO
17
B.1
Preliminaries and Notation
17
B.2 Equivalence Proof for Optimization Objectives
17
B.3 Convergence Analysis
18
Experimental Details
19
C.I Training Details
19
C.2 Evaluation Details
19
C33 Compared Baselines
19
C,4 Evaluation Datasets
20
More Discussion about Experiment Results
22
D.I Main Result Analysis
22
D.2 In-depth Analysis of Optimized Planning
23
E
Instruction Templates in AGENTFLOW
24
E.1 Modules and Memory
24
E.2 Toolset Metadata
29
E.3
LLM-based Judging
34
F
Case Studies
35
F.1
Example Efficient Search for Simple Tasks
F2
Example 2:
Spontaneous Brute-force...
36
F3
Example 3: Good Initial Plan Essential
38
F.4
Example 4: Robust Self-Correction and Adaptation
40
F.5
Example 5: New Combo: Retrieve with Specific URL
42
F6
Example 6: Rapid and Correct Physics Calculation
44
F.7
Example 7: Multi-Source Cross- Verificeiiic
46
15

---

Published as a conference paper at ICLR 2026
A TRAINING ALGORITHM OF AGENTFLOW
We provide a flowchart of the overall training algorithm of AGENTFLOW ($3) in Algorithm
Algorithm In-the-Flow Optimization for AGENTFLOW
Require: Dataset D, Action Planner policy ne. Tool Executor ε, Executive Verifier V, Solution Generator G, Toolset K, and Shared Evolving Memory M
Ensure: Optimized Action Planner parameters 0*
1: for each training iteration do
2: for each query-label pair (4, y²) D do
3:
1. IN-THE-FLOR ROLLOUT GENERATION
4: Initialize: + 1, M <q
5: repeat
6:
a' ~ To(a' q, K, MM) {Plan Action]
7:
el E(e" a. K) (Execute Action)
8:
of V(r' q,ee, M³ (Verify Resuls}
9:
M Imm a', e,, v²) ",e,v') {Update Memory)
10:
t -+= 1-t- t+1
until termination condition met
12:
~ ((o) q. MT) [Generate Final Solution)
13:
2. REWARD COMPUTATION
14:
R(a') R(o,,,17 v"),
VI = 1,..,...,7
15:
3. POLICY UPDATE
16: Update the Action Planner policy he by maximizing the Flow-GRPO objective (Eq. 5)
17: end for
18: end for
19: return optimized parameters l*
16

---

Published as a conference paper at ICLR 2026
B THEORETICAL ANALYSIS OF FLOW-GRPO
B. PRELIMINARIES AND NOTATION
We adopt the notation from the paper formalize our analysis. Definition B.1 (Core Components). Here we list core definition of variables.
Symbol and Description
me Tea
The trainable planner policy, parameterized by The behavior policy used to sample trajectories. The state turn i, defined s' (q. K, M,). The action (a sequence tokens) generated at state s', where a' ~ no(: {) A trajectory of states and actions over T time steps, defined
s!
{(s' d'))
R(T)
The outcome- based reward trajectory т, where R(T) E 10,1}.
Ar
The group-normalized advantage for trajectory т. A crucial property is that the advantage constant for all timesteps within trajectory defined in Eq. 7: a Ar, a(s",a")En
pig
The token-level importance sampling ratio, defined
Adaiss 8) (i.-111)11)
Pi.
Teaa (aLs 16 1/15--1))
(p A)
The PPO clipped objective term, defined Leup (p. A) min(pA,
Definition B.2 (Objective Functions). The global policy objective is the expected trajectory-level reward:
O(0) == Br [R(T]]-
(8)
The single-turn optimization objective for given state s² is defined as:
la
Frocal (0; s') =- Eg EainTa (:lat)
Letip (Ph At)
(9)
la
j=1
The full Flow-GRPO objective function the multi-turn setting is given by:
T
lati Latp Lamp Ai) - (Tel(TT))))) (10)
JFRoW-GRPO (0) = )~D ~oB
(T.) (
B.2 EQUIVALENCE PROOF FOR OPTIMIZATION OBJECTIVES
Theorem B.I. In Flow-GRPO, maximizing the global multi-turn objective is mathematically equiv- alent to maximizing the expected loken-level local objective at each time step under the on-policy induced state distribution, given standard sampling assumptions (Irajectories sampled i.i.d. from the policy with fixed finite turn T)).
Proof. Let's denote the clipping part of the Flow-GRPO objective as (e).
group of G trajectories. Since the trajectories (T:) are sampled independently and identically (i.i.d)) from the behavior policy Bay the expectation of their average is equal to the expectation over single trajectory.
First, the linearity of expectation, we simplify the expectation
of Latip (pi.g AH)
(0) B<G
E B i
(11)
T
aa
la'1
=B(gge
B (19)
Tectip (rij Ar)
(12)
T
la
17

---

Published as a conference paper at ICLR 2026
Here, == ('ss,a'))[<> represents single, arbitrarily sampled trajectory with advantage A,.
Next, we can re-interpret the expectation over trajectories as an expectation over the state-visitation distribution induced by the policy TT Let a be the on-policy distribution of states visited, where each state s² in a trajectory of length is weighted by I/T. The expectation can rewritten
la Lelpp (pj. At)
]
Jeup (gg yaD
E Bo
Batwre (155))
(13)
a
Note that A the advantage corresponding to the trajectory from which s was sampled.
We now recognize that the inner expectation is precisely the definition of the local, per-state objec- tive, (0; s²).
Jelp (0) = 18 (g D-D, aa [Tiocal (0; s²)]
(14)
Adding the KL-divergence term back, we arrive at the final equivalence:
Trw www..GRO (0) = (~,"">>> Rna [Ou s²)]
(15)
This proves that maximizing the global multi-turn Flow-GRPO objective is equivalent to maximiz- ing the expected token-level local objective at each time step under the on-policy induced state distribution.
B.3 CONVERGENCE ANALYSIS
Having established the structural validity of the objective, we now analyze its convergence proper- ties. The analysis builds on the monotonic improvement guarantee provided by trust-region meth- ods (Schulman et al., 2015).
Lemma B.2 (Policy Performance Difference). For two policies Ty and Te the difference in ex- pected return can be expressed as:
O(0) J (Ooia) ETwnT
Σ
Ansa (s', ,a))
(16)
where Abaa is the advantage function under the old policy.
This lemma enables the construction of lower bound on policy improvement.
Theorem B.3 (Monotonic Improvement Guarantee). Define the surrogate objective
(a() Mosa (at (s()s) A (8", a²)
La (0) = E EE
(17)
Then the performance improvement satisfies the lower bound
J(A) - (Bosa) 2 Loasa (8) C DKL( (To To):
(18)
where C 0 is constant depending on the horizon and reward scale, and Drg. denotes the average Kl--divergence between the two policies.
By optimizing the right-hand side of the above inequality, we are guaranteed to improve the perfor- mance of mg. Therefore, for policies T and T obtained from iterations f and f. 4 1, we have:
I(0++1) J(0').
(19)
Conclusion. This analysis establishes that Flow-GRPO optimizes valid surrogate objective and guarantees monotonic policy improvement, thereby converging reliably to a locally optimal policy.
18

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL DETAILS
C.I TRAINING DETAILS
We provide further details on the training setup for AGENTFLOW. Our Flow-GRPO implementation uses learning rate of 1 x 10-6, The Action Planner generates actions with a sampling temperature of 0.5 to balance exploration and exploitation. To prevent policy collapse and stabilize training. we incorporate KL-divergence penalty against reference policy with coefficient 3 0.001. The maximum output length for the planner is set to 2048 tokens to ensure complete exploration during rollouts.
To accelerate the training speed, we limit the maximum number of turns per rollout to 3. The final- outcome reward signal (Eq. 4) is provided by LLM-as-judge, for which use GPT-4o. All tool calls are executed synchronously with a 500-second timeout to handle external service latency robustly. The LLM engines within the tools are and stable outputs. The full training process was conducted on 8 NVIDIA A100 GPUs. Further details on agent prompts and the memory update mechanism are provided $E.1.
temperature of 0.0 to ensure deterministic
C.2 EVALUATION DETAILS
Here, we outline the specifics of our evaluation protocol. For evaluation, we increase the maximum
number of turns per rollout to T = 10 to allow for more extensive and deeper reasoning. The planner's sampling temperature set to 0.7 encourage diverse solution paths. Unless otherwise specified, all tool LLM engines are initialized with Qwen2.5-7B-Instruec:
For fair and consistent evaluation, we adopt the previous work's methodology while standardizing tools: we replace search tools in search-enhanced models with our Google Search tool and code tools code-enhanced models with our Python Coder tool. We use GPT-4o as an LLM-based judge to determine the correctness of final answers. This approach provides robust measure of semantic and numerical equivalence, which is critical for complex reasoning tasks. The specific judging prompt is detailed in $E.3, and additional information evaluation datasets be found in $C.4. To mitigate randomness, we report the average accuracy with standard deviation across three trials for all experiments.
C.3 COMPARED BASELINES
Proprietary LLMs:
Qwen2.5 Series (Yang et al., 2024a), created by Alibaba, comes in multiple configurations. These models undergo training on multilingual corpora covering 29 different languages, demonstrating superior performance in cross-lingual applications. Furthermore, Qwen2.5 showcases robust pro- ficiency in programming and mathematical domains.
Llama-3 Series (Dubey et al., 2024), created by Meta AI, encompasses various iterations. Each model configuration within the Llama family provides dual versions: foundational and instruction-follooinn variants. Training incorporates diverse dataset combinations spanning mul- tiple domains and linguistic varieties. The Llama model family demonstrates excellent results in logical reasoning, software development, and cross-lingual comprehension evaluations. Through progressive enhancements in fine-tuning methodologies and expanded sequence lengths, these models become more applicable to practical deployment scenarios.
GPT-4o Series (Hurst et al., 2024), produced by OpenAl, includes several model variants such as GPT-4o and GPT-4o-mini, with training leveraging extensive multimodal datasets encompass- ing text, vision, and audio modalities. The series achieves outstanding performance in complex reasoning tasks, creative generation, and multimodal understanding benchmarks with continuous refinements in alignment techniques and enhanced processing capabilities.
Reasoning LLMs:
SFT (Zeng et al., 2025b) serves as our basic baseline following Search-RI (Jin et al., 2025). We fine-tune models using supervised fine-tuning GPT-4o-generrted reasoning chains.
19

---

Published as a conference paper at ICLR 2026
SimpleRL-Zoo (Zeng et al., 2025b) investigates zero reinforcement learning training across 10 diverse base models spanning different families and sizes using GRPO algorithm with simple rule-based rewards, achieving substantial improvements in reasoning accuracy.
Zeenn-eeasoner-Zero (Hu et al., 2025a) presents the first open-source implementation of large- scale reasoning-orienn RL training using PPO with GAE and straightforward rule-based wards, without KI. regularization. The framework demonstrates that minimalist design can suc- cessfully scale both response length and benchmark performance.
Geasral-Reasoner (Ma al., 2025) extends LLM reasoning capabilities beyond mathematics to diverse domains using RLVR through a 230K verifiable reasoning questions dataset spanning physics, chemistry, and finance.
LUFFY (Yan et al., 2025) addresses limitations in on-policy RLVR by introducing an off-policy framework that augments training with external reasoning demonstrations using Mixed Policy GRPO and regularized importance sampling.
Gearch-Integraan Reasoning LLMs:
Iter-RetGen (Shao al., 2023) addresses limitations in retrieval- -ugmenned language models by introducing iterative retrieval-generation synergy, where a model's 's previous response serves as context for retrieving more relevant knowledge in subsequent iterations.
Search-R1 (Jin et al., 2025) represents reinforcement learning approach that develops model from the ground up to invoke search functionality throughout the reasoning process.
ZeroSearch (Sun al., 2025) addresses high API costs in RL-based search training by using an LLM to simulate search engines, employing lightweight supervised fine-tuning to transform an LLM into retrieval module that generates both useful and noisy documents. The framework combines this with a curriculum-based rollout strategy that progressively degrades document quality, achieving better performance than real search engine-based methods while incurring zero API costs.
ReSearch (Chen et al., 2025) proposes a reinforcement learning framework that trains LLMs to integrate search operations components of the reasoning chain without supervised data reasoning steps, treating search decisions as guided by text-based thinking.
StepSearch (Wang et al., 2025d) addresses the sparse reward problem in multi-hop reasoning by training search LLMs using step-wise proximal policy optimization with intermediate rewards and token-level process supervision based on information gain and redundancy penalties.
VerlTool (Jiang et al., 2025) addresses fragmentation and synchronization bottlenecks in Agentic Reinforcement Learning with Tool use by introducing a unified modular framework that extends beyond single-turn RLVR paradigms, providing upstream VeRL alignment and unified tool man- agement with asynchronous rollout execution achieving near 2x speedup.
Code-Inter Code-Integrated Reasoning LLMs:
TIR (Yang et al., 2024b) is a basic baseline that demonstrates the model's ability to generate code for tool utilization. our implementation, we directly prompt the model write code that calls the programming interpreter and processes the returned results to generate the final answer.
ToRL (Li et al., 2025b) is a code-enhanced architecture developed via reinforcement learning that empowers models to independently activate code execution environments for mathematical reasoning tasks.
Training-free Agentic System
AutoGen (Wu et al., 2024) introduces agentic conversation framework that enables developers to build LLM applications through conversable agents that can operate using combinations of LLMs, human inputs, and tools.
C.4 EVALUATION DATASETS
We provide a detailed introduction to the search-intensive and agentic benchmarks in our experi- ments follows:
20

---

Published as a conference paper at ICLR 2026
Bamboogle (Press al., 2023) presents demanding multi-step reasoning dataset containing manually constructed questions requiring up to four inferential steps. The dataset evaluates mod- els' capacity for intricate compositional reasoning across intercomnected facts.
2Wiki (2WikiMultihopQA) (Ho et al., 2020) constitutes a comprehensive multi-step QA cor- pus combining structured Wikidata knowledge with unstructured Wikipedia text. The dataset encompasses varied question formats and annotated reasoning chains to facilitate interpretable sequential inference. We randomly sample 100 examples as a test set for efficiency.
HotpotQA (Yang et al., 2018) represents a widely-adopted question answering corpus featuring multi-step queries constructed from Wikipedia entries. We randomly sample 100 examples test set for efficiency.
Musique (Trivedi et al., 2022) comprises multi-step reasoning corpus requiring sequential in- ference where each reasoning stage depends on information derived from preceding steps. We conduct evaluations using the development partition of this particularly challenging dataset. We randomly sample 100 examples
test set for efficiency.
GAIA (Mialon et al., 2023) constitutes a benchmark engineered to assess general Al systems and agents, demanding capabilities including sequential reasoning, web navigation, and compre- hensive tool utilization skills. We utilize the text-exclusive portion of this dataset, designed to challenge base language models in our experimental setup.
Furthermore, we also conduct series of experiments on math and scientific reasoning benchmarks:
AIME24 (Art of Problem Solving, 2025) A collection of 30 demanding mathematical problems sourced from the 2024 American Invitational Mathematics Examination (AIME), encompassing algebra, geometry, number theory, and combinatorics. Each JSONL-formatted record contains the problem identifier, question text, comprehensive solution methodology. and the final numer- ical result. Created to assess large language models' sophisticated mathematical reasoning abil- ities, the dataset presents substantial difficulty, systematic multi-phase solutions, and distinctive unswers-establishnng as a robust benchmark for evaluating advanced analytical capabilities.
AMC23 (MAA, 2023) contains mathematical problems derived from the 2023 American Mathe- matics Competition, emphasizing areas such as functional equations and complex analysis.
Gfm4Of24 (Lile, 2024) derives from the traditional numerical puzzle known as 24 (alternatively called the 24 numbers game). The challenge requires utilizing four given numbers with fundamen- tal arithmetic operations (addition, subtraction, multiplication, division) to create an expression yielding 24. For instance, with numbers 9, 10, and 13, a correct solution would be "(10 - 4) x (13 9) = 24". Successfully solving requires computational proficiency along with iterative attempts validate potential solutions. Each challenge formatted as open-ended inquiries.
GPQA Graduate Level Google-Pront Proof Q& Benchmark (Rein et al., 2024) comprises collec- tion of demanding text-based multiple choice problems authored by subject specialists in biology, physics, and chemistry, intentionally crafted be "exceptionally challenging". We randomly sample 100 examples as test set for efficiency.
MedQA (Jin et al., 2021) features text-based multiple choice problems assembled from profes- sional medical licensing examinations. Problems encompass comprehensive medical knowledge and clinical reasoning skills.
21

---

Published as a conference paper at ICLR 2026
D MORE DISCUSSION ABOUT EXPERIMENT RESULTS
D.1 MAIN RESULT ANALYSIS
Our main results are presented in Tables and 2. Overall, AGENTFLOW consistently outperforms all baseline models across diverse domains, including search-intensive tasks, agentic tasks, and mathe- matical and scientific reasoning tasks. These comprehensive results yield several key insights:
Monolithic LLMs are insufficient for complex reasoning. While scaling up model size (from 7B model to GPT-4o) improves average performance, their monolithic nature presents limitations when facing complex tasks that require multi-turn reasoning and sub-goal decomposition. In contrast, proposed AGENTFLOW consistently outperforms these larger models. Specifically, it achieves an average improvement of 8.2% over GPT-4o search-intensive tasks (57.3% vs. 49.1% 19.1% in Table 1), and a remarkable 15.8% gain over GPT-4o on agentic tasks 13.11% vs. 17.3% in Table 1). For mathematical reasoning benchmarks, AGENTFLOW obtains a substantial improvement of 16.4% over GPT-4o (51.5% vs. 5% in Table 2). Furthermore, it surpasses the strong Llama-3.3-70B by 12.5% on scientific reasoning tasks (63.5% vs. 51.0% in Table 2). These results demonstrate that the carefully designed agentic system of AGENTFLOW, despite being built on a 7B- 1B-parameter backbone, can deliver superior and more efficient performance compared to substantially larger monolithic L.Mss.
Specialized reasoning models exhibit strong in-domain focus but limited generalizability While domain-specifie fine-tuning and tailored tool integration provide clear benefits over base LLMs, they fail to deliver robust cross-domain performance due to fundamental scaling limitations. Our evaluation across three reasoning domains substantiates these limitations. On tasks, specialized models such as Search-RI (33.3%) and VerlTool (39.0%) perform well within their narrow scope yet fall substantially short of AGENTFLOW (57.3%) shown in Table 1. Sim- ilarly, in mathematical reasoning, methods like SimpleRL.-reason significantly behind AGENTFLOOW (51.5%) in Table 2. Even in scientific reasoning, where models such as Luffy (5,5%) offer competitive results, they are consistently surpassed by AGENTFLOW (63.5%) in Table These findings demonstrate that while specialized reasoning models excel within narrow domains, their reliance on a single monolithic policy introduces poor generalization, making them brittle when confronted with diverse, cross cross-domain challenges.
(36.6%) and ToRL (37.0%) trail
AGENTFLOW demonstrates superior, versatile reasoning through its adaptive agentic system. AGENTFLLOW establishes new state-of-the-an agentic system by achieving an average accuracy of 57.3% on search-intensive tasks, 33.1 % on agentic tasks, 51.5% on mathematical reasoning, and 63.5% on scientific reasoning. Our method's advantage stems from combining an agentic system with targeted planning policy refinement via on-policy reinforcement learning in an online fash- ion. When compared to AutoGen-a general agent framework with the same backbone model- AGENTFLOW demonstrates massive improvement of 14.9% search tasks and 19.9% math tasks. This underscores that the core advantage comes from our dedicated trainable agentic system that integrates novel Flow-GRPO for in-system on-policy optimization, enabling effective agent planning and tool utilization to solve complex, long-horizon problems across diverse domains.
Ganerator
Google Search
Qwwnn2.5-7-insstrutt
GPT-4o
Search
Wikipedia Search
+13.0
+6.0
Acc:19.2%
25.2% (+6.21%)
After Finet-tuning
60
Tool Call Ratio 1%6]
Accuracy (96)
-1.5
55
38.5
13.6
13.6
.4.7
70.0
+5.2
40
61.5
+1.1
18.4
10-
34.1
3.1
-2.2
GAIA
AMC25
HelperQA
Step
32
Figure 10: Tool scaling study. AGENTFLowWs performance improves when its tools are up- graded from Qwen-2.5-7B-Be Instrruct to GPT-4o.
Figure 11: Tool call optimization on Musique. AGENTFLOW's planner increases Web Search usage after Flow-GRPO training.
22

---

Published as a conference paper at ICLR 2026
D.2 IN-DEPTH ANALYSIS OF OPTIMIZED PLANNING
AGENTFLOW adapts to inference-time tool scaling. We scale the tools-the Base Generator and Python Coder-to GPT-40-powered versions. Empirical results search and math datasets (Figure 10) show that AGENTFLOW, when using these GPT-4o-powered tools, substantially outper- forms its performance with
tools, achieving improvements 1.0% on GAIA, 6.0% on AMC23, and a notable 13.0% on HotpotQA. This finding further supports consistent trend: after in-the-flow RL. training, the planner can adaptively leverage improvements in the underlying tools to enhance the agentic system's 's overall performance.
Flow-GRPO spontaneous tool usage preference change. We further compare tool usage distri- butions before and after in-the-flow RL training Musique. Figure 11 shows that due to Musique's need for diverse source information, Flow-GRPO optimizes the planner to increase Web Search to delve deeper into the URL provided by other search tools. This maneuver presents steady performance improvement of 6.1%.
AgentFlow
AgentFlow -.ween2.5-7B-Instreee
Before tuning
Before tuning tuning
77.2
Accuracy (%)
68.8
72.3
After tuning
69.6
60
63.0
60
58.4
60.0
40
40
40.0
33.1
29.1
20
14.3
13.3
20.0
17.2
16.7
Bamboogle 2Wiki
GAIA
AIME24
Bamboogle 2Wiki
GAIA
AIME24
Figure 12: Flow-GRPO fine-tuning offers consistent gains on AGENTFI.OW as the backbone model size scales from 3B to 7B.
More evidence of training scaling in backbone size. We further investigate how the backbone LLM scale affects AGENTFLOW'S performance and the efficacy of Flow-GRPO on GameOf24, AMC23, and MedQA. We construct two versions of the system: one using Qwwn2.5-3BBInstt -nntrun and another using (rren255-7BInstruue for all four modules (planner, executor, verifier, and generator) as well as the associated tools. both versions, only the planner fine-tuned with Flow-GRPO. As shown in Figure 12, Flow-GRPO fine-tuning consistently improves performance across tasks for both backbones. These results demonstrate that our in-the-flow optimization effective across model capacities, enhancing AGENTFLOW regardless of LLM size.
23

---

Published as a conference paper at ICLR 2026
E INSTRUCTION TEMPLATES IN AGENTFLOW
E.1 MODULES AND MEMORY
E.1.1 ACTION PLANNER
Tool Metadata can be found $E.2.
Instruction for Action Planner
Task: Determine the optimal next step to address the query using available tools and previous context.
Context:
Query: {Question]
Available Tools: [Base Generator, Python Coder, Google Search, Wikipedia Search, Web Search]
Toolbox Metadata: [Tool Metadatal, Tool Metadata2, ...]
Previous Steps: {Actions from Memory)
Instructions:
1. Analyze the current objective, the history of executed steps, and the capabilities of the available tools.
2. Select the single most appropriate tool for the next action.
3. Consider the specificity the task (e.g.. calculation information retrieval).
Consider source of required information (e-g., general knowledge, mathematical computation, a specific URL).
5. Consider the limitations of each tool as defined in the metadata.
6. Formulate clear, concise, and achievable sub-goo -oal that precisely defines what the selected tool should accomplish.
Provide all necessary context (e.g., relevant data, variable names, file paths, or URLs) so tool can execute its task without ambiguity.
Response Format:
1. Justification: Explain why the chosen tool is optimal for the sub-goal, referencing its capabilities and
the task requirements.
2. Context: Provide all prerequisite information for the tool.
Sub-Goal: State the exact objective the tool.
4. Tool Name: State the exact name the selected tool (e.g.. Wikipedia Search).
Rules:
Select only one tool per step.
The Sub-Goal must be directly and solely achievable by the selected tool.
The Context section must contain all information the tool needs; do not assume implicit knowledge.
The final response must end with the Context, Sub-Goal, and Tool Name sections in that order. No additional text should follow.
24

---

Published as a conference paper at ICLR 2026
E.1.2 TooL EXECUTOR
Instruction for Tool Executor
Task: Generate a precise command to execute the selected tool.
Context:
Query: {Question}
Sub-Goal: (Sub Goal from Next Step Plan}
Tool Name: (Selected Tool from Next Step Plan}
Toolbox Metadata: (Selected Tool Metadata from Next Step Plan]
Relevant Data: (Context from Next Step Plan)
Instructions:
Analyze the tool's required parameters from its metadata.
Construct valid Python code that addresses sub-goal using provided context and data.
3. The command must include at least one call to tcol. execate (1.
4. Each tool. execute () call must be assigned a variable named execution.
5. Use exact numbers, strings, and parameters in the tool .execute call based on the context.
Output Format: Present your response the following structured format. not include any extra text or explanations.
Example 1:
Generated Command:
execution tool. execute 100 toys, masa gets how much are their togethez?")
the following porblom: "Isaac has
Example 2:
Generated Command:
execution execuex (query- "query-("Methano)) ""Methano", "function of hyperbola", "Fermat's s Last Theorem"])
25

---

Published as a conference paper at ICLR 2026
E.1.3 EXECUTION VERIFIER
Instruction for Execution Verifier
Task:
Evaluate if the current memory is complete and accurate enough to answer the query, or if more tools are needed.
Context:
Query: {Question]
Available Tools: [Base Generator, Python Coder, Google Search, Wikipedia Search, Web Search] Toolbox Metadata: [Tool Metadatal. Tool Metadata2, ...] Memory (Tools Used Results): (Actions from Mcmory)
Instructions:
1. Review the original query, the initial analysis, and the complete history of actions and results in the memory.
Does accumulated information fully address all aspects the query?
Are there any unanswered sub-questions or missing pieces information?
4. Are there any inconsistencies or contradictions between different steps?
5. Is any information ambiguous, potentially hallucinated, in need of verification?
6. Determine if any unused tools could provide critical missing information based their metadata.
Final Determination:
If the memory is sufficient to form complete and accurate answer, explain why and conclude with "Conclusion: STOP". If more information is needed, clearly state what missing, suggest which tool(s) could help, and con- clude with "Conclusion: CONTINUE".
Rules:
The response must end with either exactly "Conclusion: STOP" or "Conclusion: CONTINUE"..
Do not include any text after the conclusion statement. Your justification must be concise and directly tied the query and memory.
26

---

Published as a conference paper at ICLR 2026
E.1.4 SOLUTION GENERATOR
Instruction for Solution Generator
Task: Generate a concise final answer to the query based on all provided context.
Context:
Query: {Question]
Initial Analysis: (Query Analysis}
Actions Taken: {Actions from Mcmory)
Instructions:
1. Carefully review the original user query, the initial analysis, and the complete sequence of actions and their results.
2. Synthesize the key findings from the action history into coherent narrative.
Construct clear, step-by-step summary that explains how action contributed solving the query.
4. Provide direct, precise, and standalone final answer to the original query.
Output Structure:
1. Process Summary: A clear, step-by-step breakdown of how the query was addressed. For each action,
state its purpose (c.g., "To verify X"") and summarize its key result or finding in one sentence.
2. Answer: A direct and concise final answer to the query. This should be self-contained statement that fully resolves the user's question.
Rules:
The response must follow the exact two-part structure above.
The Process Summary should informative concise. focusing on logical flow the solution.
The Answer must be placed at the very end and be clearly identifiable.
Do not include any additional sections, explanations, or disclaimers beyond the specified structure.
27

---

Published as a conference paper at ICLR 2026
E.1.5 EVOLVING MEMORY
Example Memory Entry
"Query": : Where the largest shopping mall besides Tokyo's biggest metzopolitan station?
"Action Turn 1": {
"Tool Name": "eikipedia Search",
"Sub-Goal": "Retrieve detailed information about Tokyo netropolitan azea from Kikipedia.",
"Command": "execation tool. (query-"Tokyo metropoliten area details")"
"Result"; "The Greater Tokyo Area the largest metropolitan the world...",
"Verification Status":
Brief Review of the Query, Initial Analysis, and Previous Memory. Assessment of Completeness and Accurscy,
Conclusion: The memory is complete and accurate enough the query. Additional tools are needed verify or generate more solations Final Determination: CONTINUE"
}
"Action
{
},
"Action Turn t": {
"Verification Status":
Brief of the Query, Initial Analysis, Previcus Memory. Assessment of Completeness and Accuracy. [Including Time Dilation calculation, Geographic Precise, Inconsiatencies or Contradictions, Unit Conversion, etc.
Conclusion: memory is complete and accurate enough query. No additional tools are needed to verify or generate more solutions. Final Datermination: STOP"
the
Our shared evolving memory system creates a deterministic, structured record that captures the reasoning process across three integrated agents: the Action Planner, Tool Executor, and Execution Verifier. By sequentially stacking crucial information from each action step, the system enables transparent state tracking, controllable behavior, and bounded context growth.
The memory reading and matching process employs regular expressions to parse outputs generated by different system components, adhering to standardized formats defined in their respective com- ponent instructions. For the Action Planner, we use a relatively permissive regular expression to extract key information. Specifically, matches the content immediately following: Sub-Goal as the sub-goal and the content following: Tool Name as the selected tool. This extracted information then used to populate the next memory entry. For the Tool Executor. the regular expression de- signed to capture the entire Command line starting with execution Additionally, the value passed to the Query parameter within this command is parsed and saved into the memory for future reference. All results returned by the tools are directly stored in the Result field the memory. The Verification Status is extracted from Execution Verifier, including a brief analysis of the current tool result and previous memory, and then it gives conclusion whether the loop needs to be CONTINUE or STOP.
= tool execute .).
28

---

Published as a conference paper at ICLR 2026
E.2 TOOLSET METADATA
This section details the implementation and metadata of the tools used in our main results. We employ suite of specialized tools, each designed for distinct tasks. Below, we present metadata for each tool, including its functionality, input/output schema, limitations, and best practices.
E.2.1
BASE GENERATOR
Tool Metadata Base Generator
Description: A generalized tool that takes query from the user, and answers the question step step the best of ability. It can also accept image.
Input: query: str The query that includes query from the user to guide the agent to generate response.
Output: The generated response to original query
Demo Commands:
Command:
execution tool execute (query=" "query="Summrrii "Summarize the following few lines")
Description: Generate short summary given the query from the user.
Limitation
The Base Generator may provide hallucinated incorrect responses.
Best Practice
1. Use it for general queries or tasks that don't require specialized knowledge or specific tools in the toolbox.
2. Provide clear, specific query.
3. Use it to answer the original query through step by step reasoning for tasks without complex multi-step reasoning.
4. For complex queries, break them down into subtasks and use the tool multiple times.
5. Use it starting point for complex tasks, then refine with specialized tools.
6. Verify important information from its responses.
LLM Engine Required: True
29

---

Published as a conference paper at ICLR 2026
E.2.2
PYTHON CODER
Tool Metadata of Python Coder
that generates and exccutes simple Python code snippets for basic arithmetical caleu- lations and math-related problems. The generated code runs in a highly restricted environment with only basic mathematical operations available.
Description:
Input: query: clear, specific description of the arithmetic calculation math problem to be solved, including any necessary numerical inputs.
Output: dict A dictionary containing the generated code, calculation result, and any messages.
Output prompt: Given query, generate Python code snippet that performs the specified operation on the provided data. Please think step by step. Ensure to break down the process into clear, logical steps. Make sure to print the final result in the generated code snippet with descriptive message explaining what the output represents. The final output should be presented in the following format: python
<code snippet>
Demo Commands:
Command:
execution fcol.execute "quiry""Find the of prime numbers up to 50"; Generate Python code snippet find the sum of prime numbers up to 50.
Description:
Command:
query=" Given the st 2, 4,
7, 10], cal culate the of
squares odd numbers"
execut fan tool execute (query=query)
Description: Generate a Python function for mathematical operation on a given list of numbers.
Limitation
1. Restricted to basic Python arithmetic operations and built-in mathematical functions.
2. Cannot use any external libraries or modules, including those in the Python standard library.
3. Limited to simple mathematical calculations and problems.
4. Cannot perform any string processing, data strueture manipulation, complex algorithms.
5. No access to any system resources, file operations, or network requests.
6. Cannot use 'import' statements.
7. All calculations must self-contained within single function or script.
8. Input must be provided directly in the query string.
9. Output limited numerical results or simple lists/tuples of numbers.
10. Output should be kept single numerical result simple list/uuple of numbers.
11. DO NOT generate loop output.
Best Practice
1. Provide clear and specific queries that describe the desired mathematical calculation.
2. Include all necessary numerical inputs directly in the query string.
3. Keep tasks focused basic arithmetic. algebraic calculations, simple algorithms.
4. Ensure all required numerical data included in the query.
5. Verify that the query only involves mathematical operations and does require any data pro- cessing complex algorithms.
6. Review generated code to ensure it only uses basic Python arithmetic operations and built-in math functions.
LLM Engine Required: True
30

---

Published as a conference paper at ICLR 2026
E.2.3 GOOGLE SEARCH
Tool Metadata of Google Search
Description: web search tool powered by Google Search that provides real-time information from the internet with citation support.
Input: query: str The search query find information the web.
Input:
add citations: bool Whether to add citations to the results. If True, the results will be formatted with citations. By default, it is True.
Output: str - The search results of the query.
Demo Commands:
Command:
execution tool. execute Iquezy="What the capital France?")
Description: Search for general information about the capital of France with default citations enabled. Command:
execution tool. execute Iquezy-"Who won the ease 2024?", add.ittat.ons=Falee)
Description: Search for information about the Euro 2024 winner without citations.
Command:
execution too...xeeuee exxccuee (query="Physica cs and Soot ety article arXiv August 11, 2016", add cicaticcc.................. ons=True)
Description: Search for specific academic articles with citations enabled.
Limitation
1. This tool is only suitable for general information search.
2. This tool contains less domain- speciifc information.
3. This tool is not suitable for searching and analyzing videos on YouTube or other video platforms.
Best Practice
Choose this tool when you search for general information about topic.
2. Choose this tool for question types of query, such as "What is the capital of France?" or "Who invented the telephone?".
3. The will return summarized information.
4. This tool is more suitable for definition, world knowledge, and general information search.
LLM Engine Required: False
31

---

Published as a conference paper at ICLR 2026
E.2.4 WIKIPEDIA SEARCH
Wikipedia search will first call Wikipedia API retrieve relevant URLs with snippets. Then the RAG Retrieval-Augmentee Generation) process begins by extracting raw text content from the given webpage URL, cleaning remove HTML elements and retain only meaningful text. This content is then split into overlapping chunks of approximately 200 words each, with 20-word overlap to preserve context across segments from the first 1M words in each URL. Next, both the user's query and the document chunks are embedded into the vector space using the OpenAl text-embeddi --- smal 11 model. The system computes the cosine similarity between the query embedding and each chunk embedding to rank the chunks by relevance. We set that the top 10 most similar chunks are selected and passed forward as context. And base LLM engine will summarize the extracted context.
Wikipedia search will first call Wikipedia API to retrieve relevant URLs with snippets.
Tool Metadata of Wikipedia Search
Description: tool that searches Wikipedia and returns relevant pages with their page titles, URLs, stract, and retrieved information based on a given query.
Input: query: str The search query for Wikipedia.
Output: dict dictionary containing search results, all matching pages with their content, URLs, and metadata.
Demo Commands:
Command:
execution tool. execute (quezy-"hbat the exact mass kg the moon")
Description: Search Wikipedia and get the information about the mass of the moon.
Command:
execut inn = tool. eexecute (query="Funts i on of human kidnay")
Description: Search Wikipedia and get the information about the function of the human kidney.
Command:
execution tcol. execute (quezy= "Ween was the first moon landing?")
Deseription: Scarch Wikipedia and get the information about the first moon landing.
Limitation
1. It is designed specifically for retrieving grounded information from Wikipedia pages only.
The returned information accuracy depends on Wikipedia's 's content quality.
Best Practice
1. Use specific, targeted queries rather than broad amhiguous questions.
2. If initial results are insufficient, examine the "other-pages" section for additional potentially rel- evant content.
3. Use this tool part of multi-step research process rather than a single source truth.
4. You can use the Web Search to get more information from the URLs.
LLM Engine Required: True
https://platffomm.oog http://pw..../cccc//////////ttte ng-
32

---

Published as a conference paper at ICLR 2026
E.2.5
WEB SEARCH
Web search will directly access the URL in the query. Then the RAG (Retrievll-Augmentc Gener- ation) process begins by splitting content from the page into overlapping chunks of approximately 200 words each, with a 20-word overlap to preserve context across segments from the first 1M words in each URL. Next, both the user's query and the document chunks are embedded into the vector space using the OpenAl lext-embedding-3-snee between the query embedding and each chunk embedding to rank the chunks by relevance. We set that the top 10 most similar chunks are selected and passed forward as context. And base L.LM engine will summarize the extracted context.
model. The system computes the cosine similarity
Tool Metadata Web Search
Deseription: A specialized tool for answering questions by retrieving relevant information from given website using RAG SeetrievalAuugmente Generation).
Input: query: str The search query for the website.
Input: url: The URL of the website retrieve information from.
Output: str The answer to user's query based on information gathered from the website.
Demo Commands:
Command:
execution = tool. execute (query="What is the exact mass in kg of the moon?", url="https:/ / /en. wikipedia org/wiki /Moor")
Description: Retrieve information about the moon's 's mass from Wikipedia.
Command:
execution cool.xxecuter iquery- the main features of Python programming language?", url="https: //www python org/ aatp///bo///ppp/""
Description: information about Python features the official website.
Limitation
Requires valid URLs that are accessible and contain text content.
2. May not work with JavaScript-heary websites those requiring authentication.
3. Performance depends on the quality and relevance the website content.
4. May return incomplete inaccurate information if the website content is not comprehensive.
5. Limited by the chunking and embedding process which may miss context.
6. Requires OpenAl API access for embeddings and LLM generation.
Best Practice
1. Use specific, targeted queries rather than broad questions.
2. Ensure the URL is accessible and contains relevant information.
3. Prefer websites with well-structured, text-rich content.
4. For complex queries, break them down into smaller, specific questions.
5. Verify important information from multiple sources when possible.
6. Use it part of multi-step research process rather than single of truth.
7. It is highly recommended to use this tool after calling other web-based tools (e.g., Google Search, Wikipedia Search, etc.) to the real, accessible URLs.
LLM Engine Required: True
openaa openai httpaaa..cm/cc////aa////tttteeee haatbbedddnng-----
33

---

Published as a conference paper at ICLR 2026
E.3 LLM-BASED JUDGING
We employ GPT-4o as our judge model using a two-step "analyze-thennjudge"" instruction paradigm ensure both accuracy and efficiency.
Reward Function Instruction Training
Task: Determine if the Model Response is equivalent to the Ground Truth.
Instructions:
Extract: Isolate the final answer from the Model Response, ignoring all reasoning steps. Look specif- ically for content within ... the concluding statement.
Normalize & Compare: Assess equivalence after normalization:
3. Mathematical Answers: Must be mathematically identical (e.g., is equivalent to 0.5).
Numerical/Textual Answers: Ignore formatting (commas, spaces), case sensitivity, and extraneous units/currency (e.g.. "1,000" == "1000", "Paris" = "PARIS").
5. Multiple Choice Questions (MCQ): The answer match either the correct option's content (e.g.- "Paris") or identifier (e.g.. "A" "Ist").
6. Verdict: Return "True" only if the normalized answers are semantically mathematically equivalent.
Inputs:
Question: {Question}
Model Response: (Final Response from Solution Generator)
Ground Truth: {GT}
Output Format:
Present your response in the following structured format. Do not include any extra text explanations. Lanalysisen Brief analysis of the comparison. <true false>: "True" or "False".
34

---

Published as a conference paper at ICLR 2026
F CASE STUDIES
In this section, we conduct a case study to demonstrate how our AGENTFLOW, coherent with Flow-GRPO, enhances problem-solving performance with greater elegance, efficiency, and robust- ness. We present solution comparisons showing brief outputs from memory the Action Planner (Qwen2.5-7B-Inse
before (w/o) tuning by Flow-GRPO and after (w/) Flow-GRE tuning, with the methodology detailed in $3.2.
F.1 EXAMPLE 1: EFFICIENT SEARCH FOR SIMPLE TASKS
This case demonstrates that, with Flow-GRPO tuning, the Action Planner can effectively leverage the search engine to retrieve correct answers for simple tasks in a highly efficient manner-unlike the untuned baseline, which requires multiple trials.
Example from GameOf24
Using the numbers [1, 1, 13), create expression that equals 24. You basic arithmetic operations (+, -, s, /) and parentheses.
Question:
Answer: (13+1)((++1)
Success Case: AGENTFLOW w/ Flow-GRPO
Action Step 1:
Tool: Google Search
Sub-goal: Find valid arithmetic expression that equals using numbers [1, 1, 13].
Command:
tool execute (query="[L, 1, 1, 13] arithmetic expression to get 24") Result: Here's arithmetic expression using the numbers 1, and 13 to get 24: (13 (1 =4
Execution Verifier: PASS, STOP
Solution Generator: The arithmetic expression is ((13 (13-1) x (1 1))
Failure Case: AGENTFLOw w/o Flow-GRPO
Action Step 1:
Took: Python Coder
Sub-goal: Generate and test all possible combinations of the numbers [1. 1, 1, 13] using basic arithmetic operations and parentheses find an expression that equals 24.
Command:
tcol .execute (query="[1, 1, 1, 131 execution
(13
1)1")
Result:
& Define the values
value.list
1, 131
operatior_result
Print the result with a descriptive message
print (he result of the operation is: operation.reaule")
The result of the operation is: 11
Action Step 2:
Tool: Python Coder
Sub-goal: Generate and test all possible combinations of the numbers [1. 1, 13] using arithmetic operations and parentheses to find an expression that equals 24.
Command:
tcol. execute (query=" [1, 1, ->
1)
Result: Error: unexpected indent (<stringg>, line
Action Steps 3-8:
Tool: No matched tool given (Tool Not Found)
35

---

Published as a conference paper at ICLR 2026
Sub-goal: Generate and test all possible combinations of the numbers [1. 1, 13] using basic arithmetic operations and parentheses find expression that equals 24.
Command: Not command is generated due to tool not found.
Result: Not result is generated due to the tool not found.
Note: These steps are identical repetitions where the agent failed to find a matching tool.
Execution Verifier: STOP, Tool Not Found
Solution Generator: The agent incorrectly concludes that (1 13 24.
F.2
EXAMPLE 2: SPONTANEOUS BRUTE-FORCE
This case demonstrates that, when tuned with Flow-GRPO, the Action Planner first attempts several solutions, recognizes their ineffecttvenness, resorts to brute-force approach, and finally verifies the result using a search engine.
Example from # GfmeOf24
Using the numbers 6, 9], create an expression that equals 24. You must basic arithmetic operations (+, -, &, /) parentheses.
Question:
Answer: (1+1)99+6
Success Case: AGENTFLOW w/ Flow-GRPO
Action Step 1: Initial Attempt
Tool: Python Coder
Sub-goal: Generate and test possible arithmetic expressions using the numbers exactly
once check any combination equals 24.
Command:
teol. exoluexe (query-"Using the numbers [1, 1, 6, 91, create expression
that equals 24 using basic arithmetic operations and parentheses.")
Result:
4 Siven numbers
numbers = [1, 1, 6, 9] Calculate the expression
result (numbers [0] numbers [1]) (numbers [3] numbers [2])
Print the result
(f*The result of the expressien (numbers (0) numbers [1]
(numbers [3] numbers [2]) is: result")
The result of the expression (1 +1) is:
Action Step 2: Tool Error
Tool: Python Coder
Sub-goal: Generate and test all possible permutations of the numbers with different operator place-
ments to find an expression that equals 24.
Command:
tcol .execute (query="Us' ng the numbers [1, 1, 6, 91, create expression
with +, that equals s 24")
Result: Error: eval() arg must he a string, bytes or code object
Action Step 3: Second Attempt
Tool: Python Coder
Sub-goal: Generate possible combinations of four numbers with every permutation oper-
ators (t.-.*/) *.) to find valid equation resulting exactly 24.
Command:
tool.execute execute (query-"Using the numbers [1, 1, 6, create expression with +,-,*,/ that equals 24
Result:
# Given numbers numbers - [1, 1, 6, 91
36

---

Published as a conference paper at ICLR 2026
t Ca callcllen ate the expression
reault (numbers numbers [0]) (numbers [3]
numbers (11)
numbers
Print the reault
print (h" result of the expression is: result")
The result of the expression is: 45.0
Action Step 4: Systematic Search (Breakthrough)
Tool: Python Coder
Sub-goal: Generate and evaluate possible permutations of four numbers with every permuta- tion of operators (+,+,*) to find valid equation resulting in exactly 24.
Command:
tcol. executee (query="Using the numbers [1, 1, that equals 24 using -,-,/,*")
create expression
Result: Successfully found valid expressions: The expression (1+1)89+6 equals The expres-
sion 6+(1+1)89 equals 24. The expression 64(99(1++)) equals 24. The expression 9*(1+1)+6 equals 24.
Action Step 5: Confused Verification
Tool: Python Coder
Sub-goal: Simplify and verify any minor adjustments to the existing promising expression can achieve an exact match of 24.
Command:
tcol. execute (query-"Using
numbers [1, 1, 6, and basic arithmetic
operations create an expression that equals 24: 16 (1 11))*
Result: The result of the expression (9 (1+11 1) 6) is: 108
Action Step 6: Final Verification
Tool: Google Search
Sub-goal: Search online for examples discussions related creating expressions using given numbers that equal target value.
Command:
tcol.execute (query="asing the numbers [1, 1, 6, 9] create an expression that equals 24", vnd..citatioos=free)
Result: Here's expression using the numbers that equals 24: + 1)++)
Execution Verifier: PASS, STOP (via exhaustive search and verification)
Solution Generator: After a multi-step process of trial, error, and verification, final correct expression is (I 9 6
Failure Case: AGENTFLOw w/o Flow-GRPO
Action Step 1:
Tool: Python Coder
Sub-goal: Generate and test Python code snippets to find an arithmetic expression using the numbers
[1. 1. 9] exactly once that equals 24.
Command:
tool.execute (query="Using the numbers [1, 1, 6, 91, create an expression
that equals 24 using basic arithmetic operations ons and parentheses."'
Result:
Desine the numbers
numbers [1, 1, 9]
& Calculate the expression
result (6 9) ((1 1) 15)
Print the result with descriptive message print ("The result of the expression is:", result)
The result the expression is: 24
Execution Verifier: STOP
Solution Generator:
The agent incorrectly concludes the answer is (6 x 9) - ((1 + x 15), failing notice it used the number 15, which was not in the allowed input set.
37

---

Published as a conference paper at ICLR 2026
F.3 EXAMPLE 3: A GOOD INITIAL PLAN ESSENTIAL
This case demonstrates that well-crafted initial search with a highly relevant query is far more effective than issuing numerous wrong paths. When tuned with Flow-GRPO, the Action Planner in AGENTFLOW can identify the optimal search engine and formulate the most effective query, leading to correct and targeted answer in single trial.
Example from GALA
Question: Assuming scientists in the famous youtube video The Thinking Machine (Artificial Intelligence in the 1960s) were interviewed the same year, what is the name of the scientist predicting the sooner thinking machines robots? Answer using the format First Last name.
Answer: Claude Shannon
Success Case AGENTFLOw w/ Flow-GRPO
Action Turn 1:
Tool: Google Search
Sub-goal: Search for details on "The Thinking Machine" YouTube vidco and identify key figures making predictions about AI.
Command:
famous youtube videa Thinking Machine (Artificial Intelligence the 196Cs) were interviewed the same year, what is the name of the scientist predicting sooner
tcol. executee (query-"Assuming scientists
thinking machines
zobots?")
Result: The scientist who predicted the sooner emergence of thinking machines robots in "The Thinking Machine" video from the 1960s was Claude Shannon. He stated, "I confidently expect that within 10 15 years we will find emerging from laboratories something not too from the robots of science-fiction fame." This prediction was made in the 1961 documentary, which was produced by CBS to honor MIT's 100th birthday.
Execution Verifier: PASS, STOP
Solution Generator: The scientist who made the prediction is Claude Shannon.
Failure Case: AGENTFL.Ow w/o Flow-GRPO
Action Turn 1:
Tool: Search
Sub-goal: Retrieve information the YouTube video "The Thinking Machine" (Artificial Intel- ligence in the 1960s) to identify the name of the scientist who made the prediction about thinking machines or robots.
Command:
tool. cxolexxecute (query="Who predicted the sconer thinking machines or rebots in the YouTube video 'The Thinking Machine' (Artificial Intelligence in the 1960s) ?", orl="https:/ /tw:///wwww.. htw.://///wwww./bbb/eeee.
www.hhhyyyyuu _video_id"" id")
Result: The reference information provided does not contain any details regarding predictions made about thinking machines robots the YouTube video "The Thinking Machine' (Artificial Intelli- gence the 1960s). Therefore, cannot provide an answer to the query.
Action Turn 2:
Tool: Google Search
Sub-goal: Find the name of the scientist who predicted the sooner development of thinking machines robots in the YouTube video "The Thinking Machine" (Artificial Intelligence in the 1960s).
Command:
execute (query-"In the YouTube video The Thinking Kachine (Artificial Intelligence in the 1960s), which scientist predicted the sooner development of thinking machines robets?", addleitations-Irue Result: In the YouTube vidco "The Thinking Machine (Artificial Intelligence in the 1960s)," Jerome Wiesner, then Director the Research Laboratory Electronics MIT, predicted the sooner
38

---

Published as a conference paper at ICLR 2026
velopment of thinking machines. He stated, "if come back in four or 5 years I'll say "Sure they really do think."
Action Turn
Tool: Google Search
Sub-goal: Find the name of the scientist who predicted the sooner development of thinking machines
robots in the YouTube video "The Thinking Machine" (Artificial Intelligence in the 1960s).
Command:
tool. execute (query-"In the Yourabe video The Thinking Kachine (Artificial Intelligence in the 1960s), which scientist predicted the aooner robots?",
development of thinking machines Result: In the YouTube video "The Thinking Machine (Artificial Intelligence in the 1960s)," Pro- fessor Jerome Wiesner, director of the research laboratory of electronies MIT, predicted the sooner development of thinking machines. He stated, suspect you come back four 5 years I'll say 'Sure they really do think."" "" Another scientist in the video confidently expected "within a matter of 10 or 15 years something will emerge from the laboratories which is not too far from robot science fiction fan."
Action Turn 4:
Tool: Google Search
Sub-goal: Find the exact year of the interview in the YouTube video "The Thinking Machine" (Artificial Intelligence in the 1960s) where Jerome B. Wiesner made the prediction about thinking machines robots.
Command:
tool cool.execooe execute (query="In the YouTube video The Thinking Machine (Artificial Intelligence in the 1960s), which scientist predicted the sooner development of thinking machines Result: In the YouTuhe video "The Thinking Machine (Artificial Intelligence in the 1960s)," scientist confidently predicted that "within matter 10 15 years something will emerge from the laboratories which is not too far from the robot of science fiction fame." While the video features interviews with several pioneering scientists in artificial intelligence, including Jerome Wiesner, Oliver Selfridge, and Claude Shannon, the specific speaker for this particular prediction about the emergence of robots within to 15 years not explicitly named in the provided transcripts. Jerome Wiesner, one the co-hosts, also made a prediction, stating he suspected that "four or years" machines would "really do think."
robets?", add_eitattion--ruee
Action Turn 5:
Tool: Google Search
Sub-goal: Find the exact year the interview the YouTube video "The Thinking Machine" (Artificial Intelligence the 1960s).
Command:
tcol eexccute (query-"When
the YouTube video The Th Thinking Mach nee released?", add citations-Irue)
Result: The original television program titled 'The Thinking Machine" aired on the TV net- work on October 26, 1960. It was a special program produced by CBS News in conjunction with the Massachusetts Institute of Technology (MTT). This documentary explored carly concepts of artifi- cial intelligence and computers. While the original broadcast was in 1960, various versions of 'The Thinking Machine' have been uploaded to YouTube at different times.
Action Turn
Tool: Google Search
Sub-goal: Find the exact year of the interview in the YouTube video "The Thinking Machine" (Arti- ficial Intelligence in the 1960s) where Jerome Wiesner made his prediction about thinking machines robots.
Command:
tool. execute (query-"In which year did Jerome Wiesner make his prediction about thinking machines rebots in the YouTube video 'The Thinking Machine' (Artificial Intelligence in 1960s) addleltations-Truee
Result: Jerome Wiesner made his prediction about thinking machines robots in the YouTube vidco "The Thinking Machine' (Artificial Intelligence the 1960s) 1960. The documentary, which features Wiesner, originally aired on October 26, 1960, as part of a CBS series titled "Tomor- row". In the video, Wiesner stated his confident expectation that "within a matter of 10 or 15 years something will emerge from the laboratories which is too far from robot science fiction fan".
39

---

Published as a conference paper at ICLR 2026
Execution Verifier: STOP, Initial Tool Failure & Inefficient Search Path
Solution Generator: The answer is Jerome B. Wiesner. However, the initial direct video search failed, leading multiple, somewhat redundant Google searches extract information.
F.4 EXAMPLE 4: ROBUST
AND ADAPTATION
This side-by-side comparison illustrates the critical impact of Flow-GRPO tuning on strategic tool usage. The trained AGENTFLOW agent demonstrates adaptive planning recovering from failed searches, refining input formulations, and ultimately achieving correct solution in single effec- tive trial. In contrast, the untrained agent, despite accessing the correct information early, fails to properly utilize the Python Coder tool and becomes trapped in repetitive error loop, unable to learn or adjust. This highlights Flow-GRPO's s role in enabling not just tool selection, but strategic resilience and goal-directed reasoning.
Example from GAIA
Question: Compute the check digit the Tropicos ID for the Order Helotiales would have if it were an ISBN-10 number.
Answer:
Success Case: AGENTFLOW w/ Flow-GRPO
Action Step 1:
Tool: kipedia Search
Sub-goal: Retrieve the Tropicos ID from Wikipedia page on Helotiales.
Command:
teol. execute (Toopcsss ID of Order Relotiales")
Result: No results found for query: Tropicos ID of Order Helotiales
Action Step
Tool: Google Search
Sub-goal: Retrieve the Tropicos ID Order Helotiales from a reliable online source.
Command:
tcol. execute (query="Tzopicos of Order Helotiales")
Result: The Tropicos ID for the Order Helotiales is 100370510.
Action Step
Tool: Python Coder
Sub-goal: Write and execute a Python script to calculate the check digit for tropicos.id as if it were ISBN-10 number.
Command:
tcol. mool.exeee execute (query-"Calcules the check digit for the ISBN-10 number tropicoo cos.id)
Result: Error in execute tool command: name 'tropicos id' is not defined
Action Step 4:
Tool: Python Coder
Sub-goal: Write execute a Python script to calculate the check digit for tropicos as if
were an ISBN-10 number.
Command:
txol.execute (query="Conpute the check digit for 100370510 ISPN-10 number*)
Result:
def calculate.che www.eecchcccccooee (isbn):
Extract the first digits of the ISBN-10
isbn.digits (int (digit) for digit in iaba[:9]]
Calculate the som of products
40

---

Published as a conference paper at ICLR 2026
total sum sum iposition digit. for position, digit in enumerate (isbn digits, start-1))
Determine the check digit check,digit toual.sum 11 if check.digiz == 10: return 'X'
else: return stz (check.digit)
Given ISBN-10 number 1sba_10 "100370510"
Compute the check digit check.digit.result = cal.oulate.cheek.diig. (isbn.10)
Print the final result
print Th* check digit for the ISBX-10 number isbn_10 is check digit result")
Printed output: The check digit for the ISBN-10 number 100370510 is
Action Step 5:
Tool: Python Coder
Sub-goal: Execute the existing Python script to calculate and print the check digit for tropicos.id as an ISBN-10 number.
Command:
tcol .execute (query="Corpute chack df digit tropicos id 100370510 as
ISBN-10 number")
Result:
def calculate 1sbe10 check digit (tropicos id)
Step Extract the first digits of the Trepicos isbn.digits tropicos.id[:91
Step 2: Initialize the sum for the check digit calculation total.sum
Step 3: Calculate the of the products of each digit and its position
for i, digit in enumerate (isbn_digits, start-1): total sum sum +- i int (digit)
check digit total-sum the check digit 10, is represented if check.digi: 10:
check.digit - 'x'
Step 5: Format the ISPN-10
isbn.10 fiishh.dipisscchec.diee
return isbr_10
Siven Tropicos ID tropicos "100370510"
i Scep 6: Generate the IS3N-10 and print the result 1sbn.10 taleulate..bmmitccheek..t
(tropicos.id)
print (f" The ISBN-10 number for the Tropicos tropicos.id isbn.10")
Printed output: ISBN-10 number for the Tropicos ID 100370510 1003705103
Execution Verifier: PASS, STOP
Solution Generator: The check digit is 3, resulting in the full number 1003705103.
Failure Case: AGENTFLOW w/o Flow-GRPO
Action Step
Tool: Nikipedia Search
Sub-goal:
Retrieve the Tropicos ID of the Order Helotiales from Wikipedia.
Command:
tcol. eexecute (query="Tzopicos
for the Result: results found for query: Tropicos the Order Helotiales
ales")
41

---

Published as a conference paper at ICLR 2026
Action Step 2:
Tool: Google Search
Sub-goal: Retrieve the Tropicos ID the Order Helotiales from reliable web source.
Command:
tcol. execute (qupry="Tzopicos ID for the Order Helotiales",
add_citat. inss-rrre)
Result: The Tropicos ID for the Order Helotiales is 100370510.[1][]
Action Steps 3-9:
Tool: Python Coder
Sub-goal: Calculate the check digit for the Tropicos ID 100370510 if were an ISBN-10 number.
Command:
tcol. execcute "etery="Calcules" the check digit for ISBN-10 number 100370510")
Result: Error: name 'isbn' is not defined
Note: Action Steps 3 through 9 all have identical subgoals, commands, and error results-the agent
is stuck in repetitive error loop, unable to fix the variable naming issue.
Execution Verifie: STOP, Stuck in Repetitive Tool Error
Solution Generator: Python tool failed repeatedly with the same 'isbn' variable error across 7 consecutive attempts (Steps 3-9). The agent was unable to recover from this error and failed to calculate the check digit.
F.5 EXAMPLE 5: NEW COMBO: RETRIEVE WITH SPECIFIC URL
This case highlights how both agents eventually succeed, but with markedly different efficiency and strategy. The Flow-GRPO-tunes AGENTFLOW agent learns to refine its queries effectively and und- recognizing the limitations of Wikipedia search-switche geted and the most task-solving relevant web search, achieving success with minimal redundancy. In contrast, untrained agent persists issuing dense, ineffective queries within the same tool de- spite diminishing returns, only escaping the loop by eventually switching to Google Search. While both reach the correct answer, the latter exhibits inefficient exploration and delayed adaptation; furthermore, with path consistency, underscoring Flow-GRPO's role in fostering not just correct- ness, but strategic and timely tool transition.
tools strategically to tar-
Example from # HotpotQA
Which part of Handel's Messiah covers the birth child who, according to the gospels Luke and Matthew, was born in Bethlehem?
Question:
Answer: Part
Success Case: AGENTFLOW Flow-GRPO
Action Steps 1-2: Initial Failures
Tool: Nikipedia Search
Sub-goal: Retrieve information about the part of Handel's Messiah that refers to the birth of Jesus in Bethlehem.
Command:
tool .exccute (query="Whi ch part of Handel s Messi iah refers to the birth of Jeaus in Bethlehem?")
Initial long-form queries failed to find relevant pages (returned irrelevant pages like "His- torical Jesus", "Advent", "Passion of Jesus").
Result:
Action Step 3: Query Refinement
Tool: Wikipedia Search
Sub-goal: Identify the specific part of Handel's Messiah that mentions the birth of Jesus in Bethle- hem.
Command:
42

---

Published as a conference paper at ICLR 2026
rcol.execute (query="Jandell Messlah birth of Jesus Bethlehem"; Result: This successfully identified relevant pages like "Structure of Handel's Messiah" and "Mes- siah Part l".
Action Steps 4-6: Further Search
Tool: k iieeia Search Sub-goal: Retrieve the specific part of Handel's Messiah that mentions the birth of Jesus in Bethle- hem.
Command:
tcol. tool.execute (query="Which part Handel's s Messiah refers to birth of Jesus in Bethlehem?")
and
(query-"birth of Jesus in Mandel'ss Messiah")
Result: Agent gets briefly confused, re-runs failed query, but then another refined query again locates main "Messiah (Handel)" Wikipedia pages.
Action Step Switching (Breakthrough)
Tool: Neb Search (on specific URL.)
Sub-goal: Retrieve the specific part of Handel's Messiah that mentions the birth of Jesus in Bethle- hem.
Command:
(query-"Which part of Handel's a Messiah covers the birth of Jesus in Bethlehem?",
url="https:/ wikipedia a httg/wwkk/// Baikkii/aandllle 427 MMsssiah")
Result: Success! The tool extracted the answer from the page: The part Handel's Messiah that covers the birth Jesus Bethlehem is found Part specifically scene includes the prophecy of Christ's birth. This section features the well-known chorus "For unto us a child is born," which celebrates the nativity of Jesus.
Execution Verifier: PASS, STOP
Solution Generator: The relevant section is Part I, which includes the famous chorus "For unto a child born".
Suceess Case but stuck dense information: AGENTFLOW w/o Flow-GRPO
Action Step Initial Wikipedia Search
Tool: 8 k ipedia Search
Sub-goal: Retrieve information about the part of Handel's Messiah that refers the birth of Jesus in Bethlehem.
Command:
tool.execuse execute (query="Which part of Handel's s Messiah refers to the birth of Jesus in Bethlehem?")
Result:
Returned irrelevant pages (e.g., "Historical Jesus", "Advent"), (No direct answer)
Action Step 2: Refined Wikipedia Search
Tool: Wikipedia Search
Sub-goal: Retrieve information about the specific part of Handel's Messiah that mentions the birth of Jesus in Bethlchem.
Command:
tcol.ese exeluexe (query="Which part Handel's Messiah refers to birth (Hint found, but no direct answer)
Jeaus in Bethlehem?")
Result: Still returned mostly irrelevant pages, but "Messiah Part I" appeared in other suggestions.
Action Step 3: Keyword Wikipedia Search
Tool: Kiedia Search
Sub-goal: Retrieve information about the specific part Handel's Messiah that mentions hirth of Jesus in Bethlehem.
Command:
tcol.es execute (query="aandel Messiah birth of Jesus Bethlehem";
Result: Still direct relevant pages, but "Structure of Handel's Messiah" and "Messiah Part r"
again appeared. (Still no direct answer)
43

---

Published as a conference paper at ICLR 2026
Action Steps 4-S: Persistent Wikipedia Queries
Tool: Nikipedia Search
Sub-goal: Retrieve the specific part Handel's Messiah that mentions the birth Jesus in Bethle- hem.
Command:
(query="Which part Handel's Messiah refers the birth of Jesus in Bethlehem?")
tcol. scol.execute (query-"Randel Messiah ah birth of Jesus Rethlehem")
Result: Consistently returned irrelevant pages like "Historical Jesus" and "Christmas Oratorio". (Wikipedia tool stuck/ineffective for direct answer)
Action Step 6: External Search (Breakthrough)
Tool: Google Search
Sub-goal:
specific part of Handel's Messiah mentions birth Jesus Bethlehem.
Command:
exocuuee (query-"Which part of landel's Messiah covers the birth of child who, according to gospels of Luke and Matthew,
born
Bethlehem?", add.citatiion--rre))
Result: Successfully found the answer: Handel's Messiah addresses the birth child born in Bethlehem primarily in Part I of the work. Key elements within Part include the chorus "For unto us a child is born" and the scene depicting the annunciation to the shepherds.
Execution Verifier: PASS. STOP (via query refinement and external search after Wikipedia limita- tions)
Solution Generator: The part of Handel's Messiah referring birth Jesus in Bethlehem found in Part I, particularly the chorus "For unto us a child is born" and the scene depicting the annunciation to the shepherds.
F.6 EXAMPLE 6: RAPID AND CORRECT PHYSICS CALCULATION
This GPQA example reveals fundamental difference in reasoning quality between the tuned and untuned agents. The flo--GRPO-enhaneed AGENTFLOW correctly identifies the core chal- lenge-relativisticc time dilation interstellar distances-and applies the appropriate physics- based computation minimal steps, arriving at the correct answer (81 years) efficiently. In contrast, the untrained agent misinterprets the astronaut's age as the travel duration, leading cascade of erroncous calculations across multiple tool calls. Despite eventually retrieving the distance via search, fails integrate this information coherently or recognize its conceptual mistake. This highlights that Flow- GRRP not only improves tool usage efficiency but also promotes correct prob- lem formulation, enabling the agent distinguish between proper time, coordinate time, and mission constraints-4 critical capability for complex scientific reasoning.
Example from GPQA
An intelligent civilization in the Large Magellanic Cloud has engineered extraordinary spacecraft capable of traveling substantial fraction of the speed of light. The average lifetime of these aliens is roughly 150 solar years. Now, having Earth as their destination in mind, they are determined travel with this spacecraft a constant speed of 0,999999877c, where e is the speed light. Approxi- mately, how long will take for their 22 years old astronaut (from the point of view of the astronaut) to reach the Earth using this incredibly fast spacecraft?
Question:
Choices:
81 years
B. 77 years
C. 72 years
D. The astronaut will die before reaching to the Earth.
Answer: A. 81 years
44

---

Published as a conference paper at ICLR 2026
Success Case: AGENTFLOW w/ Flow-GRPO
Action Step I:
Tool: Python Coder
Sub-goal: Calculate the time takes for the astronaut to reach Earth from the perspective of the astronaut.
Command:
tcol .exccute (query="A spacecraft travels at speed 999999987*c, where с is the speed of light. The distance to Barth is 165000 light-years. If the astronaut is 22 years old, how long will it take for the astronaut to reach Earth from the perspective of the astronaut? Using the formula time distance speed, calculate the time dilation effect with the velccity given and determine the perceived time for the astroraut.")
Result:
Import math
Constants
speed.of.light 1 & Speed of light in arbitrary units
distance.to_earth = 16500C & Distance ta Earth in light-years
astronaut age 22 Astronaut's s age in years
velocity factor 0.99999987 # Velocity light
fraction speed
Step 1: Calculate the time it takes for the spacecraft to travel Earch
travelttmee distance.to.earth velocity
Calculate the Lorent2 factor farentz.factor math. velocity.factor-.2)
$ Step 2:
Step 3: Calculate the perceived time for the astronaut perceived.time travelitime lorentz.factor
# Print the results
print The spacecraft will take travel.tims:.. .2 years to reach Earth. ") print Df*Due to tine dilation, the astronaut will perceive perceived,t
years passing.")
The spacecraft take 165000.02 years to reach Earth. to time dilation, astronaut will perceive 323591675.56 years passing.
Note: The calculation contains an error in the formula application. The correct approach should use the inverse relationship for proper time calculation.
Execution Verifier: PASS, STOP
Solution Generator: Correct physics approach properly calculated relativistic travel consider-
ing distance and time dilation effects. Therefore, the correct choice is: A. 81 years.
Failure Case: AGENTFLOw w/o Flow-GRPO
Action Step 1:
Tool: Python Coder
Sub-goal: Generate and execute Python code to caleulate the time experienced by the astronaut where the travel time from stationary observer's
using time dilation formula: t'
perspective, and v is 0.99..999999999 0,99999987 Command:
tcol.execute (query-"Caleui using the time dilation formula: t' / sqrt v22/e"2) 2), where t is approximately 22 years (the lifespan of the astronaut) and v is 0. 9999999977 ")
time experienced by an astronaut
Result:
45

---

Velocity of the astronaut fraction of the speed Published as a conference paper at ICLR 2026
# Step 1: Define the constants
299792458 Speed of light in meters second
0.99999987
1ight
t Step 2: Calculate the value inside the square reat value inside aqrt
Step 3:
Compute the time experienced by the astronaut
t.prime 22 / melue_inside.sqtt & Li Lifespan of the astronaut in years
Step 4: Print result print The time experienced by the astronaut t.prime: years.")
The time experienced by the astronaut is 84615390.151914 years. The time experienced by the astronaut is 84615390. years.
Action Step 2:
Tool: Google
Sub-goal: Find scientific references discussing similar scenarios regarding travel times under rela- tivistic conditions.
Command:
tcol.execute (query="relatiiiiss" travel time calculations for speeds close the speed light", aad.citaations-reee
Result:
Retrieved detailed information about time dilation formulas and Lorentz factors, but this
theoretical knowledge was not properly integrated with the problem's specific requirements.
Execution Verifier: STOP, Conceptual Error & Inefficient Multi-Step Process
The answer is 84615390. 151914 years. However, the approach incorrectly treated the astronaut's age travel time, leading multiple redundant calculations and fundamen- tally flawed methodology.
Solution Generator:
F.7 EXAMPLE 7: MULTI-SOURCE CROSS- VERIFICATION
The comparison highlights the effectiveness of a multi-tool, systematic reasoning approach en- abled by Flow-GRPO. In the success case, the model leveraged sequential tool usage-starting with Google Search, followed by targeted Wikipedia and Web Search to accurately identify Gülçişek Hatun Olivera Despina's 's mother-in-laww through verified historical sources. Each step built upon prior findings, ensuring robustness and precision. In contrast, the failure case without Flow-GRPO relied on single, improperly executed Wikipedia query without task decomposition that resulted in timeout and no meaningful output, leading to premature termination. This demonstrates that Flow-GRPO enhances reasoning trace reliability, tool coordination, and overall task completion in complex knowledge retrieval scenarios.
Example from 2Wiki
Question: Who is the mother-in-law of Olivera Despina?
Answer: Güllççeek Hatun
Success Case: AGENTFLOW w/ Flow-GRPO
Action Step 1:
Tool: Google
Sub-goal: Perform a Google search "Olivera Despina biography" to relevant information
about her and possibly identify her husband's family members.
Command:
tcol. execute (query="olivera Deapina biography*, addLeitations--zss
Result: Olivera Despina, also known as Mileva Olivera Lazarević Despina Hatun, Serbian
princess born around 1372. She was the youngest daughter Prince Lazar Serbia and Princess
46

---

Published as a conference paper at ICLR 2026
Milica. In 1389, shortly after the Battle of Kosovo, Olivera was given in marriage to the Ottoman Sultan Bayezid as peace offering between the Lazarević and Ottoman dynasties.
Action Step
Tool: wikk kipedia Search
Sub-goal: Identify any mention of Olivera's 's parents or in-laws in Wikipedia articles.
Command:
tcol. execute (query-"Who the mother-in-las of Olivera Deapina")
Result: Returned pages about Bayezid and other related historical figures, but direct answer relevant pages.
Action Step 3:
Tool: Google Search
Sub-goal: Perform Google search to find more detailed biographical information about Olivera
Despina and her family members.
Command:
teol execute (query="Who is the mother-in-law of olivera Despina")
Result: Olivera Despina's mother-in-law was Gülçççekk Hatun. Olivera Despina was Serbian
princess who married Ottoman Sultan Bayezid after the Battle of Kosovo in 1389. Gülçiçek Hatun was the first wife Sultan Murad and mother Bayezid
Execution Verifier: PASS, STOP
Solution Generator: Multi-tool verification approach successfully identified Gülçiçek Hatun as Oliv- era Despina's 's mother-in-hee through systematic research.
Failure Case: AGENTFLOw w/o Flow-GRPO
Action Step 1:
Tool: Google Search
Sub-goal: Search for information about the film "Pugacher" (1978) and identify the director and any awards they received.
Command:
tool. Ccol.execuue execute (query="What is the award that the director of film Pugachev
(1978 Film) received?", aaddcitaaiion--Pee
Result: Alexey Saltykov, the director of the 1978 film "Pugachev," received the title of People's Artist of the RSFSR in 1980 and the Order of the Badge of Honour for his contributions cinema. While film "Pugachev" itself was honored with special prize the All-Union Film Festival 1979, this award was given to the film rather than specifically to Saltykov for his direction of it. Note: This step shows the agent working on completely different question than asked.
Action Step 2:
Tool: Wikipedia Search
Sub-goal: Retrieve information about the personal awards received by Alexey Saltykov for directing the film "Pugachev" (1978).
Command:
tcol. execute (query=AAlxxey Saltykov awards for Pugachev 1978")
Result: Returned pages about "Alexey Saltykov (director)". "Catherine the Great", and "Imperial Russian Army" no relevant information about the original question.
Execution Verifier: STOP Agent worked on wrong question, no useful information gathered for original query
Solution Generator: Tool attempts failed to address the actual question about Olivera Despina's mother-in-law. The agent became confused and worked unrelated question about the Pugachev film director.
47


Reviews:
[{"review_id": "7xQJxcig5q", "reviewer": "Reviewer_VTBo", "summary": "The paper introduces AgentFlow, a trainable agentic framework designed to address the challenges of tool-augmented reasoning in large language models (LLMs). By decomposing tasks into specialized modules (planner, executor, verifier, generator) and optimizing the planner through in-the-flow, on-policy training, AgentFlow achieves significant improvements in long-horizon tasks. The proposed Flow-GRPO algorithm tackles the sparse-reward, multi-turn credit assignment problem by converting it into tractable single-turn updates, aligning local decisions with global outcomes. Empirical evaluations across ten benchmarks demonstrate the framework’s effectiveness, with substantial accuracy gains over existing baselines and even larger proprietary models like GPT-4o.", "questions": "1. How does the Flow-GRPO algorithm scale with increasing model size or when dealing with a significantly larger number of tools? Are there any bottlenecks or limitations observed during training?\n2. How robust is AgentFlow in noisy or adversarial environments where tools may fail or provide incorrect outputs? Does the verifier module effectively mitigate such issues?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The introduction of AgentFlow as a trainable and modular agentic framework is innovative, particularly in how it addresses key limitations of monolithic policies and static, offline training approaches. The decomposition into specialized modules is a well-motivated design that aligns with the principles of scalability and adaptability to diverse tools and tasks.\n2. The paper provides a clear and robust methodological foundation. The Flow-GRPO algorithm is a well-constructed solution to the long-horizon, sparse-reward problem, and the use of trajectory-level outcomes to guide local updates is both elegant and effective.\n3. The experimental results are impressive, showcasing consistent gains across a wide range of benchmarks, including search, agentic, mathematical, and scientific tasks. The 14.9% average improvement in search tasks and 14.5% gain in mathematical tasks highlight the framework’s practical value.\n4. The clarity of the code and modular design of AgentFlow suggest strong potential for adoption by the community. Open-sourcing this framework would likely facilitate further research and development in agentic LLM systems.", "weaknesses": "1. While the framework demonstrates strong performance across selected benchmarks, the paper could provide more discussion on how well AgentFlow generalizes to entirely unseen tools or tasks, especially in zero-shot or low-resource settings.\n2. The on-policy training approach, while effective, may introduce additional computational complexity compared to training-free or offline methods. A discussion on the trade-offs between performance gains and computational costs would strengthen the narrative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a3yy6mWcXG", "reviewer": "Reviewer_YoHg", "summary": "This work introduces AgentFlow, a trainable, in-the-flow agentic framework that coordinates four modules through an evolving memory, optimizing the planner on-policy within the multi-turn reasoning loop. It further proposes Flow-GRPO, a novel reinforcement learning algorithm that converts long-horizon, sparse-reward multi-turn optimization into tractable single-turn updates by broadcasting a trajectory-level reward to all turns. Experiments across benchmarks demonstrate substantial gains over state-of-the-art models.", "questions": "It's a well-written and comprehensive paper. Some potential improvements please refer to the Weakness section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of in-the-flow reinforcement learning for agentic systems is both useful and interesting.\n2. The proposed Flow-GRPO provides a stable, elegant formulation for long-horizon credit assignment.\n3. The authors conduct comprehensive evaluation across multiple domains, outperforming competitive baselines.", "weaknesses": "1. The authors conduct comprehensive experiments. It's better to add discussion on computational cost and training stability.\n2. The proposed method performs good on text-based tasks. How about the results on dynamic environments or multi-modal settings?\n3. Minor: It's better to add some task description presented in appendix in Figure 1 or 2 to make the solution more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5tONEMypI5", "reviewer": "Reviewer_1fZQ", "summary": "This paper introduces AGENTFLOW, a trainable agentic system that enables multi-turn reasoning and tool use by optimizing its planner in the flow of execution. Traditional tool-augmented LLMs train a single, monolithic policy that struggles with long-horizon reasoning. AGENTFLOW instead decomposes reasoning into four interacting modules that coordinate through a shared evolving memory.\nThe key technical contribution is Flow-based Group Refined Policy Optimization (Flow-GRPO), an on-policy RL algorithm that converts sparse, long-horizon reinforcement learning into tractable per-turn updates by broadcasting a single trajectory-level reward to all turns and using group-normalized advantages for stability. This design allows for end-to-end optimization of the planner within the live agentic loop.", "questions": "First of all, I would like to thank the authors for their work.\nI would like to ask how reward hacking is avoided when global rewards are distributed directly to each turn? In such cases, even undesirable behaviors within a successful trajectory could be reinforced, potentially leading to unnecessarily long or inefficient trajectories. Additionally, could the authors comment on the computational cost of the training process?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The paper addresses the sparse reward problem in LLM-based reasoning and enables the training of modular, multi-tool systems in dynamic environments\n\n- strong empirical results", "weaknesses": "- representation should be improved; it is super hard to follow\n\n- The reliance on LLM-as-a-Judge rewards risks evaluation leakage. I also find the direct distribution of the global reward across turns unconvincing.\n\n- Although the results are mostly qualitative and good, the paper does not report the computational cost, which raises concerns about efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)