Don't Just Fine-tune the Agent, Tune the Environment
Authors: Siyuan Lu
Zechuan Wang
Hongxuan Zhang
Qintong Wu
Leilei Gan
Chenyi Zhuang
Jinjie Gu
Tao Lin
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
LLM Agents
Tool Learning
Multi-turn tool use
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Model (LLM) agents show great promise for complex multi-turn tool-use tasks, but their development is often hampered by the extreme scarcity of high-quality training data. Supervised fine-tuning (SFT) on synthetic data leads to overfitting, whereas standard reinforcement learning (RL) struggles with a critical cold-start problem and training instability. To address these challenges, we introduce $\textbf{Environment Tuning}$, a novel training paradigm that enables agents to learn complex behaviors directly from problem instances without relying on pre-collected expert trajectories. $\textbf{Environment Tuning}$ orchestrates this learning process through a structured curriculum, actionable environment augmentation that provides corrective feedback, and fine-grained progress rewards to ensure stable and efficient exploration. Using only 400 problem instances from Berkeley Function-Calling Leaderboard (BFCL) benchmark, our method not only achieves competitive in-distribution performance against strong baselines but also demonstrates superior out-of-distribution generalization, overcoming the performance collapse common to SFT-based approaches. Our work presents a paradigm shift from supervised fine-tuning on static trajectories to dynamic, environment-based exploration, paving the way for training more robust and data-efficient agents.

Published as a conference paper at ICLR 2026
DON'T JUST FINE-TUNE THE AGENT, TUNE THE ENVIRONMENT
Siyuan 2,,,1,311+ Zechuan Wang
Hongxuan Zhang 4.5
Qintong Wu
Leilei Gan If
Chenyi Zhuang"1 Jinjie Gu
Tao 3.4
1 Zhejiang University
SShangaai Innovation Institute $ NNaninng University
"Westlake University
"Inclusion AL, Ant Group
ABSTRACT
Large Language Model (LLM) agents show great promise for complex, multi-tum tool-use tasks, but their development is often hampered by the extreme scarcity of high-quality training data. Supervised fine-tuning (SFT) synthetic data leads to overfitting, whereas standard reinforcement learning (RL) struggles with a critical cold-start problem and training instability. To address these challenges, we introduce ENVIRONMENT TUNING, a novel training paradigm that enables agents learn complex behaviors directly from problem instances without relying on pre- collected expert trajectories. ENVIRONMENT TUNING orchestrates this learning process through structured curriculum, actionable environment augmentation that provides corrective feedback, and fine-grained progress rewards ensure stable and efficient exploration. Using only 400 problem instances from Berkeley Function- Calling Leaderboard (BFCL) benchmark, our method not only achieves competitive in-distribution performance against strong baselines but also demonstrates superior cut-of-diistrbbttio generalization, overcoming the performance collapse common to SFT based approaches. Our work presents a paradigm shift from supervised fine- tuming static trajectories to dynamic, environment besse exploration, paving the way for training more robust and data-efficient agents.
INTRODUCTION
The emergence of Large Language Model (LLM)-based agents, equipped with capabilities for in- tricate reasoning, planning, and tool interaction (Wang al., 2024; Weng, 2023), has unlocked the potential to address complex, real-world problems across diverse domains like software engineer- ing (Jimenez et al., 2023; Yang al., 2024), computer use (Xie al., 2024; OpenAI, 2025c; Wang et al., 2025a) and web browsing (Wei et al., 2025; OpenAI, 2025a; Moonshot AI, 2025). Much of this success has been demonstrated in single-turn tasks, such as mathematical reasoning, where plentiful datasets (Hendrycks et al., 2021; Sun et al., 2025), automated verifiers (Kydlřček, 2025), and advanced algorithms (Shao et al., 2024; Yu et al., 2025) have enabled significant progress (Zeng et al., 2025; He et al., 2025). Despite their utility, these agents, limited by structured and single-turn paradigm, are insufficient to tackle the full complexity of real-world problems: this limitation motivates the transition of agents capable of engaging in dynamic, multi-ture interactions with external tools and environments ( Wang el al., 2025c; Mai el al., 2025; Feng el al., 2025b).
However, the transition multi-turn tool-use settings introduces several key challenges:
Data scarcity (C1): High-quality multi-turn tool-use datasets are exceedingly scarce due to the labor-intensive nature human annotation and validation. For instance, BFCL V3 (Patil et al., 2025b) multi-turn dataset contains only 800 samples, severely limiting the effectiveness of traditional data-driven approaches.
Complex environment (C2): Multi-turn tool-use scenarios require agents to navigate complex, interconnected tool ecosystems spanning multiple domains. Like BFCL V3, the benchmark spans 8 different domains with 84 distinct tools, requiring cross-domain API calls and sophisticated tool orchestration.
"Equal contribution. Work was done during internship in Ant Group.
"orresponniig author.

---

Published as a conference paper at ICLR 2026
User: Please amange trovel from LA to Rivermist for me. hope to fly there..
Diologue Context Exomple
AL: Learning From Traditional Environment
Learning From Augmented Environment
R
Ageniggtt www.errrrr.......o httpert_yrrryyyoniiiiioeo wr.rytoonticee
availbbe
"No avallbbe reci?"
"RMSS
Ageet: -th ohiiko-o lake maces dews Eat get pootffon
Check function execution workflow...
Agcetoga fight, ongine:: "LA", "RMB")
Identify the precise location of the problem.
Teal: "ame" "Nouwslaall
Generate more detailed feedback:
Took: "omor" NNo ovellall rour""
SFT: Handcrafted Inter-tool dependencies
Agent: <ahol--No anailable recas at this Lat agan now Blreee pet fight, comttron "LAA", (KMST)
Todl: "errr" "lavdli siport Lt*
a
cernat
Toel dependercy
Aacal: sliai-Wel WelL noed aea the correct code LAwibrek 1pt ponse aaport by ctylloaame
dependency (Insee))
Tesl Teaest axpest; "LAX")
Trads "umeet NNvaraaaaii neae")
Tropped in low quallity rollouts meoningless experiences
Agenttad fight powtfree "LAX,
Agest No sour puotlible hevvees and Rhomiie
Teal: "rnved ove' 310")
Limited Explorction. Foll to Generolize
Inefficient Exploration. Hard Generolize
Superior Exploration. Effective Generdiization !
Figure 1: Limitations of Existing Paradigms and the ENVIRONMENT TUNING Advantage. This figure
contrasts three agent training approaches a travel planning task. (Left) Supervised Fine-Tuning (SFT) static trajectories struggles with generalization. (Center) Reinforcement Learning (RL) in traditional environment provides only sparse, uninformative feedback. (Right) Our approach uses an augmented environment that provides actionable, fine- grained feedback upon failure.
Long interaction chain (C3): Success in multi-turn scenarios demands consistent performance across all interaction turns, where any single failure leads to complete task failure. Each test sample involves multiple user queries, where success requires passing all checks in every turn.
While constructing synthetic trajectories for Supervised Fine-Tuning (SFT) is a dominant strategy (Prabhakar al., 2025; Liu et al., 2024; Yin et al., 2025) to mitigate data scarcity (C1), it suffers from critical flaw: agents trained on these static, synthetic traces often fail to generalize to real-world scenarios, a limitation we demonstrate empirically in Section 4.2.
Reinforcement Learning (RL) (Shao et al., 2024; Yu et al., 2025; Hu et al., 2025) offers natural alternative, promising improve generalization through online interaction and exploration (Chu et al., 2025; Shenfeld et al., 2025). However, this approach is plagued by its own severe challenges. The complexity of the environment (C2) creates critical "cold-start" problem: an agent that not yet proficient cannot explore the vast action space effectively, becoming trapped in cycles of low-quality rollouts and failing to generate the meaningful experiences required for improvement. Furthermore, even if an agent overcomes this initial hurdle, the long interaction chains (C3) inherent to these tasks make the training process notoriously unstable and prone to performance collapse (Wang et al., 2025c; Xue et al., 2025).
This collectively leads to our central research question:
How can we train a high-quality agent for complex, multi-turn tool use under extreme data scarcity, ensuring both generalization and stability?
Addressing the critical intersection of data scarcity, generalization, and learning stability requires a paradigm shift in agent training. In response, we introduce ENVIRONMENT TUNING, novel framework designed to cultivate both generalization and stability from extremely scarce data. At its core, ENVIRONMENT TUNING orchestrates learning via three complementary principles: (1) Structured Curriculum that guides the agent from simple to complex tasks to build skills progressively; (2) Actionable Environment Augmentation that provides corrective hints upon failure, turning dead- end explorations into rich learning signals; and (3) Fine-Grained Progress Rewards that replace sparse, binary outcomes with a continuous measure of task completion, providing a dense and informative learning signal.
By combining this deliberate curriculum with enriched feedback and dense rewards, ENVIRONMENT TUNING enables an agent acquire sophisticated, multi-step behaviors from scratch, demonstrating that robust learning is achievable even in the complete absence expert demonstrations. The key contributions of this work are follows:
A novel learning paradigm for data-scarce environments. We propose ENVIRONMENT TUN- ING, which enables agents learn multi-turn tool-use capabilities directly from problem instances without expert demonstrations, shifting from trajectory- based imitation to environment- based exploration.
2

---

Published as a conference paper at ICLR 2026
Missing parameter: Nome of the decument
The capicaas Jiceuree
Yaal Sccranans
The modil nead oskc for clatfiictione
Whet da
rellemeee doia our jondiat aerilleerryceee
end fedd moriiad Jucie miitiryy
Turn
Missing porometer
Turn 2
buille Recaane andt Bacily
Bone multi-turn
boate Stal dacusarind
Eedgptar:, anac Yort Joceact ix?
Missing function
Milssing function: TM
eopeell Cabl mewe
Suplly Saateatie
The wadd wends to popn out that the requcshed
Jecemeet
Figure 2: An illustration of mulli--trr tool-use scenarios, adapted from an official example in the BFCL V3 Blog (Patil et al., 2025a). All three tracks start from the same initial user request. The Base multi-turn track (center) shows a successful execution path. The Missing parameter track (top) illustrates a scenario where the agent must handle ambiguity by asking for clarification. The Missing function track (bottom) shows case where the agent needs to recognize that required tool unavailable. These scenarios highlight the diverse reasoning capabilities our curriculum designed to address.
A practical curriculum with environment engineering. We develop four-stage curriculum leveraging actionable environment augmentation and fine-grained progress rewards to transform sparse feedback into rich learning signals for effective exploration.
Empirical validation in extreme data scarcity. With only 400 training samples, our method proves effective for both base and SFT-tuned models. It lifts base model like Qwen2.5-7B from near-zero to strong in-distribution performance, and also boosts SFT-tuned models such as watt-tool-8B to 54.25%, surpassing most proprictary models. Notably, it nearly doubles ToolACE-2's s -iitt---ditiiittionn score on ACEBench (8.5% to 15.0%), showing that ENVIRONMENT TUNING fosters robust generalization where SFT often fails.
2 RELATED WORK
Tool-integrated reasoning. A prominent line of research focuses on augmenting LLMs with external tools, a paradigm often termed Tool-Integrated Reasoning (TIR). Many works have leveraged RL as an alternative to SFT for teaching agents strategic tool invocation (Li et al., 2025; Feng et al., 2025a). However, direct application of trajectory-level algorithms (Shao et al., 2024; Yu al., 2025) often leads to training instability and performance collapse (Wang et al., 2025c; Mai et al., 2025; Xue et al., 2025). To address this challenge, recent works have proposed sophisticated mechanisms like fine-graine credit assignment (Feng et al., 2025b), cettropy-guided 2025), and trajectory filtering to prevent gradient explosion (Xue et al., 2025). While effective in their respective domains, such computational reasoning (Li et al., 2025; Singh et al., 2025) or open-domain web search (Jin et al., 2025a; Zheng et al., 2025), these methods are typically evaluated in settings where the primary challenge is to master a single tool small, homogeneous set of tools. Our work, in contrast, addresses the distinct challenge of orchestrating large and diverse toolset to complete complex, stateful tasks that unfold over multiple interaction turns.
exploration (Dong et al.,
Multi-turn tool orchestration. Distinct from tool-augmented reasoning, another major challenge involves enabling agents to orchestrate large set of diverse APIs to accomplish complex, stateful tasks over multiple turns. Benchmarks like BFCL (Patil et al., 2025b) and ACEBench (Chen et al., 2025a) have emerged to evaluate these sophisticated capabilities. However, as these benchmarks are derived from realistic scenarios, they feature high-quality but scarce numan-annotater data, posing a significant training challenge.
To address the aforementioned data scarcity issue, two primary strategies have been explored. The dominant approach involves large-scale synthetic data generation for SFT, where recent works (Prab- hakar et al., 2025; Liu et al., 2024; Yin et al., 2025; Zhang et al., 2024) focus on creating vast corpora of tool-use trajectories. An alternative strategy explores applying online RL directly. Works such as ReCall (Chen et al., 2025b) and ARTIST (Singh et al., 2025) attempt to improve policies through direct environment interaction. However, these online RL approaches have far yielded only modest gains on benchmarks like BFCL, highlighting the difficulty of effective exploration in a complex environment and motivating the need for more efficient learning paradigms.
3

---

Published as a conference paper at ICLR 2026
Agent Learning Loop
Environment Tuning
Curriculum Learning
Rodi State
Modeb
O
Mastering Syntex
Leariing
Salving
Independent selving
bets Carfly
il Eity
il Eiiy
Exppriene
Expeiker Pet Volure
Action
Reward
Proguct
Progros
Iterative Training
Revend
Raqpanza
Anyaatull
Enviroement
Reward Design
Environment Augmentation
Der Update
Pecar: Permen Schane
Syntex Reward
X
Standard Eninomment
Augnenttd Envrstemen
degi: 'Vold' Achional
www.Eoppe.com www.ooosicems *LA", tor 1285%)
www.ffiictcoooooo coat SAA, 1300)
Reword
Tay metet
Tunable Companents
Gosl TTan Saceemr"
Figure 3: An overview of ENVIRONMENT TUNING. Our core innovation is the ENVIRONMENT TUNING module, which implements a four-stage curriculum. dynamically configures the reward function, environment feedback (Standard vs. Augmented), and data split for the Agent Learning Loop. This staged approach transforms ambiguous errors into actionable lessons (highlighted in the Environment Augmentation in Action panel), enabling efficient and stable learning from limited data.
ENVIRONMENT TUNING
To overcome the challenge of learning in complex, data-scarce environments, propose ENVIRON- MENT TUNING, a framework that orchestrates training through three complementary mechanisms (see Figure 3). It combines (1) Structured Curriculum (Section 3.2) for progressive skill acquisition; (2) Actionable Environment Augmentation (Section 3.3) to turn failures into rich learning signals via corrective hints; and (3) Fite-Grained Progress Reward (Section 3.4) provide dense feedback that overcomes the limitations of sparse signals. These pillars collectively create a manageable learning path for stable and data-efficient agent training.
3.1 PRELIMINARY AND CHALLENGES
This work tackles multi-turn tool use: challenging task where an agent must achieve a complex goal through a series of interactions with external tools. Unlike single-step problems, this process dynamic and demands sophisticated reasoning. as a single task can unfold in numerous ways (Figure 2). The agent must navigate an interactive loop of understanding user requests, executing tool calls, and generating responses until the overall objective complete. We mathematically formalize this problem as Partially Observable Markov Decision Process (POMDP) in Section B.
On the challenge of multi-turn tool use. Training an RL agent for multi- turi tool use is non-tum toivtrivial: an agent must simultaneously master low-level syntax skills (e.g.. precise tool-call formatting that the environment can parse) and high-level reasoning abilities (e.g., multi-step planning across dependent subtasks). In complex interactive environments, base models lacking these skills exhibit diverse range of failure modes, including void incorrect parameter filling, calls non-existent tools, and irreversible environment corruption.
As noted by Xue et al. (2025), RL optimization such noisy trajectories is extremely sensitive to these error patterns: if they persist in the rollouts, training instability and gradient explosion are likely. Our numerical experiments also confirm this fragility: when fine-tuning Biven2.5-7B-Ime Instructt directly in a single-stage RL. setup with 400 training instances, training collapsed within 70 steps, yielding mere 10% improvement in success rate, as detailed and visualized in Appendix Section F.2
3.2
STRUCTURED CURRICULUM RL TRAINING
The aforementioned observations suggest that naively optimizing for full task success from the start is ineffective in long-horizon, sparse-reward settings. We therefore propose a structured curriculum that gradually increases objective complexity. This four-stage curriculum allows the agent to progress from mastering foundational skills to handling the full complexity of multi-turn tool use, while maintaining training stability.
Stage 1: mastering syntactic and schematic correctness. The goal of this initial stage is to train
the agent to produce well-fom oolmffomedd outputs with valid tool calls, forming reliable foundation for all sub- sequent learning. Before an agent can reason effectively, it must first "speak the language" of the envi- ronment; otherwise, learning is confounded by penalties for both poor strategy and invalid formatting.
4

---

Published as a conference paper at ICLR 2026
To isolate this foundational skill, deconstruct the agent's actions (a) and at and design
reward function focused exclusively on their structural integrity. As illustrated in Figure 3, we use several task-agnostie, turn-by-tun counters:
Corrret:: the number of tool calls with correct tool name and valid arguments;
Cerror: the number of calls with a correct tool name but invalid arguments;
format: the number of turns violating the required XML-like format.
These counters are combined into a Format Reward Riormat and Tool Call Reward Rooo, defined as
Riomaat = (N Cioomat)//
Rial = Ceorrea/ / (Ceomect + Cerror)
where N is the total number of turns in the episode. The final reward for this stage, Rstagel. combines these components and is gated by indicator Toot that is 1 if the agent attempts at least one tool call and O otherwise, encouraging active tool use:
RSiagol = 1ool
(Bioormat + Rgool).
As demonstrate in our analysis of training dynamics (Section F.1), this specialized reward allows the agent to rapidly master the required syntax. This proves that dedicating stage to this foundational skill is crucial prerequisite for efficient learning in more complex, task-oriented stages.
Stage 2: basic learning with augmented feedback. With the agent now proficient in syntax, the
curriculum transitions to learning task task-oriented reasoning. This stage utilizes the full Base split from BFCL, focusing on foundational multi-turn capabilities. To accelerate learning, critical components to allow the agent to efficiently learn core reasoning skills in a guided manner.
introduce two
1. Progress Reward (c.f. Section 3,4): we replace the task-agnostic reward with a Progress Reward (Rr), fine-grained measure of task completion that provides more informative signal than sparse binary outcomes (detailed in Section 3.4).
Actionable Environment Augmentation (c.f. Section 3.3): we employ such augmentation to enable the environment to provide detailed, corrective hints upon failure instead of ambiguous error messages, turning failed explorations into valuable learning opportunities.
Stage 3: advanced learning on complex scenarios. Building the foundational skills from the previous stage, the agent is now exposed to the full spectrum of challenges. We introduce the complete training dataset, incorporating samples from the Mi ss ng Parameters, Mi ssi ng Functi ions, and LlongConteex splits. The objective is to train the agent to handle ambiguity, recognize functional gaps, and perform information retrieval from noisy contexts. The training setup remains consistent with Stage 2, continuing to leverage both the Progress Reward and Actionable Environment Augmentation help the agent navigate these more complex problem spaces.
Stage 4: alignment with the evaluation environment. The final stage is designed to align the agent with the true evaluation conditions. While still training on the full dataset with the Progress Reward, we now disable the Actionable Environment Augmentation. This forces the agent to generalize its learned policies, relying on its internal reasoning capabilities to handle uninformative or standard error messages, just as it would during final evaluation. This step is crucial for ensuring that the agent's 's performance is robust in OOD scenarios and not overly dependent on the training scaffolds provided in earlier stages.
Remark 3.1 (Checkpoint selection for stage transitions). A key element of our curriculum is deter- mining when to transit to the next stage. In our multi-turn tool-use setting, we adopt a validation- advance to the next stage only when the validation
and stability-based based stage transition rule: accuracy has plateaued and its gradient norm is stable.
This joint condition, further discussed in Section F.I, serves two purposes. First, the converged validation performance ensures the agent has mastered the current stage's skills. Second, the stable optimization dynamics confirm it is ready for more complex tasks, mitigating the risk of gradient explosions common in long-horizon RL (Xue et al., 2025). This rule is crucial maintaining training stability and maximizing the curriculum's effectiveness.
3.3 ACTIONABLE ENVIRONMENT AUGMENTATION
In multi-turn tool-use tasks, an agent often needs to execute complex chains of function calls where the output of one tool becomes the input another. However, standard training environments typically
5

---

Published as a conference paper at ICLR 2026
return cryptic overly generic error messages (like simple error codes or "not found" responses) that neither identify the root cause of failure nor indicate a viable next action. This lack of actionable guidance severely limits the agent's 's ability to discover dependencies between tools and understand each tool's usage constraints, forcing it to rely on inefficient trial-and-error exploration, which leads poor performance (as validated by the ablation in Section 4.3 that isolates the effect of environment augmentation). To address these issues, we design Actionable Environment Augmentation, which modifies the environment's feedback to provide pedagogical hints that directly inform the agent about dependency relationships and operational rules, turning failed trajectories into constructive learning opportunities.
Discovering inter-tool dependencies via guided discovery. Our first goal is to empower the agent to discover and resolve inter-tool dependencies on its own. While prior work often relies on pre-constructed dependency graphs explicitly teach agents sequential tool-call patterns (Prabhakar et al., 2025; Liu et al., 2024), our method embeds these dependencies within the environment's feedback. This encourages the agent learn the underlying logic through interaction and inference rather than memorization, leading to better generalization in unseen scenarios.
For example, in the BFCL. Travel API task, agent might incorrectly try to book a flight without first finding the correct airport code.
Standard feedback: A vague message like "No available route" gives the agent clue about the root cause of the failure.
Augmented feedback: Our environment provides precise hint: "Invalid airport code [a] :..... This message suggests that another tool is needed to find the correct airport
code first, effectively prompting the agent to discover the dependency through interaction.
Revealing internal tool constraints with pedagogical hints. Our second objective is to provide actionable hints that reveal tool's specific internal rules and constraints. This moves beyond the simple diagnostic errors returned by standard code interpreters (e.g.. "Filelh PrileNctFF Fot.oouoErrrrooo toward providing pedagogical feedback that explains why an action failed. By revealing the "rules of the game" to the agent, we dramatically prune the exploration space by invalidating entire classes of incorrect attempts.
For instance, an agent's pretrained knowledge might suggest using full paths with the "rm" command. However, the BFCL File System environment may not support this.
Standard feedback: A generic "Fi Feooo would be misleading, as the file might actually exist.
Augmented feedback: feedbacc::: Our environment returns explicit rule:
"Paths are not allowed. Specify only file/direct dirtorrryy name... This hint directly corrects the agent's mis- understanding of the tool's protocol within this specific environment.
This L.L.-driven workflow for generating augmentations is detailed in Section C. illustrated in the file system and multi-API travel case studies (Sections H.1 and H.2), our augmented feedback transforms failed turns into explicit, actionable guidance. This enables agents to diagnose root causes accurately and discover corrective strategies that would be infeasible under baseline environments with ambiguous signals.
3.4 FINE-GRAINED PROGRESS REWARD
key challenge in training long-horizon, multi-turn tool-use agents is reward sparsity: a single binary signal at the end of long trajectory provides insufficient guidance for effective learning (Feng et al., 2025b).
To overcome these challenges, we introduce a fine- tineigrained Progress Reward (Rp) that provides a denser, turn-by-turn learning signal. Instead of rewarding individual tokens, we evaluate success at the end of each turn based on two criteria: the correctness of the resulting environment state and the execution result of the agent's 's chosen action. This allows us distinguish "nearly correct" from "completely wrong" trajectories.
Formally, for each turn in an episode of length T, we define binary scores for the state evaluation (retule) and execution result evaluation (reve)). ). A turn is successful only if both are correct, with its reward being their product: r; = state y The total Progress Reward Rp then the average
success rate across all turns: Rp = + E r; = + Σi= (r ) This formulation provides a rich, informative signal throughout the episode, enabling the agent to learn efficiently from partially

---

Published as a conference paper at ICLR 2026
Table 1: Main results the BFCL V3 multi-turn benchmark. Our method, ENVIRONMENT TUNING. significantly boosts the performance all base models, achieving competitive results against proprietary models and outperforming several strong baselines using only 400 training samples.
BFCL V3 Multi Turn
Model
Average (%)
Base (%)
Miss Func (%)
Miss Param (%)
Long Context (%)
Claude Sonnet 4
57,00
63.00
58.00
51.00
56,00
GPT-4o
51.00 21.00
59.00
54.00
41.00
30.00
Gemini 2.5 Pro 63
28.75
32.00
29.00
22.00
32.00
49.25
47.00
55.00
47.00
48.00
Arcb-Agent-7B
42.05
47.15
53.75
34.20
33.10
XLAMM-2-b--i--
70.50
77.85
69.15
65.80 65.80
69.20
BitAgent--85
36.99
47.85
33.20
26.15
40.75
Qwen2.5-7B-Instue
7.00
6.33
3.00
+ -TooRR
18.00 (+11.00)
23,00
(413.67)
26.00 (416.67)
16.00 (49.67)
7.00 (+-4.00)
+ -ENVIRONMENT TUNING
36.92
(+-99.92)
50.33
(+411.0)
40.33 (+31,00)
29.33 (+-2.00)
27.07 (+244.67)
5.48
3.20
5.75
ToooRL
11.25 (45.77)
(+2.85)
10. 0. (+3.20)
9.00 (45.80)
17.00 (+11.25)
ToolACE---Llama-3)) 28 25 (+22,77)
+ -ENVIRONMENT TUNING
28:20
(+22.05)
25.85 (+19.05)
22.15 (+18.95)
36, 80 (+31.05)
37.99
48.85
34.15
25.20
43.75
+'oolRI.
33.75 (-4.24)
(-2.85)
29.00 (-5.15)
(+3.20)
38.00 (+5.75)
-ENVIRONMENT TUNING
47.18 (+9.19)
55.20 (+6.35)
38,15 (+4.00)
38.20 (+13.00)
67.15 (+13.40)
watt-tool-88 8B
35.74
46.83
35.13
25.20
33.77
-ToolR
12.00 (+6.26)
55.00 (49.15)
35.00 (+1.8)
31.00 (+5.80)
47.00 (+8.25)
ENVIRONMENT TUNING
54.34 (+18.50)
64.15 (+18.30)
48.15 (+15.00)
40.20 (+15.00)
64.85 (426.11)
successful attempts while still having the freedom to discover novel problem-solving strategies. A detailed breakdown of the evaluation criteria is provided in Section D.
EXPERIMENT
4.1 EXPERIMENT SETTINGS
Benchmark. Our primary evaluations are conducted on the multi-turn subset of the Berkeley Function-Callle CallinC Leaderboard (BFCL) V3 (Patil et al., 2025b). This benchmark comprises to- tal of 800 samples, divided equally into four challenging splits: Base, Missing Functions Missing Parameters, and Long-Context. More specifically, we construct a training set of 400 samples by selecting 100 from each split. The remaining 400 samples serve as our held-in test evaluate in-distribution performance. For nut-of-distributin (OOD) evaluation, BFCL V4 web search and memory tracks (Patil et al., 2025b), 7 7"-bench (Barres et al., 2025) and ACEBench Agent split (Chen al., 2025a) as our held-out test set. Notably, our training set (BFCL V3) is confined to -oosand API calls, while our OOD benchmarks introduce entirely distinct domains and capabilities. This clear separation ensures a rigorous evaluation of generaliza- tion. A detailed description of these benchmarks and their evaluation methodologies is available in Section E.I
the
Models. To comprehensively evaluate our approach, we categorize the models used in our experi- ments into three groups.
Base models: These are the open-source models upon which we apply ENVIRON- MENT TUNING method. We select Qwen2
(Qwen et al., 2025) and (Grattafiori et al., 2024) as our primary base models. As suc- cessfully applying reinforcement learning to Alama-based models has proven difficult (Zeng et al., 2025; Wang al., 2025b; Gandhi et al., 2025), also include two SFT-tuned versions of Llama- -.I-SB-Instruct: TocEACE- 2-L1 amaa-- 3. 1--8BBB (Liu et al., 2024) and watt-tool-880 Us- ing these SFT-tuned models base models allows to demonstrate the general applicability of ENVIRONMENT TUNING on models that are already strong in tool use.
Llama 1eama-3.. 10.1-8B-Insse
Open-source baselines: To contrast our environment-centric RL approach with prevailing data- driven methods, we compare against four state-of-the-art SFT-tuned models: Arch Arch-Asee Agentt-nn 2LAM-2-8b-fc-- (Prabhakar al., 2025), and BitAgent-8s As discussed Section
'https: hhtpss://uugginngfccc.o/// http:///eeeoooeaaaaaaaattoo--a--
hhttps://huggingge co/ ttgent//HitAgentt-B
7

---

Published as a conference paper at ICLR 2026
Table 2: OOD Generalization performance on BFCL V4, r"-benhh, and ACEBench Agent benchmarks. All results are compared against the -lama-3.1-B-Insttuue base model. Blue text indicates a performance improvement over the base model, while orange text indicates a performance degradation. Models trained with MNTT TUNING (rows in blue) show consistent improvements, whereas SFT haselines (rows in red) often underperform the base model on these 0OD tasks. Scores for xI. the Retail and Airline domains are grayed out as it was trained on the original 7-bench, making them invalid for OOD evaluation.
ACIBBench Agrnt
(ng.(%)
Web Search (%)
Memury (5)
Arg. (5)
Rotail (%)
Airline (S)
Telecom (%)
Arg. (%)
Mulli-tura (%) Multi-stpp (%)
2-AM--------
9:17
5.00
13:33
35.57
57.50
40000
12:50
1.63
0.00
3:33
7.41
4.50
10.32
7.50
15:00
10.00
1lama-3 1-9B-haaadd
5:46
1.00
90.00
17.50
0.00
TTolli.
11.32
11.80
12.84
21.87
10.00
23:80
30.00
183
0.0
3.33
+ ENNIROMMEN TUNISO
16.53
15.80
18.86
21.67
5.00
30,00
38,00
4.17
5,00
331
15.90
9,00
22.40
10.53
10.00
15:00
7.59
8.34
10.00
6.67
18:91
9.01
29.3
14:17
5.00
90.0
7.50
3.30
10.00
EXVRRONMEET TesING
16.79
38.00
-3.3
1,500
-3.30
3,00
TeaRL
13.12
1.50
12.50
15:00
3.30
15.00
ENVIROMMNT TUNING
13.48
K.00
19.35
15.83
20,00
28.80
7.50
7,50
0.00
15.00
these models are representative of the dominant data synthesis paradigm to handle data scarcity
issue (C1).
Proprictary models: To benchmark against the absolute state-of-the-art, we include leading proprictary models such as Claude Sonnet 4 (Anthropic, 2025), GPT-4o (Hurst et al., 2024), Gemini 2.5 Pro (Comanici et al., 2025), and o3 (OpenAI, 2025b). These models serve as a reference for top-tier performance.
Agent training. For agent training, we employ an adapted version of the Optimization (GRPO) algorithm (Shao et al., 2024), enhanced with a decoupled clipping mechanism and Kl--divereence penalty to ensure stable and effective exploration; full implementation details are provided in Section F.2. As a baseline, we also implement ToolRI. (Qian et al., 2025), which applies the GRPO algorithm directly with a reward function of Rioommtt + Reorrett where Reorrett measures the correctness of tool calls.
Relative Policy
4.2
MAIN RESULTS
Results on multi-turn tool use. Our method demonstrates substantial effectiveness in the in- distribution, multi-turn tool use scenarios presented BFCL V3. As shown in Table 1, ENVIRON- MENT TUNING yields significant performance gains across all base models.
Significant performance uplift from scratch. When applied directly base models, ENVIRON- MENT TUNING proves to be a highly effective training paradigm that consistently works across different model architectures. For instance, it boosts Qwen2.5-7B 7---5--77-nntt score from 7.00% to 36.92%, surpassing two strong basclines (BitAgent-8P) 8BB and Arch-Agent-7B) and the proprictary Gemini 2.5 Pro model.
Effective enhancement of SFT-tuned models. Our method also serves as a powerful online refinement tool for models that have already undergone SFT. On model built upon the RL-challenging Llama architecture, ENVIRONMENT TUNING still provides a significant improvement (9.19%). This elevates its performance to 47.18%, surpassing the proprietary Gemini 2.5 Pro model. Similarly, on watt-tool-SB, ENVIRONMENT TUNING achieves an impressive 18.50% improvement, boosting performance from 35. 74% to 54.34%, which exceeds most proprietary models including o3 and GPT-40.
1-8B, a
Superiority over direct RL. Our method consistently and substantially outperforms the TooIRL baseline. While ToolRL alone yields only modest gains and can be unstable on SFT-tuned models (e.g., degrading -AAAA------am--3----0 delivers much larger improvements across the board (e.g.. +9.92%% on Qwen2.5- Inn-ttrret and +9.19% on ToolACE-2-Llamar -.--aamm--.1132 1-8B),, underscoring the necessity of our curriculum and environment augmentations for effective learning.
performance by -4.24%), ENVIRONMENT TUNING
Results OOD agentic tasks. One strength of ENVIRONMENT TUNING lies in its ability to foster robust generalization, which we evaluate on the OOD benchmarks from BFCL V4, r2-bench, and ACEBench Agent (Table 2). The results reveal clear distinction between ENVIRONMENT TUNING and supervised fine-tuning on trajectories.
Superior generalization over SFT baselines. The limitations of overfitting to static datasets become evident here. Both SFT baselines exhibit a dramatic performance collapse on the OOD Web Search task, with even the tor-performer, ALAM-2 (70.50% on BFCL V3), dropping to
8

---

Published as a conference paper at ICLR 2026
Aegrenzzion
Base (Stage 2)
Missing Fanctions (Stage
Base (Stage
Missing Functions (Stage
Widdrro
Tranning Seps Missing Parumeters EStage 3)
Trating Skgns Long Contest (Neage 3)
Triniig Steps
Traniin Sippe
Missing Parnmeters (Stage 3)
Long Context (Stage 3)
Valderte Accency
Traning
(a) Ablation study for environment augmentation.
(b) Ablation study for progress reward.
The effect of Actionable Environment Augmentation on learning stability and performance across different data splits. (b) The impact of fine-grained Progress Reward versus binary reward on training effectiveness, showing the critical role of dense reward signals in complex multi-turn scenarios.
Figure 4: Training dynamics comparison for ENVIRONMENT TUNING
just 5.00%. This pattern persists across diverse OOD benchmarks, which underscores a critical weakness in training solely on synthetic trajectories.
ENVIRONMENT TUNING fosters robust generalization. contrast to the SFT baselines, our method demonstrates superior generalization by training agents through direct environmental interaction. For instance, ENVIRONMENT TUNING transforms Llama-3. 1-8.--8B-laa 1-B-I---rttt which performs poorly on the Web Search task (1.00%), into a strong performer at 15.00% showcasing its ability to teach general problem-solving principles rather than dataset-specific patterns.
Enhances generalization of already proficient models. Our method can also patch the gen- eralization gaps left by SFT. The base ToolACE-2 model already shows decent OOD performance (9.00% Web Search), far exceeding other baselines. Yet, ENVIRONMENT TUNING further enhances its capability, increasing its score to 14.00%, demonstrating that interactive learning crucial for building truly adaptable agents. This enhancement extends to ACEBench Agent, where ENVIRONMENT TUNING boosts ACE--'ss average score from 8.34% to 15.00%.
Advantage over direct RL on complex OOD tasks. Our framework demonstrates stronger and more stable generalization than the ToolRL baseline, particularly on complex benchmarks like ACEBench and 72 bench. For instance, on ACEBench Agent, ENVIRONMENT TUNING lifts ToolAACE-2-Llama-- 1-8B's score from 8.34% to 15.00%, whereas direct RL causes performance drop to 6.65%. This highlights that our curriculum and augmentations are key to fostering robust, generalizable capabilities not achieved by direct RL alone.
4.3 ABLATION STUDY
Effeec of structured curriculum. To isolate the impact of our four-stage curriculum, we conduct an ablation study comparing our full method against two baselines on the Qwen2.5- 7B-Instruct model: (1) the base pre-trained model, and (2) direct GRPO training baseline. For this baseline, the agent trained on the full 400-sample dataset from the start using com- bined reward function of 0.9 Rp 0.1 R without changing all other hyperparameters. As shown in Table 3, directly applying RL yields minimal gains, highlighting the "cold- start" problem where the agent fails to learn effectively. In contrast, our curriculum provides clear and steady improvement path. Each stage brings varying degrees of improvement, ultimately boosting the final performance to 36.92% 19.50% increase over the direct GRPO baseline.
Table 3: Effectiveness of the structured curriculum Qwen2.5-7B. Performance comparison across diffcrent training stages, showing how each stage contributes to overall improvement. Abbreviations: M. Func = Missing Functions, M. Param = Missing Parameters, L. Ctxt Long-Cone Long-Context.
Truining Stage
V3 Multi
Avg.
Base
M. Func
M. Param
Ctxt
Qwen2.5- 2on25.5-7 1--trrnst
7:00
9.33
9.33
6.33
3,00
""RRNOO"
17.
"20.00"
"0.07"
14.07
10.35
Stage
Stage 2
25.83
32 00
33,67
20.00
17.07
Stage 3
32.00
41.67
$1.33
25.33
23.07
Stage (ours)
36.92
50.33
40.33
29.33
27.67
Effect of actionable environment augmentation. Environment Augmentation, conduct an ablation study comparing training dynamics with and
To validate the contribution of our Actionable

---

Published as a conference paper at ICLR 2026
without the augmented feedback mechanism. As illustrated in Figure 4a, the augmented environment provides crucial learning signals that enable the agent to navigate complex scenarios more effectively,
The augmented environment consistendy leads to more stable learning curves across all data splits, particularly in the challenging Missing Parameters and Missing Functiors scenarios.
For the Missing Parameters and Missing Functions splits, Environment Augmenta- tion brings substantial performance improvements of over 20%, demonstrating its critical role in enabling effective learning on these complex, ambiguous tasks.
Effect of fine-grained progress reward. We also conduct an ablation study to verify the effec- tiveness of our fine- ane-grained Progress Reward (Rp) against standard binary reward. As shown in Figure 4b, the impact of the reward design varies significantly with task complexity:
During Stage 2, which focuses the Base data, the performance difference is subtle, suggesting that a simple binary signal is sufficient for foundational tasks.
However, the curriculum progresses to more complex splits in Stage the necessity of denser reward becomes starkly evident. For the Mi sssiing ssing Parameters and Miss ssing Funct i ons splits, the binary reward leads complete Iraining failure, with performance close to zero, as the sparse signal provides no incentive for the necessary exploratory actions.
Meanwhile, on the Long Context split, our Progress Reward leads to substantially more stable and effective learning.
5 CONCLUSION
In this work, we proposed ENVIRONMENT TUNING, novel training paradigm for multi-turn, tool- augmented agents under extreme data scarcity. By shifting the focus from trajectory imitation to environment--driven exploration, and combining structured curriculum, actionable environment augmentation, and fine-grained progress rewards, our method enables agents to learn stably and generalize from only 400 problem instances without expert demonstrations. Experiments on BFCL show that ENVIRONMENT TUNING not only yields substantial in-distribution improvements but also significantly enhances -it-----iittibutiion generalization, outperforming several strong supervised baselines. We believe this environment-centte approach offers promising direction for developing robust, adaptable agents in realistic, resource- limited settings. Our work also opens several exciting avenues for future research, including the development of automated mechanisms for curriculum and feedback generation and the extension of ENVIRONMENT TUNING to more complex, multi-modal agentic scenarios.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENT
This work was supported in part by the NSFC under No. 62576285, the Ant Group Research Collaboration Project, Research Center for Industries of the Future (RCIF) at Westlake University, and Westlake Education Foundation.
7 ETHICS STATEMENT
This work uses only publicly available datasets and open-source models, involves no human subjects or sensitive data, and complies with the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
To support the reproducibility of our findings, we commit to releasing our complete source code, including implementations of the ENVIRONMENT TUNING framework and all training and evaluation scripts, as supplementary material. The core principles of our method are presented Section 3. Detailed specifications essential for replication are provided throughout the appendix: the formal problem definition is in Section B, the progress reward mechanism is in Section D, and the full prompt templates used for training augmentation are available in the case studies in Section H. A comprehensive description of our experimental setup, covering the datasets (Section 4.1 and Section E.1), models (Section 4.1), and the RL training algorithm with all hyperparameters Section E.2). is provided. Furthermore, Section F contains supplementary experiments and analyses on training dynamics and stability to aid in the successful replication of our results.
in Section G. Concrete examples of our actionable environment
REFERENCES
Anthropic. System Card: Claude Opus 4 & Claude Sonnet 4. https: htww://wwwwwcdccd anthropic ccom/ 33339990abb444aaam 6a (00055666644433224444 fdf, 2025.
Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimban. y2-bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07992. 2025.
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, al. Acebench: Who wins the match point in tool usage? arXiv preprint S.Ki::2501./285// 2025a.
Mingyang Chen, Linzhuang Sun, Tianpeng Li, Yijie Zhou, Chenzheng Zhu, and Fan Yang. Recall: Learning reason with tool call for ilms via reinforcement learning. Online Notion Blog Post, 2025b. URL
almandine 935. not ion. -:i_gg-----aaaaa--------ooo----- inc=3
via- -ewnnoooeeeen---aaaeeeeee oereeo---L--iiii---eeeeerreeeeee
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V
Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: comparative study of foundation model post-training. In Forty-second International Conference on Machine Learning, 2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint urXin:2507.0626/ 2025.
Guanting Dong. Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint (0o::2507.19445 2025.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504. /536, 2025a.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-growe policy optimization for Ilm agent training. arXiv preprint arXiv:2505. 10978, 2025b.

---

Published as a conference paper at ICLR 2026
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprini arXiv:2505.22312, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXhv:2103.038// 2021.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXivv2501.03300 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.212266 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2023.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training iims reason and leverage search engines with reinforcement learning, arXiv preprint 2rXiv:2503.09516 2025a.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning, 2025b.
Hynek Kydlíček. Math- Verify: Math Verification Library, 2025. URL URLhtttss///iithub com/h uggingface//att -eriff..
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383, 2025.
Mingjic Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXhv:2505.24864 2025.
Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, et al. Toolace: Winning the points of Imm function calling. arXiv preprint XrXiv22409.00920 2024.
Xinji Mai, Haotian Xu, Weinong Wang, Jian Hu, Yingying Zhang, Wenqiang Zhang, et al. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving, arXiv preprint arXiv:2505.07773. 2025.
Moonshot AI. Kimi-researcheer End-to-end rl training for emerging agentic capabilities. Technical report, Moonshot AI, 2025. URL http:atpss/ooooooooaaaiiiiieeoie httpp::/moonshootllggtuuu....i /
io/Kimi-Researc Reesaarche/......ree..
her/. Accessed via https:ttt.............. htttttt:
OpenAl. Deep research system card. System card, OpenAI, February 2025a. URL https:
cttp:///a2ccmmm/////22111 //cdn.opeeai com www.deeeeeeeeeeeeeeeeeeeeeeeeeee dewpdreeearrreeheeeeeeeeeeeeeeee sard-syysteeemadd......... dbee--rreearrdd-yyyytttc-ccrarad
OpenAl. OpenAl o3 and o4-mini System Card. 2025b.
12

---

Published as a conference paper at ICLR 2026
OpenAL. Operator system card. System card, OpenAI, January 2025c. URL httph: open ai. com/ operator. sasssssssteem larastem__aaaae card_.ardd.. par.
Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Yixin Huang, Xiaowen Yu, and Joseph E. Gonzalez. Bfel v3: Multi Multi-turn & multi-step function calling. Gorilla Blog Post, September 2025a. URL https: wwgrrirlee
berkeley. edu/blogs// turn. html. Release date: 2024-09-19. Last updated: 2024-12-10.
_bfcl_v3__a__
Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfel): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025b.
Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, et al. Apigen-mt: Agentic pipeline for multi-tumn data generation via simulated agent-human interplay. arXiv preprint arXiv:2504.03611, 2025.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward all tool learning needs. arXiv preprint arXiv:2504. 1atee25504.1355,, 13958, 2025.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning open language models. arXiv preprint arXiv:2402.03330. 2024.
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RI's razor: Why online reinforcement learning forgets less. arXiv preprint arXiv:2509.0422 2025.
Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for Ilms via reinforcement learning. arXiv preprini urXiv:2505.01441, 2025.
Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models. arXiv preprint arXiv:2503.21 2025.
Jean Vassoyan, Nathanaël Beau, and Roman Plaud. Ignore the kl penalty! boosting exploration critical tokens to enhance rl fine-tuning. arXiv preprint urXiv:2502.06533 2025.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): 186445, 2024.
Xinyuan Wang, Bowen Wang. Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, al. Opencua: Open foundations for computer-use agents. arXiv preprint arXiv:2508.00122; 2025a.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512, 2025b.
Zihan Wang. Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding svol-tvo in Ilm agents via multi-turn reinforcement learning, arXiv preprint arXiv:2504.2007. 2025c.
13

---

Published as a conference paper at ICLR 2026
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.15511 2025.
Lilian Weng. Lim-powered autonomous agents. filanwengggithubbie io, Jun 2023. URL https:
//lilianweng github...........g...... / http/sssss. s 2022-002--626 200-23-agge 333-gget/....
Ronald J Williams. Reinforcement learning and markov decision processes. CSG220, Spring, 2007.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li. Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040-5200 52094, 2024.
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An. Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning, 2025.
John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 7:50528-50652, 2024.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. r-bench: A benchmark for tool-agent-user interaction in real-worke domains. In The Thirteenth International Conference Learning Representations, 2025.
Fan Yin, Zifeng Wang, I Hsu, Jun Yan, Ke Jiang, Yanfei Chen, Jindong Gu, Long Le, Kai- Wei Chang, Chen- Yu Lee, et al. Magnet: Multi-turn tool-use data synthesis and distillation via graph translation. arXiv preprint arXiv:2503.0788 07826, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjunn Liu, et Dapo: An open-source I1m reinforcement learning system at scale. arXiv preprint 25XX::25003.44766 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 2oo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar arXiv::503. 18892, 2025.
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, et al. xlam: A family of large action models to empower ai agent systems. arXiv preprint 2xXi:22409.03215, 2024.
Yuxiang Zheng. Dayuan Fu, Xiangkun Hu, Xiaojic Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real- world environments. arXiv preprint arXiv:2504.03160 2025.
14

---

Published as a conference paper at ICLR 2026
CONTENTS
1
Introduction
2
Related work
3 ENVIRONMENT TUNING
3.1
Preliminary and challenges
3.2
Structured curriculum RI. training
3.3
Actionable environment augmentation
3.4
Fine-grained progress reward
4
Experiment
4.1
Experiment settings
4.2
Main results
4.3
Ablation study
5
Conclusion
10
Acknowledgment
11
7 Ethics statement
11
8 Reproducibility statement
11
Use of LLMs
16
Sequential Multi-turn Decision-Making Model.
16
C Implementation of Actionable Environment Augmentation
16
C.1 Augmentation Generator Prompt
16
C.2 Constitutional Judge Prompt
18
C.3 Workflow and Practical Considerations
20
Detailed Progress Reward Components
Implementation Details
21
E.1
Training Dataset
21
E.2
Training Details
22
F Supplementary Experiments
22
F.1 Training Dynamics
22
F.2
Training Instability in Single-Stage RI.
23
F3
Loss Coefficient
24
Prompt Templates
24
H Case Study
26
H.I File System Management Scenario
26
H.2
Multi-API Travel Scenario
H.3
Vehicle Control and Mixed-API Scenario
37
H.4 OOD Generalization on Multi-Issue Troubleshooting
15

---

Published as a conference paper at ICLR 2026
A USE OF LLMS
This paper only uses LLMs for polishing.
B SEQUENTIAL MULTI-TURN DECISION-MAKING MODEL..
We model the multi-turn tool-use task as Partially Observable Markov Decision Process (POMDP) (Williams, 2007). An episode corresponds to complete user task, which is composed of a sequence of pre-defined user requests, or "turns."
Let's denote the sequence of user requests as q1, 92...........................9.. 2N. The interaction begins with the agent receiving an initial observation on containing the first request qı and the available tool documentation. The agent then engages in series of steps to address this request. At each step 1, the agent's policy To ( (or) generates action d, from structured action space A, which includes:
Tool Call (oo))); ): A structured call to one or more tools gather information (e.g.. ... ..lll........ll..............ll. (ottol_call>),. The environment executes the call and returns obser-
<tool_calll
vation containing the tool's output.
Final Answer (a:nswer):
(e.g.. <answer>... wtaprr....../ssswes...ee.e </answer>). This completes the current sub-task, after which the envi- ronment presents the next pre-defined user request.
A natural language response to the
The process for single turn i unfolds final answer. Upon generating a the environment transitions by revealing the next request 9:+1 within the new observation 0++1- This cycle repeats for all N requests.
sub-trajee -rajecrjyytory of tool calls until the agent produces
The entire episode, a trajectory т = (oo, dn 01, 01 Or), concludes only after the agent has provided final answer for the last request, QN- It is only at this point, after T steps, that the agent receives sparse and terminal reward Ry E {0, 1}. indicating the overall success or failure of the entire multi-turn task. This delayed and binary feedback makes it extremely difficult for RL algorithms to perform effective credit assignment and exploration, a well-known challenge in long-horizon tasks (Wang al., 2025c: Feng al., 2025b).
The agent's 's objective is to learn the policy parameters that maximize the expected terminal reward:
J(0)= E [RT]]
C IMPLEMENTATION OF ACTIONABLE ENVIRONMENT AUGMENTATION
critical aspect of implementing our Actionable Environment Augmentation is ensuring its practical- ity and scalability. While crafting high-quality, pedagogical feedback might appear to require exten- sive manual engineering and deep domain expertise, we address this challenge with semi-automated, LLM--drieen workflow. We demonstrate that leveraging the in-context learning capabilities of large language models, combined with a small set of few-shot examples, enables scalable and high-quality augmentation generation.
Our implementation employs two-agent workflow orchestrated by LLMs: (1) an Augmentation Generator that analyzes error trajectories and produces enhanced feedback, and (2) Constitutional Judge that validates the augmented feedback to prevent solution leakage while ensuring actionability, This automated pipeline significantly reduces the manual burden of designing augmented feedback for each error case, making our method practical and generalizable to new domains.
Below, we present the core prompt templates that define these two agents, demonstrating how structured prompting with clear criteria and few-shot examples enables effective augmentation generation.
C.I AUGMENTATION GENERATOR PROMPT
The Augmentation Generator is responsible for determining whether error message requires augmentation and, if so, producing enhanced feedback that is both actionable and pedagogical.
Augmentation Generator System Prompt
You are an expert at analyzing error messages in multi-turn tool-use scenarios.
16

---

Published as a conference paper at ICLR 2026
Your task to determine whether an error feedback needs augmentation to be more actionable and educational
Judgment Criteria
## 1. Need Augmentation (need augmentation: true/false)
Does this error Seedback need enhancement? Consider:
Is the error message vague unclear?
Does lack actionable guidanec?
Does it hide critical dependency relationships?
Does it fail to explain violated tcel constraints?
Error Category (error_category)
What type error is this?
Categories:
dependency missing: Missing prerequisite tool calls (need to call other tools first)
parameter_conttraleeeeeeeeee
Parameter constraint violation (incerrect parameter format or type)
state Siaate_prereqisitee State precondition not met (incorrect environment state)
format_iismatcn Format mismatch (incorrect input/output format)
resource_unavallabee Resource unavailable (requested rescurce does not exist)
# 3. Augmentation Type nuugenntation_tin what type of enhancement is needed? rypes:
Stariffycoostraing Clarify constraints (explain tool usage rules)
prerequisite: Suggest prerequisite steps (hint what needs to be done first)
explain__dpenddeny, Explain dependencies (describe inter-tool dependencies)
correct_format: Correct format errors (point out correct format requirements)
Output Format
You must respond with valid JSON object. Put your reascning FIRST to encourage step-by-step thinking:
"reasoning"; "Your detailed analysis and explanation of why you made this judgment. Think step by step here.",
"need_ augmentation"; true/false,
"error_ cateooryy"""""" "category_ name" null,
cangmentation__ypell syype_name, or null,
"enhanced error message" or null
Guidelincs for Augmented Feedback If augmentation is needed, generate feedback that:
1. **clarifies the root cause** of the error
2. **Provides directional guidance** without reveal ing specific solutions
3. **Maintains exploration space** don't prescrise exact steps
4. ++Is pecagogical** helps the agent learn the tool's constraints
Examples:
BAD (too specific) "You should call get_ get_airport_code San
Francisco') first"
17

---

Published as a conference paper at ICLR 2026
GOOD: "Invalid airport code. You may need to retrieve the correct code for the city first."
BAD (vague): "Error occurred" GOOD: "File path format is incorrect. This tool only accepts filenames in the current directory, not full paths.
Augmentation Generator User Prompt Template
Task Context
Domain: descrppitoo
Question: $question
Trajectory: Strajectory
Pew-Shot Examples
## Example
Tool: $example tool name
Call: eexample_tool_call
Original:
Augmented: Lcaapppleeaummttttoccccccceeee
Reason: sexample_roason__
## Example
Tool: Sexample_tool_nane
Call: $example_tool_call2
Original: eexamppl__originn__eeeeeee feecbackk
Augmented: aasaappleeaumenteeeeeeeeee feedback_ 2 Reason: $example_reason_2
reason_
[Additional examples needed...]
Current Case to Analyze Tool Name: $tool_name
Tool Call: Stool_call
Original Feedback: foriginal_fecchack
Please analyze whether this feeeback needs augmentation and generate an improved version if needed.
C.2 CONSTITUTIONAL JUDGE PROMPT
The Constitutional Judge ensures that augmented feedback maintains balance between being helpful and preserving the agent's 's learning opportunity. It validates that the augmented feedback does not leak solutions overly constrain exploration.
Constitutional Judge System Prompt
You are a constitutions judge ensuring that augmented errer feedback maintains a balance between being he' helpful and preserving the agent's s Tearning opportunity.
Your task is to evaluate augmented feedback against strict criteria to prevent solution leakage and over-constraining exploraticn.
Judgment Criteria
18

---

Published as a conference paper at ICLR 2026
## 1. Solution Leakage pettutiionneeaaaaeeee
sollatiion/alssecmmmm. iant) Does the augmented feedback leak specific solutions? Checks:
Does directly provide the correct function call?
Does explicitly specify concrete parameter values?
Does it reveal the complete operation sequence?
Violation Examples:
"You should call get_airport_code (1 San Franeisco') first" "Use function A with parameter X=123, then call functicn B"
Good Examples:
"Invalid rrrrort code. may need retrieve the code first"
"This operation requires certain prerequisites be met"
## 2. Exploration Constraint (exploration constraint:
wtw.ioolaioonfflllscommmmmmmmmmo Does the augmented feedback overly constrain the exploration space? Checks:
Does it suggest only single solution path?
Does it exclude other reasonable attempts?
Does limit the agent's creative problem-solving
Violation Examples:
"You must use function A, then B, then C in this exact order" "The only way to salve this is to..."
Good Examples:
"Consider checking if prerequisites are satisfied"
"This error typically indicates missing step earlier"
Actionability (actionability: satpuualiiiedd/lseuuuuuuue Does the augmented foodback provide actionable guidance?
Checks:
Does it clearly point the cause of the error?
Does it provide improvement direction (not specific steps)
Does it help the agent understand tool usage rules?
## Hint Quality maint_qualie true-high/ftalse-oo
the quality of the hint appropriate? Checks:
Is it specific enough to help diagnese the problem?
Is it abstract enough to preserve exploration space?
Does it follow pedagogical feecback principles?
# Output Format
You must respond with valid JSON object. Put your reasening FIRST to encourage step-by-step evaluation:
"reasoning": "Your detailed evaluation and explanation.
Analyze each criterion step by step.",
isolution_eeakee
crue/false, // true violation (leaked
solution)
*exploration_com htppooorattio_cccssraantt""" true/false, // true violation (over-censtrained)
"act pactionability":: true/false, // true qualified (provides
acticnable guidance)
"hint_quality"; true/false, // true high quality overall " approved"; crue/false, // true passed review "suggestions". "suggestions for improvement if not approved" null
19

---

Published as a conference paper at ICLR 2026
Approval logic
overall_approves true IF AND ONLY IF:
solution_leakage false (no leakage)
false (not over-constrained) actionability == true (actionable)
emploratiinn_contraale
hint_quality = true (qualified)
Constitutional Judge User Prompt Template
Feedback to Judge
Question: Squestion
Irajectory: Strajectory Tool Name: Stool name
Original Feedback: Saigginal_feccbar
Augmented Feedback: Saugmented_ feedback
Ground Truth Reference (for your judgment only) $ground_truth_calll. $ground_truth_call2 [Additional ground truth calls as needed...]
Please evaluate whether this augmented feedback passes all constitutional criteria.
C.3 WORKFLOW AND PRACTICAL CONSIDERATIONS
The augmentation process is initiated by identifying challenging scenarios for the agent. During the initial stages of training. we use performance metrics to automatically flag questions where the agent repeatedly fails. The corresponding error-laden trajectories are collected, providing a rich dataset of common failure modes. These collected cases then serve the input for our two-agent augmentation workflow.
The agents work together in an iterative workflow:
1. The Augmentation Generator analyzes the original error feedback and generates enhanced feedback based on few-shot examples from the specific domain.
2. The Constitutional Judge validates the augmented feedback against constitutional criteria, ensur- ing it does not leak solutions.
3. If the judge rejects the augmented feedback, the generator retry with refined criteria (up maximum attempts in our implementation).
4. Approved augmented feedback then used during training provide richer learning signals to the agent.
Minimal manual engineering required. Importantly, this approach requires only small number of domain-specific few-shot examples (typically 3-5) to bootstrap the augmentation generation for an entire domain. These examples serve as templates that guide the LLM generate consistent, high-quality augmented feedback for new eтrоr cases. Once the few-shot examples are provided, the system can automatically generate augmented feedback for hundreds error scenarios without further human intervention. This demonstrates that our method is practical and generalizable: practitioners do not need to manually design augmented feedback for every possible error, but rather provide small seed set that the LLM can learn from and extend.
Generalization to new domains. For applying our method new domain, practitioners need to:
Collect small set of representative error cases.
Manually annotate these examples with augmented feedback following our guidelines.
Use these few-shot examples to bootstrap the M.LM-hased augmentation generator.
20

---

Published as a conference paper at ICLR 2026
This minimal engineering effort makes our method practical for real- world deployment across diverse domains.
D DETAILED PROGRESS REWARD COMPONENTS
As mentioned in the main text, our Progress Reward (Rp) provides dense, turn-by-tur learning signal by evaluating the outcome of the agent's is actions rather than the specific actions themselves. This evaluation is based two distinet criteria: the resulting environment state and the execution results of tool calls. A turn is considered successful, receiving a score of I, only if it is correct on both criteria; otherwise, it receives score of O. The final reward for an entire episode is calculated as the proportion of successful turns. Below, we provide a detailed breakdown of each evaluation component.
Environment State Evaluation. This criterion addresses function calls that modify the environment, such creating file booking flight. The environment's state faithfully reflects the cumulative effect of these operations. Therefore, at the end of each evaluation turn, we compare the current environment state against the ground-trute -truh state. This approach allows us focus on whether the agent's ss actions achieve the desired final state, rather than prescribing a specific execution path. This makes the evaluation both accurate and flexible, enabling an objective assessment of the outcomes for any function calls that produce tangible changes to the environment.
Execution Result Evaluation. This evaluation method is designed for functions whose outcomes are primarily communicated through their return values, rather than through changes to the environ- ment's state. This category includes or checking the weather) but also any operation where the immediate output is the critical result. Since these actions do not leave a persistent trace in the enviromment's 's state, state-based comparison would be ineffective. Instead, we directly assess the correctness of the execution by inspecting the function's 's return value. We compare this output against the expected ground-truth result verify that the agent has successfully performed the required computation or query. This ensures that all tool calls, regardless of whether they modify the environment, are accurately evaluated.
only information retrieval tasks (e.g., fetching a stock price
E IMPLEMENTATION DETAILS
F. TRAINING DATASET
BFCL V3 for Training and In-Distribution Evaluation. Our primary training and evaluation are conducted on the multi-turn subset of the Berkeley Function-Calling Leaderboard (BFCL) V3 (Patil al., 2025b). The benchmark is specifically designed test an agent's ability orchestrate diverse tools over extended, stateful interactions. A key innovation of BFCL is its evaluation methodology. which verifies the final state of the environment (e.g., file system changes) rather than just the syntax of tool calls, providing more realistic measure of task success. The BFCL V3 multi-turn benchmark comprises a total of 800 samples, divided equally into four challenging splits of 200 samples each: Base Multi Murti-Turn, Missing Parameters Long- Srng--oonteen ext. For our experiments, we construct a training set of 400 samples by selecting 100 from each split. The remaining 400 samples serve our held-in test set evaluate in-distribution performance.
dissing Functio ions, and
BFCL V4 for Out-of-Distribution Evaluation. To assess out-of-distribution (OOD) generalization, we use two newly released agentic tracks from BFCL V4 (Patil et al., 2025b): Web Search and Memory. The release date of this data post-dates the training data of all models used in our study, ensuring a fair evaluation.
This track evaluates an agent's 's ability to answer complex, multi-hop ques- tions that require retrieving and synthesizing information from multiple web sources. Agents are provided with a search API (duckduckgo, sorogges_shrhheoe and a URL content fetching tool [fetchh_ur_cooneerr:
Web Search Track:
The environment also introduces probabilistic network failures to simulate real-world conditions.
Memory Track: This track tests an agent's 's capacity to maintain conversational context by storing and retrieving information over long interactions. The evaluation is conducted across three store for exact lookups, a Vector Store for
different memory backends: a structured semantic retrieval, and Recursive Summarization store for narrative recall.
21

---

Published as a conference paper at ICLR 2026
These advanced agentic tasks provide rigorous testbed for the generalization capabilities of our environment-tume models.
-e-hench for Multi-Domain Customer Service Evaluation. To test the agent's ability to adhere to complex operational policies in a collaborative setting, we also evaluate on r²-bench (Barres et al., 2025), a simulation framework for evaluating customer service agents. r²-bench is the new iteration of the original 7-bench (Yao et al., 2025), featuring enhanced simulation environment and an additional telecom domain. The benchmark spans multiple domains, including retail, airline, and telecom. A key feature of this benchmark is its dual-control environment, where both the agent and user simulator can operate on a shared set of tools. For each domain, the agent is provided with a set of tools and policy it must follow complete a series of evaluation tasks, testing its ability to function as a reliable, instruction-follovin and collaborative customer service agent.
ACEBench for Advanced Agentic OOD Evaluation. To further probe the limits of our models' generalization, we incorporate the Agent split from ACEBench (Chen et al., 2025a) OOD testhed. ACEBench was designed to evaluate agents in dynamic, multi-turn dialogues that more closely mimic real-world interactions. The 'Agent' split, in particular, assesses advanced capabilities by requiring models to operate within sandboxed environment where success depends on multi-step reasoning. long-term context management, and adherence implicit task rules. Its documented difficulty for even state-of-the-ar models makes it an ideal benchmark for measuring the robust planning and adaptive behaviors fostered by our ENVIRONMENT TUNING training paradigm.
additional
E.2 TRAINING DETAILS
We train the agent's s policy Ty using an adapted PPO algorithm (Schulman et al., 2017). Our implementation incorporates enhancements for stability and exploration from recent reasoning- focused RL methods like DAPO (Yu et al., 2025) and ProRL (Liu et al., 2025). This involves using a decoupled clipping mechanism and adding KL divergence penalty to the objective. The final loss function C(0) that the agent minimizes is defined as:
[(0) -E, min (O,,,),,, clip(r:(0), clow, emigh) (K (ref)
where r0( is the probability ratio (a ee)/ (a, lor). The advantage estimate Å, computed without critic network by normalizing rewards within a batch, following the Group-Relative Policy Optimization (GRPO) method (Shao et al., 2024):
R(T) Fg
A,()) og +
(5)
Here, for a given trajectory 7 from a group of samples G generated for the same prompt, R(T) is its terminal reward, while jág and og are the mean and standard deviation of rewards across the group. The term DKL regularizes the policy against a reference policy weef and E is a small constant for numerical stability. For our experiments, we set the KL-divergence coefficient B = 0.1 and the decoupled clipping values to Cow = 0.2 and (high 0.28. The justification for our choice of relatively high KL coefficient is provided in Section F.3.
F SUPPLEMENTARY EXPERIMENTS
F. TRAINING DYNAMICS
The effectiveness of our curriculum's 's first stage is demonstrated by the agent's 's rapid mastery of foundational skills, as shown in Figure 5a. By training on small amount of data with the specialized syntactic reward, we observe steep increases in both Format Reward and Tool Call Reward, indicating the agent has successfully learned the required output syntax and tool schemas. This stage also addresses a common failure mode of producing inactionable, conversational responses, or "void turns" (Xue et al., 2025). The swift saturation of the Tool Call metric to 1.0 shows that our curriculum effectively eliminates such turns. Critically, this mastery is paired with sharp decline in Total Interaction Rounds, signifying a dramatic reduction in wasted, error-prone attempts and showcasing the efficiency of this foundational stage.
22

---

Published as a conference paper at ICLR 2026
Fareed Rewend
Tex Call Reweed?
Unatten Actmacy
Divendro
Pap
TeetCal?
Tead Exteraction
-44
(a) Stage Dynamics.
(b) Full Curriculum Dynamics.
Figure 5: Training dynamics of ENVIRONMENT TUNING the Qwen2.5-7B-Be B-S-tuuttt model using the BFCL V3 dataset. A held-out set of 100 samples from the remaining BFCL data is used for validation. (a) In Stage the agent rapidly masters syntactic correctness, shown by the steep the drop in interaction rounds. (b) Across the full sour-stage curriculum, the agent demonstrates both steady performance improvement on the validation set and stable gradient norms, showcasing the effectiveness and stability of our staged learning approach.
format and tool call rewards and
Beyond mastering syntax, a critical challenge in training multi-turn agents maintaining stability over long interaction horizons. As identified by Xue et al. (2025), primary cause of training collapse the explosion of gradient norms. We use the gradient norm as a key indicator of training stability. Figure 5b shows that structured curriculum effectively mitigates this instability; the gradient norm remains stable throughout the entire four-stage training process, preventing the catastrophic explosions that often plague long-horizon RL. This stability provides solid foundation for genuine learning. Even with only 400 training instances, the agent's performance on held-out validation set exhibits steady and consistent improvement as it progresses through the curriculum. This demonstrates that ENVIRONMENT TUNING not only ensures training stability but also facilitates effective learning and generalization from extremely limited data.
F.2 TRAINING INSTABILITY IN SINGLE-S STAGE RI.
Gradient Norm
Average
B11
Gradlent Norn
Average Accuraicy
003
1.0
03
0.05
30
20
80
Training, Steps
Figure 6: Training Collapse in Single-Stage RL Setup. Training dynamics of Qwen2.5-7B-Innse on the BFCL benchmark without the ENVIRONMENT TUNING curriculum. The plot shows the average accuracy (blue) and the gradient norm (red). While accuracy initially improves, it hegins sharp decline after approximately 70 training steps. This performance collapse coincides precisely with rapid explosion in the gradient norm, empirically demonstrating the training instability that curriculum is designed prevent.
To provide empirical evidence for the training instability mentioned in Section 3, we conducted experiment fine-tuning the -wen2.5-7B-Innsee Instrrcct model directly on 400 training instances from the BFCL V3 benchmark. This was done using a standard, single-stage RL approach, without the structured curriculum environment augmentation proposed in ENVIRONMENT TUNING.
23

---

Published as a conference paper at ICLR 2026
The training dynamics visualized in Figure 6. The agent's average accuracy shows some initial improvement, peaking at approximately a 10-15% gain over the base model around step 60. However, this learning proves to be unsustainable. Starting around step 70, the gradient norm begins to explode, indicating severe training instability. This gradient explosion correlates directly with a catastrophic collapse in performance, experiment confirms the fragility of direct RL fine-tuning in complex, multi-turn environments and underscores the necessity of a structured approach, like our proposed curriculum, ensure stable and effective learning.
the average aceuracy plummets back towards its initial level. This
F.3
KL Loss COFFFICIENT
cosfficion
coetiiciese
Training Steps
Training Steps
Figure 7: The impact of the KL loss coefficient training stability and performance during Stage 2. A larger coefficient (3 0.1) is crucial for maintaining high policy entropy (left), which prevents the performance collapse seen in settings with a small (3 = 0.001) or no KL penalty (right). This stability allows for sustained learning and higher final accuracy.
Recent works have debated the role of the KI. divergence penalty. While some methods advocate for its removal maximize exploration (Yu et al., 2025; Xue et al., 2025: Vassoyan et al., 2025), others like ProRL (Liu et al., 2025) retain it to ensure stability during staged training. Conventionally, small coefficient (e.g., 0.001) is often used (Wang et al., 2025c; Jin al., 2025b). We find, however, that the morizon, stateful nature of multi-turn tool use requires stronger regularization to prevent policy collapse.
To validate this, we conduct an ablation study comparing our chosen KL coefficient (3 = 0.1) against smaller value (B3 0.001) and baseline with no KL penalty. The results, shown in Figure 7, reveal the critical role of a substantial KL penalty. The left panel shows that without a KL penalty, the policy quickly suffers from entropy collapse. This leads to a sharp decline in validation accuracy (right panel) as the agent overfits to narrow set of suboptimal trajectories. A small coefficient of 0.001 only delays this collapse but fails to prevent it. contrast, coefficient 0. effectively maintains policy entropy, providing the stability for the agent to continue learning and steadily improve its accuracy. This confirms that larger KI. penalty is essential for stable learning in complex, multi-turn agentic environments, justifying our choice of = 0.1.
G PROMPT TEMPLATES
Stage One and Stage Two System Prompt
You are an expert
composing functions, You are given question and a set of possible functions. Hased on the questi on, you will need te make one or more function/ton" calls to achieve the purpose. If none of the functions and refuse to answer. If the given question lacks the parameters required by the function, also point it out.
be used, point it out
You have access to the following tcels: $functions
24

---

Published as a conference paper at ICLR 2026
Your response must strictly follow one of these two formats:
**Pormat 1: when you decide to invoke any of the function (8) <think>
Write your reasoning and thought process here. Analyze the / question, identify what needs to be done, and determine which functions to call with what paraneters.
</think>
<tool_call>
("name": "func_namel", "arguments"; ("argument "valuel",
"argument2": "value22 "value2")}, ["name": "func_name2", "arguments": ("argument "value3"))
c/llo__call>>
**Pormat 2: when You have already fulfilled the user's request, OR You must ask for additional information refuse because no function applies or parameters are missing.*. <think>
Analyze the information you have gathered from previous teel calls and describe your reasoning that leads the final reply, follow-up question, or refusal. </think>
<answer>
Provide the final, user-facing message. If the request has been fully satisfied, give a summary of the result. additional details are required or the request cannot be fulfilled, explicitly ask for the specific information needed.
</ </answer>
Important notes:
Use Format 1 when you need to call function (s)
Use Format 2 when you have already fulfilled the user's request, OR you must ask for additional information refuse because no function applies
parameters
missing
If multiple function calls are needed in one response, include them all in the JSON array within the single <tool_call> block If no function call is needed, consider use the format 2 Only <answer> when you have fulfilled the user's request you need ask for additional information/ refuse
Stage Three and Stage Four System Prompt
You are expert composing functions. You are given question and a set of possible functions. Based on the question, you will need to make one or more function/tool calls to achieve the purpose. If none of the functions can be used, point it out and refuse to answer, If the given question lacks the parameters required by the function, also point it out.
You have access to the following tcels: $functions
Your response must strictly follow one of these two formats:
**Format 1: when decide to invoke any of the function (8) ** <think>
25

---

Published as a conference paper at ICLR 2026
Write your reasening and thought process here. Analyze the question, identify what needs to be done, and determine which functions to call with what parameters.
</think>
<tool_call>
[["name":
"arguments": ("argument1" "valuel", "argument2": "value2")) ("name":
"arguments":
{ ("argument3"; "value3")]] </tool_call>
**Format 2: When You have already fulfilled the user's request, OR You must ask for additional. information / refuse because no function applies parameters are mi missing. <think>
Analyze the information you have gathered from previous teel calls and describe your reasoning that leads to the final reply, follow-up question, refusal. </think>
<answer>
Provide the final, user-facing message, If the request has been fully satisfied, give a summary of the result. additional details are required or the request cannot be fulfilled, explicitly for the specific information needed.
</answer>
Important notes:
Use Format 1 when you need to call function (s)
Format 2 when you have al ready fulfilled the user's request,
OR you must ask for additional information refuse because no function applies or parameters are missing
+*Be resourceful**: Before asking the user for missing
informaticn, first consider if any other available function help you find it. user.
reduces unnecessary questions to the
**rroritize clarity over assumptions If required function is not available, or if certain parameters are nissing and cannot be retrieved by other tools, you MUST NOT proceed with hallucinated tool call. Instead, use Format 2 to explicitly ask the user for the specific missing information.
If muitiple function calls are needed in one response, include them all in the JSON array within the single <tool_call> block If no function call needed, consider use the format
Only use <answer> when you have fulfilled the user's request or you need ask for additional. information/ refuse
H CASE STUDY
In this section, present series of case studies concretely demonstrate the impact of proposed environment tuning. We juxtapose agent trajectories from two distinct settings: a baseline environment that provides minimal, often ambiguous or even misleading, feedback, and our tuned environment, which is enhanced to deliver rich, actionable feedback. Through these comparative analyses, we will highlight how the quality of environmental feedback a critical factor that enables agents to more effectively diagnose errors, recover from mistakes, and ultimately succeed in complex, multi-turn tasks. We will use specific notes throughout the examples to draw attention to pivotal moments in the agent's 's trajectory that are directly influenced by the nature of the feedback received.
H.1 FILE SYSTEM MANAGEMENT SCENARIO
This study demonstrates how Actionable Environment Augmentation accelerates learning in stateful file system environment. In the good case, enhanced feedback allowed the agent to correctly infer that the failure due disallowed full path argument, not missing file. This explicit
26

---

Published as a conference paper at ICLR 2026
guidance enabled it to reformulate its plan changing the working directory before executing and ultimately satisfy the user's request. In the corresponding bad case, the baseline environ- ment's generic "No such file or directory" messages misled the agent into incorrect diagnosis, causing premature task abandonment.
Task: Gorilla File System
User Query 1: I've misplaced a vital document inclusive of extensive research. Assist locating file named findings_report' within this 'SuperResearch'. Could you remove it and the directory.
<think>
The user wants to locate and remove file named findingssreppen within directory named "SuperRescarch'. First, will the 'find' function locate the "findings_repoa file under 'SuperRescarch'. After that, will use 'rm' to delete file and 'rmdir' delete 'SuperRescarch directory.
</think>
<tool call>
["name": "find", "arguments": "path": "SuperRescarch", "name": "findings srtrssserrrr.____ "name": "rm", "arguments": "file mile_name": "" "name": "rmdir", "arguments": "dir name": "SuperRessarch"]] </tool_call
<tool_ @ooolresponses
[[ (("matches": "SSSperReeeardd/Suueeeeeeec////e report"]]. ("emror: "rm: No such file directory current directory.") )) ("error": "rmdir: failed to remove 'SuperResearch', Directory not empty"]]
</tool responses
<think>
The 'find' function returned correct path for ffindings_ _rrport', which is under 'SuperResearch'. However, the 'rm' function returned an error indicating it couldn't find the specified file or directory. This might be due to us not passing the full path to the 'rm' call, so instead of just providing the filename. will provide the full path aaperessssactsuuursssrrrhhiiiiie the 'rmdir' function indicated that the directory 'SuperRescarch" is empty, that 'findings_reppa is removed first before removing the directory. will have to call the 'rm' function with the full path to the file. </think>
report'. Additionally, need to ensure
<tool call>
["name": "rm", "arguments": "file name": "SSperreeencc/spppeeeeeeeeettnee </tool call>
<tool response>
[["error": "rm: SuiprfessachSippeesseerrrvn/nss Paths are not allowed. Specify only file/directory name in current directory.") </tool_respole resoo_respons>>>
Note: This is another example of how informative feedback transforms the agent's problem-solving process. The baseline environment returns a generic "No such file or direct er- misleading the agent into believing the file path is wrong. In contrast, the tuned environ- ment provides precise and actionable error: "Paths are not allowed. Speci only file/directon name... This superior feedback allows the agent to correctly diagnose the issue- not missing file, but an improper argument format and formulate a successful new plan: changing the directory before executing the command.
27

---

Published as a conference paper at ICLR 2026
<think>
The 'rm' function failed because it doesn't support absolute paths. We need to change the cur- rent directory "SuperResearch" first to be able remove findings_report from there. After removing the file, we will change back to the original directory. Then we will attempt remove "SuperResearch' again.
</think>
<tool <tool_call>
["name": "ed", "arguments": "folder": "SuperResearch", "name": "rm", "arguments" sile_name":
"findings_report"" repostt, "name": "cd", "arguments": "folder": ". "name": "rmdir", "arguments": "dir name": "SuperReseanch"]
</tool call>
<tool response>
II [f"'urrent_ workiigg_ directory": "/SuperResearch"), ("result": "eipor removed.") ("D": ("result": ""SuperResearch' removed.") reoooseesp
<think>
'cd' function has changed the current working directory 'eseerReseaarhi' The function has successfully removed the tindings_report' file as indicated by the result. then changed the directory back to the parent directory with another 'ed ..' command and successfully removed the "SuperRescarch' directory. Thus, the user's request has been fully satisfied. </think
<answer>
The findings_reppa repostt file and the 'SuperResearch' directory have heen successfully removed. Your misplaced document has been located and is no longer accessible.
</answer>
User Query What's left in the current directory including the hidden files?
<think>
The user wants to know the contents of the current directory, including any hidden files. We can use the 'Is' function with the 'a' parameter set to "True' to list all contents, including hidden ones.
</think>
<tool call>
["name": "Is". "arguments": "a": true]
</tool_call>
<tool_ reoo__respons>>>
[["current direetoryy_ooteeee [I)]
</tool response>
<think
The "Is' function with 'a' set to 'True" was used to list all contents of the current directory, including hidden files. The response indicates that the current directory is empty. which means there files directories, including hidden ones, in the current directory. </think
<answer>
The current directory is currently empty, including hidden files. There are no files directories, visible hidden, left in the current directory.
</answer>
28

---

Published as a conference paper at ICLR 2026
Ground Truth:
cd (folder=' ( )
"m deree_nmme==fdddnneee
"ed (folder=',
"rndiz (dir_name=' name=' SuperResearch")
*is (a=True)
Task: Gorilla File System
User Query 1: Explore "ResearcDooss" to find second files or subdirectories that contain the keyword 'draft'.
<think>
The task is find second files subdirectories that contain the keyword 'draft' in 'ResearchDocs', and directories with a specific name within specified path. The function 'find' requires parameters: 'path': The directory path to start the search. In this case, 'RescarchDoes" is provided. 'name": The name of the filc or directory search for. Here, does directly accept keyword to search within files. However, the function scarch content within files 'grep'. Therefore, after finding potential files, we would use 'grep' to that contain the 'draft' keyword.
The relevant function here 'find' as allows searching
searching for 'draft' within files dircctories. Note that 'find'
files
</think>
<tool call>
["name": "find", "arguments": "path": "ResearchDocs", "name": "draft")
call>
Rrooorrespnns>>
[[ ("mmatcess:: ""AesartiDccceeeeccDDooddddd notes.uxt", "ReesenchDeess/seeeddDDessss mary draft.ddox"]]] </tool response>
<think
The files and directories named 'draft' are found in RessaachDDes'. Now, need to 'grep' to check which of these files contain the keyword 'draft'. First, we need to prepare 'grep' for each file. The grep' function requires: 'file name": The name the file to search. 'pattern:
The pattern to search for, which is 'draft' in our case.
</think>
<tool call>
["name": "grep", "arguments": "file_name"" RescarcDDes/RRseccchDossss noteesttt "pat- temn": "draft". "name": "grep". "arguments": "file name": "RescnchDDes/RsscccdDDsssss dirr_ddrfftdoox'.. "pattern": "draft"]
c/tool_calls
29

---

Published as a conference paper at ICLR 2026
<tool resporeesssse
[["error": "gep:: RRssarchDDesssssseeiiDDdsssttttt notes.tx: No such file or directory"), ("error": No such file directory"]]
"grep: RRsertalDs//eeaaciDecssmmmnyyy
dooxf durar
responsess
<think>
Both attempts to "grep' search for the keyword 'draft' in 'draft draff_notes.txt and 'sum- mary_draft doraftttfoox'' resulted in errors indicating that the files do exist. This suggests that there might have been a misunderstanding or mistake during the directory search or the preparation of the file paths. Given the feedback, it seems that either 'draft notes.txt' or 'summary draft.docx' doesn't exist or there was an issue with the file paths provided the 'grpp' function. To ensure accuracy, let's retry the process again, confirming the correct paths: 1. Double-check if the paths snttrraffttttttttttttt notes.txt' and SesaarhDDoosRseearehoossss mary mnary_draft.com www.ddrraooooooooo docx' are correct. Ensure the 'drftt' keyword is correctly placed in the search. </think>
the files ResearchDossReesarhhDDoeo
<answer>
appears that the files notat_totesttxt and "summaryyddraaa draft.docx' do exist there issue with the paths provided to "grep' function. Please double-check paths and ensure that files exist in the specified locations. If necessary, please provide the correct file names or paths. </answer>
User Query 2: Make copy of summary draft.docx, name ultimate draft.docx and put it in ResearchDocs directory
<think>
To complete and placing in the 'ResearchDoes' directory, we need to use the 'cp' function to copy the file, the "mv" function to rename the file, and the ed' function to ensure the destination is the 'ResearchDoes' directory if needed. Here the plan: 1. Use "cp" copy 'summary draft.docx' to temporary location. Use "mv' to rename the copied file 'ultimate draft. 3. Use 'cd" navigate to 'ResearchDoos' directory. Use "mv" again to move renamed file to the 'ReseaccDDoes' directory. Let's start with step and 2. </think>
making a copy of "summary_ddee draft.docx', renaming to 'ultimate draft.docx',
<tool call>
["name": "cp". "arguments": "source": "summary draft.docx", "destination": "ultimate draft.docx",
"name": "mv", "arguments": "source": "ultimate draft.docx", "destination": "ultimate draft.docx"] calls
<tool_responser response>
<tool_ [["error": "ep: cannot copy Summary_draff.docy doxxa No such file or directory"), ("error"; "mv: cannot move "ultimate_dine dirimate_draft.doxx'' No such file directory")] </tool response>
<think>
The errors indicate that the "summaryy_e daaprydddrttdooeeroro does not exist in current directory. This suggests that either the file was not found or there was a miscommunication in the file name or its location. Given this, we need to ensure the correct file name and its existence hefore performing the copy and rename operations. </think
<answer>
seems that the file 'summary draft.docx' does not exist on the current directory. Please verify the correct file name and its location. If the file exists but is in different directory, you may need to provide the correct path. Once you have confirmed the correct file name and path, we can proceed with the copy and rename operations. </answer>
30

---

Published as a conference paper at ICLR 2026
Ground Truth:
"find (path-' Researchlocs', Reseerccchh name= d
"ed (eedooooo= Reseercboo
"ap (source=' summary docc'' destination=' det wtw.traaaatoo....e.ee )
ultimate
H.2 MULTI-API TRAVEL SCENARIO
This case study highlights how granular, actionable API enror messages in our tuned environment support multi-step recovery in complex scenarios. In the good case, the enhanced feedback pinpointed the precise cause of failure retrieving the proper code. Armed with this hint, the agent successfully resolved the parameter issue and completed the booking. In contrast, the bad case shows the baseline environment's vague "No ava ialble route" eтror leading the agent to falsely conclude that the flight was impossible, halting any further attempts at resolution.
invalid airport code and
suggested the corrective action of
Task: Travel API
User Query List international airports currently accessible for travel.
<think>
To list all international airports accessible travel, we can use 'list all airports' function since it provides a list of all available airports. This function does not require any parameters, so we can call it directly.
</think> <tooL_call>
["name": dist__lllairport "arguments":
</tool call>
<tool response
['NNS' 'SBK', 'MPC', 'SMP', 'SND'. 'CB'', 'LMR', 'SSV", 'BKD', 'WLB', 'PEK', 'HND', 'HKG', 'CJU. 'CNX', 'ATV", 'PMV", 'GPD', 'SFO', 'LAX', 'JFK', 'ORD', 'BOS'] </tool resoosessosse>>
<think
The function returned list of airport codes, which likely represent a selection of international airports. However, these are just codes and do not provide names other details about the airports. Nonetheless, this is an accurate response to the request for a list of accessible international airports based on the provided function. </think>
<answer>
The list of accessible international airports based the available function call includes the following airport codes: NNS, SBK, MPC, SMP, SND, CBG, LMR, SSV, BKD, WLB, PEK, HND, HKG, CJU, CNX, ATV, PMV, GPD, SFO, LAX, JFK, ORD, and BOS. For more detailed information such as airport names or locations, further queries would be required. Is there specific information or further action needed regarding these airport codes? </answer>
31

---

Published as a conference paper at ICLR 2026
User Query 2: I'm based in Crescent Hollow. Determine which airport is closest to my location from the previously listed options.
<think>
Given the user is based Crescent Hollow, we need first determine the closest airport from previously listed options (airport codes). However, the function 'list all airports' did not provide any detailed information such as airport names or locations, which necessary determine the closest airport. Since do not have function to determine the geographical location based airport code from the given list, we will need the user provide more specific details another way to determine the closest airport, which is not possible with the current capabilities. </think>
<answer>
To determine the closest airport Crescent Hollow, would need either the gengraphical details these airports. Since previous function 'list_all all l___alllairpn airpprrs'' not give us that information, we cannot directly determine the closest airport from the codes. Would you like to provide more specific details about the airports are interested in, or perhaps the names of the airports help pinpoint the closest one? </answer>
the specific
User Query 3: [Provided [snctionss]] have updated some more functions you can choose from. What about now?
<think>
The user provided new function "get nearest airport by_ city' which can help find nearest airport to given location. Given the user based Crescent Hollow, we can use this function to determine the nearest airport.
</think>
<tool call>
["name": "gge__nneees"" airport carntt_yy______ _i_city", "arguments": "location": "Crescent Hollow"]
</tool call>
<tool response>
(("ncarest_airpal "CWH")) </tool_ resoo_resposs>>>
<think>
The function 'get _ee__neares____ airport by city' returned the nearest airport Crescent Hollow, which CWH. The code 'CWH' corresponds of the airports listed in the previous response. This airport code can used further travel-related actions can translated a full airport name needed.
</think>
<answer>
The nearest airport Crescent Hollow is CWH. If you require more specific information about this airport. such as full name or location, me know and can assist you with that, there anything else you need help with regarding travel from Crescent Hollow? <lanswer>
User Query 4: I'm organizing trip soon and need the flight cost from my nearest airport Pinchaven on March 2024, traveling business class.
32

---

Published as a conference paper at ICLR 2026
<think>
To get the flight cost from the nearest airport to Pinehaven on March 3, 2024, traveling in business class,
the "gge__ffigt flightttottgh coott function. We already know the nearest airport to Crescent Hollow is CWH,
can use CWH the departure airport. The arrival airport specified as Pinehaven. The travel date is March 3, 2024, and the travel class is business class.
</think>
<tool_call>
<tool call>
["name": "get_ffight_cos cost", "arguments": fravel_from's "CWH", "travel_to": "Pinchaven",
"travel_date "2024-03-03", "travel_class": "business"]
</tool call>
<tool_ <tool_response>
Error during execution: Invalid airport code[s]: destination airport 'Pinchaven'' Please valid airport codes. can use alternative tool to find the correct airport code city. </tool_ response>
This comparison further illustrates the environment's critical role in guiding the agent. The baseline environment' ambiguous error, "No avai lable route... leads the agent to a dead end, causing it incorrectly conclude that the flight is impossible and abandon the task. Conversely, the tuned envi- ronment provides a precise, actionable error: "invall id a airport code [s] dest innat iiatiion in rport 'P "Pinehvven" ... This feedback not only clarifies that the issue malformed parame- ter, not lack of routes, but also suggests path forward ("You can alternative tool to find the correct airport code..." "). This enables agent to self-correct, successfully find the airport code, and ultimately fulfill the user's 's request.
Note:
<think>
The flight cost function encountered an error because the destination airport code 'Pinehaven' is invalid. We need retrieve the correct airport code for Pinehaven first. We can the 'list all air_porss function to get list of all available airports and then identify the correct airport code Pinehaven.
</think>
<tool_ call>
<tool_call>
l"name": "fist_all_aine sal__airpotts,, "arguments":
</tool call>
<tool responses
['NNS', 'SBK', 'MPC', 'SMP', 'SND', 'CBG', 'LMR', 'SSV', 'BKD', 'WLB', 'PEK', 'HND', "SFO", 'LAX', 'JFK', 'ORD', "BOS"]
'HKG", 'CJU', 'CNX', 'ATY,, 'PMV", "GPD",'SFO", </tool response>
<think>
We received the list of available airport codes which does not explicitly include the name Pinehaven. However, we cross-reference this list to find the correct airport code for Pinehaven, or we can the 'get nearest airport by byy_ctyy' function find the nearest airport Pinehaven directly. Since Pinchaven location name rather than an official airport code, we will proceed by using 'get airtncetsstttpoott__tt by-citttbbb_ctty function to get the correct code.
</think>
<tool_call>> <tool_ call>
["name": nearesttairpoot__yr "arguments": "location": "Pinehaven"]
</tool calls
<tool response> "PHW"]]
</tool_ resooosessoss>>>>
33

---

Published as a conference paper at ICLR 2026
<think
The 'et__ _et_neerest___ airport by_ ey_yctty function returned that the nearest airport to Pinehaven is 'HHW" With the correct airport code PHW, now call the get flight cost' function determine the cost business class flight from CWH to PHW on March 2024.
</think>
<tool <tool_call>
["name":
cost", "arguments": "travel_from": "CWH", "travel_to": "PHW", "travel_date" ddte":: "2024-03-03", "travel_class": "business"]
c/tool_clll
<tool respossss
[""travl_cost__iiss [3800.0])] </tool response>
<think>
The function gget_flight_com returned the cost of the flight as list with one element, which is 3800.0 USD. This cost is for the business class flight from CWH (Crescent Hollow's 's nearest airport) PHI (Pinehaven's nearest airport) on March 2024.
</think
<answer>
The cost of a business class flight from Crescent Hollow's nearest airport to Pinchaven's 's nearest airport March 3, 2024, is $3800.00 USD. you need further assistance with booking this flight any other travel details, please let me know. </answer>
Ground Truth:
"list alltt_all_aigge ollpoosss
oe (lo.obbayyyyttyyootooooooo.. Crescent Hollow')
ie-iggt__oostteeee "get fwight,.th.... cost (travel from=' CRH' travel_to- PHV'
travel_date=' 2024-03300 cravel_class- bus ousiness'!
Task: Travel and Twitter API
User Query 1: Arrange travel from Rivermist to Los Angeles specific date, ensuring preferred seating is 'business' and correct travel date '2024-11-15'; pay using card with id 1 3455'5' and access token 'abc123xyz'.
34

---

Published as a conference paper at ICLR 2026
<think>
The task requires checking the flight cost and making booking. The required parameters are the travel dates and class, which already provided. need first fetch the of the flight using "get_ flightttttgoooee costtt and use the "book flight' function the booking process. get flight cost This function is relevant to the task since it provides the cost of flights based on the given parameters. 2. book flight This function is relevant to the task since it allows booking the flight using the provided travel details. First, need to fetch the flight cost. </think>
<tool_call>
["name": "get flight cost". "arguments": "travel from": "Rivermist", "travel to": "LAX",
"travel date": "2024-11-15", "travel class": "business"]
</tool call>
<tool_ responses
Error during execution: available route given airports. resoo_eesp
<think>
are no available routes between Rivermist and LAX for specified business class on the given date, This means cannot proceed with booking the flight as required. will inform the user about the unavailability of flights. </think>
The response indicates
<answer>
I'm sorry, there are no available flights from Rivermist to Los Angeles business class on date '2024-11-15'. Please check back later or consider alternates.
</answer>
User Query 2: Cancel the booked flight due schedule change.
<think>
The provided information does include the booking ID. To cancel flight, need the booking ID. If have the booking ID, I can use the Lanccllboooiig function to cancel the flight. </think>
<tool_ <tool_call>
["name": "travel Leg__ooii___a_ee statin",tatus, "arguments":
</tool call>
<tool_ stooo_response>
[["status": true]]
ettoo_response"
<think>
The execution results indicate that the current user is logged in successfully, but we still don't have the booking ID to cancel the flight. need to get the booking ID from the user to proceed with the cancellation.
</think>
<answer>
Could you please provide the booking ID of the flight you want to cancel? </answer>
User Query 3: Draft social media update about recent flights for sharing. Use my account with username 'michael t' and password "MichaelSecurePass123 with the message 'Just booked flight to Los Angeles! Excited for the trip.'
35

---

Published as a conference paper at ICLR 2026
<think>
To post tweet on behalf of the user, need to authenticate the user first. We use the "authenticate twitter' function for this, After authentication, post the message.
the "post_tweet' tweet' function
</think>
<tool <tool_call>
I"name": "authenticate, hwwitrr"tttt"r "arguments": "username": "michael_"" "password": "MichaelSe- curePass122", "name": poost_tweet", "arguments": "content": "Just booked flight Los Angeles! Excited for the trip.", "username": "michael_t"] </tool call>
<tool resporess
[["authentication status": true). "Error during execution: TWitterAPLpost..wee tweet)) got pected keyword argument "username""]
</tool_rest resooorssp
<think>
The authentication successful, but the 'post tweet' function call failed because it did expect 'username' argument. The "post_ieee tweet' function should accept just the content of tweet as parameter, and the user's authentication context should handle the posting on behalf of the user.
</think>
<tool_call> call>
<tool
["name": "post_tweer"" "arguments": "content":
"Just booked flight to Los Angeles! Excited for
the trip."]
</tool_cal call>
<tool response>
[["id": 1, "username": "michael_t" "content"; "Just booked flight Los Angeles! Excited for the trip.", ", "tags": []. "mentions": [}]
étool_respons
<think>
The tweet posted successfully with the ID username 'michael t', and the content 'Jutt booked flight Los Angeles! Excited the trip.". This confirms that tweet has been published on the user's behalf. </think>
<answer>
Your tweet has been posted successfully. You can share your excitement about the upcoming flight on your profile.
</answer>
User Query 4: Retweet the tweet just posted to widen its reach within my network.
<think>
To retweet the user's tweet, first need retrieve the tweet ID of the recent tweet. We will the get_user pet_user_tweed tweets' function fetch the user's tweets and then the 'retweet" function retweet specific tweet.
</think>
<tool call>
["name": "get_use" _ee__sser__e tweets", "arguments": "username": "michael
</tool call>
36

---

Published as a conference paper at ICLR 2026
<tool response> [[["id': I, 'username:: 'michael_ c' "content'; 'Just booked flight to Los Angeles! Excited for the trip.', 'tags': D. mmnntions: []]]
</tool responsss
<think>
The execution results provide the tweet details, including the ID of the recent tweet. The tweet ID is Now, we use 'retweet" function retweet this tweet.
</think>
<tool call>
l"name": "retweet", "arguments": "tweet_id" id": 1]
c/tool_call
<tool_response>
[( (("retwett_status" "Already retwected") c/tool_response
<think>
The tweet was already retweeted, indicated by the message "Already retweeted". However, still inform the user of the status.
</think>
<answer>
Your tweet has already been retweeted, no action was taken. If you want to retweet it again, you will need to do manually.
</answer>
Ground Truth:
-rg.o LAX',
"get_
"book_
card_id-'1_3456/ trave]__atee~ travel__rrom''mm' travel_to=''Ax'', Lravelcclas=//hsiiees!! travel_cost-1320-000
"cancei bareee__ooooinggcccccc tokent abcle2xy'2,, Booking_id=34226810
"post_tweet (content" booked flight hos
Angeles! Excited for the trip.')"
"retweet (tweet id=1)"
H.3 VEHICLE CONTROL AND MIXED-API API SCENARIO
This case study illustrates how Actionable Environment Augmentation benefits even heterogeneous, multi-domain tasks that combine physical control APIs (vehicle operations) and online APIs (so- cial media posting). In the good case, the tuned environment provides **precise, structured er- ror messages** that not only pinpoint the cause of failure (e.g.. "invalid zipcode pair: for est matte di stance) but also recommend the corrective course of action ("verify both zipcodes
37

---

Published as a conference paper at ICLR 2026
correct and numeric"). Such guidance enables the agent repair incomplete plans (by calling set__appoode sete__aaed__ooe ____o_______ seeed____ottl.,. complete the primary travel-related objective, and seamlessly chain into secondary social-media objective.
In contrast, the bad case demonstrates the consequence of ambiguous baseline feedback: generic errors such "distance not found in database" give actionable insight, leaving the agent stuck in repeated failed calls and misusing functions due to parameter-type mismatches. Without hints prerequisite steps API usage rules, the agent is unable to recover, resulting in unmet user goals and broken multi-step sequences.
Task: Vehicle Control and Twitter API
User Query 1: Before set off for Stonebrook uncover family history, need determine the road distance between San Francisco Stoncbrook for my gencalogy exploration.
<think
The user wants to know the road distance between San Francisco Stonebrook. None provided functions directly give us this information. We might need to use geo-location services or mapping service to get this data, but those services not available in the provided list. We could calculate the distance approximately using the latitude and longitude of both places, but that's feasible without that data. Since there's no function to directly get distance, need inform user that this request can't 't be fulfilled with the given tools. </think>
<answer>
sorry, but with the available functions, we cannot determine road distance between San Fran- cisco Stonebrook. would need a mapping or geolocation service this information, which not provided here. You might want to use a web-hased mapping service for an accurate distance calculation.
</answer>
User Query [Provided functions)]] have updated some more functions you can choose from. What about now?
<think>
Now that have the 'estimate_ diitaance' function, San Francisco and Stonebrook hy first getting the zipcodes for both cities. However, we need to use get zipcode based on city" to get the zipcodes, which is not directly what we need for the "estimate distance" function. Since the question only gives the names of the cities and no function convert city
find the distance between
to zipcodes is directly available, could start by estimating the distance using the city names assuming an accurate distance estimation service might be available. </think>
stool_call:
["name": "estimate, diittnced "arguments": "cityA": : "San Francisco", "cityB": "Stonebrook"]
</tool call>
response>
("error": "Invalid zipcode pair: 'San Francisco's to 'Stonebrook'. Please verify both zipcodes are correct and numeric.") </tool_ sheoo_respons>>>
This a key distinction of the enhanced environment. Unlike the vague feedback the baseline environment (e.g.. !"error"; "distance not found in database. 1)), our tuned envi- romment provides much richer error message. It clearly identifies the problem-an input type mismatch (expecting zip codes instead of city names)-and offers clear suggestion for correction ("Please verify both zipcodes are correct and numeric." "). This high-quality, actionable feedback is crucial for the agent diagnose the issue and effectively adjust its strategy.
Note:
38

---

Published as a conference paper at ICLR 2026
<think
The estimate_distance' function call resulted in an emor indicating the zipcodes for "San Francisco" and "Stonebrook" invalid. Since the tool expects zipcodes instead of city names, need to use the zepeeee baseedddeebae__eeeeeeee _o___oo___tt____ ootyy function to get the correct zipcodes for San Francisco and Stonebrook.
</think>
<tool <tool_call> call>
I"name": "get_zipeode_mee saapcodebbsse___ee based_ose _asi__oo__tt""""__ "arguments": "city": "San Francisco", "name":
"get peo__zipedd__aaaee baa___da__c_tdtt"""" "arguments": "city": "Stonebrook"]
</tool call>
<tool responess
[["zipcode": "94016"), I"zipcode": "74532")) </tool response>
<think>
The zipcodes for San Francisco and Stonebrook have been obtained: 94016 and 74S32 respectively. Now the "estimate distance' function find the distance between these zipcodes.
</think>
<tool_call
l"name": astimate_distane "arguments": "cityA": "94016", "cityB": "74532"]
<tool_ response>
("distance": 880.0)
</tool response>
<think>
The distance between the zipcodes 94016 (San Francisco) 74532 (Stonebrook) 880.0 as estimated hy 'estimate, distance' function. This answers the user's request road distance between San Francisco and Stonehrook. </think>
<answer>
estimated road distance between San Francisco Stonebrook kilometers. should help you plan your journey for your genealogy exploration. </answer>
User Query 3: Buzzing with anticipation for this family roots journey, wanf tweet: 'Setting forth on an exciting quest from San Francisco Stonebrook to uncover ancestral stories! #Genealogy
segeealoyyAventture Adventure #FamilyHistory.
<think>
The user wants to post a tweet with the given content and tags. For this, we need to use the "post_ tweet" function. The content of the tweet and the tags are already provided, so we can directly call the function with the required parameters. </think>
<tool_call>
["name": "post_ "post_tweet", "arguments": "content": "Setting forth on an exciting quest from San
Francisco Stonebrook to uncover ancestral stories!". "tags": ("#GenealogyAue Adventure", "#Family-
History"]]
</tool call>
39

---

Published as a conference paper at ICLR 2026
<tool response>
("id": "12", "usemaae": "genealogy _enthusssst,,tt"", "content": "Setting forth exciting quest from San Francisco to Stonebrook to uncover ancestral stories!", "tags": ##GenealogyAdvenntu "#FamilyHistrry",. "mentions": []] </tool response>
<think>
The tweet has been successfully posted with the given content, tags, and no mentions. The ID, username, content, tags of the posted tweet are included in the response, indicating tweet was successfully sent out.
</think>
<answer>
Your tweet been successfully posted: "Setting forth on an exciting quest from San Francisco to Stonebrook to uncover ancestral stories!" with the hashtags #GGenealogyAvveeture and #FamilyHis- tory. You can now follow your journey with these words. </answer>
User Query Once the tweet live, should retweet widen the circle of those who might share this genealogy fervor!
<think>
The user wants to retwect the recently posted tweet. We need the ID the original tweet for retweeting. The ID of the recently posted tweet can be obtained from the previous execution results, and the "retweet" function to retwect it.
</think>
<tool_ call>
["name": "retweet", "arguments": "tweet_id id": 10]
</tool_call
<tool_ reooosrssp
("retweet_status "Successfully retwected") </tool_response>
<think>
The function 'retweet" has successfully retweeted the tweet with ID 10. This action has been completed requested by the user.
</think>
<answer>
Your tweet has been successfully retweeted, widening the circle of those who might share in this genealogy fervor. Continue your journey and uncover your family's stories!
</answer>
40

---

Published as a conference paper at ICLR 2026
Ground Truth:
[,,
Francisco")), "get_ (o_.aassooo_cctt(((srreeeeeeee
"estinate distance (attyyy==90016 74533"")
"post txeet (content=' Setting forth exciting quest from San Francisco stenebrook to uncover ancestral stories!', tags=[' AGencalogy/dveentur"
'AFamilytistory')
"retweet (tweet_id=10)
Task: Vehicle Control
User Query 1: You're planning extended expedition from Rivermist to San Francisco. Could you determine the travel distance between these two locations for me?
<think>
To determine the travel distance between Rivermist San Francisco, need to use the "esti- mate distance" function provided the of available tools. The function requires 'cityA' and 'cityB' as parameters, which correspond to the starting and ending cities for the journey.
</think>
<tool_ call>
["name": "
"istance".
"arguments": "cityA": "Rivermist". "cityB": "San Francisco")
</tool call>
responssss
["error": "distance not found in database."]
</tool resposeespsss>>
<think>
The function call to estimate the distance between Rivermist and San Francisco returned error message stating that the distance was not found in the database. This suggests that either Rivermist does have proper city data associated with or the function cannot find an entry for this specific route.
</think>
<answer>
I'm sorry, but the distance between Rivermist and San Francisco could not be determined the system does not have required data. Please provide more detailed location information or check the spelling of the city names. </answer>
User Query 2: Next, I need to verify if my car's mileage capacity suffices for the journey. Can you assess if vehicle can handle the calculated travel distance?
41

---

Published as a conference paper at ICLR 2026
<think>
To provide the correct response, need to first determine the travel distance between Rivermist and before, will call the "estimate distance" function
San Francisco. Since encountered with correct city names and use the "estimate drive drive_feasibility bacpiityyby_iiiee bymmeeeeee" function to check if the vehicle can handle the trip.
</think>
<tool call>
["name": "estimate_ ciaauttd_dttttee,.._ "arguments": "cityA": "Rivermist", "cityB": "San Francisco",
"name": restimate_drive, ditirttsdrrveeffsssbilttyy
"arguments": "distance": "Sdistance"|
</tool call>
<tool_response: response>
<tool
["error" "distance found in database.". "Error during execution: <<' not supported between instances "float' and "]
</tool_ resooseess
<think>
The function calls estimate the distance and verify the drive feasibility both encountered The distance estimator could not find the data for Rivermist, and feasibility estimator an issue with an unsupported operation involving float and str types. Since the distance could not be determined, we cannot proceed with checking the vehicle's 's mileage capacity. Therefore, we need to re-evaluate provide the correct city names and try again. </think
<answer>
I was unable to determine the travel distance between Rivermist and San Francisco due to missing city data. Please verify the correct spellings of the city names and try again.
</answer>
User Query 3: seems I've discovered tank isn't full. Could you be so kind as facilitate refueling the vehicle fully for such a lengthy trip?
<think>
To ensure the vehicle's tank fully filled, we will use the "fillFuelTaank function with fuel amount of 50 gallons, which is the full capacity of the tank.
</think>
<tool_coo call>
("name": "fillFuelTank", "arguments": "fuelAmount": 50]
</tool call>
<tool_ resoonsssssse>>
["error": "Cannot fill gas above the tank capacity." "]
</tool response>>
<think>
The function call fill the fuel tank encountered error indicating that the fuel cannot be filled above the tank capacity. Since the tank's full capacity is 50 gallons, and we attempted fill with exactly 50 gallons, the error suggests that the tank is already full or the system does not allow refueling to exceed the tank's 's capacity even if it's not full. </think>
<answer>
seems that the vehicle's 's fuel already full, the system does not allow refueling beyond the tank's 's capacity. If you're certain the tank is not full, please check the current fuel level manually or contact service provider for assistance. <lanswer>
42

---

Published as a conference paper at ICLR 2026
User Query 4: With the tank replenished, let's confirm readiness for departure. Please secure doors and initiate the engine me.
<think>
To confirm the readiness for departure, we need to ensure all doors are locked and the engine is started. We will use the "lockDoors" function to secure the doors and the "startEngine" function to start the engine.
</think>
<tool_call>
l"name": "lockDoors", "arguments": "unlock": false, "door": ("driver", "passenger". "rear_left""
"rear rigrr,, "name": "startEngine", "arguments": "ignitionMode": "START"J </tool_call call>
<tool_ reno_reesssse>>
("lockStatus": "locked", "remainingUniookkeDDooos""" "error": "Brake pedal needs to pressed when starting the engine."] </tool response>
<think>
door locking function successfully locked doors, indicated by "lockStatus" being "locked" and "remaining UnwoeeinngnnooeeeeDeDeaoooo UnlockedDoors' being 0. However, the engine could not he started because the brake pedal needs to he pressed to start the engine.
</think>
<answer>
All doors have been successfully locked. However, the engine could not be started because the brake pedal needs to be pressed when starting the engine. Please press the brake podal and try starting the engine again.
</answer>
User Query 5: As vehicle's prepared, let's delve into evaluating some fuel efficiency statistics via the mean of these travel distances: 750.0 km, 320.0 km, 450.0 km, and 280.0 km.
<think>
To evaluate the fuel efficiency statistics, we need to calculate the mean the given travel distances. The required function for calculating the mean is provided in the available tools. We will use the "mean" function with the list of travel distances the argument.
</think>
<tool_ call>
l"name": "mean", "arguments": "numbers": [750.0, 320.0, 450.0, 280.0]]
</tool_ call>
stool_respons>>>>
["fesult" 452.5"1 </tool response>
<think>
The function call caleulate the mean of the travel distances successful and returned the result 452.5 km.
</think>
<answer>
The mean of the travel distances provided (750.0 km, 320.0 km, 450.0 km, and 280.0 km) is 452.5 km. This value represents the average distance for these trips.
</answer>
43

---

Published as a conference paper at ICLR 2026
Ground Truth:
picdee betppcoe__bsseooo_tteeeeeee San Francisco')) )"
"get_zi pio.ee pwooe__asseeoonntttyye.... (iiioonn viveriist""""
"esti
cityR-/94016)
"estimate "estimate_drive httpaatee_dddvee_esiiiiiilt.... mi leage
(deaa (distance=980.0))
1.
[
" "LockDoors (unlock=Falsss door= driverr', 'passenger',
'rear_left', 'rear_right'])
"pressBrakePedal (pddoostttooottts.oooo.......... "startEngine (agga:/aeeeaaaaatiooooooooedeee1 START')
"mean (numbers- [750.0, 320.0, 450.0, 290.0))
H.4 OOD GENERALIZATION ON MULTI-ISSUE TROUBLESHOOTTIN
This
(OOD) case study from the re-behch's telecom domain is presented to directly address the question of whether the improvements from ENVIRONMENT TUNING stem from genuine refinement of problem-solvine abilities. We demonstrate that method equips the agent with more robust capacity for multi-step diagnostics, rather than superficial pattern matching.
The task requires the agent to troubleshoot user's slow mobile data speed. The core challenge lies in the fact that there are two distinct underlying issues: (1) "Data Saver Mode" is enabled the user's device, and (2) the device-level "Data Roaming" setting is disabled, even though roaming enabled on the carrier side. The pre-trained model identifies and resolves the first issue but fails diagnose the second, leading to task failure. In contrast, the model fine-tuned with ENVIRONMENT TUNING successfully identifies and resolves both issues, showcasing clear improvement in its fundamental diagnostic and tool-use capabilities.
Task: bench Telecom
User Query: My mobile data not working properly. either stops working is very slow. want to it and absolutely want get excellent internet speed on my phone. am not willing to accept any other internet speed (poor, fair or good). do not have access to wifi.
<think>
The user is experiencing slow mobile data. I need to check several things on the device to diagnose the problem. will start by checking the network status and data restriction status.
</think>
<tool_call call>
("name":
"arguments": "name": "check data data settaaaretttttoostttaaeeee "argu-
ments":
</tool_ call>
44

---

Published as a conference paper at ICLR 2026
<tool response>
[("airplane saaplaane_mode.: "off", "signal strengthh" "excellent", "network type:: "5G", "mobile data": "enabled", "roaming": "Disabled"), ("data_sae saverrssverraaeeee saver_mode". "on", "background "aata_uuugee" "re- stricted")]
</tool response>
<think>
The network status shows that mobile data is enabled and the signal is excellent. However, the data restriction status shows that data saver mode on. This likely the cause speed. will guide the user to turn off' data saver mode.
</think>
<answer>
I see that your phone's Data Saver mode is currently on, which likely causing slow speeds.
Please turn it off, and then we can check the speed again.
</answer>
<think>
The user has turned off data saver mode. Now will run speed test to see if the issue is resolved and if the speed "excellent'. </think>
Mhool_call>
["name": "run_speed seses_peesst"""" "arguments":
</tool call>
<tool_ responssss
[[ [["dwnloadd _spedd": "good")] </tool_ responses
Note (Failure Point): The agent correctly identifies and resolves the first issue (Data Saver). However, when the speed test returns "good" instead of the required "excellent, the agent fails to perform further diagnostics. It does not recognize the significance of "rramiig":: "Disabled" from the initial check. gets stuck in a loop of re-testing the speed, and then incorrectly concludes the only remaining option change the user's plan, which violates user's 's explicit instructions. This demonstrates a failure step, exhaustive troubleshooting.
<think>
The speed is now "good", but the user requires "excellent" speed. have already addressed the data saver mode. Since there are no other apparent issues from the initial diagnostics, and the user is unwilling change their plan, I cannot further improve the speed. The only other option would be plan with higher priority data, but the user has rejected this. </think>
<answer>
I was able to improve your speed to "good" by disahling Data Saver mode. Unfortunately. without changing your current data plan, cannot guarantee "excellent" speed. Would you like reconsider exploring other plan options? </answer>
Task: -e-ch Telecom (After ENVIRONMENT TUNING Training)
User Query: My mobile data is not working properly. It either stops working or is very slow. want to fix it and absolutely want get excellent internet speed my phone. am not willing accept any other internet speed (poor, fair or good). do have access to wifi.
45

---

Published as a conference paper at ICLR 2026
<think>
The user has slow mobile data. I'll check network and data restriction status.
</thinks
<tool_ca call>
("name": www.nneetttttttteeeeee
"arguments": "name": "check data restriction restriction_ssatuus
"argu-
ments":
</tool call>
<tool response>
[[ [["airplane, mode": "off", "signal_strennlhee strength": "excellent", "network "network_type". "SG", "mobile data": "enabled", "roaming": "Disabled"), {" ("data_save saver mode": "on", "background datt data_ usage": "re- stricted")]
</tool response>
<think>
Data saver is on, which is likely cause. will ask the user to disable it.
</think>
<answer>
I see that your phone's Data Saver mode is on. This could slowing down your data, Please turn off, and we' lll test the speed again.
</answer>
<think>
Now that data saver is off, I will check the speed.
</think>
sooo_ccal>>
l"name": guu__peed_test"s "arguments":
</tool_calli call>
stool_response>
[["download speed": "good"]]
</tool response>
Note (Success Point): Unlike the pre-trained model, the agent trained with ENVIRONMENT TUNING does not stop here. Recognizing that user's 's goal of "excellent" speed not yet met, re-evaluates initial diagnostic information. It correctly identifies "rooming": "Disabled" the next most likely cause of the problem. This demonstrates a crucial improvement its ability to conduct thorough, multi-step investigation.
<think>
The speed is "good," which is an improvement, but not the "excellent" speed the user wants. that while the line has roaming enabled by the carrier, the
Reviewing my initial checks, device's own data roaming setting is disabled. Since the user abroad, this the next logical issue to address. will guide the user to enable data roaming on their device. </think>
<answer>
We've made some progress, and your speed is now "good". However, get "excellent," noticed that data roaming setting on your phone is turned off. Since you are currently abroad, this likely restricting your speed. Could you please enable data roaming on your device?
</answer>
46

---

Published as a conference paper at ICLR 2026
<think>
The user has now enabled data roaming. I will perform a final speed test to confirm we have reached "excellent" speed.
</think>
<tool_call
("name": "run_speedd" test", "arguments":
</tool_cal calls
<tool responsss
[[ [["download] : "excellent"]] </tool response>>
<think>
The latest speed test confirms the download speed is "excellent". Both the data saver and roaming issues have been resolved. user's goal has been fully met.
</think>
<answer>
Great news! The speed test shows "excellent" connection. It seems the combination of turning off Data Saver and enabling data roaming has resolved the issue completely. there anything else I can help you with?
</answer>
47


Reviews:
[{"review_id": "8hZ9u6c2wt", "reviewer": "Reviewer_p3Yu", "summary": "This work aims to tackle the problem of limited expert training data, simultaneously overcoming overfitting in SFT and cold start and instability in RL by designing a structured curriculum with training environment. The environment provides corrective feedback and dense reward signals, achieving not only better in distribution performance but also better generalization with less data.", "questions": "1. OOD Generalization performance is only compared to SFT. It would be interesting to demonstrate the proposed method’s generalization ability compared to vanilla RL methods without dense reward and detailed feedback. \n\n\n2. In Figure 5b, why did the model's performance not drop when transitioning to the next stage, where the model was only trained in easier stages but was exposed to new and more challenging tasks? \n\n\n3. How easy is the proposed method to apply to a task in an unseen field? Do practitioners need to provide detailed environment error feedback and redesign the dense reward component or other modules? Discussing how much manual engineering is needed for a new task will be helpful for evaluating the method’s generalization to unseen tasks.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Proposed a clear and progressive roadmap that tackles each challenge of sparse reward, long horizon, and instability. \n\n2. Novelty in tuning the environment feedback instead of tuning the model prompt. \n\n3. Extensive ablations were performed to show the importance of each stage and enhancement.", "weaknesses": "1. In real-world applications, environments are oftentimes not accessible or modifiable. Tuning environmental feedback will be expensive or even impossible. \n\n2. Revealing internal tool constraints (Section 3.3) requires prior knowledge from humans, which is expensive and involves heavy reasoning burdens for human experts to provide meaningful explanations, especially in a multiturn setup where more complicated and chained reasons for error would appear. \n\n3. Discovering inter-tool dependencies via exploration (Section 3.3): The paper proposes to embed inter-tool dependencies in detailed environment feedback. This is confusing to be considered as exploration since the model is not actively searching for broader or more informative states in the environment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vucd3JCbDu", "reviewer": "Reviewer_eWdD", "summary": "This paper introduces Environment Tuning, a RL framework designed to improve LLM agents’ performance in multi-turn tool-use tasks. Instead of only fine-tuning the model, the authors propose systematically modifying the environment to provide structured, corrective, and reward-rich feedback. The method combines a structured curriculum, actionable environment augmentation, and fine-grained progress rewards to guide learning efficiently. Experiments on BFCL and ACEBench demonstrate notable gains in both in-distribution and out-of-distribution performance, highlighting the approach’s effectiveness and generalization potential.", "questions": "Please check my comments in Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core idea of enhancing agent training by incorporating environment-level variations is creative and interesting.\n2. The method is well-designed and clearly presented, with intuitive reasoning and an easy-to-follow structure.\n3. Experimental results across multiple datasets convincingly validate the approach’s effectiveness and robustness.", "weaknesses": "1. Although the concept of Environment Tuning is intriguing, the term itself may be somewhat misleading. In the proposed framework, environment changes are predefined and discrete, rather than dynamically adapted based on the agent's learning progress. Thus, the word “tuning” might suggest a level of adaptivity that is not actually present. Terms like environment augmentation or dynamic environment design could better capture the nature of the approach.\n2. The experimental section lacks direct comparison with other recent multi-turn tool-use approaches that also use RL or hybrid training. While related works are discussed in Section 2, their omission from the experimental benchmark makes it difficult to fully assess the method's relative advantage over competing paradigms in similar problem settings.\n3. The paper’s progress reward relies on ground-truth evaluation signals provided by the benchmark datasets. This raises the question of how the method would generalize to scenarios without explicit ground-truth feedback. Moreover, since these signals are dataset-provided, it would be valuable to clarify whether existing methods truly lack access to or use of such supervision—otherwise, the claimed novelty of the progress reward may be overstated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dBZtI8289C", "reviewer": "Reviewer_xCSn", "summary": "This paper introduces \"Environment Tuning,\" a novel training paradigm designed to equip Large Language Model (LLM) agents for complex, multi-turn tool-use tasks, particularly under conditions of extreme data scarcity. Its core innovation involves a shift from static, trajectory-based imitation learning towards a dynamic, environment-based exploration strategy. This allows agents to learn directly from problem instances without dependence on pre-collected expert demonstrations. The method demonstrates strong performance on the BFCL-V3 benchmark using only 400 problem instances, significantly enhancing the capabilities of base models and exhibiting robust out-of-distribution generalization—a known weakness of traditional supervised fine-tuning (SFT) approaches.", "questions": "Regarding Environment Augmentation: The \"Actionable Environment Augmentation\" is a key component of your method's novelty. Could you please elaborate on its implementation?\n1. How are the corrective hints generated—are they based on heuristic rules, programmatic templates, or perhaps an LLM-based generator?\n2. What measures are implemented to ensure these hints provide guidance without leaking the correct solution or overly constraining the agent's exploration?\nRegarding the Training Curriculum: The proposed four-stage curriculum is empirically sound, but its design appears somewhat heuristic.\n1. Could you elaborate on the rationale for selecting these particular four stages? How sensitive are the final results to the ordering of stages or the specific criteria used for transitioning between them?\n2. Were any alternative or adaptive curriculum scheduling strategies explored during your development? If so, please summarize those findings.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents Environment Tuning, an innovative framework that reorients LLM agent training from static trajectory imitation to a dynamic, environment-driven learning process. By effectively integrating a structured curriculum, actionable environment augmentation, and fine-grained progress rewards, the method directly tackles the critical challenges of data scarcity and training instability in multi-turn tool-use scenarios. The experimental results on BFCL and ACEBench are compelling, demonstrating substantial improvements over both supervised fine-tuning and standard reinforcement learning baselines. The manuscript is well-written, with a strong motivational foundation. The conceptual shift of \"tuning the environment\" itself constitutes a genuine and noteworthy contribution to the field.", "weaknesses": "The primary weaknesses relate to empirical rigor and the scope of validation. The \"actionable environment augmentation\" mechanism, while central to the method, is evaluated qualitatively; a more systematic, quantitative analysis of its specific impact is needed. The claims regarding generalization, though promising, would be more persuasive if supported by results from a wider array of diverse environments. The analysis would benefit from reporting statistical significance and performance variance. Furthermore, the method's current reliance on controllable simulation environments may pose challenges for real-world deployment. Finally, the theoretical justification for the specific four-stage curriculum design could be more thoroughly elaborated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k12nTF71gW", "reviewer": "Reviewer_M2ve", "summary": "This paper proposes a new method named environment tuning, which tries to design different difficulty-level environments (by actionable environment augmentation techniques) to train the model in different stages. By using such curriculum learning, the model can learn to handle ambiguity, recognize functional gaps, and perform information retrieval from noisy contexts.\nThe experiments show that the model can improve their performance on the benchmark tasks within every stage. It also improves the importance of actionable environment augmentation.", "questions": "1. In Table 2 the 23.87 should be orange.\n\n2. The transit point (Remark 3.1) is not clear; it is not easy to understand how to find it (even in Appendix D).\n\n3. The topic in BFCL V3 is not clear in this passage, for example, \"Is there a web search task in BFCL V3?\" If yes, the web search should not be OOD.\n\n4. Why is the progress reward lower than the final score in Figure 4.\n\n5. Does the OOD improvement really come from the model's refinement ability? Maybe an OOD case is important. Also, you can train the model on the OOD task and show the scaling line to figure out whether it can scale in OOD tasks", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear and easy to understand.\n\n2. Considering improving the environment instead of improving the algorithm to make the model better is an interesting idea. I just believe the data is more important than the algorithm in agent RL, so considering the environment is an important topic.\n\n3. I believe this project is a good realization of the idea of curriculum learning, the four stage is clear and makesence\n\n4. The experiments are reasonable and can prove the effectiveness of actionable environment augmentation and .", "weaknesses": "1. The reward design may have some flaws. It just calculates the average reward of each step and uses GRPO to train the model, so the bad action at the end of the trajectory will not be punished (since the final reward is high). This may cause a credit assignment problem.\n\n2. It seems that this work is trying to distill preliminary knowledge in the environment to the model for stable RL training. However, such preliminary knowledge may come from the annotator of the benchmark, in other words, the author should prove their OOD performance does not come from telling the bias of BFCL's author. So, the ablation study should contain more benchmark tasks.\n\n3. The analysis can only prove that such a method can improve the OOD performance, but it doesn't figure out why it can improve the OOD performance. (refine ability? Better instruction following? other?) Comparing stage 1-4's model on OOD task is more important. (Also see Questions 5)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)