Don't Just Fine-tune the Agent, Tune the Environment
Authors: name: Siyuan Lu; name: Zechuan Wang; name: Hongxuan Zhang; name: Qintong Wu; name: Leilei Gan; name: Chenyi Zhuang; name: Jinjie Gu; name: Tao Lin
Published: 2025-10-11 12:35:15

Abstract: Large Language Model (LLM) agents show great promise for complex, multi-turn
tool-use tasks, but their development is often hampered by the extreme scarcity
of high-quality training data. Supervised fine-tuning (SFT) on synthetic data
leads to overfitting, whereas standard reinforcement learning (RL) struggles
with a critical cold-start problem and training instability. To address these
challenges, we introduce Environment Tuning, a novel training
paradigm that enables agents to learn complex behaviors directly from problem
instances without relying on pre-collected expert trajectories.
Environment Tuning orchestrates this learning process through a
structured curriculum, actionable environment augmentation that provides
corrective feedback, and fine-grained progress rewards to ensure stable and
efficient exploration. Using only 400 problem instances from Berkeley
Function-Calling Leaderboard (BFCL) benchmark, our method not only achieves
competitive in-distribution performance against strong baselines but also
demonstrates superior out-of-distribution generalization, overcoming the
performance collapse common to SFT-based approaches. Our work presents a
paradigm shift from supervised fine-tuning on static trajectories to dynamic,
environment-based exploration, paving the way for training more robust and
data-efficient agents.