AgentGym-RL: An Open-Source Framework to Train LLM Agents for Long-Horizon Decision Making via Multi-Turn RL
Authors: Zhiheng Xi
Jixuan Huang
Chenyang Liao
Baodai Huang
Jiaqi Liu
Honglin Guo
yajie yang
Rui Zheng
Junjie Ye
Jiazheng Zhang
Wenxiang Chen
Wei He
Yiwen Ding
Guanyu Li
Zehui Chen
Zhengyin Du
Xuesong Yao
Yufei Xu
Jiecao Chen
Tao Gui
Zuxuan Wu
Qi Zhang
Xuanjing Huang
Yu-Gang Jiang
Venue: ICLR 2026 Oral (2026)
Keywords: large language model
LLM-based agent
decision-making
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present AgentGym-RL, a unified open-source framework for training LLM agents from scratch across diverse and realistic environments, and propose ScalingInter-RL, a staged training strategy for stable long-horizon RL training.

Abstract: Training LLM agents for complex multi-turn decision-making tasks requires extensive exploration within their environment, with reinforcement learning (RL) as a natural way. However, the open-source community currently lacks a unified RL framework capable of training agents from scratch across diverse and realistic environments. To bridge this gap, we introduce AgentGym-RL, a modular and decoupled framework specifically designed for RL-based agent in multi-turn decision-making tasks. It offers high flexibility and extensibility, supports mainstream RL algorithms, and spans a broad range of real-world scenarios. To effectively train agents for challenging tasks, we argue that they are required to expand external interactions with the environment, rather than relying solely on internal reasoning. Nevertheless, training agents for long-horizon interaction with vanilla methods often faces challenges like training instability. To this end, we propose ScalingInter-RL, a staged training approach for stable long-horizon RL training. It starts with short-horizon interaction to establish foundational policies and progressively expands them to encourage deeper exploration. Extensive experiments show that agents trained with our method achieve performance on par with—or even surpass—commercial counterparts like OpenAI o3 and Gemini-2.5-Pro across 27 tasks in diverse environments. We share key insights and release the full framework, including code and datasets, to empower the community in building the next generation of intelligent agents. Our framework is available at https://github.com/WooooDyy/AgentGym-RL.

Published as a conference paper at ICLR 2026
AGENTGYM-RL: AN OPEN-SOURCE FRAMEWORK TO TRAIN LLM AGENTS FOR LONG-HORIZON DECISION MAKING VIA MULTI-TURN RL
Zhiheng Xit.*.1, Jixuan Huang".", Chenyang Liao?:*, Baodai Huang', Jiaqi Liu', Honglin Guo Yajie Yang', Rui Zheng', Junjie Ye Jiazheng Zhang', Wenxiang Chen', Wei He!, Yiwen Ding', Guanyu Li Zehui Chen², Zhengyin Du Xuesong Yao?, Yufei Xu², Jiecao Chen', Tao Gui 133,4,t Zuxuan Wul.3,4 Qi Zhang! 1.3..t,
Xuanjing Huang .34 Yu-Gang Jiang 1..4
FFudan University "ayteDance Seed SShangaai Key Laboratory of Multimodal Embodied Al "Shanghai Collaborative Innovation Center of Intelligent Visual Computing
zhxi 22 @m. fudan. edu. en, {tgui, @z]dfudan, edu. cn
ABSTRACT
Training L.M agents for complex multi-turn decision-making tasks requires ex- tensive exploration within their environment, with reinforcement learning (RL.) as natural way. However, the open-source community currently lacks a unified RL framework capable of training agents from scratch across diverse and realis- tic environments. To bridge this gap, we introduce AgentGym-RI, and decoupled framework specifically designed for RL-based agent multi-tur turt decision making tasks. It offers high flexibility and extensibility, supports main- stream RL algorithms, and spans broad range of real-world scenarios. fectively train agents for challenging tasks, we argue that they are required to expand external interactions with the environment, rather than relying solely on internal reasoning. Neverthcless, training agents for long- horgooorizon interaction with vanilla methods often faces challenges like training instability. To this end, we propose ScalingInter-RI, staged training approach for stable long-horizon RL training. It starts with short-horizon interaction to establish foundational policies and progressively expands them to encourage deeper exploration. Extensive ex- periments show that agents trained with our method achieve performance on par with-or or even cappass-commerriii counterparts like OpenAl o3 and Gemini- 2.5-Pro across 27 tasks in diverse environments. We share key insights and re- lease the full framework, including code and datasets, to empower the community in building the next generation of intelligent agents. Our framework is available at
modular
ef-
INTRODUCTION
As Large Language Models (LLMs) rapidly advance (OpenAl 2023 Anthropic 2024 DeepSeek- al. to autonomous agents addressing long-horizon real-world decision-making tasks (Xi al. 2025a
Al al, 2024 Team et al. 2023 Yang et al. (2225), their applications have extended from chatbots
Moonshot Al 2025). Analogous to human cognitive development, LLM agents are expected to
acquire new knowledge and skills by actively exploring with the environment (Xi et al. 2025b
OpenAl 2025).
Reinforcement learning (RL) is a natural choice for achieving this, demonstrating success in LLM reasoning (DeepSeek-Al al. 2025 Jaech et al.) 2024 Xi al. 2024a TTunn et al.) 2024 Team et al. 2025b; He et al. 2025). While recent efforts have sought to extend RI. methodologies to
develop L.M agents with multi-tum interaction capabilities (Zhou et al. 2024h) Chen al. 2025
"Equal contribution.
1 "Correspnniing authors.

---

Published as a conference paper at ICLR 2026
Average Percentile
90
80
Overal Accuracy (%)
50
40
10
TestCraft
2
2
2
#Parameters (Billion)
Qwer 5.78
Qwert 23500-220
Gemini2 Saace
Ozendt
Overd 525
Ovent-8 Ovenb-253--225
Usmad 5-00 DeepS Curs-7
Ower
Laamzs11 88
2.5
Closud Medes
Owend Qeura
Ower 5 48
Lam25.1-705
Owen Mox
Frincing Modes
Figure I: Left: Performance of proprietary models, open- open-source models, and our RL models across different agentic tasks. Right: Performance model scale.
Wang et al., 2025 Qi et al. 2025 Jin et al. 2025b; Cao et al.) 2025), they still struggle with lim- ited task complexity and insufficient environmental diversity. Critically, the open-source community lacks unified RL framework capable of training agents from scratch across diverse, realistic envi- ronments.
To bridge this gap, we introduce AgentGym-RL ($3,, a unified framework designed for training tasks (Figure 2}. With a mod-
LLM agents through RL in multi-turn interactive ular and decoupled architecture, AgentGym- enables clean separation of agents, environments, and learning algorithms, offering high extensibility and flexibility for diverse research needs. The framework supports mainstream RL algorithms, and covers wide range of real-world scenarios, e.g., web navigation (Zhou et al., 2024a Yao et al.] 2022), deep search (Wei et al., 2025 Jin et al., 2025b), digital games (Prasad et al. 2024 Fan et al.] (2022), embodied tasks (Chevalier- Boisvert et al. 2019 Shridhar et al. 2021 1). and scientific tasks (Wang et al. 2022 Starace et al. 2025).
Furthermore, enhance agents' ability to tackle challenging tasks, argue that expanding their interactions with the environment is crucial, rather than relying solely on internal reasoning. How- ever, our preliminary experiments show that directly training agents for long-horize interaction often faces instability. To address this, we propose ScalingInter-Rl ($4) based on AgentGym-RI This progressively scaling interaction enables the agent to avoid repetitive and unproductive actions, enhance deeper exploration of environments, and ultimately achieve more effective and efficient task completion while maintaining training stability.
Extensive experiments ($5) demonstrate that ScalingInter-RI within AgentGym-RL framework de- livers significant performance gains across tasks spanning 5 diverse scenarios (Figure Left)). Open-source models. e.g., Qwen-2.5-7B ( (Yang et al., 2024], achieve an average improvement of 33.65 points, matching or even surpassing larger commercial models such as OpenAl-o3 (OpenAl, 2025) and Gemini-2.5-Pro (Comanici et al. 2025). In addition, we conduct extensive analytical experiments to provide key insights (66). showing that scaling both post-training and test-time in- teractions holds substantial potential for advancing agentic intelligence (Figure Rigght). We hope our work will serve as a valuable contribution to the community's 's progress.
PRELIMINARIES
2.1 FORMULATION
In this work, we study the multi-turn interactive decision-making tasks, i.e., agentic tasks, and model them as a Partially Observable Markov Decision Process (POMDP) (4,S,A,0,T, like A + S,r: U x S IR represents
Xi et al. 2025b Zhou et al. 2024b), where S, O, the instruction space, the state space, the action space, the observation space, the deterministic state transition function, and the reward function, respectively.
2

---

Published as a conference paper at ICLR 2026
OBJECTIVE: Tell me the full names of the repositories where made contributiins and they got than 100 stars?
K-turn Rollout for times
(
K-+1
T(()
(I)
(
Science
4g
1
OK
LLM Agent
(1)
HTTP
k
k
Search
dg
Env Client
Env Server
3
Trujectory
Update Policy
T(6)
7(f)
(i) O
{6}}
Ok
x+1
Embodied
Games
Code
<thinkz Click the 'Yours' tab see the repositoriies where we have made contributions.. </thinke-
Since we
repositories and none of them have than </thnk../
>(rr) =1
Suets click [411] where [41]] is the
sop <tart>
Figure 2: Overview of the AgentGym-RL framework. It features decoupled, flexible, and exten- sible architecture, comprising three primary modules- environment, the agent, and the training module. It supports diverse scenarios, environments, and algorithms.
U, the agentic task requires the LLM agent generate sequence of actions af ~ (T(-(s)) based on its policy parameterized by 9 to complete the given task, where aş E A, and Sy E S, and T is the reasoning path (Yao et al. 2023). The agent then receives an observation Oę E O from the environment, and the state is then transitioned to T(ss, (xk) = 8k+1- Finally after turns of interactions, the environment provides an outcome reward r(r) E [0, to describe the completion of the multi-turn interactive decision-making tasks.
Given task instruction
2.2 POLICY GRADIENT
We utilize policy gradient (PG) methods (Sutton et al. 1999) that optimizes our policy agent. They perform gradient ascent according the objective (0). which is function of the policy param- eters 0. Specifically, (0) represents the expected cumulative reward the agent anticipates receiv- ing when following policy Ty and interacting with the environment. Mathematically, this is ex- pressed as the expectation of the total reward r(r) over trajectories generated by the policy: J(() E,-so [r(r)]- To perform optimization on (()), we require the policy gradient (g)( In the vanilla policy gradient methods, the policy gradient can be estimated by:
JoJ ETaoT
r(T)
Vo log (ak(ssk)
k=0
where is the policy parameterized 6, represents trajectory consisting of a sequence of states and actions, ag and 8ą are the action and state at time step k; and r(r) is the reward of the trajectory Mainstream RL algorithms for training LLMMs include PPO (Schulman et al. (007)) GRPO (Shao) et al. 2024), and REINFORCE+ Hu 2025
all of which are integrated into our framework.
3
THE AGENTGYM-RL FRAMEWORK
3.1
ARCHITECTURE OVERVIEW
AgentGym-RI adopts modular design with well-defined responsibilities for each module, allowing for extensibility, As shown in Figure 2 the framework is organized into three core modules.
Environment module. In this module, each environment encapsulated as an independent ser- vice with the option of deploying multiple replicas to support parallel requests. An environment
3

---

Published as a conference paper at ICLR 2026
Stage 1: Generate
Stage Brsch_froooroo_ prron_oolouut howlootthannderssnneee (hanlerss
experience
task_ids Ids expand(task sanple_nun)
batch gen, batch_
envs caaatt_en__cieenssteee
"webarena", base_unl)
batch aetor.conpure parpuute__o___eeeeeee _og_prenn(batch)
batcc reference.canpuse ref log (ratchhh)hh)))
Do in parallel:
batch corpute advantages
nethod-"grpor
(env, td) zip(envs, task_ids):
env.reset(task__d)
# Stoge 3: Actor tratntng acton.updato wwt.:-uppttttttttttoo per_acttoorattte
handlers -f
Rollouthaandler().. useer (es.ssseee((eeeosseree(()) for env in envs]
Parallel
range(max rounds)
prompts th.gett_romppte for h in handlers]
State & Reward
responses aaccoo. (ettr:generste(rronptt))
results ttheedd_sete _i5tt_it((____ parallel:
Agent
Environnent
for (env, response) in zip (envs, responses):
results.append www.steprreesposs)
Action
ra (handders,,, responses, results):
h.add_assiss pws.sssstttttt.... conesstatt_nessgeeG
h.add usen wes.sageerrs..stttnee h.score res.score
Update
Rollout
Training
(hlnder)(andier)):
Figure 3: Pseudocode demonstrating the example usage of our proposed framework (provided APIs marked orange), alongside a simplified diagram illustrating the agent-envirroment interaction and training pipeline.
client communicates with the environment server via HTTP and exposes APIs to the agent, includ- ing /observation to get the current observation, / aavailable act ions to get the currently available actions, / /step to perform action, and / /reset to reset the environment. Currently, AgentGym-Re RL covers five major scenario categories. This modular server-clien design allows new environments to provide comprehensive environment and data support for LLM agent training.
Agent module. The agent module encapsulates the ceasoning-action loop of LLM-based agents. receives observations from the environment, performs reasoning over multiple turns, and outputs actions (e.g., invoking provided APIs). The module supports different prompting strategies and sampling configurations.
Training module. The training module provides unified reinforcement learning (RL) pipeline that supports both online and offline algorithms, offering researchers a flexible foundation for large- scale LLM agent training. The module manages the entire RL lifecycle: trajectory collection, ad- vantage estimation, policy optimization, and reward shaping.
Workflow. The overall workflow and pseudocode are shown Figure Given batch of queries and initial environment states, the framework initializes multiple parallel environment clients. Each client serves single agent, ensuring isolated execution. At every step, the agent generates an action, the environment returns the updated state and reward, and the trajectories are collected concurrently for training updates.
The entire training pipeline can be distributed across multiple nodes, leveraging both multi-process and multi node parallelism. Efficient batching and asynchronous logging utilities ensure that system throughput scales with additional compute resources.
3.2 FEATURES AND CHARACTERISTICS
The AgentGym- RL framework is built on AgentGym ( et al. 2025b). which provides several basic interactive environments for LLM agents. We have further extended diversity of environments, algorithm support, engineering optimizations, open-source availability, and interaction visualization.
Diverse scenarios and environments. To build LLM agents capable of multi-turn decision- making, AgentGym-RL provides five heterogeneous environments spanning web navigation, deep search, digital games, embodied control, and scientific tasks. They exhibit significant variance in state space, action space, and reward structures. This cross-domain heterogeneity creates testbed
4

---

Published as a conference paper at ICLR 2026
0.6
Rownis 8
Revards
0.5
0.4
Mat Interacioon Rumde-10
ScalingloteeRL
Mat Intercttoon Romis-s
0.3
100
Truining Steps
Training Steps
Figure 4: Training dynamics under different maximum interaction turns Deep Search environ- ment. Our ScalingInter RL method progressively increases the interaction horizon, and ultimately achieves higher and more efficient long-tent performance.
for training and evaluating research artifacts across diverse environments. more detailed introduc- tion of the environments included is shown in Appendix
Comprehensive algorithm support. While the original AgentGym (Xi et al. 2025b focused pri- marily on SFT, AgentGym-RL, places online reinforcement learning at the core of its training stack. It allows agents adapt through continual interaction with the environment and move beyond static demonstration corpora. The framework unifies mainstream RL algorithms such as PPO (Schulman al. 2017), GRPO (Shao al.] (024), RLOO (Chen al. 2025) and REINFORCE++ (Hu 2025) under a single interface, while also supporting complementary offline paradigms including SFT (Peng et al. 2023), DPO (Rafailov et al., 2023), and self-improvement (Xi et al. (025bb).
Engineering optimizations. AgentGym-RL incorporates targeted engineering optimizations to support large-scale reinforcement learning research, with a focus on extensibility, scalability, and reliability. For extensibility, the framework adopts modular plug-and-play design, allowing new environments to be integrated by simple inheritance from base classes. For scalability, we enhance both computational parallelism and long-horizon training efficiency by introducing optimizations architecture and refined environment initialization routines. For reliability,
like subprocess-basee we address critical issues such memory leaks and flawed recursive implementations. more detailed description of the engineering optimizations is shown in Appendix C
Open-source availability and Visualization support. AgentGym-RL provides a unified frame- work with consistent evaluating metrics and reproducible training pipelines. It also offers turnkey scripts that automate the workflow from environment setup to final assessment, enabling reliable replication. Additionally, interactive graphical UI (See Figure TO in Appendix C supports visu- alization of step-by-step inspection and replay of full trajectories.
4 SCALINGINTER-RL SCALING INTERACTIONS FOR LLM AGENTS
Motivation. Inference- compute scaling in LLM reasoning shows that additional compu- tation offers better performance (DeepSeek-Al
Deep Scardh
ScWWorld
Ammurayy (50
et al. 2025 Jaech et al.) 2024). However, given the interactive nature of agent tasks, gue that effective progress requires expand- ing external interactions with the environ- ment, not merely internal reasoning. To val- idate this, we investigate the impact of increas- ing the maximum number of interaction turns available to the agent, using several baseline models Deep Search and SciWorld environments. As shown in Figure 5 all models show im- provement as the number of interaction turns increases, demonstrating that long-horizon interaction and sufficient exploration contribute to enhanced agentic performance. However, the performance gains of the baseline models plateau as the number of interactions continues to grow, indicating their limited capability to solve complex tasks through long-horizon interactions.
Alnieraction
Alnteraction
SeltagheeKL.0
Figure 5: Scaling rest-time interaction turns.
5

---

Published as a conference paper at ICLR 2026
Baby/Al
TeaiCraft
SelWorld
WedhArana
SearchiAA
A M
Reaco
Renook
210g
Stap
Figure 6: Training rewards in different environments leveraging AgentGym-RI framework with the Scalinginter-R8 method.
To address this limitation, we further explore leveraging RI. to enhance agents' capabilities in long- horizon scenarios. Specifically, we vary the maximum number of interaction turns during RI. roll- outs and analyze the resulting training dynamics (Figure 4. We find that larger interaction horizons (e.g., 10 turns) enable deeper exploration but introduce training instability, often leading to training collapse, with the model exhibiting redundant interactions and unnecessary repetition. In contrast, shorter horizons provide stability but cap performance due to limited interaction turns. Therefore, our core motivation is how to scale interactions at train-time in stable and effective way.
Method. To this end, we introduce ScalingInter-RL. stably optimize LLM agents for chal- lenging tasks that require long- morgzooizon interactions. The central idea of ScalingInter-B RL lies in progressive horizon-scaling strategy that gradually increases the number of interaction turns during RL training, as illustrated in Figure 9 (Appendix B).
Specifically, the objective to maximize the expected final reward under a constrained interaction budget:
J(0) ET (T)],
where each trajectory T (al 01, ................................ - .f. ok) is sampled from the current policy Te, with K representing the total number of interaction turns, T representing the reasoning path. To prevent the training collapse observed in the previously mentioned long-turn setting, begin training with a short interaction horizon. By initially limiting the horizon, the agent focuses on exploitation, mastering fundamental task-solving skills through simpler tasks. This lays solid foundation for stable training as the horizon gradually extends in later stages.
As training progresses, introduce a monotonic schedule (h1 < h2 ... Nn), where h defines the maximum number of interaction turns allowed during phase t:
(T ha), subject to K, S ht-
The horizon he is updated every A training steps according curriculum schedule:
h4-1 = h4 + 6n,
where 5n is an adaptive increment. As the horizon expands, the agent is encouraged to explore the environment more deeply, thereby enhances the ability to efficiently acquire and leverage in- formation through more interactions. This staged scaling approach allows the agent to make more intelligent decisions, enabling deeper exploration of the environment, and ultimately results in more effective task completion while ensuring training stability.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Scenarios, Environments and Tasks. As mentioned before, we include five scenarios in
AgentGym-RL. Specifically, we include WebArena (Zhou et al. (022a)) for web navigation, a RAG- based environment (Jin et al.] 2025b Joshi et al., 2017 Ho et al. 2020 Kwiatkowski et al., 2019 Mallen et al., 2022, Trivedi et al. 2022 Yang et al. 2018 Press al. (2023) for deep search, TextCraft (Prasad et al. 2024) for digital games, BabyAl (Chevalier-Botsvent
Boisvert et al. 2019) for em- bodied tasks, and SoiWorld (Wang et al. 2022) for scientific tasks.

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation results on Deep Search benchmark. For each group, the best result is in bold. and the second- best is underlined. MaarchRR-ittvO... baseline uses Cearch-R1-v0.. models (Jin et al., 2025a). See Appendix for results of tasks on other scenarios.
Model
NQ
TriviQQA
PopQA
HotpotQA
2Wiki
Musique
Bamboogle
Overall
Proprictary Models
GPT-40
Hurst
30.0
10.0
Owen-Kex Tang o 2024
24.0
52.0
26.0
24.0
16.0
17.0
36.0
29.5
Gemini-2..
30.0
16.0
8.0
34.0
23.5
OpenAE 04 ot-ming OPRAU 2025
22.0
68.0
50.0
38.0
28.0
62.0
42.5
OpenAI
03
Open A 2025
28.0
70,0
56.0
46.0
64.0
29.0
74,0
49.5
Gemf
2125
22.0
62.0
38.0
28.0
48.0
19.0
36.5
Open Open-seurced Models 1008
Oven3-235D-A220
Yane al 2
28,0
$4,0
30,0
32,0
22.0
14.0
32.0
28.3
DeepSeex-V2-0S22 DeepSeck 2024
28.0
11.0
ReepSeex-R1- RD5-
DeepSeek -l 2125
32.0
68.0
42.0
44.0
50.0
21.0
44.0
40.3
Open-sourced Modelly 10GB
089n2.3-UB-Instrrae
8.0
0sen2_5-75-Tnstrice
Yall aF 2754
18.0
$4.0
20.0
18.0
6.0
4.0
26.0
18.8
06an2.3-28BInatz22
Yong
22.0
52.0
24.0
28.0
24.0
12.0
38.0
26.5
Owen3-43
20250
18.0
26.0
5.0
22.8
@wen3-82
20238
26.0
44.0
26.0
22.0
32.0
10.0
32.0
25.3
Owen3-325
20250
24.0
54.0
22.0
36.0
28.0
11.0
20.0
25.8
1.1am=3.11- 10551188--mms Puct
Dubey er
16.0
c1smm-3..1-70-- 11.11-000Innttuuu Instruct
Dirxy LE R
2024
20.0
44.0
22.0
22.0
18.0
9.0
32.0
22.0
SearchR1-it-33R-2
20.0
30.0
32.0
14.0
1it---1--tt---000 3oreo
c. 20236
24.0
52.0
30.0
22.0
34.0
6.0
26.0
25.0
Our RL. Models
Agent.6ym-RT.-36
30.0
$0.0
30.0
30.0
46.0
4.0
12.0
25.8
Aggen.Gym-Rul Ry
44,0
64.0
32.0
40.0
15,0
26.0
34.0
Scalinglater-1B
$2.0
70.0
46.0
42.0
44.0
14.0
24.0
38.3
Baselines and backbone models. We leverage Qwen-2.5-3B and Qwen-2.5-7B (Yang al. 2024)
as our backbone models. Additionally, we introduce closed- source commercial models and strong open-source models as baselines, shown in Table T Both training and evaluation are
ducted using ReAct (Yao et al. 2023) paradigm.
Detailed settings of each environment. Different environments have distinct observation spaces, action spaces, and reward structures. Due to space limitations, we provide detailed descriptions of the tools, APIs, and experimental settings for each environment in Appendix
5.2
MAIN RESULTS
The main results are shown in Figure and the detailed results Deep Search shown in Table I See Appendix D for detailed results of tasks on other scenarios.
Reinforcement learning generally improves agentic intelligence of open-source LLMs, bring- ing them on par with proprietary models. As shown Figure [ our RL model outperforms other open-source models by a large margin. It also leads in average success rate over closed-soures models like GPT-4o and Gemini-2.5-Pra across five different scenarios. This demonstrates the fectiveness of our framework in enabling models to learn and make decisions in complex tasks, narrowing the gap between open open-source and proprictary models
ScalingInter-RI. significantly and consistently boosts performance. We set phase transition points based the total optimization steps in the RL process, rather than performing extensive hyperparameter tuning, as it has already proven effective. Scalinglnter-RU consistently outperforms the baseline across various environments. For example, it improves WebArena performance by over 15 points, bringing it closer to closed-source source commercial models. It also boosts TextCraft scores by nealy 50 points, achieving tfut--of-the-r results. These improvements show that our method effectively balances exploration and exploitation, enabling the model to interact more intelligently with the environment, adapt, and complete tasks.
Post-training and test-time compute show higher scaling potential than model size. As shown in Figure [ (right), ScalingInter-BB with 7B parameters achieves average success rate of 61.8%, significantly surpassing larger models like Llama3.1 (46.9%) and Qwen2.5-72B (42.8%). This shows that simply increasing model size provides limited performance gains, while increasing post-

---

Published as a conference paper at ICLR 2026
training and anference-time compute offers better results, providing insights for future scaling strategies.
The environment plays a key role in the efficiency of reinforcement learning. The effectiveness of Agen(Gym-RL depends on the environment and the type of feedback provided. In simulated worlds with clear rules and direct cause-and-effect relationships, such as TextCraft, Baby and SciWorld, RL achieves the greatest performance improvements. For instance. SciWorld's score jumps from 1.50% to 50.50%, a remarkable increase of almost 50 points. On the other hand, in more open-ended environments like WehArena and Deep Search, the performance gains from RI. more limited, due to the challenges of task complexity and potential noisy feedback. This provides valuable insights for the design of environmental feedback and reward structure in the future.
DISCUSSION
6.1 TEST-TIME SCALING FOR AGENTS
Scaling interaction turns. As shown in Figure 5 all models improve with more turns, showing that long-horizon interaction and sufficient exploration contribute to enhanced agentic performance. Moreover, the ScalingInter- CealinglnterRR--traineee agent consistently surpasses the baseline by substantial margin, further highlighting its ability in long-horizon scenaios and the effectiveness of our method.
Scaling parallel sampling. As shown in Fig- 7 increasing the number of samples yields a marked improvement in @ass@K performance, signaling the downstream optimization potential of each model. The ScalingInter-Rl trained model surpasses the baselines even with a small sampling budget, and as sampling increases, it continues to outperform the baseline in a stable and significant manner. Notably, in SciWorld, the ScalingInter- RL model's Pass even surpasses all baselines' Pass@64 64, showcasing the compute--fficienny and superior optimization capability of our method.
Deep
Number of Sample
Number of Sample K
Figure 7: Pass @ K performance.
6.2 PERFORMANCE OF DIFFERENT RI ALGORITHMS
Table 2: Evaluation results of different RL algorithms.
We compare two mainstream RL algorithms for LLM post-training, i.e., GRPO and REINFORCE+++ As shown in Table 2 GRPO consistently and substantially outper- forms REINFORCE+ on the TexICraft, BabyAl, and Deep Search benchmarks. Notably, 3B-GRPO model even surpasses the 7B- REINFORC++ model, highlight- ing an algorithmic advantage beyond model scale.
RL. Algorithms
TextCraft
BabyAI
SearchQA
Owen2, 5D30-
ERPO
75.00
93,33
25.75
REINZORCE-+
28.00
70.00
13.25
Ow0 70--777777- To---natrre
GRPC
89.00
92.22
34,00
RETNPOROR**
73.00
84.44
24.00
The performance difference can be attributed to the way each algorithm calculates the advantage. GRPO calculates a baseline as the average value of multiple trajectories for query, and then perform normalization, which helps reduce the impact of outliers from individual trajectories, leading to more robust optimization. In contrast, REINFORCE++ normalizes within batch, which can lead to high- gradients.
6.3 ABLATION STUDY FOR SCALINGINTER-RL Table 3: Ablation study of ScalingInter-RL.
We conduct detailed ablation studies on Deep Search task with respect to the initial number of interaction rounds, the stage transition frequency, and the inter- action interval for ScalingInter-RL. The results are shown in Table 3 We find that ScalingInter- RL is not sensitive these hyperparameters.
Interact Turn List
Stage Transition Frequency
Performance
100
38.3
[5, 101
[5, 3.1
125
38.5
[3, 8, 13]
100
36,8
10,10, 12]
100
37.6
15]
100
39.1
71
8

---

Published as a conference paper at ICLR 2026
6.4 EFFICIENCY ANALYSIS OF SCALINGINTER RL
We analyze the efficiency of ScalingInter-RL. by examining the training reward, the cumulative number of interaction rounds during training, and the total training time. As shown in Figure X can observe that, thanks to the stage-based design, Scalinglnter- Ritter-RLii is able to achieve relatively high rewards with comparatively high efficiency and reduced time and resource consumption.
Cumulated Rounds
3000
Cumulated Time (s)
15000
0.6-
Reward
2000
0.5
10000-
0.4
1000
3000
0.3
100
200 Steps
300
100
200
300
1000
200
300
Steps
Steps
AgunnGym XL Wed 10)
AgensGym XL (Fixed))
Scalinglitter RL
Figure 8: Analysis of computational resources and efficiency.
6.5 EXPERIMENTS OF APPLYING SCALINGINTER-RI TO MORE ALGORITIMMS
We apply ScalingInter- RL to other algorithms and present the results in Table We can observe performance differences across algorithms, which is consistent with previous work Hu (2025); (Shao al. (2024). In addition, is clear that ScalingInter-RI brings performance improvements across different algorithms.
Table 4: Applying Scalinglnter-RI. to algorithms.
RL Algorithm
Method
TextCraft
BabyAI
SciWorld
Base Model
42.00
66.67
1.50
AgentGyymRR1-78
68.00
86.66
10.83
PPO
Scalinglnter-78
71.00
90.00
25.69
AgentGym-RLe RL-7B
73.00
84.44
13.63
REINFORCE++
ScalingInter-77
77.00
87.77
26.14
6.6 CASE STUDY
We provide series of case studies different tasks that highlight both the shortcomings of the base agent and the improvements achieved by our reinforcement learning agents in Appendix G
RL agent vs. Base agent. RL-trained agents consistently outperform base agents by complet- ing tasks more strategically. They can avoid unproductive loops and adapt to challenges. In the WebArena environment, Figures 15 and [6 show how RL optimization enhances web navigation. While base agents repeatedly click on ineffective interface elements without making progress, RL- trained agents recover from mistakes, escape deadlocks, and ultimately complete the task. In the BabyAl environment, Figures 12 and B illustrate improvement in navigation capabilities. Unlike the base agent which exhibits repetitive movements, the RL agent demonstrates strategic backtrack- ing, superior spatial reasoning, eventually accomplishes the task.
Exception Cases. To provide a balanced perspective, we also include two representative failure cases-in scientific reasoning and in efficient web navigation underscore areas for improve- ment. In the SciWorld environment, Figure 17 shows that while the RL agent can reach task-relevant states, still struggles with execution. Two main issues are identified: substituting factual recall for necessary experimental procedures during debugging, and prematurely ending exploration by focusing solely on one animal. These failures demonstrate the agent's 's insufficient procedural un- derstanding required for scientific analysis. In the WehArena environment, Figure 18 illustrate that

---

Published as a conference paper at ICLR 2026
though the RL agent successfully reaching the correct target websites, it performs redundant actions such as unnecessary clicking, hovering and scrolling. These behaviors hinder effective information extraction, revealing gap between state-reaching ability and precise, efficient action selection.
7 RELATED WORK
Developing agents with large language models.
With the advancement of large language models
(Achiam al.) 2023 Anthropic
2024 leam et al.
2023), researchers have explored building agents
for multi-turn decision-making
(Xi ct al. 2025a
Yao 2024; Shen et al. 2026 Liu et al., 2026
Yang ct al. 2026: Team et al.
2025a Tang ct al.
2025:
Ye et al.
2025b b). Current approaches
mainly use prompting to invoke tools (Qin et al.
2025
Ye et al.
2025a c). often enhanced with
self-reflection (Shinn et al.] 2023 Xi et al.] [024c|
Xie
et al.] [025|]
Renze & Guven 2024 Yuan
et al.
2025). long-horizon planning (Liu et al. 2023
Nayak et al.
2024
Prasad et al. 2024 Sun
al.
2023), and self-correction (Kamoi al.
2024
Kumar et 2025
Multi-LLM workflows
assign specialized roles to different models (Liang et al. 2024| Wu et al. 2023 Talebirad & Nadiri
2023 Hong et al., 2024] Guo al., 2025), but usually depend on proprictary models (e.g., OpenAl
03) and lack intrinsic agentic training. Another direction collects expert trajectories for imitation
learning (Zhang et al. 2024 Zeng al., 2024, Chen et al., 2023 2024b), which grants skills like API use and planning but is costly, hard to scale, and limits self-improvement.
Reinforcement learning for large language models. Reinforcement learning is a crucial post- training technique for LLMs, supporting preference alignment (Ouyang et al. 2022] Zheng et al.
2023 Xia et al. [222]] [Chen et al. [024a]] Ji et al., (023)) improved reasoning (Jacch et al. 2024
Trung et al. 2024,]X Xi et al. 2024a, DeepSeek-AI et al. 22025; Owen Team 2025 He et al. 2025
Xi et a 2024b; (025c), and new scaling strategies (DeepSeek-Al Al et al. 2025). ). Algorithms such as
al. 2017), GRPO (Shao et al., 2024), RENFORCE++ (Hu) 2025), and RLO0 (Chen et al.) 2025) have been widely adopted. Yet most efforts like DeepSeek- RI focus on single- turn tasks, limiting multi turn interaction with complex environments. Recent advances extend RL to self-reflection (Xie et al.) 2025), tool use (Ye et al.) 2024), and long-horizon interaction (Zhoul let al., 2024b) [hen et al. 2025; Wang et al.) 2025 Qi et al. 2025) Un et al.) 2025b; Cao et al. 2025 but face challenges in scalability, task diversity, and optimization stability. To address this, we present unified RL framework for multi-turn decision-making across diverse environments, and introduce ScalingInter-RI an interaction-scaling method that stabilizes training and enhances agent performance.
PPO (Schulman
Scaling inference compute for language models. Increasing inference compute both at test time
and during RL rollouts yields strong scaling effects (Jaech et al.; 2024 DeepSeek-AA et al.) 2025
XAI 2025; Snell et al. 20244] Xi et al. (025d).. Techniques like long-chain-ol -fthuutttt reasoning
(Snell al. 2024; Xi et al. 2024c), majority voting (Li et al. 2024 Wang et al. 2023). best-
of-N sampling (Chow et al.
2025; Jinnal et al. 2024). beam search (Xie et al. 2023] Zhu et al.)
2024). and Monte Carlo tree search (Chi et al., 2024 Gan al.] 2025). Zhu et al. (2025) address
inference-scaling for LLM agents but they do not investigate inference scaling in RL. TTI (Shen ct al., 2025) teaches compute allocation via rejection sampling. By contrast, we use on-policy RL. (c.g., GRPO, REINFORCE++) to scale interactions without restricting compute "thinking" or "acting", letting the agent adaptively allocate extra compute to improve exploration, skill acquisition, and performance.
8 CONCLUSION
In this work, we present AgentGym-RI, a unified reinforcement learning framework for training LLM agents in long-horizon, multi-turn decision-making tasks. The framework offers diverse en- vironments and scenarios, integrates mainstream RL algorithms, and provides a high degree of ex- tensibility, making it versatile and powerful resource for the community. Building on this, we introduce ScalingInter-RL. RL,, a staged training approach that progressively scales enent-environment interactions and achieves strong final performance. Extensive experiments demonstrate the effec- tiveness of both the framework and the method. We hope our work offers valuable insights and supports the development of next-generation intelligent agents.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This paper presents AgentGym-RLL unified framework that enable stable reinforcement learning of LLM agents for diverse real-world multi-turn tasks. further proposes ScalingInter-RL, training approach designed for exploration-exploittiii balance and stable RL optimization. We firmly state that this work is intended for ethical and constructive purpose. While no immediate societal harms are identified, proactive measures should ensure responsible deployment to mitigate potential misuse or unintended consequences.
REPRODUCIBILITY STATEMENT
We claim our detailed experiment setting Appendix E addition, we upload anonymized ver- sions of our data and code in Zip file with a Readme file to ensure easy reproduction of all reported results.
ACKNOWLEDGEMENT
The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by the Science and Technology Commission of Shanghai Municipality (No.24511103100) National Natural Science Foundation of China (No.62476061, 62576106, 62376061), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (232R1403500)
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Al Anthropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 1(1):4, 2024.
Shiyi Cao, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, Jiayi Pan, Xingyao Wang, Akshay Malik, Graham Neubig, Kourosh Hakhamaneshi, Richard Liaw, Philipp Moritz, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Skyrl-vO: Train real-world long-horizon agents via reinforcement learning, 2025.
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fire- act: Toward language agent funing. CORR, abs/2310.05915, 2023. doi: 10.48550/ARXIV. 2310.05915. URL htwph:tttt//ttt org/10 0.e..445555 48550/arXiv 2310 05915
Kevin Chen, Marco F. Cusumano-Towner, Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl. Reinforcement learning for long-horizon interactive
LLM agents. CoRR, abs/2502.016000, 2025. doi: 10.44550AARXI..2200000000 URL https:
//doi.org//1 410.11..8555///aaa/
01 610
Lu Chen, Rui Zheng, Binghai Wang, Senjie Jin, Caishuang Huang, Junjie Ye, Zhihao Zhang, Yuhao Zhou, Zhiheng Xi, Tao Gui, Qi Zhang, and Xuanjing Huang. Improving discriminative capability of reward models in RLHF using contrastive learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. EMNLP 2024, Miami, FL, USA, November 12-16, 16, 2024, pp. 15270-15288 Association for Computational Linguistics, 2024a. doi: 10. 10.18653/V1// http:///.222...2...
852.
URL
wro./r/g/// /00 1165188661v//
emn p- 852
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meet- ing, August 11-16, 2024, pp. 9354-9366. Association for Computational Linguistics, 2024b.
11

---

Published as a conference paper at ICLR 2026
doi: 10.663N1/2044FNDDNNS....... URL https://aoi.org//// 2024. Findings-acll
Maxime Chevalier-ooisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform study the sample efficiency of grounded language learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: // //openreview. net/foormm d=rJexCo0eyX
Yizhou Chi, Yizhang Lin, Sirui Hong, Duyi Pan, Yaying Fei, Guanghao Mei, Bangbang Liu, Tianqi Pang, Jacky Kwok, Ceyao Zhang, Bang Liu, and Chenglin Wu. SELA: tree-search enhanced LLM agents for automated machine learning. CoRR, abs/2410. 17238, 2024, doi: 10.48V50/ARXIV. 2410. 17238. URI // /tg
2410 17238
Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Aviral Kumar, Rishabh Agarwal, Sridhar Thiagarajan, Craig Boutilier, and Aleksandra Faust. Inference-aware fine-tuning for best-of-n sampling in large language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. net www.://www.e/eeeueeneeeemm -d- 77g0dd-7777hh
URL htttt::///ppeeeeeeeeeeee
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.063: 2025.
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang. Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li. Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang. Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. I.. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, and Wangding Zeng. Deepseek- v3 technical report. CoRR, abs/2412. 19437, 2024. doi: 0044550/ARRIV.24122.00 19437. URL hittps://dos www.do....gg...... 48550 araii 2412. 19437
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. Li. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. CoRR, abs/2501. 11000.122244 12948, 2025. doi: 10. 48550/ARX/V.2501. 12948.
or0/10.. 48550 arwiv 2501 12948.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony
12

---

Published as a conference paper at ICLR 2026
Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloc Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Ko- revaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, 0oos/2407.217855 2024. doi: 008555/ARXXV.240.2272333 URL https://doo /ttp::::///ii.or/1..... 48550/ / arxiv. wwtp550/.wxxiv........ 21783
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, An- drew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.). Advances Neural Information Processing Systems 35: Annual Conference
Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December URL http: / /ttpppaapp/aaaaeaaaaaaaeeaaaaaaa nips. cc cc/paper.p.p......eeeeoe files/paper/20221 Socc /tt.a//c22cccccc1111133333aeae33 -awpaeeeeeeeeeea-0aaaeaaeaeee Datasets_ance _nd
9. 2022, 2022.
Benchmarks. html
Bingzheng Gan, Yufan Zhao, Tianyi Zhang, Jing Huang, Yusu Li, Shu Xian Teo, Changwang Zhang, and Wei Shi. MASTER: A multi-agent system with LLM specialized MCTS. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Na- tions of the Americas Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, NAACL 2025 Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 May 4, 2025, pp. 9409-9420 Association for Computational Linguistics, 2025. doi: 10.18653//1/2025..0 Tww.:/22..AAAALLLLNN.....O URL /tt.://///t////oo..o/////0/.0111
naacl oog...g......................... 4
Honglin Guo, Kai Lv, Qipeng Guo, Tianyi Liang, Zhiheng Xi, Demin Song, Qi Zhang, Yu Sun, Kai Chen, Xipeng Qiu, and Tao Gui. Critiq: Mining data quality criteria from human preferences. CoRR, abs/2502. 2025, doi: 10.48550/RRXI.2202.22222 10. 10.48550/ar22 101500/aXii1 25022...2552...22 19279
URL/http:://dot /rtg:///////o///////////o/
Jujie He, Jiacai Liu, Chris Yuhao Liu. Rui Yan, Chaojie Wang, Peng Cheng. Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen. Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner technical report. CoRR, abs/2505.22312, 2025. doi: 10445500/ARXI11255223311 2505.22315
URL https : doii 4rg//10.4855/arxx0
Xanh Ho, Anh- Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing A multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Núria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computa- tional Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020. pp. 6609- 6625. International Committee on Computational Linguistics, 2020. doi: 10. 16653//222/2000 COLING-MAIN.580 URL
580
Sirui Hong, Mingehen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi-agent
13

---

Published as a conference paper at ICLR 2026
collaborative framework. In The Twelfth International Conference Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:
Jian Hu. REINFORCE++: A simple and efficient approach for aligning large language models. CoRR, 01//250100226. 2025, doi: 10.48550/ARXI.2550...0 20..222........ httth:tps:///// org/ 10. 00.48550/azz0 105500/axxiv: 2501 03262
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Aaron Jaech. Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 2024.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng. Juntao Dai, Xuehai Pan, Aidan O'Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang. Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao. Al alignment: A com- prehensive survey. CoRR, abs/2310.19852. 2023. doi: 10.48550/RRXI.2310.0909
URL
1ttp://..i...//.....0.. 48550/ wttp://xxivv.... 111..880000211.221.1111012 19852
Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Serean Ö. Arik, and Jiawei Han. An empirical study on reinforcement learning for reasoning-searee interleaved LLM agents. CoRR, abs/2505.
2025a. doi: 0044550//ARIV.25555.555 151177 http:::ts:///cc http:////oo.oog/....30... 2505. 15117
www/o0.g/1000 48550/ar855
Bowen Jin, Hansi Zeng. Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- rl: Training Ilms to reason and leverage search engines with reinforcement learning. CoRR, Abs/2503.09516 2025b. doi: 10.45500ARXV255000055 URL httthh:tt/:/oo http::///ooooog///.1.. org/10.
48550/arXiv 2503.
Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, and Kenshi Abe. Regularized best-of-n sampling to mitigate reward hacking for language model alignment. CoRR, abs/2404.01054, 2024. doi: 10. 48550/ARXIIV2440000055. URL htt.ps::://.i.orrttttt
(08550/arXii10 240.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 1601-1611. Association for Computational Linguistics, 2017. doi: 10. 11.186//111111711177. URL https:
7/dei. 1153//1//111111011
Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can llms Actu- ally correct their own mistakes? A critical survey of self-correction of llms. Trans. Assoc. Comput. Linguistics, 12:1417-1440, 2024. doi: A, 00713. URL
https:
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D. Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M. Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal M. P. Behbahani, and Aleksandra Faust. Training language models to self-correct via reinforcement learning. In The Thirleenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URLhttps: https: https/////ppeeeeie.eeeeom net forum?id C CiWERcAU7w
Tom Kwiatkowski. Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: benchmark for question answering research. Trans. Assoc. Comput. doi: 10.1162/TACL) A) 00276. URL https: doi. org/ 10. 1t..116tttcll _0206_6___02666111
Linguistics, 7:452 466, 2019.
14

---

Published as a conference paper at ICLR 2026
Junyou Li, Qin Zhang. Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. forum?id=
Trans. Mach. Learn. Res., 2024, 2024. URL httpp::/ http::///peeeeiew...coo net/ net
bgzUSZ8aeg
Tian Liang Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang, Rui Wang, Yujuu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi- agent debate. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNIP 2024, Miami, FL, USA, November 12-16, 2024, pp. 11889-17904 Association for Computational Linguistics. 2024. doi: N0.18653/V1/20000 WTW.://22..EELLMAAAIN...
URL https:: htw.:oiiorrggg// 18653/
v1/2024. emn 1p ma main. 992
Dongrui Liu, Qihan Ren, Chen Qian, Shuai Shao, Yuejin Xie, Yu Li, Zhonghao Yang, Haoyu Luo, Peng Wang, Qingyu Liu, Binxin Hu, Ling Tang, Jilin Mei, Dadi Guo, Leitao Yuan, Junyao Yang, Guanxu Chen, Qihao Lin, Yi Yu, Bo Zhang, Jiaxuan Guo, Jie Zhang, Wenqi Shao, Huiqi Deng. Zhiheng Xi, Wenjie Wang. Wenxuan Wang, Wen Shen, Zhikai Chen, Haoyu Xie, Jialing Tao, Juntao Dai, Jiaming Ji, Zhongjie Ba, Linfeng Zhang, Yong Liu, Quanshi Zhang, Lei Zhu, Zhihua Wei, Hui Xue, Chaochao Lu, Jing Shao, and Xia Hu. Agentdog: A diagmostic guardrail framework for agent safety and security, 2026. URL https: arxiv wwtp://wwrgg/bbb///eeeea 18491
Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang. Caim- ing Xiong, and Silvio Savarese. BOLAA: benchmarking and orchestrating llm-augmented au- tonomous agents. CoRR, abs/2308.059000 2023, doi: 10.48555/ARXXIV230088 05960. URL. https://dof www.rrrg..............ooo......o wtt.5///axii...m 2308 05960
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannanch Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. CORR, abs/2212. 10511, 2022. doi: 10.48500/ARXIV2222.0011
URL https:://d/o wttg/o.....00000
48550/ arxiv. wttpaiiiiii..iii.............
Moonshot AI. Kimi k2: Open agentic intelligence. pttps://monstotaai..itts www.:oosshhaii.gttu...bm
Kimi kkm] 2025.
Accessed:
2025-07-15.
Siddharth Nayak, Adelmo Morrison Orozco, Marina Ten Have, Jackson Zhang, Vittal Thiru- malai, Darren Chen, Aditya Kapoor, Eric Robinson, Karthik Gopalakrishnan, James Har- rison, Anuj Mahajan, Brian Ichter, and Hamsa Balakrishnan. Long-horizon planning for multi-agent robots in partially observable environments. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Systems 2024, NeurIPS 2024. Vancouver, BC, Canada, December 10- 15, 2024, 2024. http:ttppp/pppes.tmttt mips. www.::ii....cccppeopce faaeipeerriieeeeeeeeeee fwwes/paper//// /iles//ppppe////////asss ors
774e5888854442223433333335353333 htt.::/atttttttttooooooeeeeeeeaa html
OpenAI. GPT-44 report. wbw.oRR.ab//2330.87744.22222 doi: 10.48550/ARX122230.00777
OpenAl.
Openai o3 and
o4-mini
system
card.
https://odn
openai http:///w..22221111000000000-000
o3- 33-andcoof-mmill-syytterrrrrr-p 4789 8005 6775-377333372211
2025.
URL
https://odr.
ogenaa..comm/pff/22221877----000 o3-and-on a4--n-o---aa -io---min-sssteemmm sawaaii-sstttemcrrd... card.pdf
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feed- back. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- fion Processing Systems 2022, NeurlPS 2022, New Orleans, IA, USA, November 28- December
15

---

Published as a conference paper at ICLR 2026
9. 2022, 2022. URL htt.:////aaaaaaaaa........aaaaaa
haash/
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. CoRR, abs/2304.033777, 2023. doi: 10448500AAXX..23300037777 URL https:
7/doi. org/10 4 48550/855X0/. 2304, 03277
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mo- hit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with lan- guage models. In Kevin Duh, Helena Gómez-Adomo, and Steven Bethard (eds.), Find- ings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 4226-4252. Association for Computational Linguistics, 2024. doi: 10. 10.16663/1//22..77DISS----AA 2024. wwn.innggngnnal...2222.
URL https://doi. http://ooo..org////// org.10..g11..o 1ttp///....115///11 18653/v1/
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Mea- suring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 5687-5711. Association for Computational Linguis- tics, 2023. doi: 10. 11//663//1/22223 WWW.FFIIIIIINGNNNLLLLI 18653 vl / 2023. findings emn 1p.nl.... 378
URL httpt::///..i..ooo
Zehan Qi, Xiao Liu, lat Long long, Hanyu Lai, Xueqiao Sun, Jiadai Sun, Xinyue Yang, Yu Yang, Shuntian Yao, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training LLM web agents via self- evolving online curriculum reinforcement learning. In The Thirleenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
URL https: /ttp:r////ppeeeviie.ee net/ wtwp:eewwwwwweeeeeennneee. forum? OVKEAA JEqv
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Xuanhe Zhou, Yufei Huang, Chaojun Xiao, Chi Han, Yi R. Fung. Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang. Jing Yi, Yuzhang Zhu, Zhemning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang. Tongshuang Wu, Heng Ji, Guoliang Li, Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models. ACM Comput. Surv. 57(4):111:1-01:444 2025. doi: 10.1145/3704435.
URL
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https: http///wwl......immm..mmmmm oo/blolbooooooooo /oollo///// 32bb
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10- httpee/eeppppee///aaaaaeee
16, 2023, 2023. URL htwp: wwt.perrs.iippp..mmmm.... 2r55::/w......aa..ee.e.eeaaaeeee
html
Matthew Renze and Erhan Guven. Self-reflection in LLM agents: Effectt on problem-solving per- formance. CoRR, abs/2405.06682, 2024. doi: 10.85550/ARXX1.2405..66666 URL https:
//doi oww. (ta.g//1...888111 48550 2405. 2405.06682 06682
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 703406377, 2017. URL http::/arxiv. org.
1707. 06347
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300. 2024. doi: 10-8550/ARXXI.22222.3300. URL https://dos httpp::://ooo.rg/11... wttp///..450//rrr..../ http://awwv.......... 24e...2222220 2330000300000000
16

---

Published as a conference paper at ICLR 2026
Junhong Shen, Hao Bai, Lunjun Zhang. Yifei Zhou, Amrith Setlur, Shengbang Tong, Diego Caples, Nan Jiang, Tong Zhang, Ameet Talwalkar, and Aviral Kumar. Thinking vs. doing: Agents that reason by scaling test-time interaction. CoRR, abs/2506.07976, 2025. doi: 10.48550/ARXIV. 2506.07976. htttss/t::///dt wtt.:ooogg1...85000000 wtt.:/.www...6.7799....
Yujiong Shen, Yajie Yang, Zhiheng Xi, Binze Hu, Huayu Sha, Jiazheng Zhang, Qiyuan Peng, Junlin Shang, Jixuan Huang, Yutao Fan. Jingqi Tong. Shihan Dou, Ming Zhang. Lei Bai, Zhenfei Yin, Tao Gui, Xingjun Ma, Qi Zhang, Xuanjing Huang, and Yu-Gang Jiang. Sci- agentgym: Benchmarking multi-step scientific tool-use in Ilm agents, 2026. URL https: watxiv.org/abs//250a.. 12984
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: language agents with verbal reinforcement learning. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Informa- tion Processing Systems 2023. NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: 1b44b878bb 048.8777877222554488886822000000
in Neural Information Processing Systems 36: Annual Conference
nips. cw/paperpppppppeepe files/ www.s//appeeeee/////e/
http//ppppe//2////peeeeee htmm html
Mohit Shridhar, Xingdi Yuan, Mlrc-Alex Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https::/opperevieew net forum? ?id= 0IOX0YcCdTи
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute opti- mally can be more effective than scaling model parameters. CoRR, abs/2408.03314, 2024. doi: 10. 48550/ARXIV.2408.0 03314. htttthttps://ddde
org/10. 48550 wwt.50//rxivv.. 240.xx....40..... 03314
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai's ability to replicate AI research. CoRR, doi: 10. 10-48500ARXIW2550000444. URL https::/doot org/1c.
abs/2504.01848, 2025.
48550/arXiv 250. 01848
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023, URL http::/ppppps...pppppp nips cttpspercpppppppppppp files htteee//pppe//2///asssss http//ppprrfflsseppppepepeeee html
Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour. Policy gra- dient methods for reinforcement learning with function approximation. In Sara A. Solla, Todd K. Leen, and Klaus-Robert Müller (eds.), Advances in Neural Information Process- ing Systems 12, JNIPS Conference, Denver, Colorado, USA, November 29 December 4, 1999], pp. 1057-1063. The MIT Press, 1999. URL http:: nips wwwp.pcpppppp.......ppppppppppp. 1113-policy-greee aer--poooo-grrcciee--eeeeeeeee
- for www.forcemnntteeeeeeeeeeeeeeeeee -earnnngt--aagnngggggnn-nn with hug.::/ooooattt--ffuui--aa-apppo -uwciiot-ppppooxiiiaat
Yashar Talebirad and Amirhossein Nadiri, Multi-agent collaboration: Harnessing the power of intelligent LLM agents. CoRR, abs/2306.03314 2023. doi: 10.4850/ARXIV2230600022 URL
https::/doi.orgg/ http://...455///aa/
03314
Liujian Tang, Shaokang Dong. Yijia Huang, Minqi Xiang. Hongtao Ruan, Bin Wang. Shuo Li, Zhi- heng Xi, Zhihui Cao, Hailiang Pang, Heng Kong, He Yang, Mingxu Chai, Zhilin Gao, Xingyu Liu, Yingnan Fu, Jiaming Liu, Xuanjing Huang, Yu-Gang Jiang, Tao Gui, Qi Zhang, Kang Wang, Yunke Zhang, and Yuran Wang. Magicgui: A foundational mobile gui agent with scalable data pipeline and reinforcement fine-tum tne-tuning, 2025. URL TRLhhtppa:// arxiv org/ abs/2508. 03700
17

---

Published as a conference paper at ICLR 2026
AGI Team, Yuxuan Cai, Lu Chen, Qiaoling Chen, Yuyang Ding, Liwen Fan, Wenjie Fu, Yufei Gao, Honglin Guo, Pinxue Guo, Zhenhua Han, Zhengfu He, Hanglei Hu, Kai Hu, Shengjia Hua, Tianyu Huai, Baodai Huang, Li Ji. Zhen Jiang, Zhikai Lei, Bufan Li, Jiahang Lin, Lizhi Lin, Jinxiu Liu, Shichun Liu, Ziming Liu, Yuchen Ni, Pengfang Qian, Yujiong Shen, Qingyun Shi, Wentao Shu, Peng Sun, Yiran Suo, Tian Tang. Boyu Tian, Guoteng Wang, Junzhe Wang, Peixin Wang, Zhiheng Xi, Hang Yan, Jie Yang, Zhixiong Yang, Tianchu Yao, Guangze Ye, Qianxi Yu, Shuo Zhang, Xinyue Zhang, Yiqi Zhang, Jiarong Zhao, Miao Zheng, Rui Zheng, Enyu Zhou, Jiazheng Zhou, Maosen Zhou, Yuhao Zhou, Tao Gui, Yining Zheng, Xinchi Chen, Jie Zhou, Siyuan Feng. Qin Chen, Liang He. Qi Zhang, Xuanjing Huang, and Xipeng Qiu. Nex-nl: Agentic models trained via a unified ecosystem for large-scale environment construction, 2025a. URL https::/arxiig orge bargg//bbsa 221.2.112.44499
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang. Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang. Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong. Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang. Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang. Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang. Zhiqi Huang. Zihao Huang, Ziyao Xu, and Zonghan Yang- Kimi k1.5: Scaling reinforcement learning with llms. CoRR, abs/2501. 12599, 2025b. doi: 10. 3S550/ARXIV.250.12225
htthhtts::// /ttttp::///i....oo wtwp:/o........... 48550/arXiv. arxiv. 2501
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multi- hop questions via single-hop question composition. Trans. Assoc. Comput. Linguistics, 10:539- 554, 2022. doi: 10. 10.1162/TACL) A|000455. URL httppsss:: /ooi wtw.oii.orrro....0 11120..11111ccc 00475
Luong Quoe Trung Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Rea- soning with reinforced fine-tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 7601-7614. As- sociation for Computational Linguistics, 2024. doi: acl- ong 410
URL
httpss://doi..o wrg. /00 1883386333/2 v
Ruoyao Wang, Peter A. Jansen, Mare- Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than 5th grader? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, EMNLPP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 11279-11298 Association for Computational Linguistics, 2022. doi: URL httpp::///dooo org 11. 1ttgg//1111111311/1/00000
emnlp mamn.an..... 775
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representationss ICLR 2023, Kigali. Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL Forum?iq- 111--11LIIIMMMM
Zihan Wang, Kangrui Wang, Qineng Wang. Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. RAGEN: understanding self-evolution in LLM agents via multi-turn reinforcement learning. CORR, abs//254..20773, 2025. doi: 10.48550/ BRXIV.2504.20073 U https: //do:////ooo.o .org / 011110.4850/888 225.://0.....22....00.. 20073
18

---

Published as a conference paper at ICLR 2026
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: simple yet challenging benchmark for browsing agents. CoRR, abs/2504. 2025. doi: 10.48550/ ARXIV.2504.1255 12516. URIhttps:: httpa:://ddoo.oo www.:/o.rg/1100 48550/arxiv 2504. 12516
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. CoRR, abs/2308 0112223080811155 08155, 2023. doi: 10.48350ARXXIV23000. 08155. StLh::tts://ddt http:///oo...gg//// 48550/
2305.08155
xAI. Grok 4.
2025.
Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang. Peng Sun, Tao Gui, Qi Zhang. and Xuanjing Huang. Training large language models for reasoning through reverse curriculum reinforcement learning. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024a. URL /ttttpss:/openreeeeee http//pperrrveeeeee ew net/ forum witiiiiidittttt f
fmRtk
Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang, Peng Sun, Tao Gui, Qi Zhang, and Xuanjing Huang. Training large language models for reasoning through reverse curriculum reinforcement learning, 2024b. URL https: //tpii////xvv..o/// org/ abpp//bs/ 2402. 05808
Zhiheng Xi, Dingwen Yang, Jixuan Huang. Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Dou, Wenyu Zhan, Xiao Wang. Rui Zheng. Tao Ji, Xiaowei Shi. Yitao Zhai. Rongx- iang Weng, Jingang Wang, Xunliang Cai, Tao Gui, Zuxuan Wu, Qi Zhang, Xipeng Qiu, Xuanjing Huang, and Yu-Gang Jiang. Enhancing L.LM reasoning via critique models with test-time and training-time time supervision. CoRR, abs/2411.16599 2024c. doi: 10.4850/ARXI22411.6655
ww1.a.iww....................... 16147....7799 16579
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong. Ming Zhang, Jun- zhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang. Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, Qi Zhang, and Tao Gui. The rise and potential of large language model based agents: survey, Sei. China Inf. Sci., 68(2), 2025a. doi: 10. 10.1007/S11432-024-4222 111432-024-4222
URL 0ttp://000..000111011......
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang. Chenyang Liao, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Agent- Gym: Evaluating and training large language model-based agents across diverse environments. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- une 1: Long Papers), pp. 27914-27961, Vienna, Austria, July 2025b. Association for Compu- tational Linguistics. ISBN 979- 979-8-89176-251-0 doi: 10.18653/v 10/65///1/2222.... 1tt.://.22-2aaaai acl- 13557
URL
Zhiheng Xi, Jixuan Huang. Xin Guo, Boyang Hong, Dingwen Yang, Xiaoran Fan, Shuo Li, Zehui Chen, Junjie Ye. Siyu Yuan, Zhengyin Du Xuesong Yao, Yufei Xu, Jiecao Chen, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. Critique-rl: Training language models for critiquing through two-stage reinforcement learning, 2025c. URL htw.hhttss:///t
abs http:/g/a////555/ 24320
Zhiheng Xi, Chenyang Liao, Guanyu Li, Yajie Yang, Wenxiang Chen, Zhihao Zhang, Binghai Wang, Senjie Jin, Yuhao Zhou, Jian Guan, Wei Wu, Tao Ji, Tao Gui, Qi Zhang, and Xuanjing Huang. Agentprm: Process reward models for I1m agents via step-wise promise and progress, 2025d. URL U httpt:://////t. pttttp://arxiwwe org/ abs./5551111 08325
19

---

Published as a conference paper at ICLR 2026
Han Xia, Songyang Gao, Qiming Ge, Zhiheng Xi, Qi Zhang, and Xuanjing Huang. Inverse- qt: Token level reinforcement learning for aligning large language models without prefer- ence data. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, Novem- ber 12-16, 2024, pp. 8178- 8188. Association for Computational Linguistics, 2024. doi: 10. WTW.///222.FFNINSN-EEEEEN 2024. Findings emnlp. 478
URL https: http:://doooorg//////// 18663//v...66/11
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Yen Kan, Junxian He, and Michael Qizhe Xie. Self-evaluation guided beam search for reasoning. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference
Neural Infor- mation Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http://papers.nigg http:///aapppppppppp...////eeeea http://iileeeaape///22222.22 81fde95c4dc 888888888662226222 0b Ab Cow.ctttoooofffeeeeeoc html
Zhihui Xie, Jie Chen, Liyu Chen, Weichao Mao, Jingjing Xu, and Lingpeng Kong. Teaching lan- guage models to critique via reinforcement learning. CoRR, abs/2502.03499 2025. doi: 10. 48550/ARX1V25000034499
URLhts:s:://ddo
org/10 48550 arwiv. 2ww.r55522300.0
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu. Mei Li, Mingfeng Xue. Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, 155/2412.151155 2024. doi: 10.44550/ARXIV.24122.0 15155 URL https: /do.. www..orrggl... 485.0/45550/50 2412. 15115
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CORR, abs/2505 003888 2025a. doi: 10.44550/ARXIV.2500. 009388. U
pontpps://oo http:://doooooggg/// www. 000g//11.855500 48550/ xrx 2505. 09388
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang, Jingren Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang. Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang. Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, abs/2505 09388, 2025b. doi: 10.48550/ARXIV25505 .93388. URL httts:://:/d org/l0...............oo. 48550/ arxiv 2505. 09388
Jie Yang, Honglin Guo, Li Ji, Jiazheng Zhou, Rui Zheng, Zhikai Lei, Shuo Zhang, Zhiheng Xi, Shichun Liu, Yuxin Wang, Bo Wang, Yining Zheng, Tao Gui, and Xipeng Qiu. Abc- bench: Benchmarking agentic backend coding in real-world development. 2026. URL hctps:
7/arxiv. org/ wtx.g//bb/26000 11077
Zhilin Yang, Peng Qi Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse. explainable multi-hop question answering. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun ichi Tsuji (eds.), Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brus- sels, Belgium, October 31- November 4, 2018, pp. 2369-2380 Association for Computational
20

---

Published as a conference paper at ICLR 2026
Linguistics, 2018. doi: 10.18653/V1/D1.. 1259. URL UBLAALpp://00b..02//110.01 d18-1259
Shunyu Yao. Language Agents: From Next-Token Prediction to Digital Automation. PhD thesis, Princeton University, 2024.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scal- able real-world web interaction with grounded language agents. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL
www.niiss.ccppppppm facerer_flllspppppeee html
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023. htth:ttp:: httpts::///peeeeieeeeeee
net forum? -
Junjie Ye, Yilong Wu, Sixian Li, Yuming Yang, Tao Gui, Qi Zhang, Xuanjing Huang, Peng Wang, Zhongchao Shi, Jianping Fan, and Zhengyin Du Tl-training: A feature- framework for training large language models in tool use. CoRR, abs/2412. 15495, 2024, doi: 10.48550/ARXIV. 2412.15495. URL http:: /ooi. org/10......ooo...oooo.r1o...1 48550/ arxiv. 2412. 15495
Junjie Ye. Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang. Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, and Jiecao Chen. Toolhop: A query- driven benchmark for evaluating large language models in multi-hop tool use. CoRR, 0as/2501.025066 2025a. doi: 0044550/ARXV.2501.000 18550/arXi:0 2001. 02506
URL httpth:tt////t /ttp:://ooooogg1.//
Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang. Sining Zhu, Zhiheng Xi. Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, and Jiecao Chen. Toolhop: A query-drivent benchmark for evaluating large language models in multi-hop tool use, 2025b. URL
https://arxi::
org/ abs///bs///// 2501. 02506
Junjie Ye, Changhao Jiang, Zhengyin Du, Yufei Xu, Xuesong Yao, Zhiheng Xi, Xiaoran Fan, Qi Zhang, Tao Gui, Xuanjing Huang, and Jiecao Chen. Feedback-driven tool-use improve- ments in large language models via automated build environments, 2025c. URL hetps:
08791
Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. Agent-r: Training language model agents to reflect via iterative self-training, 2025. URL httphttts://aaii....
abs/2501 11425
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agent- tuning: Enabling generalized agent abilities for llms. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 3053-3077. Association for Computational Linguisties, 2024. doi: 10.18653/V1120000 FFO5:///I/222FFNNNNGGS--A //doi.org/ 10.
URL https:
181
Jianuu Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liangwei Yang. Yihao Feng, Zuxin Liu, Tulika Manoj Awalgaonkar, Juan Carlos Niebles, Sil- vio Savarese, Shelby Heinecke, Huan Wang, and Caiming Xiong. Agentohana: Design unified data and training pipeline for effective agent learning. CoRR, abs/2402.155066 2024. doi: 10.
48550/ARXIV.2402.555 155066 URL (tthts::: doi. http//t.....g..o// orq/10. 485508550001
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng. Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of RLHF in large language models part I: PPO. CoRR, abs/2307 049644 2023. doi: 10.4855/ARXIV..307.000 04964.. URL /ttp::t:t///........ //doi. w00 48550/ 2...i..22000.0 04964
21

---

Published as a conference paper at ICLR 2026
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uni Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024a.
URL BwkL.htttps.:/ppprreveice net. worr/ dn 9c6yt
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn RL. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b. URL https: //operrevi ew. net/ wwt. formm? 1=066A0kATTT]]
King Zhu, Hanhao Li. Siwei Wu, Tianshun Xing, Dehua Ma. Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Elcanor Jiang, Changwang Zhang, Chenghua Lin. Jun Wang, Ge Zhang, and Wangchunshu Zhou. Scaling test-time compute for LLM agents. CoRR, abs/2506. 12928, 2025. doi: 10448550/ARXIW255661122
URL htt.ttt:://.i..rgg///////
48550/arXiv.205 2506. 12928
Tinghui Zhu, Kai Zhang, Jian Xie, and Yu Su. Deductive beam search: Decoding deducible ratio- nale for chain-of-thought reasoning. CoRR, abs/2401 17686, 2024. doi: 10.48550/ARXIV.2401. 17686. URL ht.p:::ttp//............lll //ooi. org/ http//.....55//xiii...ooo
17686
22

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
LLMs are utilized in this manuscript for partial grammatical checks and language polishing. The authors are fully responsible for the final content.
B
ILLUSTRATION OF SCALINGINTER-RR
Our Scalinglnter-RI is illustrated in Figure
Train with shorter horizon
Train with longer horizon
CBSS
perform repesiediy
Only able solve casy lasks.
X
Model collapse. Suboplimul performance.
hallaciatted tools
Fuble exploration
ScaleInter-RI: Progressive Scaling Interaction for Agent RL
==aap On at, .ak 0x+1). subject to K Further explore
Solve complex tasks fine
Master basic skills
di
hard
r
T
U2
R2
()
longer horizon hy
moderate horizon h;
shorter horizon ho
Training Fraction
Figure 9: Illustration of Scalinglnter-Pe RL to scale agent enveromenti interactions progressively.
C DETAILS OF THE FEATURES AND CHARACTERISTICS OF AGENTGYM-RL. FRAMEWORK
Diverse scenarios and environments. The environment's anisotropic complexity ensures that successful policies must develop agnaii-agnostic reasoning capabilities rather than task-specific heuristics, making an ideal benchmark for evaluating the generalization robustness of our Scalinglnter-RL methodology. It includes:
Web Navigation: Interacting with dynamic websites for tasks such as booking flights or extracting structured information, which requires agents to follow instructions, interpret textual and visual content, manipulate dynamic interfaces, and plan multi-step actions.
Performing multi-step, goal-directed queries with tools like browsers or Python interpreters, demanding strong information-seeking, multi-hop reasoning, long-term memory, and knowledge synthesis across sources.
Deep Search:
Digital Games: Exploring and solving problems in interactive game like environments, empha- sizing real-time decision- decision-making, strategy development, and adaptability to complex, dynamic settings.
Embodied Tasks: Controlling virtual or physical bodies for navigation, manipulation, and task execution, which calls for goal-directed planning, spatial reasoning, and robust perception-action grounding.
Scientific Tasks: Conducting experiments and solving problems in physically grounded, knowledge-intensive settings, requiring precise execution, dynamic interpretation feedback, evidence-based reasoning, and iterative hypothesis refinement.
23

---

Published as a conference paper at ICLR 2026
Extensibility is essential for advancing research, enabling framework to incorporate vironments, agent architectures, and training methods without modifying existing components. AgentGym-RL adopts modular and decoupled design, where the core components, Environment, Agent, and Training are fully plug-and-play. This extensible design allows researchers to incorpo- rate novel environments through simple inheritance from base classes (e.g., BaseEnvC1 ent.) and implementing the required methods such as reset (), step (), and observe ().
Scalability addresses the growing demands of large-scale reinforcement learning training that re- quires massive data processing and extended interaction sequences. AgentGym-RL. implements comprehensive architectural optimizations enhance both computational parallelism and training duration capabilities. For example, we replaced WebArena's ' single-browss peeeeebbbwweeee-eeeeeeeee a subprocess-base architecture enabling concurrent Chromium instance management. These op- timizations collectively enable effective scaling for large-scale training and diverse experimental requirements.
design with
Reliability ensures consistent operation during extended multi-turn agent training by preventing fail- ures and managing critical resources effectively. AgentGym-RL implements targeted optimizations to address system vulnerabilities that could disrupt long-horizon training. For example, we resolved TextCraft's 's memory leak in its recursive craft ing_tree implementation, where redundant self- replication caused exponential memory growth and training crashes by refactoring the recursion to eliminate redundant copies. These optimizations provide a stable foundation for uninterrupted operation across extended interaction sequences.
Welcome AgentCym Hub
Figure 10: An overview of the visualized user interface of our framework.
Standardized evaluation and reproducibility. AgentGym-RL designed to be user-frient frienfriendly for the community. To systematically address reproducibility challenges in LLM-based reinforce- ment learning, AgentGym-RI, institutes a standardized evaluation process and reproducible training pipelines. This design enforces uniform metrics and consistent experimental procedures ensure fair comparisons. We provide casy-to-setup reproduction scripts that automate the entire workflow, from environment configuration to final evaluation. This design enables researchers to replicate prior findings with high fidelity and significantly lowers the barrier for building upon existing work, thereby promoting verifiable research standards.
Visualized observability and analysis. An interactive graphical UI supports step-by-step inspec- tion and replay of full interaction trajectories, visualizing observations, internal reasoning, and ac- tions to reveal performance and failure modes and accelerate iterative development.
D DETAILED TASK PERFORMANCE ACROSS ENVIRONMENTS
Web navigation. As shown in Table 5 our models demonstrate highly competitive performance on the WebArena benchmark. In particular, the ScalingInter-77B model achieves an overall accuracy of 26.00% significantly surpassing top-tier proprietary models like GPT-4o (16.00%) and perform- ing on par with larger models like DeepSeek-R1 -1-ck5-----22 (28.00% and Gemini-2.5.P (28.00%). Furthermore, another 7B model of ours, AgentGym-RL-77 RL-7B, also achieved overall score of 22.00% surpassing the performance GPT-4o. This strong overall performance underpinned by Scalinglnter-775 7B's state-of-the-art proficiency in structured web navigation, where it achieved scores
24

---

Published as a conference paper at ICLR 2026
Table 5: Evaluation results on WebArena benchmark. For each group, the best result is in bold, and the second-best is underlined. In the first row, G & R means GitLab and Reddit.
Model
Shopping
CMS
Maps
G & R
Overall
Proprietary Models
GPT-40
20.00
13.33
10.00
20.00
16.00
Qwen-Max
20.00
13.33
20.00
30.00
20.00
Gemini-2. Gemini-2.5-Flash
5-Flash
26.67
20.00
10.00
30.00
22.00
OpenAI 04-ming
33.33
26.67
20.00
70.00
36,00
OpenAI o3
33.33
0.00
40.00
80.00
34.00
Gemini-2. 5-Pro
26.67
26.67
0.00
60.00
28.00
Open-sourced Models 2 100B
Qwen3 (wen3-235H-A22H
20.00
20.00
20.00
20.00
20.00
DeepSeek-V3-0322
20.00
13.33
10.00
30.00
18.00
DeepSeek-R1-052
33.33
6.67
30.00
50.00
28.00
Open-sourced Models
100B
13.33
6.67
10.00
10.00
10.00
Owen2.5-7B-Insttrun
14.29
6.67
0.00
16.67
9.76
Oven2.5-728-Instruun
13.33
13.33
0.00
20.00
12.00
Qwen3-4B
13.33
6.67
10.00
20.00
12.00
Qwen3-8B
20.00
20.00
0.00
10.00
14.00
Qwen3-328
20.00
6.67
20.00
0.00
12.00
13.33
0.00
20.00
30.00
14,00
Jlama-3.1 3.1-70B-Inssteet
26.67
6.67
20.00
10.00
16.00
Our RL Models
AgentGym-RL-SB
20.00
26.67
10.00
10.00
18.00
Agent-Gm-RL--7
20.00
33.33
0.00
30.00
22.00
ScalingInter-7B
33.33
26.67
20.00
20.00
26.00
of 33.33% in Shopping and 26.67% % in CMS, matching the best performance among all models in these categories. However, significant performance gap remains when compared to the top- performing OpenAl o3 (34.00%) and o4-mini (36.00%), a disparity almost entirely concentrated in the "GitLab & Reddit" sub-task.
Deep search. The evaluation results in Table 1 show the importance of sophisticated reasoning abilities, where proprietary models models-parrtieularly the OpenAl "o" series-currenty set the perfor- mance benchmark, with OpenAl o3 achieving the highest overall score of 49.50%. Against this com- petitive landscape, our models demonstrate exceptional performance. Specifically, our ScalingInter- 7B model achieved an excellent overall score of 38. 25. not only surpassing top-tier proprietary models like GPT-4o (26. 75%) and Gemini-2.5-PPr (36.50%) but also performing comparably to the strengths are particularly salient
strongest open-source model, DeepSeek- -1-0k28 (40.25%) in key domains: it achieved the highest score overall on the NQ task (52.00%) and tied for first place on TriviaQA (70.0%%) with GPT-4o. Furthermore, our AgentGym-RL-7B (34.00%) and AgentGym-RL-3B (25.75% models also delivered strong results, each significantly outperform- ing open-source counterparts of similar or even larger scales. These results provide strong evidence that reinforcement learning approach effectively unlocks the model's 's inherent reasoning pabilities, enabling to reach or even exceed the performance of elite reasoning models in key scenarios-crucially, without the need for explicit additional long-reasoning.
Digital game. The TextCraft benchmark effectively assesses model capabilities across a wide spectrum of difficulty, as detailed in T shallow depths (Depth 1), tasks are largely solved by top models. Conversely, the challenge becomes nearly insurmountable at maximum complexity (Depth 4), creating a performance cliff for most agents. It is at these intermediate and highest dif- ficulties that the efficacy of our models becomes particularly evident. Our ScalingInter-7I model achieves an outstanding overall score of 91.00%, puuting it on par with the top-tier proprietary and large open-source models a non-zero score at Depth 4, scoring 33.33% and demonstrating a unique robustness at maximum complexity. Our AgentGym-RRL-7B also excels with score of 89.00, surpassing prominent models
Critically, it is one of only few models to achieve
25

---

Published as a conference paper at ICLR 2026
Table 6: Evaluation results on TextCraft benchmark. For each group, the best result in bold, and the second-best is underlined.
Model
Depth
Depth 2
Depth
Depth 4
Overall
Proprietary
Models
GPT-40
100.00
87.80
64.00
0.00
83.00
Qven-Max
93.55
75.61
36.00
0.00
69.00
1..nnii-.5-FF200 B--Flasst
100.00
95.12
40.00
0.00
80.00
OpenAΣ 0f of-mini
100.00
100,00
84.00
0.00
93.00
OpenAΣ o3
100.00
100,00
84.00
0.00
93.00
Gemini 12.lni--.555
100.00
100.00
84,00
33.33
94.00
Open-sourced Models
100B
100.00
100.00
84.00
0.00
93.00
DeepSeek 13-VV3---020 V3-0324
80.65
53.66
40.00
0.00
57.00
DeepSeek R1 0528
100.00
100.00
84.00
0.00
93.00
Open-sourced Models 100B
35.48
7.32
0.00
0.00
14.00
-ven2.5--7-Inssruuu
80.65
41.46
0.00
0.00
42.00
Qwen2.5--22-Innttruee
96.77
85.37
48.00
0.00
77.00
Qwen3-4B
87.10
36.59
12.00
0.00
45.00
Qwen3-8B
100,00
78.05
40.00
33.33
74.00
Qwen3-32B
90.32
92.68
72.00
33.33
85.00
L
74.19
$6.10
4.00
0.00
47.00
2lama-3.1-70B-Insttruul
100.00
100.00
84.00
0.00
93.00
Our Models
AgentCym-RRL3BB
100.00
90.24
28.00
0.00
75.00
AgentGGmmRRI-7B
100.00
97.56
72.00
0.00
89.00
Scalinglnter-17
100.00
97.56
33.33
91.00
like GPT-4o (83.00%) The benefit of our RL training is especially dramatic for smaller models,
where AgentGym-RI-33B obtains score of 75.00% vastly outperforming similarly-sized models like Oween.55-3BIInstree (14.00%). These results showcase that our RI. approach elevates our models to achieve competitive performance on complex, sequential recision-making tasks.
Embodied tasks. As demonstrated in Table 7 our RL model achieves state-of-the-art (SOTA) performance on the BabyAl benchmark, with an overall score of 96.67 67%, which is competitive with the leading proprictary models such as o3 and o4-mini. Notably, our ScalingInter-7B model attains the highest overall accuracy of 96.67%, outperforming top-tier models such as OpenAl o3 (94.44%) and GPT-4o (86.67%) This exceptional performance is driven by Scalinglnter-7B5 consistent mas- tery of diverse sub-tasks, achieving perfect scores of 100% in GoTo, ActionObjDoor (AOD), and SynthLoc, and strong results of 80% in both FindObjS7 (Find) and OneRoomS20 (Room). Simi- larly, our AgentGym-RL-7BB and AgentGym-RL-333 models demonstrate robust capabilities, reach- ing overall accuracies of 92.22% and 93.33% respectively, and securing perfect scores GoTo and AOD tasks. Compared to other open-sourced models, such as Owen3-235B-A22 DeepSeek-R1-0522 (93.33%), our RL-based models maintain consistently high performance while effectively handling more challenging sub-tasks like Room and Find, where many LLMSs exhibit notable variability. Overall, these results highlight the strength of our RL-based approaches, partic- ularly ScalingInter-7B. in achieving state-of-the -he performance both structured navigation and object-interaction tasks the BabyAl benchmark.
(87.78%) and
Scientific Scenario. Our experiments on the SeiWorld benchmark, summarized in Table demon- strate the advanced performance of our RL-trained models. Our ScalingInter-P7 7B model establishes a new otate-of-the-art with an overall score of 57. 57.00%, which significantly surpasses all open-source and proprietary models, including the next-best proprietary model, OpenAI o3 (41.50%). This su- perior performance primarily attributed to high scores in the "Find" (88.64%) and "Test-Cond" (55.42%) sub-tasks. Furthermore, securing the second-highest overall score (50.50%) and achieving the top score "Test-Cond" (59.04%). These results highlight the effectiveness of our RL method for training agents in explo- ration and procedural execution tasks. However, our findings also identify a critical limitation shared across all evaluated models. The "Chem-Mix" sub-task proved to be intractable, with every model,
AgentGym-RL-7B model also shows strong capabilities,
26

---

Published as a conference paper at ICLR 2026
Table 7: Evaluation results on BabyAI benchmark. For each group, the best result is in bold, and the second-best is underlined. In the first row, AOD means ActionObjDoor, Find means FindObjS7, Room means OneRoomS20, SLoc means SynthLoc.
Model
GoTo
Pickup
AOD
Find
Room
SLoc
Overall
Proprietary Models
GPT-40
92.73
80.00
100.00
80.00
60.00
60.00
86.67
Qwen-Max
92.73
80.00
80.00
60.00
60.00
80.00
85.56
Gemini-2. 5-Flash
92.73
86.67
80.00
20.00
60.00
100.00
85.56
OpenAI 04-mini
96.36
100.00
100.00
80.00
40.00
80.00
92.22
OpenAT o3
98.18
93.33
100.00
80.00
60.00
100.00
94,44
Vemini-2.5-EP
94.55
93.33
100.00
40.00
60.00
60.00
87.77
Open- Open-vourced Models
Qwen3-239B-A225
89.09
86.67
100.00
80.00
60.00
100.00
87.78
DeepSeek-V3-0324
67.27
53.33
0.00
20.00
40.00
60.00
56.67
DeepSeek-R1-0522
98.18
86.67
100.00
60.00
80.00
100.00
93.33
Open-sourced Models
61.82
40.00
20.00
60.00
40.00
20.00
52.22
Oven2.5-7B-Instrru
70.91
66.67
60.00
80.00
60.00
20.00
66.67
Oven2.55-28BInst-nn
92.73
93.33
100.00
60.00
60.00
80.00
88.89
Qwen3-4B
60.00
60.00
40.00
40.00
40.00
20.00
54.44
Owen3-83
43.64
20.00
40.00
40.00
40.00
40.00
38.89
Qwen3-32B
87.27
80.00
100.00
60.00
40.00
80.00
82.22
11lem-3.11-B--Instroe
85.45
60.00
100.00
80.00
60.00
40.00
77.78
89.09
86.67
100.00
60.00
60.00
100.00
86.67
Our RL Models
Agen-GGm-RL--B
100.00
100.00
100.00
60.00
60.00
60.00
93.33
Ggn
100.00
93.33
100.00
60.00
60.00
60.00
92.22
Scaling ScalingInter-78
100.00
93.33
100.00
80.00
80.00
100.00
96.67
including our top performers, scoring zero. This uniform result indicates systemic challenge for current language models in tasks requiring complex scientific reasoning and multi-step chemical simulation, marking this as crucial area for future research.
E IMPLEMENTATION DETAILS AND SETTINGS OF EACH ENVIRONMENT
We conduct all the experiments on NVIDIA A100 GPUs and Ascend 910B NPUs. The remaining part of this section shows detailed setting of different environments.
E. WEB NAVIGATION SCENARIO
Tools and APIs. web navigation scenario, the agent simulates human interaction with web pages to ultimately complete the task. WebArena(Zhou et al., (024a) supports these interactioins through a set of tool APIs, allowing agents to perform a variety of real-world tasks, including online shopping, engaging in discussions on Reddit, collaborating on software development via GitLab, and managing store content through a CMS. In addition to these online platforms, WebArena also provides three utility-style tools: a map for navigation and location- based information search, calculator, and scratchpad for note-taking.
query case web navigation is shown below:
Web Navigation Example
You autonomous intelligent agent tasked with navigating web browser. You will be given web-based tasks. These tasks will be accomplished through the use of specific actions you issue.
27

---

Published as a conference paper at ICLR 2026
Table 8: Evaluation results on SeiWorld benchmark. For each group, the best result in bold, and the second-best is underlined. In the first row, Test-Cond. means test-conductivity. Chem-Mix means chemistry-mix,
Model
Measure
Test-Cond.
Find
Chem-Mix
Lifespan
Overall
Proprietary Models
GPT-4o
15.09
6,02
38,64
20,00
73.33
21.00
Owen-Max
9.43
0.00
34.09
20.00
40.00
13.50
Siiini-2.5-Faase
11.32
0.00
54.55
80.00
21.00
OpenAI 04-mini
20.75
14.46
47.73
0.00
100.00
29.50
OpenAI 03
47.17
25.30
56,82
40.00
66.67
41.50
Gemin1-2.5-Pro
9.43
0.00
29.55
0.00
46.67
12.50
Open-sourced Models 2 100B
Qwen3-235B-A22B
11.32
4.82
59.09
20.00
66.67
23.50
DeepSeek-V3-0324
0.00
0.00
2.27
0.00
0.00
0.50
DeepSeek-R1-0525
1.89
0.00
11.36
20.00
4.50
Open-sourced Models
100B
3.77
0.00
0.00
0.00
0.00
1.00
1.89
0.00
13.33
1.50
-wan2.5-72B-nns::
7.55
1.20
15.91
20.00
40.00
9.50
Qwen3-4B
0.00
0.00
0.00
0.00
33.33
2.50
Qwen3-8B
9.43
0.00
18.18
0.00
46.67
10.00
Qwen3-32B
5.66
31.82
0.00
66.67
14.00
1lama-3.1-80-Instzuu
9.43
0.00
4.55
20.00
0.00
4.00
1.1ama 3- Instructt
24.53
4.82
40.91
40.00
86.67
25.00
Our RI. Models
Agent&ym-Rl.-3H
20.75
28.92
0.00
0.00
66.67
22.50
AgentGGm-RRI-7B
24.53
59.04
65.91
66.67
50.50
ScalingInter-T2
33,96
55.42
88.64
0.00
73.33
57.00
Available Information:
User's objective: The task to complete
Accessibility tree: Simplified webpage representation, providing key information.
Current URL: The active page's address
Open tabs: Currently available tabs
Previous action: Last performed action
Action Categories:
Page Operations:
click
[id]: Click element with ID
type
[id] [content] [011]: Input text (I=press Enter)
hover [id] Hover over element
press
[key combl Simulate key press (e.g., Ctrl+v)
scroll
[down] up]: Scroll page direction
Tab Management:
new tab: Open new tab
tab fab_focus [tab_index]: Switch to tab
close.tab: Close current tab
URL Navigation:
goto [url]: Navigate to URL
go.back: Return to previous page
28

---

Published as a conference paper at ICLR 2026
go_forward: Advance to page
Completion:
stop [answer]: Submit final answer (or "N/A" if you believe the task is im-
possible to complete)
Homepage: If you to visit other websites, check out the homepage http:// homepage.com
Objective: Among the top 10 post in "books" forum, show me the book names from posts that recommand a single book.
Settings. We include five subtasks: E-commence, Reddit, Gitlab, OpenStreetMap (Map), and on- line store content management system (CMS), comprising total of 372 training queries and 50 test- ing queries. These are selected from the origin WehArena dataset, which contains 812 queries across three categories; Information Seeking, Site Navigation, and Content Config. To facilitate efficient parallel rollout, we exclude the Content & Config tasks, which involve insert, update and delete operations that change the state of the websites. We set the maximum number of agent-environment interactions to 15 turns in both AgentGymm- RL training and evaluation. In Scalinglnnter-R5 RL, we grad- ually increase the maximum number of interactions transition from 12 and then to 15, with each transition occurring every 80 step. We employ GRPO as the main RL algorithm with learning rate of 5 x 10-? and a KL coefficient of x 10- For each query, we sample distinct trajectories using temperature of 1.0.
E.2 DEEP SEARCH SCENARIO
Tools and APIs. The deep search senario features a search engine-based environment equipped with specialized tools and APIs supporting the interaction with search engines. These APIs en- able agents to dynamically generate search queries during the reasoning process, retrieve relevant information from external sources, and incorporate the retrieved information into subsequent reason- ing steps. This setting allows agents to engage in complex reasoning processes that involve iterative searching and information integration, thereby enhancing their capability to solve intricate problems where external knowledge is essential.
query case Deep Search is shown below:
Deep Search Example
You must always reason inside tthink>..<</hhintt first; if you lack knowledge, issue a and then stop; do not generate <information>> <answer>> yet; wait for external input between <information>>. <i//ooommttiiiooiiiioioo.. before continuing: resume only when <information>> is given; do not skip steps anticipate answers early.
Question:
Who got the first Nobel Prize in Physics?
Settings. We include queries from 7 datasets following the setup of Search-R1 (Jin et al. 2025b)
NQ (Kwiatkowski al. 2019). TriviaQA (Joshi al.) (2017). PopQA (Mallen et al. 2022), Hot-
potQA (Yang et al. 2018). 2wiki (Ho et al. (2020). Musique (Trivedi et 2022), and Bamboogle (Press al. 2023). To ensure fair comparison and balanced evaluation, we randomly sample 400 examples from the development sets of NQ. TriviaQA, PopQA, HotpotQA, 2wiki, Musique, and Bamboogle. The maximum number of agent-environment interactions is set to 10 turns eval- uation, and to 5 turns in AgentGym-RL training. In ScalingInter-RL, the maximum number of interactions is initially set to 5, increased to at step 200, and further to 10 at step 300 We employ GPRO as the main algorithm for reinforcement learning setups with learning rate of 1 10 KL coefficient of 1 10 and a sampling temperature of 1.0. We sample 8 distinct trajectories for single query.
29

---

Published as a conference paper at ICLR 2026
E.3 DIGITAL GAMES SCENARIO
Environments, Tools and APIs. As for digital games, we introduce TextCraft Prasad al. 2 a text-based game environment mirroring Minecraft. The APIs in TextCraft include crafting, inven- tory management, and dynamic narrative generation. These APIs allow agents execute predefined crafting recipes, manipulate inventory contents, navigate virtual spaces, dynamically generate quests and sub-tasks based on natural language objectives, and recursively decompose complex tasks into achievable sub-goals.
query case TextCraft can be seen below:
TextCraft Example
You are given few useful crafting recipes craft items in Minecraft, Crafting commands are of the format "craft [target object] using [input ingredients]".
Every round will give you observation, you have respond action based the state and instruction. You can "get" an object (ingredients) from the inventory or the environment, look-up the game inventory by "inventory", "craft" (target) using any of the crafting com- mands. You can use ONLY these crafting commands provided, do not use your own crafting commands. However, if the crafting command uses a generic ingredient like "planks", you can use special types of the same ingredient e.g. "dark oak planks" in the command instead.
Goal: Craft flint and steel.
Settings. In TextCraft, task difficulty measured by the maximum depth of the corresponding crafting tree. In practice, the benchmark contains tasks with crafting trees of depths 1, 2, 3, and 4. Accordingly, we divide the entire task set into four subsets based on these depths. We set the maximum number of interactions training. In ScalingInter-Ri
20 turns in evaluation, and set 30 turns AgentGym-RL we gradually increase the maximum number of interactions transition from 10 to 20 and then 30, with each transition occurring every 100 step. We employ GRPO as the main RI. algorithm with learning rate of x 10-6, KI. coefficient of 1 10-3, and sampling temperature of 1.0. We sample distinct trajectories for a single query.
E.4 EMBODIED SCENARIO
Tools and APIs. We introduce the BabyAl environment representative setting for embodied tasks. It provides APIs that allow agents to navigate a controllable grid world using natural language instructions. Through these APIs, agents can perform actions such as moving objects, unlocking doors, and interacting with the environment in response to textual commands.
query case BabyAl can be seen below:
BabyAl Example
You an exploration master that wants to finish every goal you are given. Every round I will give you an observation, and you have to respond an action and your thought based on the observation to finish the given task. You are placed in room and you need to accomplish the given goal with actions.
You use the following actions:
turn right- turn left move forward go to obj id pick up obj id
go through door id: door must be an open door.
toggle and go through door id: door can be a closed door or locked door. If you want to open locked door, you need carry key that is of the same color as the locked door. toggle: there is closed or locked door right in front of you and you can toggle it.
Your goal: Go to the red ball.
30

---

Published as a conference paper at ICLR 2026
Settings. Following the original implementation, divide the tasks into six subsets based the final goal. We set the maximum number of interactions to 20 turns in both evaluation and AgentGym-RL training. In ScalingInter- we gradually increase the maximum number of inter- actions transition from 6 to 13 and then to 20, with each transition occurring every 100 step. We employ GRPO the main RI. algorithm with learning of 10-6, KI. coefficient of X 10 and sampling temperature of 1.0. We sample 8 distinct trajectories for a single query.
E.5 SCIENTIFIC SCENARIO
Tools and APIs. SciWorld Wang et al. 2022) is an agent environment for scientific tasks. It pro- vides APIs that are designed to support scientific exploration through text-driven reasoning cycles. These APIS empower agents to conduct experiments by interacting with various scientific appara- tus and performing actions like measuring temperature, connecting electrical circuits, and mixing chemicals.
query of SeiWorld be seen below:
SciWorld Example
You are an agent for science world. Every round will give you an observation, you have to respond an action based the observation to finish the given task.
Your task to boil water. For compounds without a boiling point, combusting the substance is also acceptable. First, focus on the substance. Then, take actions that will cause it to change its state of matter.
Settings. We select 8 subsets of tasks from the original SciWorld environment. We set the max- imum number of agent-enviromment interactions to 20 turns both AgentGym-RL. training and evaluation. In Scalinglnter-RI. we gradually increase the maximum number of interactions transi- tion from 10 to 15 and then to 20, with each transition occurring every 200 step. We employ GRPO as the main RL algorithm with a learning rate of 1 10-6 KL coefficient of 10 10-3, and sampling temperature of 1.0. We sample 8 distinct trajectories for single query.
F ANALYSIS ABOUT THE INSTABILITY OF LONG-HORIZON TRAINING
From a theoretical perspective, for given query, an LLM agent performs multiple rounds of in- teraction, where each round is a ReAct step (Reasoning with an Action). Each step consists of multiple tokens, so the entire interaction produces a large volume of tokens, which makes the task very challenging for an agent that lacks sufficient environmental context and basic capabilities.
We conduct empirical analysis, shown in the Figure II We find that, in terms of training reward, long-horizon training tends to collapse, mainly due to the uncertainty introduced by long- horizon interactions and operations. At the same time, observe that this phenomenon is often accompanied by abnormal gradients, where the grad norm exhibits multiple spikes, which empiri- cally confirms its instability.
G TRAJECTORY EXAMPLES AND VISUALIZATIONS OF OUR RL AGENT
This appendix provides additional trajectory visualizations and detailed analysis across multiple en- vironments. The figures illustrate the behaviors of both baseline and RL-trained agents, highlighting the RL model's superior performance in exploration, task execution, and interaction patterns, while also revealing common failure modes that remain.
Enhanced navigation. Figure demonstrates a notable improvement in navigation capabilities within BabyAl environment. While the base agent exhibited suboptimal behavior characterized by repetitive movement patterns-gon through previously explored locations without developing a strong search strategy for completion, the RI. agent manifested more effective exploration strategy.
31

---

Published as a conference paper at ICLR 2026
80-
0.5
Grad Norm
Reward
0.4-
0.3
40-
0.2-
0.1-
0.0-
100
1.0
250
300
350
150
200
500
Steps
Steps
AyentGym
Scalingfnter
Figure 11: Empirical analysis the instability of long-horizon training.
It demonstrated strategic backtracking capabilities, systematically exiting through doorways before selecting alternative pathways, ultimately accessing a green door that provided direct access to the target blue box. This highlights the RL agent's 's superior ability in spatial reasoning and its ability to circumvent unproductive behavioral loops.
Compositional Task Mastery. Figure [4 exemplifies the successful application of reinforcement learning to complex scientific task execution. The base agent exhibited fundamental deficiencies in task interpretation, misusing non-interactive objects and generating invalid actions. In contrast, the RL-optimized agent demonstrated comprehensive task understanding through its systematic ap- proach: correctly identifying and manipulating living thing (the banana tree), executing appropriate inventory management operations, navigating multi-roon environments with obstacle resolution pabilities and successfully completing the objective by depositing the tree in the designated purple box. This highlights the RL agent's enhanced capabilities in reasoning, planning, and sequential task execution within compositional problem spaces.
Adaptive Web Navigation Strategies. Figure 15 and Figure T6 illustrates the emergence of web navigation capabilities through reinforcement learning optimization. The base agent persistently interacted with non-responsive interface elements, specifically engaging in repetitive clicking be- haviors ineffective targets without recognizing the futility of these actions. Our RL-traiiee agent exhibited markedly superior adaptive hehavior: it successfully implemented error recovery mecha- nisms when encountering a "Page found" error, subsequently utilizing the search box locate the "pittsburgh" forum, identifying contextually relevant content within trending posts, and com- pleting the subscription task deeessfully-dmmonsttaaaee enhanced robustness error handling, purposeful navigation strategies, and the ability to maintain task focus while avoiding unproductive behavioral patterns.
Limitations in Scientific Scenario. Figure 17 reveals fundamental procedural execution failures that persist in SciWorld task completion despite the RL agent's 's ability to reach task-relevant game states. These instances exemplify two distinct failure modalities: first, when confronted with inter- action failures requiring systematic debugging, the agent inappropriately substitutes direct factual recall for the intended experimental procedure; second, the agent demonstrates insufficient sys- tematic exploration, evidenced by its premature task termination after navigating to the outdoor environment and focusing only on the chameleon egg rather than analyzing all available animals that the task demands. These failures collectively indicate that the model lacks the deep procedural understanding necessary for executing rigorous scientific comparative analyses.
Over-Interaction Patterns in Web Navigation. Figure 18 demonstrates a prevalent failure mode of excessive and inefficient interaction sequences during web navigation tasks. Despite success- fully navigating to the correct target websites in both illustrated cases, the RL agent engages in su- perfluous interaction pattern--incudin redundant clicking, unnecessary hovering, and excessive scrolling that impede successful information extraction from the target pages. These behavioral patterns suggest that the reinforcement learning process failed to instill the precision and efficiency required for optimal task completion, indicating a gap between state-reachin mate-reaching capabilities and effec- tive action selection within those states.
32

---

Published as a conference paper at ICLR 2026
BabyAl
Task:
go to the blue box
In front of you this room, you can several objects: There yellow closed door stops in front of you and sleps to your right. There green closed door steps front of you and steps to your left. The room has walls around you. You are facing wall steps away. You are not carrying anything.
Available actions: ["tum left", "turn right", "move forward", "toggle and go through yellow closed dear 1", "togøle and go through green closed door I", "go to yellow closed door I", "go to green closed door 1", "chock available actions") ]
Base Model
Our RL Model
Thought To reach the blue box, noee to move foeward, but there is efficient
Thought find the blue box
yellow closed doer frout
to go through the yellow closed blocking the
Act soggle go through vellow closed door
green closed doer stepe front you stops to your right. The ruom has walls around you. You une facing wall steps away.
Act I: toggle and gu through yellow cloeed door There green closed door steps front you and stepe to your right. rocca liss walls around you. You are facing wall steps away.
Thought ... find the blue box
Act 2: green closed door I
Thought ... Now that I have gune through yellow closed door, Since I don't have any keys, I need to loggle and go through this door to find the bluc hox.
Obs 2: There green closed door right front you away.
The room has walls around you
and go through
closed doar a grey door steps
Thought find the blue
you steps
toggle and go through green closed Obs 3: There a grey closed door I stops in frunt of you and steps
to your left. The room has walls around you. You are facing wall steps away.
your
has walls you, You facing wall
stops away.
Thoughht m.. only option to togele and attompt to through.
Act 4: toggle and go through rod closed door Obs The has walls around you. You are facing wall steps
Act 4:
clased dooc
toggle and go through closed door
awaa..
Act 6: to red closed door
Act S: move forward
Act 7: namn right
through openi door go through grey open doce
right
through grey open door
Act 10: surn right
yo through grey open dooe 14: go through open door
through green open door
12: go through yellow open door
Thought 15: Given that the roomt has walls around me and I am not carrying myything, appears have reached the end of path. already explored this ans thuroughly and there are no more objects or doors interet with.
Act 13:
foggle and through green closed dioor heree blue box steps front you
left. There is green closed door 6 steps in front of you and steps to your right. The room has walls sround you. You ficing wall
cleck available octions
steps away.
14: blue
Thought 20: Verrnin left again has brought me to wall, continming
explored possible directions this Act 20: check available action
Score: 0.0
Score: 1.0
Figure 12: Comparison of our RL agent with the base agent on the BabyAI task. Our RL model significantly outperforms the base model, successfully navigating to the blue box while the base model fails to complete the task.
H COMPARATIVE CASE ANALYSIS
Figure 19 shows a case from the TextCraft environment. The short-horizon agent fails because uses incorrect quantities (violating crafting constraints) and drifts from the goal of crafting an orange bed. In contrast, our agent succeeds by adhering to constraints and managing resources strategically: it proactively checks inventory, adaptively explores alternatives (c.g., crafting intermediate items), and efficiently acquires materials to resolve shortages. This showcases stronger long-term planning and underscores the effectiveness of our method.
Figure 20 shows a case from SearchQA environment. The short-horizon agent fails because it only performs superficial searches. When encountering misleading information (such as the references to the Jefferson family), it does not conduct further verification or cross-check the family lineage. ScalingInter-PP RL. agent, on the other hand, actively adjusts its strategy after making error. Through
33

---

Published as a conference paper at ICLR 2026
Agent tecrrrie
Autol
go blue box
green
room, you
can see deveral objects: There blue box steps in front of...
[Autn] blue base
front of you in this room, you can see several objects: Thare
Figure 13: Trajectory visualization in the Baby AI task, highlighting the agent's path through the environment, action execution, and feedback.
multiple turns of search, it verifies hypotheses, filters irrelevant information, and further traces the parentage of Sally Hemings. It gradually narrows the search scope step by step, and ultimately identifies the key genealogical fact (that Sally Hemings's father is John Wayles), allowing it to arrive at the correct answer.
Figure 2T shows case from TexiCraft environment. The long-horizon agent failed because did not follow the task objective and directly craft mossy stone bricks. Instead, it attempted an action less relevant to the goal craftin mossy cobblestones-wiic actually moved farther away from completing the target task. In contrast, found the path to craft mossy stone bricks, and thus completed the task more efficiently. This case suggests that the long-horizon agent may suffer from over-exploration, engaging in redundant exploration in the environment while neglecting goal-directed progress.
agent maintained goal-relevant exploration, correctly
Figure 22 shows case from TextCraft environment. The short-horizon agent failed because did not explore the environment appropriately and sufficiently. Instead, it performed excessive internal reasoning and incorrectly assumed it needed obtain iron ingots by mining iron ore, even though this was not required by the instructions at the current difficulty level. As result, it overlooked more direct way obtain the ingredients needed craft iron chestplates. plored the environment correctly and sufficiently, recognized that it could directly obtain the required ingredients from the environment, and-based on exploration results- csults--coleected iron nuggets, then crafted iron ingots, and completed the task. This case indicates that the short-horizon agent may face an excessive reasoning issue: it relies too heavily on internal reasoning, makes incorrect assumptions about the environment state, and fails interact with the environment to explore.
contrast, our agent ex-
34

---

Published as a conference paper at ICLR 2026
SciWorld
Determine sodium chloride electrically conductive. The sodium chloride located around the workshop. First, focus the sodium chloride. If electrically conductive, place the yellow box. It electrically nonconductive, place the purple box.
Task:
hallway. you agent, substance is closed), A door the bedroom (that closed), A doer to the greenbouse (that closed), A door to the kitchen (that closed), A door the living room closed), docr the wockshop closed)
picture, see:
Base Model
Our RL Model
The takk detemmine sodium chloride is electrically conductive. The sodium chlovdde located weekshop, nead go there finst will use the "focus on" Act to signal my iineent then proceed the workshhop.
Thought I:
determine sodium conductive and piace it yellow box conductive or purple box soncoductive Let's find some sodium chloride around the workshop.. Act look around Obss 1:
Thought
workshop
Obs I: You focus worksshop.
Thought door the workshop open. through workshhop.
Thought 2:
have focused the workshup, so the sodium chluride sodium chloride
should
Act 4: go to the workshop
Act focus on sudium chloride
You
the workshop.
Obs 2:Nu sodium chloride foundi
Thought 5: have gone to the workshop. Let's see there's any sodium chloride noond here around the nrca.
Thought Sodium chloride not the workshoop yet. I'll try looking around again find it.
Act look
Ohs S: This room called the weekshop In it you sonsee .. substarce called sodium chioride...
Obs 3: You ane the hallway....
some pick it up and Socns on it to desenmine it's conductive.
Thought
chloride around workshop tats
sodium chloride
sadiums chioride
inventory.
Thought 7: picked up the sodium chioride and added it my inventory: Let's focms the sodium chloride.
focus the
chloride
Obs 7:Yon focus on the sudium chiorridei
Thought facused sodiam chloride. Sodium dhloride sonic compound
electrically noncendactive, Let's put it the
purple hov.
put sudium chloride purple Obs Semove the sediums chioride the purple box:
Score: 0.0
Score: 100
Figure 14: Performance comparison of the RI agent versus the base model on the SciWorld task. RL agent achieves a perfect score by successfully moving a banana to purple box, while the base model scores0.0.
35

---

Published as a conference paper at ICLR 2026
WebArena
Task: Open the thread of trending post on the forum "pittsburgh" and subseribe.
Obs URL: http://reddii.com (current):
[1] RootWebArea
'Postmill' focused: True
[232] link 'Forums'
[233] link 'Wiki'
[241] searchbox 'Seurch query'
[242] link 'Notifications (0)
[243] link 'Submit'
Base Model
Our RL Model
Thought I: We need to novignne to the "pitssurgh" fonum first, Act I: click [66] Obs
Thought i.... We need navigate the "pintcurrhh" forum finst. To I: click [232]
link
found"
Obs I: ...[1990 scarchbox 'Sercch query'..
Thought 2: the
"Page not found"
Thought we can see that "Asi.Redii" and
nav iigne back homepage of.... Act 2:go back Obs 2:
"relationship advice' are listed, but "pinsburgh" is not. We need to soarch for the "pussburghh' forum using the seurch type [1958] [pitsburgh]
Obs 2: ...[295]] link From Florida to Pinsburgh Is $100,000 home budget Pinsburgh disabiiity income feusibl'
Thought 3: e. chere is link to the "Forams" section which we can use navigne "pitsburgh"" fonum.
Act click
Thought Si.ce the current page nos clearly indicating a trending post, we should navigate back to the list of furums and find "pitssurgh" forum check trending posts. Act 3: goto (http://eediit..m pittsuuuug Obs 3: ... [4739) StaticTest Unilitee have gone rignificantly everywhere
... [500]
quary' ...
Thought
page lists
named "pitishungh" listed nead to search "pittshurgh" forum. Act 4: type [500] [pittshurghi [T Obs 4: (4004] link From Florida to Pinsburgh $100,000 home budget in Pinsburgh on disability income feasible?"
monchs with necessarily
thoth
gas electric), Finsburgh...
Thought St ... The search nosuls "pittsbunhh are now displayyed, and first click [4084]:
observations, recent
tranding
from loeNinetiiee with timestamp "January 12, 2023 8:54:19 AM EST". This relatively recent post, and it might be trending post. Act click (4739) Obs 4: Nothing changed.
Obs S: (8488844 button 'Subscribe No subseribers'..
open
verify.
Thought 6: The objective open the thread of trending pust om forum "pinsburgh" and subscribe. Act 6: click [8484]
click (4739)
Obs &: [8484] button 'Unsubscribee 1 subseriber!.
Obs 5-15: Nothing changedi
Thought 7: The subscription button showing that there indicating that the alreddy subscribed. stop [N/A]
Score: 0.0
Score: 1.0
Figure 15: RI. agent vs. Base Model on WehArena task. RI. agent successfully located the trending post and completed the subscription, achieving score of 1.0., while the base model scores 0.0.
36

---

Published as a conference paper at ICLR 2026
Aperoraacies
Open the thread of trending post the forum "pittsburhh and subecribe
"Subscr button located the sideber under the "pitsburggh
Browser View
currently says "Subsoribe No [7743].
Paouugh 1U000 www.vayy....iiaaeeee.eee Buabiry
To complete the taok, I will click this subscribe subreddit.
summary. the next action perform is
click [T745)
Figure 16: Trajectory visualization the WebArena task, highlighting the agent's path through the environment, action execution, and feedback.
37

---

Published as a conference paper at ICLR 2026
SciWorld
SciWorld
meking point located around the kitchen. Find, focus on themmomueer. Next, focus on the water. If the molting point of water is ahove 10.0 degroes colsins, focus the red box. the melling
Your
find the aninaal with longest life spon. The snimals are the ouuside'' location. Focus on the animal with the longest life span.
Task:
waser below -10.0 degrees
the agent sudio (thut closed) A door to the badrocm (tha is elosed) door the greenhouse (that is closed) dour the kitchen (that closed) closed)
Obs O: This room called the hailway. In substancee called Finger painting also
celsius, focus on the green box. The boxes are located around the kitchen. Obs This room is called the bedroom. In it, you see: the agent a bed. On the bed isc matiress, On the maitness white book (contrining
substance called
Foundations of the Origin of Species) litled The Foundations of the Origin Species by
door living
workssopp.
(tha is closed)
Charles closet, The door closed. painting table. nothing.
Act to the outside The
see:
hallway
open.
closed)
Thought 7: The door to the outside is not open, Lef's open before dhrough.
Thought Around here. found finger paiming. Let's
kitches to find themoomeer.
Act open the
the outside
Act 6: the kitchen
Obs 7: The door now open
The door
Thought The door the cutside
Let's through
Thought The the kitchen open. Let's open before
outside:
going through.
Act go to the outside
door to kinchen
Obs 8: You
the outside.
open.
Thought 8: The door to the kitchen new open. Let's go strough the
Thought 9: have gone to the outside. Let's sce if can find sume animals or information about animals uround here or nearby.
kitche..
8: the kitchen
Act look around
Obs You move to the kinchen.
Obs 9: This outside location called the outside. Here you seer the
sgent substance called air are chumeloon fine pit
Thought Around the kitchen, found thermometer and box. themmumeter
(comsining nothing) fountain (containing substance called substance called also see: A door to the foundry (tha: closed) A door to the greenhouse (that
my inventory.
ground parror egs buby
You
Act 10: pick up the themmometer
10:
thennometer
dour the kischen (that closed)
Thought pisked up thermometer
my inventory.
Let's focus on the thermometer.
Thought 10: Around here, found a chameleon egg, a parrol cgs. and a baby rabbiit Let's find out the life span of these animals. 10: the chameleon ega Obs move the chameleon to the inventory.
Act LI: focus un the thermometer
11: focus
thermoeneter.
Thought 12: feessed on the themometer. Let's facus on the water to
mensure the melling point.
the water
Thought picked the chameleon egg and added to inventory.
known
input.
egg.
houggh 13: fecused on the themometerr. Let's go to the sink focus
Act 11: focus on the chamclesn eggi Obs You focus the chameleon egg.
the melting point.
Obs 13: No known action matches that input
degrees Since the malring point of water is above 10.0 dagrees celsins, will focus en the red box.
The meiting point
focus
box.
Score: 0.0
Score: 0.0
Figure 17: Examples over-interaction failure on the Seiworld task for both the base model and our RL agent.
38

---

Published as a conference paper at ICLR 2026
WebArena
WebArena
Task: key aspeets customers
Zing
Task: Who gave 4 or sttes for phone casos from FYZUTAK
Jump Rope
Obs 0: [1] RoetWehAres 'Oue Stop Market focssed: True (972]
Obs [:] RoutWehhres 'Daahhbourdd Magento Admin' fucused: True
Accessories hasPopup: Thought We need to navigute to page where we can see the
menuiiess foeeZ2 Cell
[108] link "uz609 MARKETING'
[361] link 'Catalog Price Ruld
mtings for EYZUTAK's phane ceses
Thought I: The anforation provoded does not seem contain this data.
elick [972]
We navigate the MARKETING' scction there
1: [4955] link 'Cises, Holsters Sleevest item
customer feedback survey
might be
Thought "Cases, Holsters Sleeves" which likely contains the
Act 1: click [108]
EYZUTAK.
Obs [376] link 'Email Templates'
[4955]
Thought 2:
objective,
Obs 2: [4954] link 'Accssorries 1924 item
Communictions section, as it might contain customer foedback or survey Cata.
Thought We need navigate
where we can soe the
for phose coses.
Act click
Act 3: click [4954]
Obs 2: [1195] RootWebbee ebArea 'Email Templates Communications
Obs 3: [6748] combobox "ue615 Search mutocomplete both hasPopup: listbox requined: expanded: Falso
Marketing Magento Adau' focused: True
Thought 3: We
"Catalog Rule" "Cart see any related
Thought We
speciffe soction FYZUTAKK phone cases and then check the ratings.
Rules"
"Promotions"
information.
type [6748] [FYZUTAK
Act click
[9330] Descriptionn inDatill [9489] 'EYZUTAK phone case [100000000) (0000) ing 'mmagee
matching
Thought 4: TTee "Markketing" section might have other relevant sections that we have not explored yet.
Thought The saarch roultss are displayed in grid fornar, and the miings
found the product pages.
click
Act click [10000]
Obs 4: [1649] Static lost 'SEO Search'
Thought On producr the msings and reviews are typically found in section dedicated
Thought 5: The "SBO Search" section might have relevant data, such
URL that could indicate dissarisfaction.
search
hover [16300]
Thought However, the matings and reviows are not immedistely visible. need
Act St click [1649]
specific product page enstomer reciews ratrinss
[1670] StaticTest Content'
Thought
can navigste to Contant" section to
Act 7: [10000]
there any relevant data,
Act 6: click [1670]
section immedistelyy visible, scrollnng down will help to reveal it.
Thought
[1674] link 'All Reviews"
Thought 7: The seviews section is potential source customer
Act 9: scroll [down]
footback.
"Raiin""10511 generic
click [1674]
10: type [105]3] [4 ur
(2354)
pooduct's reviews to the Obs 18: [10415] myoutTable"
Thought &: We noed to nnvigate to the reviewa for 7ing Jump Rope to
Thought relevant information. Act 11: click [10115]
find relevant information.
Act click [2354]
Obs 8: [2818] link 'All Reviews'
Act 9: click [2818]
Thought 15: ratings and reviews spacifii product
[5172] grideell 'Any' requiee::
relevant the objective. we need to navigate back to the seurch resulis continue the search. focus
[5496] combsobox" "taaopppp meno expanded: Falsel
Act 10: click [S172]
click [5496]
Act type [3496] [Zing Jump Rope] []]
Thought L::: the data might not be available for Ting Jump Rope,
Act stop [NIA]
Score: 0.0
Score: 0.0
Figure 18: Instances of "ovve-interratton" failure on the WebArena task for both the base model and our RL agent.
39

---

Published as a conference paper at ICLR 2026
TextCraft
Goal: craft orange bed.
Baseline
ScalingInter-RL
Thought To make orange bed, orange wool and 3 planks. should get orange dye and white wool.
Thought 1: To emff an orange bed, need orange wool and planks. First, I need gather orange wool and planks.
Action 1: get I orange tulip
Action get orange
Observation I: Got orange tulip
Observation 1: Could not find orange wool
Thought 2: have the orange talip, now need make orange dye.
Since cannot orange wool, using white wool and orange Action 2: get orange dye Observation Could not orange dye
Thought 2:
Action orange dye using orange tulip
Observation 2: Crafted minceraftorrage dyc
Thought Crafted orange dyc, now need get white wool the bed.
Thought 3: need to gather orange dye first. Orange dye can be crafted using orange tulips.
Action 3: get 4 white wool
Action get orange tulip
Observation 3: Could not find white wool
Observation 3: Got orange tulip
(Try to craft more white woul)
... (Try
orange dye)
orange Action 14: craft orange wool using orange dyc, white wool Observation 14: Could not find enough items to craft minecrafttorange. wool
Thought Crafted white
Thought 7: I now have omange dye. With this, can craft the orange
Action 7: craft 3 orange wool using orange dyc, white wool Observation 7: Could not find enough items to craft oranee wool
(Try to craft orange dye and orange wool)
(Try to craft orange woul)
Thought 20: Crafted white wool,
make orange wool.
Action 20: craft orange wool using orange dye, I white wool Observation
Thought I have orange wool, Next, I need gather planks.
not find enough items to craft mineoraftorrage wool
Action 16: get oak logs
Observation 16: Got logs
(Try to craft orange dye orange wool again)
(Try to craft oak planks)
Thought 30: Crafted orange dye, now can make orange wool.
Action 30: cmft onange wool using orange dye, white wool Crafted minecraltoonagee.eg wool
Thought 19; I need planks craft the orange bed. have already crafted 4 oak planks, so can use oak planks for the bed. Action
orange using orange wool, planks Observation 19: Crafted minecraftrorange bed
(Reached mavimum interaction rounds)
(Task completed)
Figure 19: comparative case from TextCraft.
40

---

Published as a conference paper at ICLR 2026
SearchQA
Question: Who Harriet Hemings's maternal grandfather?
Baseline
ScalingInter-RL
The question asks maternal grandfather Harriet Hemings. know her mother Sally Hemings, but need confirmation of Sally Hemings's 's parentage.
Thought 1:
The question asks the maternal grandfather of Harriet Hemings. To answer, need information about Harrict Hemings's mother, Sally Hemings.
Thought
Action 1: search "Iarriet llemmngs maternal grandfather"
Action search "Harrict Hemings materaal grandfather"
Ohservation 1: Documents say Harriet is the daughter of Sally Hemings: Sally's 's parents are Elizabeth Hemings and John Wayles.
Observation I: Documents say Harrict is the daughter of Sally Hemings; Sally's parents are Elizabeth Hemiings and John Wayles.
Thought 2: notice Jefferson heavily involved in the Hemings lineage, he might be considered the maternal grandfather. Action answer "Thomas Jefferson"
Thought Check if Thomas Jefferson is relevant a possible Action 2: Observation Incorrect.
"Thomas Jefferson"
Observation 2: Incorrect.
Thought Verify Jefferson's lineage in case helps Action scarch "Thomas Jefferson's father" Observation 3: Documents describe Jefferson's 's ancestry, not relevant to Sally's 's maternal line.
Thought 3: Re-check the query using broader search.
Action 3: search "maternal grandfather of Harriet Hemings"
Observation 3:
Documents again describe Sally Hemings's 's family background.
(Confirm information across related family lines)...
Thought 7: Re-evaluate: Harriet's 's matemall
Thought 4: Jefferson's role might still apply.
grandfather must be Sally Hemings's 's father, who is John Wayles.
Action 4: "Thomas Jefferson" Observation 4: Incorrect.
Action "John Wayles" Observation Correct.
Action answer 'ally Hemnings"
Observation 5: Incorrect.
(Reached maximum interaction rounds)
(Task completed)
Figure 20: A comparative case from SearchQA.
41

---

Published as a conference paper at ICLR 2026
TextCraft
Goal: craft mossy stone brick slab.
Baseline
ScalingInter-RL
Thought To craft mossy stonc brick need 3 mossy stone bricks. should first craft mossy bricks using the available ingredients.
(Try to craft stone bricks)
Thought
more stone
Action I: craft mossy stone bricks using vine, 3 stone Observation Could not find enough
bricks to make mosay stone bricks. Action S: craft stone bricks using stone Observation
minceraft:mossy stone bricks
minecmaf:stone brickss
Thought seems don't have the necessury ingredients to craft mossy stone bricks directly.
crafl mossy cobblestonc and then use make mossy stone bricks.
(Try to craft mossy stone bricks)
Action 2: craft mossy cobblestone using cobblestone, vine Observation 2: Could not find enough items to craft minecraft:mossy. cobblestone
Thought 9: have crafted mossy brick. I need make mossy stone brick slab.
Action 9: crafl 2 mossy stome bricks using stone bricks, vine Observation Crafted minscraff:mossy stone bricks
(Try craft mossy cobblestones)... ...
(Try to craft mossy stone bricks)
Thought 7: Now that have both cubblesione and vine, can craft mossy cobblestone and then use make mossy stone bricks.
Thought 12: I have now crafted mossy stone bricks. can now caatt the mossy stone brick slab.
Action craft mossy cobblestone using cobblestone, vinc Observation 7: Crafted minecraftmessy cobblestone
mossy stone brick using mossy stone
bricks
Observation 12: Crafted 6 minecraft:mossy stone brick_ slab
(Try craft mossy cubblestones)...
(Task completed)
Thought appears
mossy cubblestone
despite having the necessary ingredients. This might due the game's inventory or the environment not having the required items. should chock my inventory and try again. Action 10: inventory
Obseryation 10: Inventory: [mossy cubblestone] (1)
(Try craft mossy brick slabs)
Thought 30: It seems am still unable to craft mossy stone brick slabs lack of'a direct recipe. Given
constraints, 1 should focus on what craft and to find different approach.
Action craft mossy stome bricks
using mosay
Observation 30: Could not find enough items to crift minccraft:nossy stone brick slab
(Reached muximum inferaction rounds)
Figure 21: comparative case from TextCraft.
42

---

Published as a conference paper at ICLR 2026
TextCraft
Goal: craft iron chestplate.
Baseline
ScalingInter-RL
Thought To craft an iron chestplate, need ingots. should first check have roquired ingots in my inventory.
Thought 1: To eraff an iron chestplate, need 72 iron nugget. Action I: get 72 iron nugget
Observation
nugget
Action inventory
Thought 2: need more irom ingol complete the iron chestplate.
carrying anything. Thought Since I don't have iron ingots inventory, need obtain them before crifting the iron chestplate. Iron ingots obtained by smelting However, since don't have fumaac iron noed to find iron ore in the environment and smelt it.
Observation Inventory:
Action
ingot using mugget
Observation Crafted I minecraf:trrn ingot
furacce.
Thought 3: have crafted I iron ingot får. I need more iron ingot complete the chestplate. Action 3: craft iron using nugget Observation 3: Crafted mineemfttiron ingot
Action get iron ore
Observation 2: Could not find iron Thought 3: couldn't find iron ore in the environment. need to explore or mine find iron orc. Let's assume iron ore and smelt it into iron ingots.
(Try
ingots)... ...
Thought 10: I have crafted 7 iron ingot so for. need more
Action 3: mine iron ore
iron ingot complete the iron chestplate.
Observation 3: Could not execute mine iron ore
Action 10: craft iron ingot using iron nugget Observation 10: Crafted I minccraftiiron ingot Thought have now crafted all ingot needed for iron chestplate.
(Try craft fron Ingots)
Thought
issue crulling
Action 11: craft 1 iron chestplate using 8 iron ingot Observation 11: Crafted minccraftiiron chestplate
ingots duc lack of iron nuggets, Since cannot find or obtain iron ruggets, will assume have the necessary resources and proceed craft Action craft iron chestplate using iron ingot Observation 8: Could not find enough items to crall minccrafttionn chestplate
chesiplate directly.
(Task completed)
(Try to craft iron chestplutes....
Thought 30: It there persistent issue with crafting the iron chesiplate, likely due to lack of necessary items in the environment or problem with the crifting command. Given the construints, assume
required resourves try more time to craft the iron chostplate. Action 30: cmftt iron chestplate using iron ingat Observation
find enough to cruft
minecraft:ronn chestplate
(Reached maximum interaction rounds)
Figure 22: comparative case from TextCraft.
43


Reviews:
[{"review_id": "2GbF6682ae", "reviewer": "Reviewer_M63s", "summary": "This work introduces a new framework for multi-turn LLM RL and a multi-staged training approach for long-horizon RL by progressively scaling interaction lengths. The results are strong relative to baselines, often outperforming much larger models.", "questions": "- Were any alternatives considered to the current linear schedule for the maximum number of interactions at each training step? In particular, I'd be curious to hear if this can be dynamically adapted (for instance allowing different RL rollouts to have slightly different limits and seeing which ones perform best).", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This work tackles challenging LLM RL tasks with real environment interactions, which is underexplored relative to the more common math reasoning. The proposed framework would be very useful for the LLM RL community.\n- ScalingInter-RL is well-motivated and a natural solution to the training instability of long-horizon RL\n- The results are impressive and demonstrate the strength of the proposed method/framework over strong baselines.", "weaknesses": "- The text in some of the figures is small relative to the main text (Figure 1, 5, 6, 7)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9QFQkCY2rj", "reviewer": "Reviewer_odrg", "summary": "This paper introduces AgentGym-RL, a novel and unified reinforcement learning framework designed for training LLM agents to perform long-horizon, multi-turn interactive decision-making. To address the challenge of balancing exploration and exploitation and to ensure stable optimization, the authors propose ScalingInter-RL, a method that progressively scales the agent-environment interaction horizon during training. Extensive experiments demonstrate that the framework and method deliver significant and consistent performance gains.", "questions": "1. **On the specifics of early-stage curriculum learning**: The ScalingInter-RL method emphasizes exploitation by initially restricting interaction turns. Could you detail how the agent's learning is concretely guided during these short-horizon phases? Specifically, were task-specific dense reward functions or explicit subtasks designed to ensure the acquisition of foundational skills, rather than just learning to avoid quick failure? Furthermore, was this design implemented through a universal framework applicable across all environments, or was it individually tailored for each distinct task type?\n\n- 2. **On generalization to more challenging RL tasks**: The experiments demonstrate strong performance in structured, language-mediated environments. How might the AgentGym-RL framework and the ScalingInter-RL approach be adapted to classic RL challenges that involve high-dimensional continuous state-action spaces?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- 1.The introduction of AgentGym-RL, the first modular, flexible, and end-to-end reinforcement learning framework specifically designed for training LLM agents in multi-turn, long-horizon decision-making across diverse real-world environments, filling a critical gap in the field.\n\n- 2.The proposed ScalingInter-RL method progressively increases the interaction horizon during training. This innovative approach effectively balances exploration and exploitation, leading to more stable RL optimization and enabling agents to develop richer behaviors and strategies.\n\n- 3.Extensive experiments demonstrate that the framework and method significantly enhance the capabilities of open-source models.", "weaknesses": "- 1. The proposed framework is built upon existing open-source projects like AgentGym and veRL. Its main contributions are engineering-oriented—adding diverse environments, integrating existing RL algorithms, and improving scalability and stability. These engineering improvements are certainly valuable and can significantly enhance usability and robustness; however, from a research perspective, the work represents a continuation and consolidation of existing efforts rather than a fundamentally novel contribution.\n\n- 2. The ScalingInter-RL method lacks detailed specification on how initial-stage learning is concretely guided. Crucially, it omits the design of reward functions or subtasks for short-horizon phases, making it unclear how the agent acquires foundational skills before scaling up exploration.\n\n- 3. The demonstrated success is more pronounced in structured, simulated environments with clear rules, while improvements in noisy, real-world settings are modest. This suggests the framework's effectiveness may be closely tied to environments that align well with its curriculum-based scaling approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wIhhAWPict", "reviewer": "Reviewer_gQCC", "summary": "This paper introduces AgentGym-RL, a modular open-source framework for training Large Language Model (LLM) agents on complex, multi-turn decision-making tasks using Reinforcement Learning (RL). The authors identify a key gap in the current open-source community: the lack of a unified platform for training agents from scratch in diverse and realistic environments. AgentGym-RL addresses this by providing a decoupled architecture—comprising environment, agent, and training modules—that supports various scenarios (e.g., WebArena, Deep Search, SciWorld) and mainstream RL algorithms.\nTo tackle the training instability common in long-horizon tasks, the paper proposes a curriculum learning-based strategy named ScalingInter-RL. This method begins training with shorter interaction episodes to establish a foundational policy and then progressively increases the episode length. This approach enables stable training and deeper exploration. Extensive experiments across 27 tasks demonstrate that a 7B parameter model trained with this method can match or even surpass the performance of much larger proprietary models like GPT-4 and Gemini Pro, suggesting that scaling interaction can be more effective than simply scaling model parameters.", "questions": "1. Sensitivity of ScalingInter-RL: Could the authors provide more details on the robustness of the ScalingInter-RL method? For instance, how does performance vary with different curriculum schedules (e.g., faster/slower expansion of the interaction horizon, different starting points)? A related ablation study would be highly informative.\n2. Instability in Long-Horizon Training: Could you elaborate on why training is inherently more unstable in long-horizon settings? Providing related experimental or theoretical analysis would be a valuable contribution to the community.\n3. Overhead of HTTP Communication: Is there an analysis of the time overhead associated with the environment client communicating with the environment server via HTTP? A comparison with a fully collocated setup would be very insightful.\n4. Interaction with Different RL Algorithms: The paper compares GRPO and REINFORCE++, but the main benefits of ScalingInter-RL appear to be demonstrated with GRPO. It would be beneficial to know if the stability and performance gains from ScalingInter-RL are consistent across other RL algorithms, such as PPO. This would help confirm whether the method is a general strategy for stabilizing long-horizon training, rather than a technique tightly coupled with the specific mechanics of GRPO.\n5. Computational Overhead: How do the training time and computational costs of ScalingInter-RL compare to a baseline with a fixed, long horizon? By avoiding unstable training phases, does this staged approach lead to more efficient use of computational resources, potentially achieving a high-performance policy with faster overall convergence?\n6. Comparative Failure Case Analysis: The case studies in Appendix F are very helpful. It would be even more compelling to see a direct comparison of failure modes on a complex task between a baseline agent (trained with a fixed short horizon) and the ScalingInter-RL agent. Does ScalingInter-RL specifically help mitigate certain types of errors, such as repetitive loops or failure to explore deeper parts of the state space?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The development of a unified, open-source RL framework for LLM agents is a valuable contribution to the research community. It directly addresses a practical need and has the potential to significantly lower the barrier to entry for agent research, thereby fostering reproducibility and continued innovation.\n2. ScalingInter-RL is an intuitive yet powerful method that effectively stabilizes long-horizon RL training. The paper provides a clear motivation for this curriculum-based strategy (as illustrated in Figure 4), showing how it adeptly balances the trade-off between exploration depth and training stability. Its effectiveness is validated by solid experimental results.\n3. The experimental design is a major highlight of this work. The evaluation spans five diverse and challenging environments and comprehensively compares the proposed model against a wide range of strong open-source and state-of-the-art proprietary models. This thorough evaluation lends significant credibility to the paper's conclusions.", "weaknesses": "1. Limited Analysis of the ScalingInter-RL Schedule: While the method is shown to be effective, the paper offers limited analysis of the curriculum schedule itself. The model's final performance is likely sensitive to hyperparameters such as the initial number of interaction steps, the increment size (δh), and the frequency of phase transitions (Δ). A sensitivity analysis or ablation study on these hyperparameters would provide deeper insights into the method's robustness and offer guidance for its application in new domains, thereby strengthening the paper's claims.\n2. Rigor of the Exploration-Exploitation Argument: The paper states that by initially limiting the interaction horizon, the agent focuses more on exploitation, and as the horizon expands, it is encouraged to explore more deeply. This description could be more rigorous. Even in the early stages, exploration is necessary to discover better trajectories for policy improvement. It would be beneficial to contrast this with classic exploration techniques in RL, such as curiosity-driven methods like RND or NGU, to better contextualize the exploration-exploitation trade-off within ScalingInter-RL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xjNuHXrrWO", "reviewer": "Reviewer_yP1T", "summary": "This paper presents AgentGym-RL, an open-source framework for training LLM agents via RL in multi-turn decision-making tasks. The main contributions include: a modular framework supporting diverse environments and mainstream RL algorithms, and a staged training approach named ScalingInter-RL that progressively increases interaction horizons to achieve stable long-horizon RL training. The authors demonstrate that 7B models trained with their method achieve performance comparable to or surpassing commercial models like OpenAI o3 and Gemini-2.5-Pro across 27 tasks.", "questions": "1. Could the authors provide a curriculum sensitivity study? What happens with different progression rates or starting horizons of interactions? This seems critical but is unexplored.\n2. What are the computational costs? The paper doesn't discuss training time, GPU hours, or cost comparisons with baseline approaches, making it hard to assess practical viability.\n3. Are there separate studies investigating whether the improvement mainly comes from the framework or from the ScalingInter-RL method? Would ScalingInter-RL work in other frameworks? Would the framework benefit from other training approaches?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear and meaningful. The paper tackles the genuine problem of lacking unified RL frameworks for training LLM agents across diverse environments, which is important for the community.\n2. As an open-source framework, this work provides good engineering with modular design, supporting multiple RL algorithm for diverse environments as well as various practical features.\n3. The framework demonstrates strong empirical performance. It achieves performance matching or surpassing much larger commercial models with 7B parameters, which is impressive, especially with the 33.65 point average improvement.\n4. The evaluation is comprehensive, which tests across 5 scenarios with 27 tasks. Such evaluation provides good coverage of different agent capabilities.", "weaknesses": "1. The algorithmic novelty is limited, although the effectiveness is acknowledged. ScalingInter-RL is essentially curriculum learning applied to interaction horizons. This is a straightforward adaptation rather than a novel algorithmic contribution. The progressive scaling from short to long horizons is intuitive and incremental, which lacks theoretical justification.\n2. Given that the key to ScalingInter-RL is the curriculum scaling of interaction turns, there is no ablation on the curriculum schedule. Why these specific horizon increments?\n3. I could not find quantitative report and analysis of computational costs or training efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)