Generalizable End-to-End Tool-Use RL with Synthetic CodeGym
Authors: Weihua Du
Hailei Gong
Zhan Ling
Kang Liu
Lingfeng Shen
Xuesong Yao
Yufei Xu
Dingyuan Shi
Yiming Yang
Jiecao Chen
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Agent
Reinforcement Learning
Data Synthesis
Generalizability
Primary area: foundation or frontier models, including LLMs
TL;DR: We transform static coding problems into interactive multi-turn tool-use environments, enabling LLM agents to learn through reinforcement learning and improve their generalization ability for OOD tasks.

Abstract: Tool-augmented large language models (LLMs), hereafter LLM agents, leverage external tools to solve diverse tasks and interface with the real world. However, current training practices largely rely on supervised fine-tuning (SFT) over static trajectories or reinforcement learning (RL) on narrow tasks, which generalize poorly beyond development settings and lead to brittleness with new tools and unseen workflows. Because code execution reflects many structural patterns of real-world workflows, we use coding problems as a structured substrate to build tool-use agent training environments with diverse task configurations. To this end, we introduce **CodeGym**, a scalable framework that synthesizes diverse, verifiable, and controllable multi-turn tool-use environments for agent RL, enabling LLM agents to explore and master various workflows actively. CodeGym converts static coding problems into interactive environments by extracting atomic functions or logic into callable tools, yielding verifiable tasks that span various tool-execution workflows. Models of varying sizes and chain-of-thought configurations trained in CodeGym exhibit consistent out-of-distribution generalizability; for example, Qwen2.5-32B-Instruct achieves an absolute accuracy gain of 8.7 points on the OOD benchmark $\tau$-Bench. These results highlight CodeGym as a step toward scalable general-purpose RL environments for training tool-use behaviors that align with real-world agent workflows. Our code is publicly available at https://github.com/StigLidu/CodeGym.

Published as a conference paper at ICLR 2026
GENERALIZABLE END-TO-END TOOL-USE RL WITH SYNTHETIC CODEGYM
Weihua Du Hailei Gong", Zhan Ling", Kang Liu", Lingfeng Shen", Xuesong Yao², Yufei Xu², Dingyuan Shi?, Yiming Yang Jiecao Chen
'Lanuuage Technologies Institute, Carnegie Mellon University 'ByteDance Seed
weihuadlcs emu, edu, w yimingecs. cmu. edu, jiecao. www.@hytedance..commm
ABSTRACT
Tool-augmented large language models (LLMs), hereafter LLM agents, leverage external tools to solve diverse tasks and interface with the real world. However, current training practices largely rely supervised fine-tuning (SFT) over static trajectories or reinforcement learning (RL) on narrow tasks, which generalize poorly beyond development settings and lead brittleness with new tools and unseen workflows. Because code execution reflects many structural patterns of real-world workflows, we use coding problems as a structured substrate to build tool-use agent training environments with diverse task configurations. To this end, we introduce CodeGym, a scalable framework that synthesizes diverse, verifi- able, and controllable multi-turn tool-use environments for agent RL, enabling LLM agents explore and master various workflows actively. CodeGym con- verts static coding problems into interactive environments by extracting atomic functions or logic into callable tools, yielding verifiable tasks that span various config-
tool-execution workflows. Models of varying sizes and urations trained in CodeGym exhibit consistent -itt---distribttion generalizabil- ity; for example, Swwen..5-32B-Inntte points on the OOD benchmark r-Bench. These results highlight CodeGym as a step toward scalable seneral-purpose RL environments for training tool-use be- haviors that align with real-world agent workflows. Our code is publicly available at hhttps://github com/ StiggidddCooeeeeeee CoddeeGym
achieves absolute accuracy gain of 8.7
INTRODUCTION
Large language models (LLMs) have exhibited remarkable capabilities in complex logical reason- ing, code generation, and instruction following (Jacch et al. 20244] Liu al.) 2024a) [eed et al. 2025 Yang et al.) 2025) Seed 2025b Comanici et al. 2025). but their capabilities are limited by static parametric memory (Cao al. (203bb aj Schick et al., 2023). A new paradigm, tool- augmented LLM agents, overcomes these limits by granting LLM access to external resources, such as databases (Liu al.) 2024bb Qian al., [024 Prabhakar et al. [2025). search engines (Parisi) eet al. 2022 Lu et al. 2023). and code executors (Li et al.]
2023
Wu et al. 2025). enabling them
to act with expanded problem-solving abilities (Ma et al.
2024 et 2025) and interaction
capacities (Qin et al. 2023 Yao et al. 2024).
Standard pretraining corpora lack sufficient high-quality agent interaction data, such as tool-use and workflow traces, leaving LLM agents brittle (Fu et al. 2025b). To mitigate this, previous work has constructed agent tasks and generated agent trajectories for supervised fine-tuning (SFT) (Zhou et al. 2023 Wang et al. 2024a). Although such construction improve performance designed benchmarks, the resulting trajectories often follow hand-crafted patterns and explore limited envi- ronments and task configurations leading poor generalization to distribution shifts, such new tools or unseen workflows (Huang et al. [224]] Guo et al. 2024 Li et al. (204)). This calls for train- ing environments that better capture the diversity and complexity of real-world agent workflows.
Work done BytelDance Seed

---

Published as a conference paper at ICLR 2026
Coding Problem
<CodeGym/> Environment
LLM Agent
Problem Description
Task Configuration (Unit Test)
Prompt
Coding Solution:
Initialize Env
Task Description
Functions
Action Space
Observation Space
Action Docstrings
def furt (params1)) :
[.]]
Atoric Functions
Action Feedback
Action
Multi-Turn Tool Use
def funa(ppramm22:
Funz (perome))))
Ernor Massaaee
Action Observel)
[--]
Formelia
Fun2 (peroms2))
Reward Info
Obe [Surcees]]
Main Procedure
def rain(input):
Basic Actions
Observua
Action 2: Fun Iparame-ints Statel
Funl (irput)
(bserve()
Dos [Buseas Staterx
y Fun2(x) return
Done ( (answae r)
Actian 3: Fun2/params-Noraa
Sanity
Done(answer)
Obs 3: (Faiee) Wrong Arguments
Retrieve
Compliaton Format
Verifiable Reward
Obs 4: [Bucoass] State:y
Action 4: Funzlporammmss
Web Resource
Running Tima Memory
1, annwer correct
Reward
0,, otherwise
Action Danejanswar-y"
3 Solution Exictence
Obs S: [Bucoass] Reward=1
Overview of CodeGym. We transform coding problems into interactive environments to train LLM agents. (Left) We extract atomic and reusable functions or logic from coding solutions to construct interactive environments. (Middle) CodeGym enables agents to solve tasks via multi-turn tool calls, with environment correctness verified automatically. (Right) The resulting environments support scalable RL training, improving robustness and generalization of LLM agents.
Figure I:
Beyond SFT, reinforcement learning (RL) shows promise in improving generalization (Chu al. 2025). Through active exploration and interaction with external environments, RL enables LLM agents exploit feedback from tools and dynamic contexts, learning not only from correct trials but also from failures, thereby gradually improving and adapting to novel scenarios, rather than relying solely on static teacher trajectories (Zheng al. 2025; Le et al. 2022). Recent work introduces RL training environments tailored to specific agent domains. such as coding assistants (Pan et al.) 2024) and information search (Chen et al.] (2025). However, these setups only focus on narrow tasks, limiting the potential of RI. generalize (Cobbe et al. (019).. A scalable ceneral-purposs RI. environment for improving L.LM agentic capabilities remains lacking.
To bridge these gaps, we introduce CodeGym, a framework for synthesizing large-scale, diverse, and verifiable multi-turn tool-use environments from coding problems. Code inherently embodies diverse and rigorous execution logic and naturally reflects many of the structures found in real- world workflows, making coding problems natural foundation for constructing rich tool-use envi- ronments. Using this property. CodeGym ingests raw coding problems and exploits their inherent exccution semantics to synthesize interactive environments. Reusable atomic functions and logic abstracted into callable tools, which L.MM agents invoke interactively to solve tasks instead of directly writing the full code. CodeGiym enables LLM agents to explore and adapt to unseen envi- ronments interactively rather than relying on static demonstrations, Since code encodes diverse logic and functionality, the resulting environments vary widely, not only in available tools and workflow structures, but also in the forms of tool-based reasoning agents must employ succeed.
Reinforcement learning in CodeGym exposes agents to a wide range of environments and task con- figurations, fostering adaptation strategies for real-world agent applications. We apply CodeGym to train language models of various sizes and chain-of of-thought (CoT) styles, and the trained models achieve competitive in-domain performance and, importantly, demonstrate notable generalization to out-of-distribution (OOD) settings. For example, Qwen2.5-32B. 35-32--Instrucl improves accuracy by 8.7 points r-Bench (Yao et al. 2024). These findings suggest that CodeGym promotes transferable interaction strategies, avoiding overfitting specific tasks. Our contributions are threefold:
We introduce CodeGym, scalable pipeline that transforms static coding problems into explorable and verifiable multi-tune tool-use environments.
CodeGym contains large suite of tasks with various logic and tool sets, ensuring broad trajectory coverage while providing stable and rigorous feedback.
We show that reinforcement learning on CodeGym significantly improves out-of- distribution generalization for LLM agents, highlighting the value of CodeGym for gen- eralizable agent training.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Equipped with external tools, LLMs extend their capabilities beyond intrinsic language modeling, not only improving factual reasoning through knowledge search or re- trieval (Qin et al. 2023) and program-aice aided computation (Gao et al. 2023a), but also enabling direct interaction with the world in domains such as coding (Wang et al. 2024b), customized services (Yao
LLMs as Tool-Use Agents
et al. 2024), robotic control (Ahn et al. 2022), and scientific discovery (M. Bran et al. 2024).
Synthetic Environments for LLM Agent Training For agent applications, LLMs often lack domain-specific training data, leaving them insufficiently grounded and prone to erroneous ac- tions (Qu et al., 2025). Synthetic environments have thus emerged as a promising means of pro- viding controlled, Nomain-aligned ScienceWorlde (Côté et al., 2018, Shridhar et al.) 2020 Wang et al. 2022), offered interactive text- based environments for language models to enhance instruction following and multistep reasoning, although their domain gap limits real-world transfer. More realistic benchmarks now include Web- Shop (ao et al. 2022) for online shopping, SWE-Gym (Pan et al. 2024) for code debugging, and BrowseComp- Plunp-Plus (Chen et al. (2025) for deep web search, etc. In parallel, resources such as Tool- Bench and T-Eval (Qin et al. [003] Chen et (003)) provide large-scale datasets and fine-gred -reiied evaluations of tool use capacity, but lack the evolving states and long-horizon horizon interactions of true environments. Despite these advances, broadly applicable general general-purpose tool-use environments remain scarce.
supervision. Early efforts, such as TextWorld, ALFWorld, and
Reinforcement Learning with Verifiable Reward (RLVR) Reinforcement learning has proven effective for training LLMs when rewards are verifiable, such as mathematical reasoning and code generation (Shao al. 2024 Jaech et al. 2024 He et al. (025). Based on PPO (Schulman et al. 2017], variants such as GRPO and DAPO (Shao et al. 2024 Yu et al. 2025) improve stability and efficiency during training. Tool-augmented RI. further enables models practice about when and how to invoke external tools, such as for retrieval (Li et al.) (2025) or numeric reasoning (Singh
al. 2025 Feng al., 2025). Nevertheless, scaling tool-supported RL and managing large training environments remain open challenges (Jiang et al. 2025).
3 CODEGYM
We introduce CodeGym, large-scale synthetic multi-turn tool-use environment dataset constructed from extensive coding problems available online (Section B.2). As shown in Figure [ we synthesize various agent tasks and interactive environments to support reinforcement learning for LLM agents, exploring ways to improve agent capabilities and generalization. CodeGym encompasses thousands of tools, various patterns of tool-use logic, a low-latency execution environment, and verifiable reward mechanisms. Furthermore, CodeGym is designed for scalability: Our generation pipeline (Section 3.3) can systematically convert wide range of coding tasks into interactive environments with a niorrous verification process, ensuring both the stability and correctness of environments. Finally, a series of filters, such as difficulty and trajectory complexity, applied select high- quality environments for LLM agent reinforcement training (Section 3.4
3.1 INSIGHTS
The construction of CodeGym is motivated by a key insight: code inherently embodies rigorous execution logic, which is similar to real-world workflows. For example, a loop that continues un- til condition is satisfied mirrors ilerative approval rounds in complex decision-muking workflows. Taking advantage of this property, we transform coding problems into structured tool-use environ- ments where agents use tools to solve tasks. This design bridges the gap between static datasets and interactive training, offering the diversity of real-world workflows for reinforcement learning.
Figure 3 illustrates how an interactive task is transformed from coding problem. The original problem is 'Finding the number closest to K in a sorted list of length N. From the corresponding coding solution (see Appendix C.I). extract three atomic actions: (1) observe. which returns the array length N together with the target K; (2) look look_up_____ _or_____po_______ which returns the element at index i; and (3) done, which submits the final answer. These actions form the tool set available to
3

---

Published as a conference paper at ICLR 2026
Coding Problem
(1)
(2)
Env Solution
Solution Function
CodeGym Env
Functions
Outputs
CodeGym Env
Unveniied
Verfi odi
Unit Test Inputs
Unit
(4) Compare
Served Task Configurations
Error-Prone Step
Coding Solution
Unit Test
Unit Tests
Step
Synthesis
Outputs
Verification
Figure 2: Pipeline for CodeGym Environment Generation. Coding problems reformulated into interactive environments by extracting tools, generating candidate solutions, and validating them with unit tests. The environment is deemed valid if any candidate solution passes all tests, and the resulting unit tests serve as task configurations for RL training.
the agent. The environment is initialized with specific task configuration that is hidden from the agent. The agent then interacts with the environment by invoking tools and ultimately produces an answer, whose correctness is evaluated, and binary reward assigned.
More broadly, program execution can be reimagined as a structured action sequence which agents must not only master individual tool calls but also compose them into coherent workflows. This compositional nature, coupled with the verifiable outcomes of coding tasks, makes CodeGym par- ticularly well-suited for cultivating general-purpose tool-use capabilities and robust agent training.
3.2 RESOURCE COLLECTION
Coding tasks are widely available online, and this work focuses primarily collecting com- petitive programming problems. We use the
Problem Description:
Finding the number closest to Knn sorted list of length N.
Available Action List:
KodCode dataset (Xu al., 2025) and select the category of Coding Assessment Questions as our raw corpus. Each coding problem con- tains task description paired with its corre- sponding solution code. Because code formats vary, we utilize an L.M standardize coding solutions into unified format.
del observe))
retwine me amay length ond the rarget
def checoly lnd) str
neturme vironnt
def donelons:
Example Workflow: Configuration (hidden agent]:
Arssyy
3.3 CODEGYM GENERATION PIPELINE
Amr (2. 52 20) h=s
forget K R. trutn_tnithes
Our generation pipeline (see Figure 2 consists of two complementary stages: Gym Synthesis and Gym Verification. In the synthesis stage, we extract reusable code logic from programming solutions and rewrite them into callable tools, ensuring modularity and clarity. However, be- cause large-scale generation is prone errors, we introduce verification stage that systemati- cally validates correctness and solvability. This two-step design ensures that the resulting envi- ronments are diverse and reliable.
Agent Trajectory:
checkr2l Anr2)
checkl0l Am00 =2
checktll 'Ax +
cone(9] + "ubmit anower to the envronment"
Reward Function:
for correct enswers otherwise.
CodeGym Environment Example. Given the problem description and the action list, the agent interactively solves the task and receives binary reward after submitting the answer.
Figure 3:
3.3.
GYM SYNTHESIS
We extract reusable, atomized code logic functions from programming solutions and convert them into a library tools. A tool may standalone function, a calculation utility, or frequently oc- curring code fragment (e.g., loop body). Extraction and rewriting are performed by prompting an LLM, which asks the LLM to synthesize tools with precise documentation (functionality, pa- rameters, and examples) conditioned on the source task and code solutions. Although examples
We Seed-1 1.6--11.6111 6-Thinning Seed 2025a for the CodeGym environment generation pipeline.
4

---

Published as a conference paper at ICLR 2026
generated in the synthesis step, these are withheld from the agent-facing documentation during the training stage to encourage learning through interaction and acting based on feedback.
To support reinforcement learning, synthesize environments in the OpenAI Gym format (Brock- man et al. 2016). In detail, each CodeGym environment is defined as a POMDP:
3 = (S. A,T, O),
where the state S encodes task-specific variables, the action space A consists of both generic func- tion calls (e.g., Observe, Done) and domain-specift tools, transitions T execute the correspond- ing functions, and rewards R are sparse, assigned only upon termination by comparing the submitted answer the ground truth. To discourage shortcut solutions, Observe reveals only a partial state (e.g., some task inputs are not directly accessible). reset initializes the environment with a prede- fined unit test input. The reward function returns if the agent's final answer matches the unit test output, and 0 otherwise.
This unified design provides a flexible template for incorporating various coding tasks into RL train- ing, ensuring consistency across environments while encouraging tool use and exploration. By providing a one-shot example, the LLM can amazingly follow all the format instructions in most CodeGym synthesis inferences. Details of the CodeGym environment template and the synthesis prompt are provided in Appendix C.2 and Appendix
C.3
respectively.
When used during the agent training stage, the environment exposes the task description and docu- mentation of the available tools. Agents are expected to adapt their actions based on feedback from environments (observations and error messages). Example agent prompts are listed in Appendix CC.4
3.3.2 GYM VERIFICATION
During the synthesis process, we identify two primary enrors with respect to generated environments: (1) Correctness Error, where the environment may encounter compilation failures, timeouts, or out- of-memory issues; and (2) Solvability Error, where the set of actions provided by the environment is insufficient for any agent to solve the task.
To filter out faulty environments and verify solvability, we first synthesize a collection of unit test inputs that span multiple difficulty levels and corner cases (see Appendix [D2] for details). The ground truth coding solution is then used to produce the corresponding unit test outputs. Next, leveraging the detailed tool documentation provided by the CodeGym environment, plus example outputs of tools to ensure correct grammar, we prompt an LLM generate solution functions (i.e., writing code programs that call tools to solve the environment; refer to Appendix [..I]. Although the generation of solution functions itself error-prone, we can employ the pass@ strategy: We generate K 10 candidate solution functions, and if any of them successfully passes all unit tests within the specified time and memory limits, the CodeGym environment is considered solvable. In this case, the unit tests further used task configurations for environment initialization at the RL training stage. We then denote the solution function that passes all unit tests as the oracle solution.
3.4 QUALITY CONTROL
Ensuring data quality is essential for RL training. To select high-quality task configurations from the large CodeGym dataset, we apply two filtering mechanisms: Tool-Use Complexity and Difficulty.
Tool-Use Complexity We require task configurations to exhibit non-trivial patterns of tool use, where complexity reflects both the number and the variety of tool calls. Specifically, we use oracle functions to calculate the number of tool calls needed to solve the task and filter out task configu- rations with fewer than Tmin = 10 tool calls to avoid trivial solutions and more than Tmax = 256 to remove repetitive tool call patterns, thus improving the efficiency of RL training. Moreover, to ensure that complexity does not degenerate into repeated use of single tool, we also require environments to contain least 4 distinct tools.
Tool-Use Difficulty Task configurations should not be too easy for agents to solve. To measure difficulty, we use the pass rate as a metric. Specifically, we evaluate each task configuration 4 times with 3--3B2.--32BB1 Instructt and retain only those with accuracy no greater than 25%.
5

---

Published as a conference paper at ICLR 2026
After filtering, obtain dataset of more than 80k task configurations with 13k environments. Figure presents statistics the filtered dataset regarding the number of tools and steps. The average numbers of tools and steps 6.52 and 44.07, respectively. Table 3 shows comparison between CodeGymm and previous agent training works, where CodeGym has the largest number of environments and task configurations compared to other agent training works.
Histogram
Histogram of Step Numbers (binned by
25000
20000
25000
Fequency
15000
20000
10000
15000
10000
5000
10
11
12
13
24
15
20
60 80 120 160 180 220 260
Tool Nu.mbn
Strp Number Range
Figure 4: CodeGym Statistics. The average numbers of tools and steps solve tasks 6.52 and 44.07, respectively, indicating that CodeGym encompasses diverse tools and complex logic.
3.5 DIFFICULTY AUGMENTATION
Long-CoT models sometimes solve tasks by reasoning alone they receive complete informa- tion, bypassing tool calls. To discourage this behavior, we augment the task configurations used for environment initialization increase the difficulty of pure reasoning (see Appendix Do3 details), yielding more challenging training set. In practice, we train long-CoT models on the augmented training set and short-CoT models on the original set.
TRAINING FRAMEWORK
Initialize IDnvID, Configuration, on..
Buffer
Repiny
prompt
action
Retrieve
RL Learner
Controller
Rollout
trajectory
Rollout Worker
observation
Service Ports
Environment Server (CPU-baund]
CodeGym Database
Instancces
Training
Rollout Worker N
action
"rajectory
Reward
Rollout Pipeline
Figure 5: RL Training Pipeline for CodeGym. A server provides centralized control of environ- ments, and each rollout process is allocated to service port. The rollout workers send actions to the corresponding service ports and receive observations. The rollout controller sends commands to initialize the environments and receive reward signals to form the replay buffer.
CodeGym is designed for agent reinforcement learning. To enable high throuhhruughput rollouts, implement a distributed rollout framework with CPU-bound environment server (Fig. 5). At the beginning each training epoch, the environment server receives initialization commands that spec- ify environment IDs and task configurations. Then it retrieves the corresponding environment from the CodeGym database, launches it, and establishes a dedicated service port for communication. Each rollout process is connected to of these ports, and the tool calls generated during rollouts are transmitted immediately to the server. The resulting responses are appended to the trajectory. To avoid blocking caused by repeating tool calls, allow tools be called most Tiax times in each trajectory.
Upon completion of rollout, the server computes the reward signal and returns it the replay buffer for aggregation. By decoupling the GPU-bound rollout process from the CPU-bound environment server, the framework supports stable and highly concurrent RL training.

---

Published as a conference paper at ICLR 2026
4.1 THIAL-THEN -OEEEN--VERRITEE MECHANISM
During training, the tool calls generated by LLMs can be unpredictable, particularly in the early epochs. To prevent server crashes caused by erroneous calls and bound per-step latency, adopt a trial-then-overwrite mechanism: Upon receiving a tool call, the server first serializes (pickles) the enviromment state, then executes the call in a subprocess against the serialized snapshot. the subprocess completes successfully within the time limit, we commit the resulting state back to the original environment. Otherwise, the original environment remains unchanged and returns an error as the observation. This mechanism ensures robustness during training.
5
EXPERIMENTS
Train Reward vs. Step
In- no-Domminn Validation Reward vs. Step
Rowarr
3.5
Owen1255 2- rotrud
Owen2.3 Ii-rsttu
(w442:5:344 intt
Qwe12.5 24 mnthuct
Owerz:3 228 and
Qwendd 325 inkruut
Owenz.5 5: 729 instrct
Qwenz. 723 intruct
20
3M
20
200
Training Stop
Training
Figure 6: Training Curve. Average reward during training on both the training and in-domain vali- dation environments. With binary rewards, the reward is equivalent to accuracy. The similar reward trajectories on training and validation indicate minimal overfitting. Larger base models achieve higher performance. For models smaller than 32B, three random seeds are run. The solid lines denote the mean reward across multiple random seeds. The shaded regions represent the sample standard deviation (#:1 std) across seeds.
Table Main Results. We report the performance of CodeGym-traiin trained models on held-out bench- marks spanning tool-use (7-bench and r²-bench), multi-turn interactions (ALFWorld), and rea- soning (ZebraLogic and MMLU-Pro). Models of varying sizes and CoT patterns are evaluated, and training CodeGym improve overall performance across benchmarks. Experiments = 0.7 and top-p 0.95, and results are obtained by averaging 5 inference runs per model.
Categories
Tool-Use 7-retail
Multi- Turn AW
Reasoning
Benchmarks
r-airline
-bench
ZL
MMLU-Pro
Avg.
Short-Co7 Models
Qwen2,5-7B-Inee
1288
4.5
14.9
43.6
11.3
57.9
24.2
Dwen2.5-7B-Codd00
17.3(4.51)
7.6(3.1))
15.5(0.61)
$1.3(7.71)
12.6/1.31)
57.6(0.3J)
27.0(2.81)
Qwen2.5- 1..55-144-nnnstt
17.6
32.0
20.9
59.2
19.6
66.3
35.9
Qwen2.5-14B-c 144m-odyym
21.3(371))
39.2/7.21)
19.9(100))
72.8(13.6f)
22.3/2.71)
67.2(0.91)
40.5(4.61)
Qwen2,5-3288- 328--nstuut
26.8
41.4
24.7
66.8
24.2
70.0
42.3
Qwen2.5 32B CodeGym
31.((444)
54.4/13.01)
30.7(6.01)
80.8(14.01)
29.004.81)
71.2(1.21)
49.6/7 (1)
Owen2.55-7B-IInstmaud
25.2
49.2
22.6
80.4
27.6
72.2
46.2
Qwen2:5-72B-CodeGGm 31,2(6.01)
57.007.81)
25.8(3.21)
82.8(2.41)
31.5(3.91)
73.3(1.11)
50,3(4.11)
Models QwQ-32B
Lang-CoT Models
37.6
37,7
26.1
62.4
$4.2
QwQ-32B-Codo0jg 43.2(5.61)
43.0(5.31)
30.704.6))
64.4(2.01)
76.6(3.3
81.4(0.0
56.6/2.41)
5.1
SETUP
We utilize CodeGym to train a diverse range of language models. For short-CoT models, we eval- uated the Qwen2.5 series (Qwen) 2025) with multiple model sizes (7B, 14B, 32B, and 72B). For long-CoT models, QwQ-32B (Team) 2025) is tested. For the reinforcement learning algorithm, we apply GRPO (Shao al. 2024) to train models with batch size of 512 8 (512 task configu- rations per step with each sample 8 times). Training continues until the training reward approaches saturation, which indicates diminishing returns from further updates. As shown in Figure models

---

Published as a conference paper at ICLR 2026
with greater than 32B reach a performance plateau with 100 steps. In contrast, the 72B model exhibits faster reward stabilization due to its stronger capacity, requiring only 50 steps to reach saturation. For models smaller than 32B, we train with three different seeds to evaluate stability. For larger models, we report results from a single seed due to computational limitations. Detailed hyperparameter settings are provided in Appendix F.T
5.2 TESTBEDS
We evaluated models on both the in-distribution validation set and the held-out (OOD) benchmarks. This distinction allows measure both in-distribution performance and out-of-distribution bustness. For Held-in validation, we split the CodeGym dataset into a training set and a validation set. The validation set comprises 500 CodeGym environments unseen during training, each with no more than two task configurations, for a total of 972 evaluations. For Held-out (OOD) benchmarks, we categorize the benchmarks along three distinct axes of generalization: (i) domain (tool use), (ii) pattern (multi-turn interaction), and (iii) skill (reasoning). Models are evaluated on representative benchmarks from each category listed below. Importantly, these OOD tasks are semantically dis- tinct from CodeGym's 's synthesized environments. Multi-turn tasks follow the standard ReAct (Yao) ct al. 2023 protocol, while single-turn question answering uses CoT (Wei et al. 2022) prompts.
Tool use: r-bench (Yao et al. 2024) and 72-bench (Barres al. 2025). where LLM agents interact with external tools and communicate with a user to satisty the user's request while following system instructions, rather than programming or executing code. We use GPT- 4.1 as the user simulator.
Multi-turn interaction: ALFWorld (Shridhar et al., 2020), which places agents in text- based embodied environments involving household navigation, object manipulation, and long-horizon stateful interactions that are fundamentally separate from coding tasks. We sample 50 problems from the ALFWorld evaluation dataset.
ZebraLogic (Lin et al., 2025) and MMLU-pro (Wang et al., 2024c), verify that performance in standard logical and commonsense reasoning tasks does not degrade. We sampled 200 puzzles from ZebraLogic and 1,000 problems from MMLU-pro.
Reasoning:
5.3 RESULTS
Figure 6 presents the training reward curves per step and the in-domain validation results of the Qwen-2.5 series models (since QwQ uses the hard training set, the curves are not comparable and the QwQ results are shown in Figure [6]. The reward metric is equivalent to accuracy be- cause of its binary definition. In the training set, all base models start with relatively low reward, but improve steadily, with larger models consis- tently outperforming smaller ones. Repetition experiments in small models confirm the stabil- ity of training during the initial 100 steps. In the in-domain validation set, although the envi- ronments differ from those used in training, we observe similar trends, suggesting limited over- fitting in training environments.
Avg. Tool Round vs. Step
Ave. Tool Call Round
limothims, widdendi
Arg. Otece Stille hound 43.03
Deand 5 122 22sirdieet
Oward. a28iiss
Training Step
Figure 7: Evolution of Tool Call Behavior Dur- ing Training. The average tool call number keeps increasing, suggesting improved identification of agent workflows and closer adherence them.
Table summarizes the -uito--diitribution (OOD) performance of trained models. For Short-CoT models, we observe consistent gains across all categories: tool-use scenarios, multi-turn interac- tions, and reasoning tasks. The gains in the first two categories are more pronounced because of the similarity between the synthetic environment workflows and those of the target tasks. These find- ings yield two takeaways: (i) training on CodeGym improves the generalizability of LLMs to unseen agent workflows, and (ii) the intrinsic complexity of the workflow logic in CodeGym training envi- ronments also yiclds gains in general reasoning ability. Moreover, we found that the larger models benefit more from training in CodeGym compared the smaller models OOD benchmarks. For
8

---

Published as a conference paper at ICLR 2026
Before RL (0.e0eeen22555222-1I1mttt
After RI. (Qwen-225533D--onddCymm
THOUGHT:
THOUGHT:
Locate tomato. The fridge is the most likely place. After finding heat it and discard it.
Goal: heat tomato and discard it.
Plan:
ACTION:
Search likely locations for the tomato
go to fridge
Take the tomato once found
Heat it using the microwave
Dispose of it in the garbage
Before acting, agent anticipates possible loca- tions and action constraints, deciding to check the fridge first the most likely source.
ACTION: go to fridge
Figure 8: Qualitative comparison on an ALFWorld task. After training in CodeGym, the agent performs explicit multi-step planning before acting, whereas the base model selects actions with minimal anticipation. Both models choose the same first action in the sample environment. The outputs shown here are summarized, and the full model outputs are provided in Appendix E.4
example, 10022---32--nsttrue achieves an average improvement of +7.3, whereas Qwen2.5-7B. Instruct achieves only +288. This gap suggests that larger models exhibit stronger generalizability instead of memorization.
For long-CoT models. which are heavily tuned for reasoning tasks, RL on CodeGym slightly reduces reasoning performance due to OOD training. However, these models show sub- stantial gains in tool-use scenarios and multi- turn interactions. These results motivate ex- ploring ways to combine reasoning objectives with CodeGym training, as this may provide complementary benefits to both reasoning ac- curacy and agent abilities.
Performance Comparison
81.0
Base Model
80
Distiliation (Ours)
70-
65.6
49.6
42.3
30.1
20
Figure summarizes how the average number of tool calls made by LLM agents evolves dur- ing training. The count increases steadily over the first 100 steps, indicating that agents are learning to execute longer and more structured procedures. At the same time, the gap between the LLMs and the oracle in tool call counts nar- rows, suggesting better identification and ad- herence to multi-step workflows. An additional analysis of trajectory length provided in the Appendix model,
10
In-domain
Avg. OOD
Avg.00D Tasks
Figure 9: Performance of Models Trained by Different Methods. Although SFT-based meth- ods achieve reasonable in-domain performance, they either degrade provide limited gains out-of-domain tasks.
Interestingly, the smallest trained produces the highest number of tool calls. Trajectory-level inspection shows that this arises from repetitive failure-recovery loops: the model often re-invokes the same tool with identical arguments after unsatisfactory outputs instead of revising its plan or parameters. This behavior highlights the limited error diagnosis and recovery abilities of smaller models.
E.3
Qualitative Analysis We investigate where the generalization ability comes from, as there are only very weak semantic relations between the training tasks and the OOD evaluation tasks. Af- ter examining the generated trajectories, we find that notable improvement for trained agents is that they demonstrate stronger planning capabilities and can anticipate potential scenarios before taking actions. For example, Figure @ shows comparison of the first step in ALFWorld environ- ment. Both models produce the same first action, but differ in planning quality: The trained agent anticipates the workflow before acting, while the base agent immediately commits action.

---

Published as a conference paper at ICLR 2026
5.4 ABLATION STUDY
Reinforcement Learning vs. Supervised Fine-Tuning To assess whether RL yields better OOD generalization, conducted controlled comparison. We compared our RL training with two SFT data collection strategies: (1) using ground-truth trajectories obtained from oracle solutions (men- tioned in Section 3.3.2) (Oracle-SFT); and (2) distilling trajectories judged correct from stronger LLM, geed-1.6-Thinkke Thinkinnn (Distillation). Specifically, for both strategies, we collected 10, 000 train- ing trajectories each and fine-tuned Wwen2.5-32B-Instruu on these datasets (Detailed hyperparam- eters are listed in Appendix F.2). We then evaluated the resulting models on both the in-domain validation set and OOD tasks. As shown in Figure 9 SFT approaches achieved reasonable in- domain performance but exhibited marked degradation in OOD tasks, highlighting the need for active learning to achieve generalizability. Detailed results for each method on OOD tasks are listed in Appendix E.2
Environment Filter To evaluate the effec- tiveness of our quality filters, we compare the performance of trained models filtered and unfiltered CodeGym under the same training settings and hyperparameterss using the same As shown in Table 2 the unfiltered training set performs worse than the filtered one on both the in- domain validation set and the OOD tasks. This highlights the importance of high-quality data in RL training and shows that our environment filters can improve training efficiency.
Table 2: Ablation Study on Filters. The model
trained on the unfiltered dataset performs worse compared to that trained on the filtered one, high- lighting the importance of data quality.
base model Qwen2.5533BBIInttmee
Method
In-domain
Avg. OOD Tasks
Base Model
30.1
42.3
CodeGym-Full CodeGymm Filter
75.0(44.9))
46.2(3.91) 49.6(7.31)
81.0(50.91)
CONCLUSION
We propose CodeGym, a scalable synthetic reinforcement learning environment generation pipeline for multi-turn tool-use agent training. By converting coding tasks into structured Gym environments, CodeGym enables L.M agents to actively explore and adapt to diverse environments and workflows with verifiable tasks. Empirically, models trained in these synthetic environments exhibit strong agent generalizability, achieving consistent performance improvements in both in-domain validation environments and out-of-distribution benchmarks such -Bench. We hope that CodeGym can serve as a foundation for developing more robust LLM agents capable of handling the diversity and complexity of real-world augmeneee workflows.
Ethics Statement This research does involve human subjects, personally identifiable informa- tion, or sensitive data. All experiments were based on publicly available datasets, accessible models, and widely recognized benchmarks. We believe that our work does not raise ethical concerns.
Reprodueibility Statement The supplementary material includes the complete CodeGym gener- ation and verification pipeline. along with an example subset of the environments. Our experiments use open-source models, with hyperparameters provided Appendix [ To control randomness, as shown in Section 5.1 and Table I we report results averaged over multplee training and evaluation seeds.
ACKNOWLEDGMENTS
The authors thank Prof. Sean Welleck, Yixin Dong, Ting-Han Fan, Miao Lu, Weiwei Sun, Guanghao Ye and Junjie Ye for valuable discussions and feedback on earlier drafts of this work, and Sining Zhu for support with the model evaluation infrastructure.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0166 2022.
Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. lau2-bench: Eval- uating conversational agents in dual-control environment. arXiv preprint arXiv:2506.0790 07792222.. 2025.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, arXiv preprint wrX.vv:606.015400 2016.
Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, al. T-eval: Evaluating the tool utilization capability of large language models step by step. arXiv preprint arXiv:2312.14033: 2023.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403.1288) 2024.
Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. Browsecomp-plee A more fair and transparent evaluation benchmark of deep-research agent. arXiv preprint arXiv:2508.066 06600, 2025.
Maxime
Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXivv1810.08272, 2018.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501. 150::2666..1111,,, 2025.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. In International conference on machine learning, pp. 1282-1289. PMLR, 2019.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, al. Textworld: A learning environment for text based games. In Workshop on Computer Games, pp. 41-75. Springer, 2018.
Weihua Du, Pranjal Aggarwal, Sean Welleck, and Yiming Yang. Agentic-rl: Distilled dual-strategy reasoning. arXiv preprint arXiv:2507.05707, 2025.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504. 1536, 2025.
Dayuan Fu, Keqing He, Yejie Wang, Wentao Hong, Zhuoma Gongque, Weihao Zeng, Wei Wang, Jingang Wang. Xunliang Cai, and Weiran Xu. Agentrefine: Enhancing agent generalization through refinement tuning. arXiv preprint 2rXiv:2501.01705 2025a.
Dayuan Fu, Keqing He, Yejie Wang, Wentao Hong, Zhuoma GongQue, Weihao Zeng, Wei Wang, Jingang Wang. Xunliang Cai, and Weiran Xu. Agentrefine: Enhancing agent generalization through refinement tuning. In The Thirteenth International Conference on Learning Representa- tions, 2025b.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aide aidedd language models. In International Conference on Machine Learning, pp. 10794-10799. PMLR, 2023a.
11

---

Published as a conference paper at ICLR 2026
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmente generation for large language models: A survey, arXiv preprint arXiv:2312.10997 2(1), 2023b.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, N. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: survey of progress and challenges. In International Joint Conference Artificial Intelligence, 2024. URL https:
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. Interac- tive fiction games: colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7903-7910, 2020.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.22312. 2025.
Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jian-Guang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. I, pp. 196-507, 2025.
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of Ilm agents: A survey, ArXiv, abs/2402.002716, 2024. URL http:///pp.ssmmatt.ccccooo.eeeee org/
CorpusID:267411822
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167200 2024.
Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, Tianyu Pang, and Wenhu Chen. Verltool: Towards holistic agentic reinforcement learning with tool use, 2025. URL htttt:tt////aaaa
wttg/ragaaao. abss 2509 01055
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314 2132221128, 2022.
Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with language model- augmented code emulator. arXiv preprint xxXx:23322.04474, 2023.
Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baix- uan Li, Zhengwei Tao, Xinyu Wang, et al. Websailor: Navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.0255)) 2025.
Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. A survey Ilm-based multi-agent sys- tems: workflow, infrastructure, and challenges. Vicinagearth, 2024. URL hhtpps://ag)
semanti wwchoiischhhlllo ar. http://oooouut::1111ee
Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogic: On the scaling limits of Ilms for logical reasoning. arXiv preprint arXiv:2502.0/ 100, 2025.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024a.
12

---

Published as a conference paper at ICLR 2026
Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh RN, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Systems, 37:54463-54482, 2024b.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language mod- els. Advances in Neural Information Processing Systems, 36:43447-43478, 2023.
Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelli- gence, 6(5):525-55) 02255-33,210 2024.
Yubo Ma, Zhibin Gou, Junheng Hao, Ruochen Xu, Shuohang Wang. Liangming Pan, Yujiu Yang, Yixin Cao, Aixin Sun, Hany Awadalla, et al. Sciagent: augl-augeented language models for scientific reasoning. arXiv preprint arXiv:2402. 2024.
Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engincering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139, 2024.
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205. 12255, 2022.
Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, al. Apigen-mt: Agentic pipeline for multi- turn data generation via simulated agent-human interplay. arXiv preprint arXiv:2504.0360 2025.
Cheng Qian, Shihao Liang, Yujia Qin, Yining Ye, Xin Cong, Yankai Lin, Yesai Wu, Zhiyuan Liu, and Maosong Sun. cawpiiggttt-ooooolattttee-e-ee exploit: A general strategy for inter-task agent self- evolution. arXiv preprint arXiv:2401. /3996, 2024.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toollim: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307. 16789, 2023.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji- Rong Wen. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):183333, 2025.
Qwen. Qwen2.5 technical report, 2025. URL www.ktpppe//arrdb..or//////////1
Timo Schick, Jane Dwivedi-Ye Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539- 68551, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,200 2017.
ByteDance Seed. Seed1.6 6 tech introduction. htppss//eeebbbyeeeeeee..//////ea 6. June 2025a.
ByteDance Seed. Seed-oss open-source models. http:p///ghhhbb.cco////seeeeeeee seed oss] 2025b.
ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Sced1. 5-thinking: Advancing superb rea- soning models with reinforcement learning. arXiv preprint arXiv:2504.139) 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Rumxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint 20Xiv:2402.03300, 2024.
13

---

Published as a conference paper at ICLR 2026
Mohit Shridhar, Xingdi Yuan, Mlec-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03766 2020.
Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for Ilms via reinforcement learning. arXiv preprini urXiv:2505.01441, 2025.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL
hnttps://wwenmm. htt.a//mmmm.thh....ooooooooo.. wo/blog/ qwq---------- 32b/
Ruoyao Wang, Peter Jansen, Alee-Ale Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than 5th grader? arXiv preprint WwXi::2003005400 2022.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exe- cutable code actions elicit better Ilm agents. Forty-first International Conference on Machine Learning, 2024a.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers generalist agents. arXiv preprint arXiv:2407. 16741, 2024b.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266-95290, 2024c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Junde Wu, Jiayuan Zhu, Yuyuan Liu, Min Xu, and Yueming Jin. Agentic reasoning: A streamlined framework for enhancing IIm reasoning with agentic tools. arXiv preprint arXiv:2502 04644, 2025.
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based agents across diverse environments. arXiv preprint arXiv:2406.041555 041151 2024.
Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. 2025. URL https: /arxiv...vvvv org/abs 2503. 02951
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 005.:225050093888, 2025.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 1--07744-007575 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 7-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406. 12045, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system 2025.
at scale. arXiv preprint arXiv:2503.144
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in wolld environ- ments. arXiv preprint arXiv:2504.03 03/600 2025.
14

---

Published as a conference paper at ICLR 2026
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXtv:2307.13884, 2023.
A SUPPLEMENTARY MATERIAL
The supplementary material contains the synthesis and verification pipeline for CodeGym environ- ments, as well as example CodeGym environments and task configurations. Please refer to the README in the supplementary material for details.
B CODEGYM STATISTICS
Table 3: Environment Comparison. We present a comparison between different agent training frameworks environment and task configuration quantities. CodeGym offers the largest number of environments and task configurations.
Environment
Task Con- figurations
Support RL. Training?
Construction
Environment
Type
Baby Al Chhevalier-ooisveee et al.
2018)
19
N/A!
Manual
ALFWorld (Shridhar al.
2020)
4
3,553
Manual
Jericho (Hausknecht
2020
N/A
Manual
ScienceWorld Wang et al.
2022
10
30
Manual
AgentGym Xi et al.
20
14
14,485
X
Manual
AgentRefine Fuu
2025a
N/A
64,000
Synthetic
AgentGen
(uu et al.
2025
592
7,246
Synthetic
AgentFLAN
(Chen et al.
2024)
7
34,440
X
Manual
CodeGym (Ours)
13,116
86,165
Synthetic
As shown in Table CodeGym of- fers significantly more environments and task configurations than prior agent training benchmarks, enabling large-scale reinforeement learning. Each environment is equipped with distinct toolset, with an average toolkit size of 6.52.
We present the CodeGym statistics in Figure 4 and Table 3
C CODEGYM ENVIRONMENT DESIGN DETAILS
C.1 AN EXAMPLE OF TRANSFORMATION
Figure I0 illustrates how coding problem can be rewritten into a CodeGym environment. The original problem is 'Finding the number closest to K in a sorted list of length N', whose coding solution based on binary search. From this solution, we distill three atomic actions: (1) observe, which returns the array length N together with the target K; (2) _ook__p_pos, which returns the element at index and (3) done, which submits the final answer. These actions constitute the tools available the agent. The environment is first initialized with specific task configuration (corresponding to the input of the original coding problem). After initialization, the agent interacts with the environment by invoking the available tools and ultimately produces the answer.
C.2 ENVIRONMENT DESIGN AND PROTOCOL
To allow wide range of coding tasks to be incorporated into reinforcement learning framework, we design an environment template for CodeGym environments borrowed from OpenAl Gym. This design provides flexible abstraction for the L.M generator synthesize.
Tass configuration are not pre-defined and controlled by random seeds.
The authors did report the exact number of environments.
15

---

Published as a conference paper at ICLR 2026
Problem Description:
Finding the number closest to K in a sorted list of length N.
Coding Solution:
Available Action List:
def Findtt losest Numberfer,, X]:
def observed
anay leogth target
right lenter))
def look_ up pospoc int) s-str refurns the eienent Indor
while iett <s right: mid doft right/
donelans: int)
arrmmidj
return animmis)
Varffmid] K: left mid
Example Environment Workflow:
Configuration:
right
Hidde Aray
A 12, 5, 9, 20), n= Target k
ylelt a- lenfont:: oninghht] ifright
arills]]
Agent Trajectory:
aiserve)) 'lengthes,
('abjaariif]] KI abstantrighs] AD: anileet]
up pas(2) pppppppp]
> -
look up posj0 > -400)-2"
look up _oo([]) > *A[1] 3
aritrighs)
done (1) a submit nower to the enckorment
Figure 10: Transformation Example. Transformation of a coding problem ('find the number clos- est to K') into the CodeGym environment with atomic actions.
Formally, an environment instance is defined by a POMDP:
= (S,,,,,,,
where (i) the state space S contains task-specific variables (e.g., strings, arrays, or data structures), which may be only partially observed by the agents (ii) the action space AA is instantiated from a generic set of function calls such as Observe and Done. together with task-specific actions, (iii) the transition function is implemented by executing the corresponding function of the en- vironment, (iv) the reward function R is sparse, assigned only upon termination by comparing the submitted answer with the reference solution, (v) the observation function O returns textual descrip- tions of action results.
Our template exposes a unified API consisting
reset (opt (options). ions): initializes the domain state from input task configurations;
step (action json): executes a JSON-encoded function call with arguments, return-
ing the result;
Observe (): provides interpretable state descriptions;
Done (answer): verifies the submitted solution and assigns terminal reward;
ret.ref.answer (): computes the task's reference answer from ground truth coding solution;
solve (): (optional) implements reference oracle solution using only the action API.
This abstraction enables the instantiation of new environments by specifying the state variables and extending the action set with domain-specift. functions, while preserving the overall interface. For example, in EdittDistance whose original coding task to calculate the minimal editing distance of two strings, the environment state consists of two strings and dynamic programming table, the action set includes operations such as wetStringLent CompareCha racters, and the reference solver implements the standard dynamic programming algorithm for edit distance.
SetDP TableCe and
Through this design, diverse algorithmic problems be formalized under consistent environ- ment framework, facilitating both supervised imitation (via the reference solver) and reinforcement learning (via the action interface).
16

---

Published as a conference paper at ICLR 2026
CodeGym Synthesis Prompt (Part
System:
You are an expert at transforming rode-related problems into interactive environments.
Task Description
Your task is convert the given "code problem" and its "code answer" into subclass of gymnasium, Env, making environment where an Agent can interact, explore, and complete the task.
Please output:
A clear and easy-to-unde unddersaddd task description (task), including input output examples; complete runnable Gym environment implementation code;
clear definition for each action, including its name, input parameters, and functionality descrip-
tion.
Design Requirements
Task description (task):
The task must be simple and easy to understand, describing the agent's goal:
Do not include wording like "please implement code," and do not imply coding behavior;
Do not provide any hints solution approaches, only describe the task objective;
Must include at least one input - output example;
This is an agent task. The agent interacts with the environment by calling actions, not by writing
cnde.
Environment Implementation:
The environment class must subelass gymnasi Env:
All code should runnable independently without external modules or implementations:
Must implement the following methods:
reset (sclf, options: dict): reset the environment. options is dictionary
where keys are variable names and values variable values;
#rom_env_strr (s: str): support initialization from a string the format "'En- vName@ where '...' is stringified dictionary:
get_ref_answer (self) -> Any: return the reference answer (based on the original code answer logic);
#inished: whether the environment finished, directly use implementation from
example code;
reward: the environment reward, directly use the implementation from the example code; solve: simulate the agent completing the task by calling 'step()" with actions, without directly calling internal variables reference amswer functions.
Action Design:
Each action should have the following characteristics:
An intuitive, reusable, and atomic (c.g.. 'InerementCounter' 'SelectitemByIndex]) Explicit imput parameters, no implicit dependency on environment state;
Return key state-change information
string, useful for debugging without hints or
guidance;
If structured data (e.g., list, dict) must be returned, use dumpp()) to convert it to string. The following special actions must be implemented:
Observe (): for the agent get current state:
Done (answer): : the agent submits the final answer. which is compared with the reference answer. Return "reward = 1' if correct, or 'reward = O' if incorrect. No intermediate rewards allowed.
Figure 11: CodeGym Synthesis Prompt (Part 1), The prompt for synthesizing CodeGym environ- ments.
C.3 GYM SYNTHESIS PROMPT
We designed an elaborate prompt for CodeGym environment synthesis, as shown in Figure II and
Figure 12 The prompt instructs the LLM to generate both the environment task description and the corresponding environment simultaneously. with detailed rules provided for each. Since the synthesized environments must adhere to fixed set of interfaces to support reinforcement training.
17

---

Published as a conference paper at ICLR 2026
CodeGym Synthesis Prompt (Part
Other Constraints and Requirements:
Must implement the static method trem_env_stt () for initializing the environment from a string. If complex structures (e.g., trees) are involved, implement the corresponding encoding and decoding logie;
* Use from_ent Lov__env_atr inside the iinit__ method to initialize;
* Add a self. fune__eee maaeee__apinnggggg in init mapping action names (strings) to their corre-
sponding methods;
Must include soive () method to simulate the agent completing the process by calling
step() () with actions, without directly calling internal variables or the reference answer function;
All imputs step () must be JSON string the format:
("name"; act Lon_inmm,, "parameLers"; (...))
Do not set self actiianttiieeeeeeeeeeeeeeee spaneeepaeeeaeeeee self obseevvttion_seeeeeeeee All action names must follow PascalCase (e.g., CountOccurreeces,, GetModes) for naming consistency;
The environment class name must follow the format ModeFind ModeFindingEny for unified management.
(TaskName) EEv, e.g.,
Example
Input:
prroblem>(example coding problem] </problem>
<code>[example coding solution] </code>
Output:
<camk>/cxample synthesis task] </task>>
<cnv>poxamplc synthesis environment] <lenv>
[.. Repeat Task Description... ...]
transformation the following problem and code: Input:
<problem > [coding problem] </problem>>>
<code>> <code>[coding [coding solution] </code>>
Your Output:
Figure 12: CodeGym Synthesis Prompt (Part 2). The prompt for synthesizing CodeGym environ- ments.
include one-shot example to guide the formatting. However, we observed that after reading the long example, the LLM sometimes overlooks earlier instructions. To address this, we repeat the key instructions after the example. Some prompts have been slightly modified for readability, while the raw version is available in our released code. Additionally, to support multilingual training, some examples written in Chinese, resulting in CodeGym environments that include both Chinese and English tasks.
C.4 AGENT PROMPT
As shown in Figure B the prompt of the CodeGym environment for LLM agents includes: (1) the description of all available tools with their functionality and the descriptions of arguments and returns; (2) the instruction of how to properly interact with the CodeGym environment; (3) the description of the task with an example.
D CODEGYM ENVIRONMENT VERIFICATION
D.1 SOLUTION FUNCTION GENERATION
To verify the solvability of a given CodeGym environment, we prompt the LLM to generate solution functions. As illustrated in Figure 14] the model is provided with the task description and list
18

---

Published as a conference paper at ICLR 2026
Agent Prompt
System:
Function:
def Observe
height of histogzam the cusrent index.
Args:
None
Returns:
str: Informatian containing the haight Hat of the histogram and the current index.
Function:
def PushToStack (Index: int):
gene
Push the speatiied index onto tha stack.
Args:
index (int): The index value to be pushed ento the stack.
Returns:
str: The operation result and the currant state of the stack.
More functions are omitted...
User:
Please the following question step by step according to the requirements below!
1. Do not write code answer the user's question you may only call the provided functions,
and you may call at most one function per step.
2. After you call a function, wait for the tool to return the result do not assume what the result
will be.
3. the tool's description is unclear, you can try using it first, and then adjust your function call
based on the returned result.
4. Function calls should be wrapped with
IIPunctienCalleeiiina hwaaae......aaeeteetaaaaaaaaa
and contain ISONetor list. The list should include dictionary, where each dictionary contains two parameters:
'name": : the function name
"parameters' : dictionary of key-value pairs for the arguments
Here's example of function call:
<|Functiencal latmmetiincaallleeo Begin|>> [ [+[ "name": "aommm"""uutttiii "function. cotiictionnnmmm name"
"parametera" "eyy " "vaa lue" ", "key2" 1u62") Funot Bnd]>>
Extra requirements:
Do not overthink: think briefly, then decide how to call the function.
Since you have many chances to call functions, you do not need to plan all steps in advance.
Do not try to solve the problem without using the tools.
Question:
the bar has a specific height, and the width of each bar is unit. Your task is to calculate the maximum area of the rectangle that can be formed by these consecutively arranged bars. For example, if the given list of bar heights [2, 1, 5, 6, 2, 3], the maximum rectangular that be formed is 10 (composed of two adjacent bars with heights and 6).
field data visualization, bar chart is a commonly used type of chart. Each
Figure 13: Agent Prompt. An example of the prompt for the agent, including the available tools, task instructions, and the problem definition.
of callable tools and asked to produce a corresponding solution function. To prevent leakage of internal environment states, only the documentation of the tools, added with example usages, is exposed to the LLM. The primary goal these solution functions is to assess the correctness the environment. Since of unit tests is available, adopt the pass @ K strategy: Multiple solution
19

---

Published as a conference paper at ICLR 2026
Solution Function Prompt
System:
Task Description
Given a problem scenario and its corresponding environment, you will write a solve (self) func- tion. This environment will run in pre-packaged Gym environment. The environment exposes some callable actions (i.e., function) you; you only invoke them via sel felf. stee () and therehy complete
for the problem scenario.
Notes:
What is passed into self.step() is stringified JSON, which has two keywords: name and param-
eters:
'The name keyword string whose content is the function's name;
'The parameters keyword is dictionary whose content the function's arguments;
Please wrap the solve function with <answer>-and </answer>; The solve function docs not require additional indentation.
Example Problem and Answer
Input:
<Task Description>> crrription>(Exappl |Example Task Description] </Task Description>>
<Env>[[xampa List of Callable Tools] </Env>
Output:
<answer>
def (selt):
gen*
automasically call actions environment complete process and submit the answer for verification.
returns:
str: The result information of the final answer verification.
*..
frequency_list =
in range (11)):
call Countocourrences
Erequency_ apppyll (tt.i..steppii. djsps (1' name" 'Countoccureenees'
"paramesers's : ('number'; 111)1 [1])
call GetMatt
mex_Ereg
dumps name': 'GelMaxPrequency! 'parameters': (' ('Erequency_Latt'' Exequency_list 1)
call GetModes
modes ast. ast.ri Leva (self. stt.stttpppaat.. (son,. (arjjannnnsssssse dumps (ompptl''aan Ia'mane's 'GetModes", "parametera':
(' "'requency_ frequency 'max"freq': max_freg] 1) (21)
call Done
return step dimps name': "Done", 'paranetera's (Manswer";
modes)]))
<lanswer>
Problem
Input:
Task Description>> [Task Description] <//ask Description>>
<Env>>[List of Callable Tools] </Env>>
Output:
Figure 14: Solution Function Prompt.
functions are generated, and the environment is deemed solvable if any of them passes all unit tests. In our implementation, we set K 10.
D.2 STANDARD UNIT TEST GENERATION
Unit tests are used both evaluate the solvability of the environment and to provide initialization seeds during training. Because most web resources do not supply unit tests, we synthesize them using LLMs. As illustrated in Figure [5] the prompt specifies in detail the unit test format. Mean- while, to ensure comprehensive coverage, the unit tests generated for CodeGym environments span both easy and hard scenarios, as well boundary cases. For each environment, sample unit tests
20

---

Published as a conference paper at ICLR 2026
Standard Unit Test Prompt
System:
Task Description
You are an intelligent assistant responsible for generating unit test cases for Python functions based prohlem description and gym environment. You will he given problem description and gym environment, and your task is generate 15 test for that environment. Please ensure that all test cases follow these requirements:
The input must be valid JSON string:
No Python expressions are allowed (such as |1] or 1i81l for in range (100) 1) Comments, caleulation expressions, Python syntactic sugar not permitted
Each test case must follow the a@b format, where:
is the name of the environment class
b is the dictionary of arguments, written in valid JSON format (c.g., ("argl": (...]])
Example:
ModeF ind ngEnve :scores": [1, 2, 9, 6, 10, 4, 1, 5, 8, 8, 2, 10, 3, 3, B, O. o,
Test must
variety of situations, including typical and edge cases: Different sizes, diverse structures, varying numerical distributions, etc.
Arrange lest cases in increasing order of difficulty:
The first 5 are casy
The middle 5 medium
* "The last hard (must include extreme or boundary cases)
Problem Description
<problem description>> [Problem Description] </problem description>>
Gym Environment
<gym.env>[Gy> Environment] </gym.env>
In the main function of the environment, there may exist some unit tests. They do not follow the format of the unit tests that want to generate. You may refer these unit tests, but be sure not completely copy them.
Please output one unit test per line. To reiterate, the format the test is a@b, where is name of the enviromment class, and b is the dictionary of input parameters, written in valid ISON format (for example: { ("argl": [.).
Figure 15: Standard Unit Test Prompt.
twice, with each sample containing 15 cases, resulting in a total of 30 tests. We avoid generating all 30 tests in single pass, as LLMs often produce duplicate cases when asked for too many once. After generation, the validity of the tests is verified using the ground-truth coding solution, and any invalid tests (Runtime Error Time Limit Exceeded) are discarded.
D.3 HARD UNIT TEST GENERATION
As discussed in Section 3.5 long-Coo models can sometimes bypass the intended tool-call workflow by relying solely on reasoning to produce the final answer. To mitigate this issue, we constructed a hard version of the unit tests. These hard tests are designed along two dimensions: (1) parameter values in the test cases are scaled to large magnitudes, such as long array lengths or large numerical values; and (2) solving the problem requires more intricate environment logic, such as invoking multiple functions or handling complex calling dependencies. To generate such tests, we prompt the LLM with these two difficulty dimensions to create more training instances and filter out all instances where Qwen2.5-32B Instruct has accuracy greater than 1/8. Meanwhile, the maximum allowed number of tool calls increases to T Tmax 512, thus augmenting standard unit tests with harder variants.
21

---

Published as a conference paper at ICLR 2026
E ADDITIONAL RESULTS
E. QWQ RESULTS
Due to differences in training data, we report the results of the QwQ model separately, As shown in Figure 16, QwQ trained in the hard version of CodeGym shows strong performance gains on both the training set and the in-domain validation set, similar to the improvements observed with the Qwen2.5 series (Figure [6. An interesting observation is the trend in average trajectory length: it initially increases but declines later stages of training. This may attributed to the limited context window during RL training (24K), which encourages QwQ to be more conservative in generating longer content. Another notable finding is the significant gap between the number of tool calls made by QwQ and those used in oracle solutions, even when training on the hard version of CodeGym. Developing methods to synthesize large-scale environments with theoretical guarantees that prevent LLMs from exploiting shortcuts remains important direction for future work.
Train Reward vs. Step
In- In-oomain Validation Reward vs. Stap
OwQ 328 thant)
Qme 328 (hant
37.
Roward
85-
12-
84-
25
Training Step
Training Step
Avg. Trajectory Length Step
Avg. Tool Call Round vs. Step
Camach the vindey-) OwC-328 Durd
Avg. Tho (eleecery Lenion
420
3000
Dmnoth andeennl Oiiice
7000
Training Stap
Training Step
Figure 16: QwQ Training Statistics. We report the average training reward (hard version of the training set), in-domain validation reward, average trajectory length, and Avg. Tool-Call Count (per trajectory) for the QwQ model.
E.2 ABLATION STUDY RESULTS
Table 4: Ablation Study Results. We present the performance of different training methods and datasets in CodeGym, including supervised fine-tuning on correct trajectories generated by ora- cle solutions D0wen..5-32BCCG---- training on the unfiltered environment set (Qwen2.5-32B-CG. CG-UF..... The evaluation settings are the same those in Table I
or Seed-1.6-Thinkin Brcn2.5-32B-GG-Diie as well as
Categories Benchmarks
Tool- Use 7-retail
Multi- Turn
Reasoning
airiline
2 bench
MMLU
Avg.
Qwen2.5-32B-4n 2onn2.5532B-Isstmmed
26.8
41.4
24.7
66.8
24.2
70.0
42.3
Qwen2.5.32 32B. CG SFT
39.6(2.81)
30.1(11.33)
23.2(1.5()
70.0(3.21)
24.6(0.41)
70.6(0.61)
41.3(1.01)
Qwen2.5.322 2--.25532--CG---- 22R-CG-Dissil
(4.8018.01)
48. 48.2/6.81)
23.2(1.5L)
72.8(6.01)
27. 27.4(3.21)
71.3(1.3))
47.9(5.61)
Mwen2.5-32-CGGUU
28.4(1.61)
49.0(7.7t)
23.5(1.21)
78.4(11.61)
27. 27.663.41)
70.5(0.51)
46.2(3.91)
Owen2,5-328-CO. (Ours)
31.2/4-41)
54.4(13.07)
30.7(6.01)
80.8
29.064.11)
71.2(1.11)
49.617.311
22

---

Published as a conference paper at ICLR 2026
Table shows the results of the ablation studies training methods and data filtering strategy. The ablation studies highlight two key findings. First, our RL-based training method (Qwen2.5- 32B-CG) demonstrates stronger generalization than SFT-based methods (Qwen2.5-32B-COL 3-BFCGFF and 1een225-32B-CG--iisiie
even when the supervised data are of high quality, such as being distilled from large teacher models. This suggests that reinforcement learning enables models to adapt more flexibly on diverse benchmarks. Second, the results of training on the unfiltered dataset (Owen2.5-32B-CG-UU)
show that quality control in synthetic environments crucial. Although unfiltered data can bring about some gains in specific benchmarks, careful curation of the filtering strategy yields more consistent and superior improvements across tasks.
E.3 AVERAGE TRAJECTORY LENGTH
Avg. Trajectory Length Step
4206
Figure 17 illlustrates how the average agent tra- jectory length evolves during training on the Qwen2.5 series models. The steadily increas- ing trajectory length suggests that LLM agents learn to spend soning or interaction to solve CodeGim.. This trend aligns with the findings in RL for rea- soning tasks such as mathematics, where addi- tional computation self-reflection verifica- tion leads to stronger performance (Guo et al. 2025).
Tralectoy Lengtn
4006
computation time rea-
Ava.
2300
Gwand 5-14
Gwand 5-22 intract
Cwan2 intrrcc
Training
Figure 17: Evolution of Trajectory Length. Increasing trajectory length shows LLM agents learn to spend more compute and reasoning.
E,4 FULL OUTPUTS OF THE QUALITATIVE EXAMPLE
We show the detailed model outputs of the agents before and after training in Figure [8] mentioned in Table 8 Compared to the base model, our trained model shows improved planning ability and can reason and anticipate different circumstances.
F TRAINING HYPERPARAMETER
F.1 RL HYPERPARAMETER
We used the same reinforcement-lerningg hyperparameters in all models. The actor learning rate was set to 1 x 10-6 with a linear warm-up of 5 training steps. The KL coefficient was fixed at 0. The maximum prompt and response lengths were 5,120 and 24.576 tokens, respectively. The opti- mization was performed using the Adam algorithm with 31 = 0.9, /3 0.95, and weight decay of 0.1. We adopted the GRPO algorithm with global batch size 512 x (512 training instances, each sampled 8 times), clip ratio of 0.2, and a gradient clip of 1.0. For training rollout, set the inference lemperature at 1.0 without any decoding constraints. For the in-domain validation rollout, set the inference temperature to 1.0 with top-p 0.7.
Definition of Training Step. In this work, one training step corresponds to processing a full roll- batch. Each rollout batch is generated once per step and contains all trajectories used for policy optimization. Within a training step, the GRPO update may perform multiple gradient-update itera- tions using mini-batches. Specifically, each rollout batch contains 4,096 samples, and the optimizer mini-bateh size is 512, resulting in mundient-update steps per training step.
F.2 SET HYPERPARAMETER
For the SFT experiments mentioned in Section 5.4 the number of training trajectories is 10, 000, and we set the batch size to 16 and a total training step 625. The optimization is performed with the AdamW W optimizer, using a learning rate of 10 4 with 31 = 0.9, 8z = 0.95, and a weight decay employ warm-up ratio of 10% of the total steps, after which
of 0.1. To stabilize early training, the learning rate follows a cosine decay schedule to encourage smoother convergence. Finally, we apply gradient clipping with maximum norm of 1.0.
23

---

Published as a conference paper at ICLR 2026
Solution Function Prompt
Base Model Output (Qwenn2.5-32--Instruee
THOUGHT: To complete the task, first need to locate a tomato. Fridge is the most probable place to find tomato, Once found, I need to heat it in the microwave and finally put it in the garhagecan.
ACTION: go fridge
Trained Model Output (0wen-2.5-32B-Codeeee
Okay, let's tackle this task. The goal is to heat tomato and then put it in the garbage can. First, need to find tomato. Since the available actions include going to various locations, should start by checking where the tomato might be. Tomatocs are often found in the fridge, cabinets, shelves. Let me check the fridge first.
THOUGHT:
So. I'll go to the fridge. If the fridge is open, can look inside. If not, might need open it. But the available actions don't I include opening, so maybe it's already open. Once find the tomato, need to take it. Then, have to heat it. The microwave or stove could be used for heating. Since the task says "heat," maybe the microwave is the easiest.
Wait, but the action to heat would be using the microwave the tomato. After heating. I need put it in the garhage can. So the steps
find tomato, take it, heat then put in the garbage can. First, should go to the fridge to check the tomato. If not there, check cabinets or shelves. Let's start with the fridge.
ACTION: go fridge
Qualitative comparison before and after RL training. RL training encourages struc- tured planning and anticipation prior to action selection. Although both models choose the same first action, the trained model exhibits more forward-looking reasoning.
Figure 18:
G DATASET USAGE AND ATTRIBUTION
This work makes use of the following open-source dataset(s):
Dataset Name: KodCode
-oceooooeee//ooodee/-// License: Creative Commons Attribution-NonCommeriic 4.0 International (CC BY-NC 4.0)
Source:
The dataset is used solely for non-commercial, academic research purposes. Proper credit has been given accordance with the license requirements.
In addition, our open-source CodeGym environment generation pipeline is released under the same license (CC BY-NC4.0)
H LLM USAGE
In this project, we use LLMs as tool to translate coding tasks into interactive environments. Since the resources derived from coding problems, the risk of generating sensitive inappropriate content is low. For the paper writing process, LLMs were only used at the wording level.
24


Reviews:
[{"review_id": "QhJ6Anv1ls", "reviewer": "Reviewer_cv7F", "summary": "The paper proposes CodeGym, a pipeline that converts coding problems into interactive, verifiable multi-turn tool-use environments for RL training of LLM agents. Environments are synthesized by extracting atomic functions as callable tools, validated with unit tests, and filtered by tool-use complexity/difficulty. Training (GRPO) on CodeGym reportedly improves out-of-distribution performance and scales to 13,116 environments with 86,165 task configurations.", "questions": "To what extent can the behavior trained by CodeGym be transferred beyond the coding environment? Have you tested its ability to generalize across domains to non-code-derived tool use cases (e.g., retrieval, Web API)?\n\nUnder sparse binary rewards and long trajectories, which stabilization techniques are most critical for convergence? Have you tried reward shaping, lesson learning, or entropy regularization to improve exploration?\n\nBesides the average number of tool calls, have you performed qualitative or trajectory-level analysis of what the agent has learned (e.g., improved planning, error recovery, or adaptive exploration patterns)?\n\nSince the environment generation process relies on LLM (Seed-1.6-Thinking) for code rewriting and tool extraction, how sensitive is the generated dataset to the quality of this generator? Will a weaker LLM affect the stability of validation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The observation convincingly reflects real-world workflow structures. The code naturally encodes sequences of function calls, preconditions, and validations, highly consistent with how tool-enhanced agents operate. CodeGym leverages this to create synthetic, scalable reinforcement learning environments while maintaining their verifiability.\n\nThe environment generation process in Figure 2 comprises two phases: synthesis and validation. This process employs automated LLM-assisted tools for extraction and unit testing for validation. The use of pass@K checks ensures solvability and stability, crucial for generating large-scale synthetic data.\n\nBinary rewards and deterministic unit testing eliminate noisy or subjective reward models. The CPU-intensive environment servers, deployed in a distributed system, are carefully designed for scalability.\n\nCodeGym training delivers continuous improvement across various model scales – particularly with object-oriented design (OOD) tools and multi-round benchmarking (τ-Bench, ALFWorld). Figure 6 shows smooth, non-overfitting training curves, indicating diverse and stable environments.\n\nAblation experiments directly compared reinforcement learning (RL) with SFT/distillation methods, verifying the hypothesis that \"RL has stronger generalization ability than SFT\". The results strongly support this hypothesis, with RL's OOD score being significantly higher than SFT's.", "weaknesses": "While the paper argues that code structure reflects real-world workflows, CodeGym's environment remains purely text- and algorithmic, lacking the noisy, unstructured dynamics common in real-world tools (APIs, browsers, bots). This may limit its generalization ability in non-code scenarios.\n\nAll tasks originate from coding problems (mostly algorithmic or numerical). While they exhibit logical diversity, this does not represent semantically or context-dependent tool usage, such as webpage navigation or database queries. The term \"generalizable\" should be limited to \"within the domain of code-like tools.\"\n\nThe paper presents reward curves and average tool calls but does not analyze the agent's qualitative behavior, such as the diversity of planning structures, recovery strategies, or tool choices. This makes it difficult to determine what kind of reasoning reinforcement learning (RL) actually induces.\n\nDue to sparse rewards and long paths, the efficiency and integral allocation of reinforcement learning are crucial. The paper mentions GRPO training but neglects exploration strategies or entropy regularization settings, factors that affect sample efficiency and stability.\n\nSome reported OOD benchmarks (such as τ-Bench and SWE-Gym) remain code-dependent. A clearer definition of the boundary between structural generalization and semantic generalization would help strengthen the assertion of its broad applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XDhIIhLDao", "reviewer": "Reviewer_HE2m", "summary": "The paper introduces CodeGym, a pipeline that converts coding problems into interactive, verifiable, multi-turn tool-use environments for training LLM agents with RL. The method rewrites solutions into atomic callable tools, wraps tasks as Gym-style POMDPs and verifies solvability via unit tests and a pass@K oracle-generation step before using them for RL. Experiments across Qwen2.5 (7B–72B) and a long-CoT model show consistent gains on OOD agent benchmarks (e.g., average +7.3 for Qwen2.5-32B; τ-bench improvements averaging +8.7 across domains), with analyses showing increased, more structured tool-call behavior after training. The work matters because it offers a scalable, verifiable-reward RL setting for tool-augmented agents that better reflects real-world workflows than static SFT trajectories.", "questions": "- What’s the actual real-world analogy for these “converted coding problems”? Do you see this as a proxy for software agents, or just a sandbox for multi-step reasoning?\n- I'd love to see more results from benchmarks, that's a lot different compared to tau-bench and alfworld", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper’s pipeline for turning coding problems into verified, multi-step environments is technically well thought out. The idea of checking solvability through unit tests and pass@K is clean and reproducible, and the dataset scale (tens of thousands of examples) makes it a potentially useful benchmark for RL-style training.", "weaknesses": "- Uncertain real-world mapping of “program → workflow.” The central motivation—that program structure mirrors real-world workflows—is argued conceptually, not demonstrated empirically. The environments are synthetic and limited to code-style function calls, not real API interactions or systems with state, latency, or uncertainty. The authors don’t justify why converting code problems into multi-step RL tasks helps with general agent performance. If the goal is to improve planning or tool use, why not build tasks from more realistic domains (e.g., APIs, documents, web agents)? Right now, it feels like a proxy task without a demonstrated external benefit (showing the performance improvements on more out-of-domain benchmarks compared to the training domain would demonstrate this!)\n\n- Almost all the tool-use results come from τ-bench and τ²-bench, which share design and simulator assumptions. Similarly, the CodeGym trajectory format seems very similar to Alfworld's -- i.e., Alfworld also shares design & simulator assumptions with CodeGym. These are not broad or diverse enough to show real generalization. I'd love to see more tool use evaluation beyond the scope/domain setup of tau-bench to further validate the proposed approach (e.g., on some benchmark that tests \"MCP tool calling capability\" of models, or even running agent on SWE-Bench, etc)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "skcr8cpgb3", "reviewer": "Reviewer_Pn7z", "summary": "This paper introduces CodeGym, a framework that converts static coding problems into interactive, verifiable reinforcement learning environments for training large language model (LLM) agents. Instead of directly generating code, models interact with environments by calling “tools”, i.e., atomic functions extracted from coding solutions, and receive verifiable feedback based on unit tests. The goal is to train models to develop generalizable tool-use capabilities rather than pure coding ability. CodeGym includes ~13k environments and 86k task configurations, each representing different tool-use workflows. Experiments with Qwen2.5 and QwQ models demonstrate improved out-of-distribution generalization on benchmarks such as τ-Bench and ALFWorld, suggesting that training in CodeGym enhances multi-turn reasoning and tool orchestration skills.", "questions": "see weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- The work proposes a scalable, controllable RL environment for LLMs. As scaling trial-and-error experiences becomes increasingly important in LLM training, it is valuable to see a framework that provides a controlled, verifiable setup for agent RL. Building RL environments from existing code problems makes CodeGym practical and reproducible.\n- Leveraging code as the substrate for tool environments is both natural and effective. Code inherently defines structured logic and function calls, which map well to the notion of tools in agentic workflows. The decomposition of coding problems into callable primitives offers a principled and scalable way to generate diverse and verifiable tasks.\n- The results are solid and consistent. Across multiple model scales and benchmarks, CodeGym-trained models show measurable gains, particularly in tool-use and multi-turn interaction benchmarks. The improvements over SFT baselines are convincing and empirically well-supported.", "weaknesses": "- The paper’s framing and terminology are confusing at first read. The title and abstract give the impression that CodeGym is a gym for improving coding ability, while the core contribution is about tool-use training using code-derived environments. Clarifying this early that the goal is not coding but interactive tool-use would significantly improve readability and accessibility.\n- Wording such as “synthesize” in the abstract may mislead readers. The current phrasing suggests that environments are generated from scratch, while in reality they are transformed from existing coding problems. Making this grounding explicit (e.g., “transforming coding problems into interactive environments”) would make the contribution clearer.\n- The experimental evidence for scalability is somewhat limited. Although the method is described as scalable, the RL training runs are short (around 100 steps) and already saturate in reward. This leaves open whether the framework can truly support long-horizon or large-scale training. A longer training curve or evidence of continued scaling benefits would strengthen the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SZgnRLUHfL", "reviewer": "Reviewer_W2jm", "summary": "This paper proposes CodeGym, a scalable pipeline that rewrites coding problems into interactive, verifiable, multi-turn tool-use environments (extracting atomic functions as callable tools; rewards come from unit tests), then uses RL (GRPO) to train LLM agents that generalize to new tools and workflows; models fine-tuned on CodeGym show consistent OOD gains", "questions": "How can large companies develop similar environments in real-world scenarios?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Clear problem framing: SFT on static traces and narrow RL don’t transfer, and a concrete, verifiable-reward environment design grounded in real execution logic can help\n 2. The paper proposes an automated synthesis-and-verification pipeline that scales to 13k environments with documented tools, complexity filters, and solvability checks; \n3. Thorough empirical evidence across model sizes and benchmarks showing broad OOD improvements and they also did behavior analyses.", "weaknesses": "1. Despite breadth, the environment family is still derived from coding tasks, so transfer to non-code, high-latency, or noisy real-world tools  may be limited; \n2. Binary, terminal rewards and fixed turn caps can under-shape credit assignment for long-horizon agentic tasks; \n3. Environment quality depends on LLM-driven synthesis, rewrite and unit tests", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)