GEM: A Gym for Generalist LLMs
Authors: Zichen Liu
Anya Sims
Keyu Duan
Changyu Chen
Simon Yu
Xiangxin Zhou
Haotian Xu
Shaopan Xiong
Bo Liu
Chenmien Tan
Weixun Wang
Hao Zhu
Weiyan Shi
Diyi Yang
Michael Qizhe Shieh
Yee Whye Teh
Wee Sun Lee
Min Lin
Venue: ICLR 2026 Poster (2026)
Keywords: environment
gym
llm
multi-turn
reinforcement learning
Primary area: datasets and benchmarks

Abstract: The training paradigm for large language models (LLMs) is moving from static datasets to experience-based learning, where agents acquire skills via interacting with complex environments. To facilitate this transition we introduce GEM (General Experience Maker), an open-source environment simulator designed for the age of LLMs. Analogous to OpenAI-Gym for traditional reinforcement learning (RL), GEM provides a standardized framework for the environment-agent interface, including asynchronous vectorized execution for high throughput, and flexible wrappers for easy extensibility. GEM also features a diverse suite of environments, robust integrated tools, and single-file example scripts demonstrating using GEM with five popular RL training frameworks. Along with this, we also provide a set of baselines across 24 environments using REINFORCE with Return Batch Normalization (ReBN), which---unlike GRPO---is compatible with the full RL setting of dense per-turn rewards and arbitrary discount factors. We further conduct apple-to-apple benchmarking of PPO, GRPO and REINFORCE in both single- and multi-turn settings using GEM to shed light on the algorithmic designs. GEM also functions as a convenient evaluation toolkit besides a training environment. We hope this framework can help accelerate future agentic LLM research.

Published as a conference paper at ICLR 2026
GEM: A GYM FOR AGENTIC LLMS
Zichen Liu Anya S1m, Keyu Duan Changyu Chen Simon Yu Xiangxin Zhou Haotian Xu?, Shaopan Xionge, Bo Liu², Chenmien Tan', Chuen Yang Beh?, Weixun Wang Hao Zhu?, Weiyan Shis, Diyi Yang", Michael Shieh 20 Yee Whye Teh', Wee Sun Lee², Min Lin" Sea AT Lab NUS 3Offrdd SMU Stanford 6 Northeastern "OpenRLHF &ROLL 9RL2 10 absolute Al
ABSTRACT
The training paradigm for large language models (LLMM)) is moving from static datasets to experience-based learning. where agents acquire skills via interacting with complex environments. To facilitate this transition we introduce GEM (Gen- eral Experience Maker), an open-source environment simulator designed for the age of LLMs. Analogous to OpenAI-Gym for traditional reinforcement learning (RL), GEM provides standardized framework for the environment-agenn -agent interface, including asynchronous vectorized execution for high throughput, and flexible wrappers for easy extensibility. GEM also features diverse suite of environments, robust integrated tools, and single-file example scripts demonstrating using GEM with five popular RL training frameworks. Along with this, we also provide set of baselines across 24 environments using REINFORCE with Return Batch Nor- malization (ReBN), chiihh-ullike GRPO is compatible with the full RL setting of dense per-turn rewards and offers better credit assignment. We further conduct apple-to-apple benchmarking of PPO, GRPO and REINFORCE both single- and multi-turn settings using GEM to shed light on the algorithmic designs. Lastly, GEM also functions as a convenient evaluation toolkit besides a training environ- ment. We hope this framework can help accelerate future agentic LLM research'.
Episode return Episode raturn Episode return
panesGearrhutiiterr.... panc/irresespeer (L.70
gamecHartermind (L.78)
gancllangman
garmeS (2.70)
68
200
200
400
20
40r
200
200
64
deweeThwerrrreeell (3.780
Dame:/Wordis (3.78)
rgalmple aquetien (177)
rgslimple geometry guamety (3.78)
rgichein Jum (1.78)
roteeadle baystac 0.78)
200
200
200
rgithertent path
rylleteer coartin (1.781
rgocircult logic (1.781
rgaarc (4
rgacre (8.78)
rgemiri sodaks (1.79)
12
Episode return
Policy iteration steps Policy Reration steps Policy Reration steps Policy iteration steps Policy iteratoo stops Policy iteration staps
Figure 1: Learning curves of Qwen3-hased agents across diverse environments of 5 categories: game (language
games): rg (ReasoningGymm): code (coding tasks); math (python-integnal integrated math questions); qa (search- integrated general questions). agents are learned via a simple yet general multi-turn algorithm based on REINFORCE (Algorithm 1). The comparison between two curves in each subplot illustrate the effectiveness of Return Batch Normalization (ReBN).
INTRODUCTION
Reinforcement learning (RL) (Sutton and Barto, 2018) has emerged as powerful paradigm for improving the reasoning capabilities of large language models (LLMs) (OpenAI, 2024; Guo et al., 2025). By collecting experience in interactive environments, RL allows agents to learn complex,
Equal contribution with random order. Work done during their associate membership at Sea AT L.b. Code is available https: / / thub. ccm/ axona /gem

---

Published as a conference paper at ICLR 2026
open-ended tasks without direct supervision (Silver and Sutton, 2025). This approach promises to create powerful agents for a variety of domains. For instance, an agent could develop entire software modules by writing, testing, and debugging code, while also adapting integration failures or evolving requirements. Similarly, in scientific discovery, an agent could be trained to develop hypotheses, design relevant experiments, and adjust its long-term strategy based the results.
However, current research on RL for LLMs has largely focused on single-turn tasks, such as answering math questions or retrieving specific data (Lambert et al., 2024; Guo et al., 2025). While these tasks are a valuable starting point, they significantly oversimplify multi-turn interactions (Liu et al., 2025a). This oversimplification means that algorithms which excel in the single-turn setting (e.g.. GRPO (Shao et al., 2024)) are fundamentally inapplicable to full multi-turn problems. If the goal is to train agentic LLMs capable of long-horizon planning, trial-and-error, iterative refinement etc, erucial to transition to testheds that support these more complex multi-turn interactions.
To facilitate this next step, we introduce GEM (General Experience Maker), an open-source environ- ment framework for diverse, multi-turn, long-horizon tasks. Motivated by OpenAl-Gym (Brockman et al., 2016) which catalyzed research traditional RL by providing a unified interface and stan- dardized environments, GEM aims to provide analogous foundational infrastructure for LLM agents.
GEM offers a diverse suite of environments spanning single- and multi-turn (over 100 turn)) tasks (including tool integrated responses, reasoning games etc), flexible observation and action wrappers, asynchronous parallel execution, and rich set of tools (python, search, and external MCP compatible tools). Additionally, GEM includes validated baselines and single-file training scripts showcasing seamless integration with five popular RL training frameworks (Oat, Verl, OpenRLHF, ROLL, and RL2-see Section 4.5).
Besides introducing the GEM framework, this paper also presents and discusses a simple yet effective algorithmic variant of REINFORCE (Williams, 1992) which incorporates Return Batch Normalization (ReBN), useful technique similar to advantage normalization (Andrychowicz et al., 2021: Liu et al., 2025b) that brings consistent improvements (Figure 1). Unlike GRPO and its variants, REINFORCE with ReBN is fully compatible with the multi-turn RL setting, including turn-level dense rewards and arbitrary discount factors. We further compare REINFORCE- based algorithms with multi-turn PPO (Schulman et al., 2017) and GRPO, showing its theoretical connections and empirical tradeoffs. also provide case studies on the impact the discount factor 7 on multi-turn learning, extensive results of fool-integrated RL, and performance benchmarks terminal and MCP usage of strong LLMs using GEM as a unified evaluation toolkit. We hope this framework will accelerate RL research agentic LLMS and advance progress toward more capable and autonomous AT systems.
GEM ENVIRONMENTS
This section introduces GEM's core functionality, covering its main interface (Section 2.1), the environment design (Section 2.2), and advanced features such asynchronous vectorization and modular wrappers (Sections 2.3 and 2.4).
2.1 INTERFACE
GEM employs a standardized environment interface closely following the well-established OpenAI Gym API with the main functions being zeset () and step (). basic agent-enviromment interaction loop is as follows (multi-agent interface shown in Appendix D.3):
import gem
gemmprint Lave to 1ist available environments
3 env = gem.mamm.e.. make ("game: :uene:GGuss:ae Guese hettmmereebbbbbe
observation, info env.reset
5
6 while True:
# (1) Agent acting:
8 action env, www.sample_randon_adde sandon_action!)
action agent act (observation)
scting by LLM sampling
(2) ronment exccution:
next_obs, reward, terminated, truncated, info env. (action)
(3) Agent learning: i agent, learn (observation, action, roward)
observation = next_cbs
terminated truneated:
2

---

Published as a conference paper at ICLR 2026
2.2 TASKS AND TOOLS
GEM's core environment components are tasks and tools. Each combination of task and an optional set of tools constitutes an environment that tests complex capabilities such as reasoning, multi-step planning, and tool use. These environments therefore be used to benchmark LLMs and to test and develop new algorithms. GEM currently features seven main categories of tasks:
Math: Solve math problems with chain-of-thought reasoning.
Math with image: Solve geometry math problems with images using chain-of offii--of-houuhhh reasoning.
Code: Generate code to solve competitive programming problems.
Game: Multi-turn text-based games adapted from TextArena (Guertler et al., 2025).
QA: General, potentially knowledge-intensive questions (useful for testing search tool capability).
ReasoningGym: A unified interface of ReasoningGym (Stojanovski et al., 2025) which provides 100+ single-turn verifiable tasks.
Terminal:
Perform complex tasks through a containerized terminal environment.
GEM's modular design simplifies task integration. Math (with images), code, and QA tasks can be integrated by simply providing a new dataset. Terminal tasks require a new Docker file, instruc- tions, and test cases. New games and other custom tasks can be added by inheriting from GEM's environment base class and defining their state transition and reward logic. In addition, tasks can be augmented with any combination of tools. GEM currently supports:
Python: Parses and executes code blocks, returning the stdout or execution error.
Search: Parses a query, executes search against an external engine, and returns the results. MCP: General tool calling any external servers that conform the model context protocol.
The of tools converts single-turn tasks, like Math ReasoningGym, into multi-turn tasks in which an agent can learn to call tools and adapt based on their output.
2.3 ASYNCHRONOUS VECTORIZATION AND AUTORESET
To facilitate efficient agent RL training, support parallel execution of vectorized environments via asynchronous tool calls to collect episodes in batches. In addition to the latency reduction, the of vectorized environments with autoreset streamlines the experience collection logic. Users can run a single reset () at the initialization stage and simply continue with step () in the following agent-envirrmment loop for continuous data generation. In addition, the user code can use the returned terming terminated flag to prevent value bootstrapping across episode boundaries, ensuring the correctness critic learning. An illustration of the autoreset mechanism can be found Figure 2.
.step()
stepO
stepp)
stepO)
stepO)
stepO
stepO)
parallel environments
Episode Episode
Episode
Episode
autoreset
Figure Illustration of autoreset vectorized environments. Autoresetting resets the environment automatically after termination, allowing users collect batches episodes by simply running sstep () without needing more complicated logic such as keeping track of whether individual episodes have terminated.
2.4 WRAPPERS
Like in OpenAI-Gym, GEM uses wrappers for easy extensibility. Observation wrappers, for example, control how the episode is converted into observation. Options include observing just the most recent environment output, a concatenation of all previous environment outputs, concatenation of all previous environment outputs and actions, some parsed/summarized version of this. The
3

---

Published as a conference paper at ICLR 2026
of1
a
of
a
(a) Single Token as Action
(c) Whole Interaction as Action
01
F11
...
FN
(b) Response as Action
Figure The illustration different view agentic Green nodes denote tokens responsible for loss.
Python interpreter database/web search tools also formulated wrappers which can be added on top of any specified task environment.
3 REINFORCEMENT LEARNING WITH GEM
In this section, we begin by describing the main RL formulations for LLMs, including their respective flexibilities and limitations (Section 3.1). Motivated by this, we then present our baseline algorithm which is applicable to the more flexible RL formulation (Section 3.2).
3.1 PRELIMINARY: LLMS AS AGENTS
There three main ways of treating I.L.-enviromment interactions in RL algorithms which each have different limitations and strengths:
Action Single token (Figure 3(a)): The first approach is to treat each token generated by the LLM as an individual action (Ziegler et al., 2019). This, however, means that episodes are typically very long (thousands of tokens), and it also requires specifying the reward for the addition of every token, which is difficult to evaluate. Successful applications of RL in this formulation tend to use sparse outcome reward with discount factor 7 (Guo al., 2025).
Action = Response (Figure 3(b)): To avoid these complications the second approach is to treat whole response sequence tokens until an EOS) single action (Ahmadian et al., 2024; Liu et al., 2025a). In answering math problems for example xxample-currently the most common testbed for RL for LLMs each episode contains a question and response. With this view all episodes therefore have length 1 and the RL problem essentially degenerates contextual bandits (Abe et al., 2003). This is convenient as it means sample-based advantage estimation methods such as GRPO (Shao et al., 2024) can be applied efficiently, and these have been demonstrated to be highly effective. Extending to multi-turn episodes (c.g. for games or tool use), however, results in an issue: Multi-turn interactions have episode lengths 1, meaning sample-based advantage estimation methods (e.g., Kazemnejad et al. (2025)) become infeasible (since they require collecting multiple episode completions from each turn (state) in the episode, leading to exponential complexity).
Action = Whole interaction (Figure 3(c)): One approach to make GRPO applicable to multi-tum interactions is treat the whole interaction
single action while masking the loss tool outputs. This view again degenerates the full RL problem back to one-step RL or contextual bandits, meaning GRPO etc. can be applied. However, requires two compromises: Firstly, it effectively fixes the discount factor at = 1, thus removing the incentive to solve problems quickly. This is significant, for example in Section 4.2 where we show how the optimal search algorithm is only recovered when setting 1. Secondly, this approach is limited to single trajectory- levell rewards, losing fine-grained per-turn credit assignment.
Many prior works make these concessions and use GRPO in multi-turn LLM RL (Cao al., 2025; Jiang et al., 2025; Chen al., 2025a; Jin et al., 2025; Feng et al., 2025a). However, to develop an algorithm compatible with the full RL setting, we go back to the second view (action=response) and employ a simple variant REINFORCE with Return Batch Normalization (ReBN). Unlike GRPO, this algorithm is compatible with per-step dense rewards and arbitrary discount factors (y < 1). thus making it significantly more flexible for optimizing LLMs in complex, multi-turn interactive settings.
Ignoring token-level PPO clipping which has no effect if the updates on-policy.
4

---

Published as a conference paper at ICLR 2026
3.2 BASELINE ALGORITHMS
We start from the foundational on policoy volicy-gradient method REINFORCE (Williams, 1992), which optimizes the following objective:
T
JREINFORCE (0)
Σ
Σ
cl log (a "s,"" (m)
(I)
where N the batch size, (8o: 0o; 81, 0T 0T-1
sequence of states and actions making up a "r is the
trajectory in which each 8, and a, is itself a sequence of tokens, and G, = ΣT- *
return. Though initially designed for single-turn problems (i.e., 7 1), GRPO can be extended to multi-turn tasks by sampling a group of M trajectories per initial state and normalizing the trajectory-level reward for each group*:
N
M
Trlmsn)) -1
JGRPO (0) N
Σ
A
(n.m) GRPO
Σ
log To ((.m)) (T |s| (s(m.m)))
(2)
N
m==1
1=0
where AGRPO - C 't ever, this approach has poor credit assignment for multi-turn problems because all turns in the trajectory share the same advantage estimation, and improving it typically requires tree-like sampling which leads to combinatorial explosion. To bypass the expensive sampling from each turn, we can learn value function estimate the return Gt, known critic (Sutton and Barto, 2018), which in turn guides the policy learning in the actor-critic architecture. We can compute GAE (Schulman et al., 2015) for the advantage actor-critic (A2C) objective:
Pt (n.m) im......... .M.. How- (n.m) mean (s) with R - -
N Tp()
JxZc
1
Σ
Σ
(m) log (a (s)")) (*))
(m) 1s)
N
ビ=
To retain the benefits of fine- graiggraindd and stable advantage estimation without the combinatorial explosion or learning an additional critic, we instead use Return Batch Normalization (ReBN). For ReBN the per-transition returns G, are normalized over the whole batch of transitions:
1 N T(m) -1
JREINFORCEHREEN (0)
A log ( (n))
(4)
==1 1=U
where ReBNN. (G(=)) mean (s))/std(G)) with G (G(_)) Mee T(m) Each of
these algorithms trains the agent by iterating between two main phases: (A) data collection and (B)
policy update. We present the RL loop of Equation (4) in Algorithm I in Appendix C due to space constraint.
4 EMPIRICAL STUDIES WITH GEM
In this section, we demonstrate how GEM can facilitate RL research on agentic LLMs through series of empirical studies. These include comprehensive apples-to-apples algorithm benchmarking across eight GEM environments (Section 4.1); analyses of the effects of the discount factor y and tool integration (Sections 4.2 and 4.3); an examination of cross-task generalization (Section 4.4); and, finally, a demonstration of GEM's compatibility with five RL training frameworks along with their easily accessible infrastructure benefits (Section 4.5). RL results in a vision-language environment and analysis of multi-ageni environment can be found in Appendices D.2 and D.3.
4.1 BENCIMARKIIG RL ALGORITIMSS FOR LLMS
Benchmarking has been critical for the progress of RL, with OpenAI-Gym providing standardized environments that enabled systematic evaluation of algorithms (Raffin et al., 2021; Huang et al., 2022; 2024). Following this paradigm, GEM offers unified testbed for agentic LLMs, where prior work often relied on bespoke tasks that complicate fair comparison. We benchmark all baseline algorithms introduced in Section 3.2 (GRPO, PPO*, REINFORCE, ReBN) across eight GEM environments under unified experimental protocol. All algorithms are implemented using Oat (Liu et al., 2024) with hyperparameters detailed in Appendix F. Results are evaluated by mean episode return, sample efficiency, and stability.
'Othhgonaaly, we also utilize proximal updates (Schulman al., 2017) improve sample efficiency. "This not the original GRPO because we fixed the length bias as noted by Liu et al. (2025b). SPPO in this work generally refers to turn-level PPO instead of token-level PPO commonly seen in single-tur dialogue scenarios (Ouyang al., 2022).
5

---

Published as a conference paper at ICLR 2026
rg:arc 1d (1.78)
game:Mineswesper (1.78)
rg:acre (1.78)
rg:mini sudoku (1.78)
1.5
Epizede return
0.8
0.8
0.6-
0.6
0.4
0.2
0.2
200
400
200
400
200
400
200
400
game:GusssTheNuuberr (1.78)
game:Sudoku (1.7B)
qa:MixedQAA Tool (4B)
math:DeepScalcR40K--00 Tool (48)
Epizode return
0.8
0.8-
0.5
0.5
0.2
0.2-
0.0
6.0
0.0
200
400
200
400
202
400
200
400
Policy iteration steps
Policy iteration steps
Policy iteration steps
Policy iteration steps
ERPO
PPD
REINFORCE
REINF ORCE W. ReBN (Ours)
Figure 4: Algorithm benchmarking using eight representative environments from GEM. All agents trained from Owen3- (scale) Maaal)BBBase -asse models, with sca specified in each plot. rg refers to single-tum reasoning tasks from ReasoningGym; game consists of long-horizon language games; qa and math are tool-integrated multi turn environments.
We present all learning curves in Figure 4. We first observe that in all three single-turn environments (labeled with rg), GRPO performs reasonably well, defending its effectiveness in single-step RL with verifiable rewards. However, GRPO falls short when it comes to multi-turn environments (GuessTi Teeeshheenung Number and Sudoku), where dense per-turn rewards are available and more fine- grained credit assignment is necessary for efficient policy learning, due to constant advantage estimation across all steps. Such effects are the most profound when the environment's reward structure is inherently non-sparse (qa and math is less so).
In contrast GRPO, REINFORCE and PPO are natively suitable for multi-turn RL. We find that vanilla REINFORCE is readily strong baseline in most environments, but it might suffer from suboptimal convergence (e.g., two Sudoku environments). We hypothesize that this might be because the return calculation of vanilla REINFORCE can be sensitive to reward shaping. thus hindering exploration; we defer an in-depth ablation study to Appendix D.I. On the other hand, PPO is generally performant, attaining the best episode return in the complex and long-horizon Sudoku environment. This performance advantage can be attributed to a well-learned critic, but it also deemed difficult robustly learn accurate critic (Van Hasselt et al., 2018; Kazemnejad et al., 2025) (as evidenced by the poor performance of PPO in Minesweeper), inviting future works to go in this direction.
Finally, investigate the proposed REINFORCE variant, which incorporates a simple Return Batch Normalization (ReBN) technique. Results in both Figures and 4 show that ReBN consistently improves on vanilla REINFORCE by large margin, suggesting the empirical benefits of adaptive normalization of policy gradient coefficients. Morcover, ReBN outperforms or is comparable with PPO and GRPO in all evaluated environments, rendering it the strongest baseline without expensive computations, such critic learning or extensive rollouts.
4.2 DISCOUNT FACTOR 7 MATTERS
(a) Effect of discount factor
(b) Effect of tool availability
Mean apizode length
Mean episode return
Mean Episode Succecs
math:Or257K
qa:HotpotQA
1.0
0.75
0.5
0.50
10-
0.25
0.0
200
400
200
400
200
400
0
200
400
Policy iteration
Policy iteration steps
Policy iteration
Policy iteration
Figure 5: (a) Average number of turns and episode return when trained with different discount factors. (b)
Comparative experiment results tool availability.
Next, we investigate the effect of the discount factor y. A key motivation for REINFORCE+ReBN GRPO is its compatibility with arbitrary discount factors. To investigate the effect of this

---

a consistent Published as a conference paper at ICLR 2026
trained the Owen3-1. 7B-Base model (Yang al., 2025) using REINFORCE+ReEN the Guess TheNumber environment. In this environment the agent must guess a hidden number randomly selected between 1 and 50. At each turn the agent may guess, and receives feedback to whether the hidden number larger or smaller. The optimal strategy is therefore binary search.
As shown in Figure 5(a), expected, smaller y values naturally encourage solutions with fewer turns and drive convergence to the optimal turn count (logg (50) R (c--achievable only through binary search. Example interactions are included in Appendix B. discussed in Section 3.2, the natural efficiency incentive from 7 < is not compatible with GRPO. Instead, prior works using GRPO hyperparameter tune the environment's maximum number of turns to get efficient agent behavior (Xue et al., 2025).
4.3
TOOL-INTEGRATION IN MATH AND QUESTION-ANSWERII TASKS
GEM is designed with modular support for external tools, enabling seamless integration into range of tasks. To empirically assess the impact of tool use, we focus two domains: Math and (QA).
Table 1: Math benchmark for four agents, evaluated with and without tool access and RL training. Note: scores should be interpreted relative to other values here to sensitivity to grader code (see Section 4.3).
Qwen3 Base
Base (no tool)
Base (with tool)
RL tool)
+ RL (with tool)
AIME24
10.0
6.7
16.7
30.0
AMC
39.8
50.6
49.4
67.5
MATH500
61.0
62.4
67.4
71.0
MinervaMath
36.4
30.1
40.1
40.4
OlympiadBench
29.5
31.0
33.5
39.9
Average
35.3
36.2
41.4
49.8
We first investigate the effect of GEM's Python tool on Math tasks. Starting from the base model 3wenn-4B-Bees we finetune on the math Orz57k environment, training two variants: one with Python tool integration and one without. The base model and both finetuned models are then evaluated across five distinct math environments. Hyperparameter details are provided in Appendix F, with the training curve shown Figure 5(b), and @ass@ accuracy reported in Table
The math grader used for reward and evaluation is based on HuggingTace' math_ fy library We found that even minor differences in grading logic across codebases yields substantial variation in reported performance. Thus, all results should be interpreted evaluation framework-
than absolute values. This further highlights the need for unified benchmarking, as provided by GEM.
Results Table reveal clear and consistent pattern: across all environments, performance improves substantially after RL training compared to the base model. Furthermore, the model with access to the Python tool achieves higher final performance every setting.
Table 2: QA benchmark scores for the base agent and agents trained with different RL configurations. + and * denote single-hop and multi-hop datasets, respectively.
Base RL (no tool, single env)
Base RL (no tool, mixed env)
Base RL (with tool. single env)
Base RL (with tool, mixed env)
Base (no tool)
Qwen3-4B
NQ+
15.4
15.8
35.0
TriviaQAf
35.4
43.4
44.9
69.0
71.9
PopQAt
11.3
19.0
19.9
47.1
48.1
11.1
21.1
22.1
43.2
45.5
2wiki*
10.0
26.8
30.1
44.5
46.7
Musique*
2.9
4.7
5.5
17.6
19,9
Bamboogle*
17.6
28.8
28.8
49,6
48.8
Average
10.2
22.7
23.9
43.7
45.5

---

Published as a conference paper at ICLR 2026
We also perform parallel analysis for QA tasks, this time integrating the Search tool. We train two environment compositions: qa: Hotpot QA alone, and a mixture of both qa Hotpot QA and qa Nat aatturaauusttiinss All other setting the same for the Math experiments (see above). Evaluation spans seven diverse QA environments. Results, summarized in Table 2, mirror those from the math domain: RL finetuning markedly improves performance, and models equipped with the Search tool achieve the highest accuracy in every scenario.
The consistency of these findings across both domains (mathematics and QA), tools (Python and Search), and multiple evaluation environments underscores the flexibility and robustness of GEM's approach RL LLM with tool integration.
4.4
STUDYING GENERALIZATION
Mean Episode Return
game:Sudoku-vO-eess
rgscircuit logic
rg:needle haystack
rgamini sudoku
1.00
Evel Accuracy
Eval Accuracy
Eval Accuracy
0.75
8.50
0.50
0.25
0.4
0.00
0.85
0.3
200
400
200
400
200
400
200
400
Training Steps
Training Steps
Training Steps
Training Steps
Figure Training on game: : sudcku-v--ea environment generalizes to ReasoningGym.
GEM's environments can be used for both training and evaluation. This makes ideal for investigating cross-environment generalization. For instance, demonstrate training the game sudcku-vO-ease environment, while periodically evaluating on three different environ- ments, with some encouraging initial generalization results shown in Figure 6.
4.5 INTEGRATION WITH TRAINING FRAMEWORKS
Finally, we demonstrate that GEM-which takes care of the environment side-can be easily integrated with five popular frameworks that handle the training side. There has been a proliferation of frameworks focusing the training side of LLM RL. These often rely heavily multiple other libraries (such as vLLMM for response generation (Kwon et al., 2023), and DeepSpeed for optimization (Rasley al., 2020)). The diverse range of features and design choices make it challenging for researchers to select and adapt suitable training framework to their specific needs.
To address this GEM comes with complete, single-file training scripts showing clean integration into five widely used LLM RL frameworks: Oat (Liu et al., 2024), Verl (Sheng et al., 2024), OpenRLHF (Hu et al., 2024), ROLL (Wang al., 2025a), and RI.2 (Tan et al., 2025). These validated in Figure 7(a) where we show the training curve for each. Despite minor differences due underlying design choices of the frameworks (c.g., different LLM generation engines) and RL stochasticity, all curves exhibit similar trends, demonstrating that GEM is agnostic to training frameworks and validating their implementation equivalence. Furthermore, supporting a wide range of frameworks allows to effortlessly access their advanced features. For example, enabling the asynchronous rollout RL2 gives an immediate 2x gain wall-clock efficiency (Figure 7(b)..
(a) Cross-framework comparison
(b) Efficiency comparison
Mean episode return
bame:GuussTheHuumbr
rg:LetterCounting
1.0
0.6
Sync
0.2
0.2
0.4
0.1
0.1
rell
0.0
0.0
0.0
Veri
-0.1
-01
100
200
200
400
25
50
100
10
15
20
Policy iteration steps
Policy iteration
Policy Iteration steps
Time (in hours)
Figure 7: (a) Training curves on two environments showing successful integration of GEM into five existing frameworks. (b) Asynchronous rollout improves wall-clock efficiency of training Sudoku-solving agents based Qwen3-4B-B.r Base.
8

---

Published as a conference paper at ICLR 2026
5 AGENT EVALUATION WITH GEM
In addition to RL training. GEM can serve as unified evaluation interface to test LLM agents' performance. In this section, we present two example use cases where we evaluate agents pow- ered by strong LLMs (GPT-5 (OpenAI, 2025), Gemini-2.5-Pro (Gemini Team, 2025) and Claude- Sonnet-4 (Anthropic, 2025a)) on two complex tasks: database operation via model context protocol (MCP) (Anthropic, 2025b) and terminal interaction via docker containers, both of which have been added to GEM following Appendix A.
5.1 GENERAL TOOL USE MODEL CONTEXT PROTOCOL
Modern LLM agents often need to interact with external tools, such search engines,
MCPMark Results
Terminal Banch Results
0.85
Num Der of turne
0.40
60.5
Success rate
0.3
Success rate
APIs, and code interpreters. To facilitate this, GEM is designed to be compatible with the MCP, which is an open protocol that provides standardized way for LLMs to communicate with external tools and data sources.
64-
03
Num ber
GPT-5
Gomini-22
Blaude-Sonnatl
GPP3
Gemiel-2..PP
daude Sonnat-4
Figure 8: Benchmark results on MCPMark (Postgres subset) and Terminal- -Bnnhhh (subset) using GEM unified evalua- tion toolkit.
The MCP architecture consists of an MCP host (the LLM application), an MCP client, and an MCP server (the external tool). By adopting this protocol, GEM allows for gooug-and-layy tool usage, where any tool that implements the MCP server interface can be used by an agent in a GEM environment without custom integration. This significantly simplifies the process of creating tool-augmented LLM agents and opens up a vast ecosystem of potential tools.
Using a PostgreSQI. MCP tool, we assess the agent's tool-augmented reasoning capabilities using 20 database tasks taken from MCPMark (Team, 2025a). We report the average success rate and the average number of turns required to complete the tasks the left panel of Figure 87. GPT-5 attains the best success rate with the fewest interactions, while Gemini-2.5-PPr and Claude-Sonnet-4 have slightly lower and varied performance.
5.2 TERMINAL ENVIRONMENT VIA DOCKER CONTAINER
To support a wider range tasks, especially those involving complex software dependencies and interactions with the operating system, GEM includes support for environments running inside docker containers. The integrated terminal environment provides a sandboxed unix operating system where agents can learn to perform tasks using shell commands. This approach provides a high degree of isolation and reproducibility, ensuring that the environment is consistent across different machines.
We assess the terminal mastery of LLM agents on 57 tasks sampled from Terminal-Bench (Team, 2025b), without any scaffolding. The right panel of Figure 8 reports the average success rate and the number of turns required complete the tasks. GPT-5 attains the highest success rate with the most efficient interaction, followed by Claude-Sonnets, and Gemini-2.5-Per Pro. The evaluation leverages the same interaction loop used for RL training, highlighting GEM's role as unified framework for both reinforcement learning and standardized evaluation.
6 CONCLUSIONS
GEM aims to accelerate agentic LLM research by providing decoupled and clean library that is agnostic to training frameworks, unified agent-enviromment interface and a suite of standardized environments. In this paper, we introduced the design choices of GEM, the current suite of task domains and tools, features like vectorized environment execution, a simple yet general multi-tumn REINFORCE algorithm implemented five training frameworks, a comprehensive algorithm bench- marking evaluation, and in-depth analysis on several algorithmic details. We invite the community to enter the era of experience for L.MM agent learning, and join us in both using and continuing to develop the GEM framework.
7 Our evaluation relies on the basic response generation API rather than agent frameworks (e.g., LangChain, OpenAT Agent SDK), which may lead to deviations from the original henchmark results.
9

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This project was partially supported by the Singapore Ministry of Education Academic Research Fund Tier (Award Number: T1 25 251RES2514).
REFERENCES
Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models. arXiv preprint urXiv:2311.18232, 2023.
Naoki Abe, Alan W Biermann, and Philip M Long. Reinforcement learning with immediate rewards and linear hypotheses. Algorithmica, 37(4):263-293, 2003.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace. Julia Kreutzer, Olivier Pietquin, Ahmet Ustün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint
(tt.::2222..444440 2024.
Stefano V. Albrecht, Filippos Christianos, and Lukas Schäfer. Mulli-Agemt Reinforcement Leurning: Foundations and Modern Approaches. MIT Press, 2024. URL https:ttppt... http://wwwww.a........o.
com.
Marcin Andrychowicz, Anton Raichuk, Piotr Stañczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters for on-policy deep actor-critic methods? large-scale study. In International conference on learning representations, 2021.
Anthropic. System card: Claude opus 4 & claude sonnet 4. https: /ttp:://wwwwwwwwee cdn. anthropic. com/ 07b223f99022eee.e 9fe 6ca 638e Add paf, 2025a.
Anthropic. Model context protocol. https: httpp////ggttta...c///////aaatte modelcontextproooccom 2025b.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392. 2025.
Albert Bou, Matteo Bettini, Sebastian Dittert, Vikash Kumar, Shagun Sodhani, Xiaomeng Yang, Gianni De Fabritiis, and Vincent Moens. Torchrl: A data-driven decision-makinn makinn library for pytorch, 2023. URL /atttss:/aazee
wwtg/bs/////eeeo..e/...//e.e/ee/ abs/2306. 00577.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Shiyi Cao, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, Jiayi Pan, Xingyao Wang. Akshay Malik, Graham Neubig, Kourosh Hakhamaneshi, Richard Liaw, Philipp Moritz, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Skyrl-vO: Train real-world long-horizon agents via reinforcement learning, 2025.
Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. Research: Learning to reason with search for llms via reinforcement learning. 2025a.
Wentse Chen, Jiayu Chen, Hao Zhu, and Jeff Schneider. Context-lite multi-turn reinforcement learning for LLM agents. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models, 2025b. URL https: http::::////peeeviiiee
net fott://orrmmi---CCPPP......
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in 1lms, 2025a.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-growe policy optimization for Ilm agent training. arXiv preprint arXiv:2505. 10978, 2025b.
10

---

Published as a conference paper at ICLR 2026
Google Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.0626. 2025.
Leon Guertler, Bobby Cheng, Simon Yu, Bo Liu, Leshem Choshen, and Cheston Tan. Textarena. arXiv preprint urXiv:2504.1144) 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability Ilms via reinforcement learning. arXiv preprini urXiv:2501.12948 2025.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An casy-to-use, scalable and high- eegh-performance rlhf framework. arXiv preprint arXiv:2405.11143, 2024.
Siyi Hu, Yifan Zhong, Minquan Gao, Weixun Wang, Hao Dong, Xiaodan Liang, Zhihui Li, Xiaojun Chang, and Yaodong Yang. Marllib: A scalable and efficient multi multi-agent reinforcement learning library, 2023. URL https: / arxiv org/ abs 222211117000
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality fingle-file implementations of deep rein- forcement learning algorithms. Journal of Machine Learning Research, 23(274): 2022. URL http:// jmlr. r. orgg papers v23 / 2 1 342.. html.
Shengyi Huang, Quentin Gallouédec, Florian Felten, Antonin Rafin, Rousslan Fernand Julien Dossa, Yanxiao Zhao, Ryan Sullivan. Viktor Makoviychuk, Denys Makovichuu, Mohamad H Danesh, et al. Open rl benchmark: Comprehensive tracked experiments for reinforcement learning. arXiv preprint arXiv:2402.03044 2024.
Dongfu Jiang. Zhuofeng Li. Yi Lu, Zhiheng Lvu, Ping Nic, Wenhu Chen, Tianyu Pang, and Chao Du Verltool, 2025. URL https:://// /t/pttgtttt/////// com wwt./omm//aaa..eeeee TIGER AI LLL--Lavveertoooo
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning, arXiv preprint arXiv:2503.09516. 2025.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Refining credit assignment in rl training of llms. In International conference on machine learning, 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 61 1-6622, 2023.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint 2rX1v:2411.15124. 2024.
Marc Lanctot, Edward Lockhart, Bean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis. Open- Spiel: A framework for reinforcement learning in games. CoRR, abs/1908.0944 09453,, 2019. URL http: /ttp///aavvvveeee//aaeeeeeeaaava org aww.aassss. 1908 09453.
Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi. Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, et al. Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. arXiv preprint Sttt::2506.241113 2025a.
Zichen Liu, Changyu Chen, Xinyi Wan, Chao Du, Wee Sun Lee, and Min Lin. Oat: A research- friendly framework for lim online alignment. https: httpb//tttttttttt////////tt com/ sai 1 -sg/ ca t,, 2024.
11

---

Published as a conference paper at ICLR 2026
Zichen Liu, Changyu Chen, Wenjun Li. Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1- -l-zero-like training: A critical perspective. In Conference on Language Modeling (COLM), 2025b.
OpenAl. Openai ol system card. arXiv preprint arXiv:2412.16722 2024.
OpenAL. Gpt-5 system card. https: http////dn.cee httt:://cch..ppnnii.ooo....oo/ www..///ppt---ssssee-araaadddddd 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-53 2021. URL http::/tttttttttt //t////m/rrr//////// org www.ooggggeeeeeee.o http//ppperr///// /v22/ 20 1364.1144600 html.
Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1505-3506, 2020.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint OrXiv:150e 15X.003550..2438.. 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint AXiv:1707.06347 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 2409.19256, 2024.
David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 1, 2025.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprini arXiv:2312.06558 2023.
Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, and Andreas Köpf. Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards. arXiv preprint arXiv:2505.24760, 2025.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
Fahim Tajwar, Yiding Jiang, Abitha Thankaraj, Sumaita Sadia Rahman, Zico Kolter, Jeff Schneider, and Ruslan Salakhutdinov. Training generally curious agent, 2025. URL https: /arviv. org/aba abs 250.. 17543.
Chenmien Tan, Simon Yu, Lanbo Lin. Ze Zhang, Yuanwu Xu, Chenhao Jiang, Tianyuan Yang, Sicong Xie, and Guannan Zhang. RI2: Ray less reinforcement learning. https: hitps://githull com/ Chenmi enTan RI,, 2025. GitHub repository.
The MCPMark Team. Mcpmark: Stress-testing comprehensive mep use. https: /gthhbb..t
com/eva -sys/ ssys/mppaa rk, 2025a.
12

---

Published as a conference paper at ICLR 2026
The Berminal-Bench Team. Terminal-bench: A benchmark for ai agents in terminal environments, Apr 2025b. URL https: / /tgthttttt...//////// wtw.:/aude-inssittttee////////ee
J. K. Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis Santos, Rodrigo Perez, Caroline Horsch, Clemens Dieffendahl, Niall L. Williams, Yashas Lokesh, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learning, 2021. URL
https: /aaxiv./aaxvvvvvaa org/ abs 2009 14471.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02645. 2018.
Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang. Jiaheng Liu, Zhendong Li, Xiaoyang Li, et al. Reinforcement learning optimization for large-scale learning: An efficient and user-friendly scaling library. arXiv preprini arXiv:2506.0611 06/222, 2025a.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang. Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang. Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning, 2025b.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229 256, 1992.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to Ilm reasoning; from rejection sampling to reinforce. arXiv preprint arXiv:2504.11 11343, 2025.
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru L., Zejun Ma, Bo An. Simpletir: End-to- end reinforcement learning for multi-turn tool-integrated reasoning. https::/simppee 2025. Notion Blog.
notion.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint ar arXiv:2505.0933 2025.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-use interaction in real-world domains. arXiv preprint arXhv:2406.12044 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.0859) 2019.
13

---

Published as a conference paper at ICLR 2026
A ENVIRONMENT REGISTRATION
GEM enables rapid development of new RI. environments. In this section, we illustrate two scenarios: (i) integrating additional datasets into an existing task and (ii) defining custom task, followed by the procedure for registering these environments for use.
The following code snippet shows how to add a new dataset for math environment, where the answer verification logic is predefined by GEM and can be reused.
import gam
from gem.envs eagistaaaioon import register
3
register(
"math: cak.kk
6
"gem. hatts..math.eee eav eeveeeeeeeeeee Mev::Mee''",
datoser_name= axxonrrl 11///M-k"" sugginglace local dataset path
8 question_kage kwykkkyyol-aeeoooo proo"lem",
answer_key= "answe"",
env
gem.mmm. mako ("math waMMacc # ready to use
Next, we demonstrate how to build new environment from scratch by defining the initial state distribution (in reset () and the transition and reward functions (in step () as follows.
from gem.core import
2 from gom, enve. registration import ragistar
import TERMINAT_
4 from www.uuttlssssss.... import los____________dd____________
class ReveraeS::iingenv(eim
7 def iniit (selE, str_ien: int 5):
super init_()
9
sel falffatrllen str_ler
11
def ieget_instrucciossssellll
"You are tasked to reverse given string. \nn
"You may provide your response in any manner. only the content wrapped inside \ boxed:] will be considerad your final answer. An" t"Please reverse the string: (self.gt_str)..w
15
def reset (self, sead=None)
super ()pert).reeet reset (seed)
characters atring. letters string.digits A-H, a-z, 0-9 self.gt_atr "r_join(randon choi cas (charactars, k=self.str_len)
22
return seif__.get_nnstcuuttioos (1, []
24
def step (self, actien):
cleas_act aasa_aasoon extract_laat laa_t_booed__aeeee_eeee annwerraaseeeereeee (action)
if clean_action is Nore:
27
reward
29
reward float aclean_actionil 1] -- self.gt_sta
TERKINAL STATE, zeward, True, True,
31
6 Registez environnent
regi ooer "custom: KouutoosReeessstriinng ReveraeStringenv)
36
env
make ("custom: Reverssesi
B CASE STUDIES OF LANGUAGE GAMES
This section illustrates example interactions game-playing agents trained on GuessTheNumber under different discount factors 7. With undissudieo setting (y 0.999), the agent fails to complete the game within the trial budget, as it lacks incentive to obtain the terminal correctness reward earlier as possible (see example below).
14

---

Published as a conference paper at ICLR 2026
(y - 0.999, the agent exhausts all 10 trials)
State: You playing Guess The Number. You have guess the number between and 50 (inclusive) within 50 turns. ... As you play, the history of your guesses will be appended below. Use the information complete the game before you run of guesses. Enter your first guess to start the game.
At turn 1, you guessed 14, and the target number is higher than 14. At turn 2, you guessed 22, and the target number is higher than 22.
At turn 10, you guessed 39, which has been already guessed before. Enter your next guess.
Action: Let's break down the problem step by step:
**Initial Guess (Tum 1):** The target number is higher than 14. This means the target number is between 15 and 50 (inclusive).
2. **Initial Guess (Turn 2):** The target number is higher than 22. This means the target number is between 23 and 50 (inclusive).
From the final two steps, the target number is between 37 and 50 (inclusive). The final answer is |boxed|38).
In contrast, when using a more myopic return estimate (7 = 0.9), the agent is encouraged to minimize the number of turns, discovers the binary search strategy, and converges to the optimal policy.
= the agent learns to conduct binary search)
State: You are playing Guess The Number. You have to guess the number between and 50 (inclusive) within 50 turns.
As you play, the history of your guesses will be appended below. Use the information complete the game before you run of guesses. Enter your first guess to start the game.
At turn I, you guessed 25, and the target number is lower than 25. At turn 2, you guessed 12, and the target number is higher than 12.
At turn 3, you guessed 18, and the target number is higher than 18. At turn 4, you guessed 21, and the target number is higher than 21.
At turn 5, you guessed 23, and the target number is lower than 23.
Enter your next guess.
Action: To determine the target number between and 50, we can use a binary search strategy.
which is effective for narrowing down a range when given hints like 'higher' 'lower.'
Step-by-Step Reasoning:
** Initial Guess (Turn 1):** The target number lower than 25. This means the target number
is in the range [1, 24].
2. **Second Guess (Turn 2):** The target number is higher than 12. This means the target
number is in the range [13, 24).
3. **Third Guess (Turn 3) The target number is higher than 18. This means the target number
is in the range [19, 24).
**Fourth Guess (Turn 4):** The target number is higher than 21. This means the target
number is in the range [22, 24].
5. **Fifth Guess (Turn 5):** The target number is lower than 23. This means the target number
is in the range [22, 22). Final Guess:
The target number in the range [22, 22], which means the target number is 22. Final Answer: |bxxed
15

---

Published as a conference paper at ICLR 2026
C
ALGORITHM
For completeness we include the full algorithm of ReBN in Algorithm 1.
Algorithm Multi-tum REINFORCE with Return Batch Normalization (ReBN)
Require:
Policy ne. Environment M = P. P.P). Batch size B
I: while not converged do
2: Reset batch buffer B +- 0
3: while [B] B do
Multi-turn episode collection
5: Sample the initial state ~
6:
for turn = 1. T until terminate do
7:
Ye (-(ss):)
Generate reasoning action
a. axtract_action(s))
9:
Pt - R(s,, ac)
10:
8c+1 f P(st, at)
11:
end for for
12:
0. ,...,
do km 2 - Tm
13:
Gt + > J-
Compute discounted return
14:
Add (st; Ut, Gt) to B
15:
end for end while
16:
17:
Return Batch Normalization
18:
G. + (G. mean(G)) std
19:
// PPlicy optimization
b Or proximal update for data reuse
20:
Update using Monte Carlo policy gradient Σ G. (v)(ss))
21:
end while
D EXTENDED EMPIRICAL STUDIES WITH GEM
math:Orz57K
qa:HotpotQA
math:Orz37K
qa:NotpotQA
Moan Episode success
(d)
t.3
0.5
2.4
03
0.0
1.0
103
100
425
320
302
400
520
100
200
300
<50
200 são 400 são Policy Itaration steps
16
Policy iteration steps
Policy iteration steps
Policy Iteration steps
w.. ReBN
o ReBN
Neg rew
Figure 9: Learning curves of different reward shaping strategies. (a-b) The average success rate of two environments. (c-d) The corresponding average number of turns taken solve the tasks, equal the number of tool calls minus onc.
D. IMPROVING LEARNING EFFICIENCY VIA RETURN BATCH NORMALIZATION (REBN)
As briefly discussed in Section 4.1, while REINFORCE demonstrates strong performance across environments, its convergence can be suboptimal in certain cases. To investigate this further, we present an in-depth ablation study here. Following minimalist principles, we began with the vanilla REINFORCE, algorithm and a simple reward scheme: r = for correct answers and
otherwise. This approach has been shown effective for single-turn RL training (Singh et al., 2023; Xiong et al., 2025). However, as shown in Figure 9(c) (w.o ReBN), it failed induce tool usage in multi-turn settings, despite significant amount of initial attempts.
We hypothesize that this failure arises from the absence of negative gradients under 0/1 reward shaping. which crucial for efficient learning and exploration. To address this, introduced negative gradients in two ways: (i) assigning fixed negative rewards (r = for correct and r = -1 for incorrect answers, denoted as Neg rew in Figure 9); and (ii) applying Return Batch Normalization with
16

---

Published as a conference paper at ICLR 2026
0/1 rewards, where Monte Carlo returns in REINFORCE normalized described in Algorithm 1 (denoted as ReBN in Figure 9). While both 0/1 and =1 reward schemes theoretically induce the same optimal policy, they might exhibit markedly different learning dynamics in practice.
Notably, ReBN demonstrates strong and consistent performance across environments- not only in math and QA tasks (Figure 9) but also in all other settings (Figure 1). We also observe that models can be sensitive to fixed reward shaping: for example, Neg rew fails to improve tool use in ma wathh: Orz57k, 7K, yet leads tool overuse in qa HotpotQA, both of which are suboptimal behaviors.
D.2 RL ON VISION-LANGLAGE ENVIRONMENTS
In addition to text-only environments, we support visual elements as part of the observation for the agent understand and take actions. As demonstrative example, we build visual-language environment based on Geometry3k dataser for training reasoning agent to solve geometry math problems with images input. We RL-tune Sween2.5VVL-3BB7BB-Bn
(Bai et al., 2025) using
Dr. GRPO (Liu et al., 2025b), and the learning curves are shown in Figure 10. An example reasoning trace shown in Figure 11.
Training Rewards
Validation Scores
0.7
0.45
0.4
0.30
0.25
0.1
50
1.0
230
200
250
300
33.0
400
30
100
250
200
250
300
320
400
Pollicy Reeation stop
Policy Iteration stop
Figure 10: Learning of vision-language agents. We RL-tune
using Dr. GRPO
on the math Geomet 3 environment track their training rewards (left) and validation scores (right).
Problem: Fimage--Find SXS that Sm
Response:
To determine the value of it i that the lines mill and u are paraliel, we need zusure that the conespanding angles are equal. Since the lines 1) and ( porillel, the cormopundlngg ingles that formed by the interscetion the transverall are equal. Therefore, angles 24g feirel)) and 84y"cine)) equal:
I
(2) + 2ein 84y"leire
We can set the sngles agual to each other
[
y
well side equation
side:
1 2x
22
1
63
Therefore, the value of x that makes the lines 'I and Vi parallel (iboxad]63/11.
Figure 11: An example problem and the response of a trained agent based on Qwen2.5-VL VL--VL---I-mmtt
17

---

Published as a conference paper at ICLR 2026
TAU-bench Results
(Gemint-2.0-Flash--be
-80
User Model
-6
Medium
(GPT-4o-mini)
27.8
38.3
51.3
Success
-40
20
Strong (GPT-4o)
44.3
48.7
61.7
Weak
Medium (GPT-4o-mini)
Strong (GPT-4o)
Secmini-2.0--lash---
Agent Model
Figure 12: Multi-agent evaluation on TAU-benee retail. Stronger user simulators (rows) consistently improve agent performance (columns) across model strengths.
D.3 MULTI-AGENT ENVIRONMENTS
Many scenarios considered within reinforcement learning, such as adversarial games, social dilemmae and multi-pari control systems require multiple agents by definition (Albrecht et al., 2024), This introduces whole host of different design decisions which a general multi-agent API should be able to embrace. By studying previous approaches to defining multi-agent APIs (Terry et al., 2021; Lanctot et al., 2019; Bou et al., 2023; Hu al., 2023) and condensing them into simple abstraction scheme, GEM also provides high-level API for multi-agent LLM RL.
In doing so, GEM enables convenient development of agents that can collaborate, compete, or simulate realistic interactions with other agents/entities, adding comprchensiveness its extensible platform.
Design. GEM's multi-agent API is designed with the following principles in mind:
Minimal Prescriptions. The API only imposes constraints and methods when it is helpful do so. This includes the AgentSelectos class which allows for arbitrary definitions of turn order via overriding.
One Abstraction, One API. There should be only one way to realize the environment abstraction regardless of the environment's properties.
API overview. GEM provides Mu MultiAgemn MultiAgentEmy base class that extends the standard Gym API to support multiple agents. The step () and reset () functions operate on dictionaries keyed by agent identifiers:
from gem. multiagent import sultiagen:sny
2
env mydultiAgentinvil
4 obsarvations, infos cnv.reset () Dict (agont_id, observation]
while not done:
actiens (agent_id, id: agent.act (obs) for agent_id, id, cbs observations items ()
observations, ravards, terminations, truncocions, infos env.step (actionn)) done alll:errinnttonn..aallll ())
To implement custom environment, users inherit from Mult Multife ingentEEn and implement observe (agent) and axrocess_actions (actions). The framework handles agent lifecycle management and cumulative rewards tracking. Tum coordination is managed via AgentSelect.on which supports two modes: sequential (agents act one at a time round- robin order) and parallel (all agents act simultaneously). The environment can then determine
18

---

Published as a conference paper at ICLR 2026
which agents are active at each step and automatically advance turns, enabling flexible multi-agent interaction patterns without manual bookkeeping.
TAU-bench retail integration. We demonstrate this API by integrating the TAU-bench retail benchmark ( (Yao et al., 2024), which evaluates conversational agents on customer service tasks. We formulate this as two-agent environment: an ass ant agent using tools (order lookup, product search) and a user agent simulating customer behavior via an LLM. The user simulator is initialized with task instructions and generates queries; the assistant must satisfy these requests before episode termination.
Impact of user model strength. A key question in multi-agent RL is: how does simulated user agent capability affect trainable assistant agent learning? We vary both user and assistant models across three levels: weak configurations to study user-assistant model interactions.
2.0-iii-.2---ass-iiiii medium (GPT-40-mini), and strong (GPT-40), yielding
Evaluating across all 115 tasks from the TAU-bench retail test set (Figure 12), we find that stronger user agents consistently improve overall success rates across all assistant agent model strengths. Notably, the strongest assistant (GPT-4o) exhibits the largest absolute performance gains (20% from weak to strong user), achieving 61.7% success with a strong user simulator. Interestingly, strong user paired with a weak assistant (44.3%) outperforms a weak user paired with a strong assistant (41.7%), demonstrating that improving the agent is crucial for robust conversational task completion. These results motivate us to develop multi-agent RL to co-evolve user and assistant agents to achieve scalable and autonomous learning.
E RELATED WORKS
There is a significant body of work on tool-integrated language models models-including SkyRL-vO (Cao al., 2025), VerfTool (Jiang et al., 2025), ReCall and ReSearch (Chen et al., 2025a), Search-R1 (Jin et al., 2025), Re'oool (Feng et al., 2025a), and Simple'TIR (Xue et al., 2025). A common design pattern in these methods collect multi-turn agent-envvronment interactions as single continuous sequences of tokens of agent actions interleaved with environment outputs. Training then simply involves masking the environment outputs from the loss calculation.
However, this single-sequence approach presents two significant limitations. First, the state observa- tion is rigidly defined as the complete history of actions and outputs. This restricts the ability to use alternative state representations, such as pruning "thinking" tokens or summarizing the history to avoid exceeding context lengths. Second, this formulation inherently limits the reward structure to a single, trajectory-level signal, preventing the use of finer-grained, per-step rewards, and effectively fixing the discount factor at y = 1. In Section 4.2 we demonstrate that 7 < 1 is crucial for obtaining the optimal fastest search behavior. By contrast, with trajectory-level rewards, the natural speed incentive from y < is lost, and hence other works, such as SimpleTIR, must tune and enforce a strict turn-limit to get this behavior.
To address this, our framework. GEM, is designed for maximum flexibility by collecting trajectories as sequence of individual transitions (i.e., state, action, reward, next state) in the full, unsimplified RL. formulation. This design choice enables arbitrary state observation constructions (using observation wrappers), and also preserves compatibility with per-turn rewards and arbitrary discount factors < 1. The verl-agent framework (Feng et al., 2025b) also adopts this transition wise approach, which enables its implementation of GiGPO (Feng al., 2025b), an RL method that utilizes turn-level rewards. While GiGPO collapses to trajectory-level GRPO when observations are unique, it is an example of type of algorithm that is now straightforward implement with GEM's infrastructure.
There are multiple popular frameworks that focus on the agent training side (c.g.. Oat (Liu et al., 2024), Verl (Sheng et al., 2024), OpenRLHF (Hu al., 2024), ROLL (Wang al., 2025a), and RL2 (Tan et al., 2025)). Currently, many works that build on these, including verl-agent, RAGEN (Wang et al., 2025b), Verlog (Chen et al., 2025b), and many of the works above, add environments by directly modifying the source code. This results in tight coupling between training and environments, and makes it difficult to maintain and reuse the environments for future research. As a result, each codebase tends to support only small, ad-hoc collection of environments, making it hard to compare different methods. Even environments with the same name are often inconsistent between codebases. GEM addresses this by dealing with all the environment infrastructure, including providing
19

---

Published as a conference paper at ICLR 2026
diverse suite of environments, and corresponding baselines. This makes it easy to keep training and environments decoupled, with the aim of freeing researchers from cumbersome environment development and setup, and thus enabling quicker prototyping and evaluation of new ideas.
We also note that there are early works (Abdulhai al., 2023; Tajwar et al., 2025) incorporating text games the evaluation toolkit for language model fine-tuning However, they did not focus standardized suite of RL training environments nor did they investigate different RL algorithms.
F EXPERIMENTAL SETTINGS
All our experiments are performed on 8 x A100 GPUs and finished in about one day. The detailed experimental configurations shown in Table
Table 3: Hyperparameter configurations used in all experiments.
Parameter
Value
ACTOR
Maximum response length per turn Sampling temperature, train Sampling temperature, evaluation (top P. top k)
4096 tokens
1.0
0.0
(1.0,
LEARNER
Optimizer
AdamW
Adam parameters (31, (2)
(0.9, 0.95)
Weight decay
0.0
Gradient norm clipping
1.0
Learning rate scheduler Learning rate Inner proximal update epoch
Constant
1 10-6
2
KL loss coefficient
0.0
KL penalty coefficient
0.0
Policy clipping parameter
0.2
Discount factor
0.9 (game.qa); 1.0 (otherwise)
GAE
0.95
Steps
500
20


Reviews:
[{"review_id": "qSRMujj4Mg", "reviewer": "Reviewer_5brH", "summary": "The paper introduces an RL environment interface for LLM agents supporting multi-step (phrased as multi-turn in the paper) tool use. It aggregates tasks including ReasoningGym, text-based games (TextArena), coding, math, and QA, and integrates tools such as Python interpreters, search engines, and the MCP. The authors also benchmark a reinforce with Return Batch Normalization (ReBN) baseline across these environments.", "questions": "- Would it be possible to include open-ended environment design with more practical real-world usages in the current framework? If so, If so, what specific engineering/coding improvements would be necessary?\n- Training for long-horizon tool use suffers from many specific issues like temporal credit assignment. The proposed ReBN only serves as generic regularization, could the authors discuss more principled algorithmic designs specifically tailored to address more specific challenges inherent in agentic RL?\nI am happy to raise my scores if the concerns raised can be sufficiently addressed.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Engineering quality: The paper/report is well-written and represents a solid engineering effort, particularly in offering a unified interface compatible with five major training frameworks (Oat, Verl, OpenRLHF, ROLL, RL2).\n- Clarity: The motivation for transitioning from single-turn to multi-turn agentic workflows is clearly presented.", "weaknesses": "In general, I believe this is a solid engineering work which is helpful to many practitioners, however, the novelty of both the environment and the algorithmic design may limit its contribution to the academia.\n- Limited novelty (environments): The contribution is largely an aggregation of existing tasks and datasets (e.g., GSM8K, TextArena, ReasoningGym) rather than a fundamental/novel research contribution in environment design. The differentiation from prior tool-integrated frameworks is minimal, and the scale of environments and experiments are limited.\n- Limited novelty/relevance (algorithm): The proposed ReBN algorithm is presented as a contribution, but normalizing returns/advantages is a standard technique in the RL literature and does not constitute an original algorithmic contribution. It may be more meaningful to propose algorithms that focus on the special challenges of multi-step tool use scenarios in the real world.\n- Please place related works in the main body of the paper instead of downweighting them in the appendix. It is important to properly give credits to prior works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8558dbafzG", "reviewer": "Reviewer_hvWT", "summary": "This paper introduces GEM (General Experience Maker), an open-source environment simulator for agentic RL. Apart from a standardized framework for the environment-agent interface, GEM features a diverse suite of environments, robust integrated tools, and single-file example scripts demonstrating using GEM with five popular RL training frameworks. Additionally, authors also propose REINFORCE with Return Batch Normalization (ReBN), designed for multi-turn agentic LLM tasks.", "questions": "Please refer to Weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I appreciate the work authors did for building such an integrated environment simulator, which I believe will facilitate future research on agentic LLM.", "weaknesses": "1. The proposed REINFORCE with ReBN, seems like a trivial extention of GRPO to multi-turn tasks. Further practical investivation and theoretical foundation are required to fully justify its effectiveness.\n2. While the unification of existing benchmarks is a valuable contribution, this work would have been even more solid and appealing had the authors also analyzed the deficiencies of current benchmarks and built new ones based on that analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T3BnDkZ0hw", "reviewer": "Reviewer_LQDp", "summary": "This paper introduces GEM, a open-source framework designed to facilitate the training and evaluation of agentic RL (LLM-based). Drawing an analogy to OpenAI-Gym's role in traditional RL, GEM provides a standardized environment-agent interface, a diverse suite of multi-turn environments, and features for efficient experimentation like asynchronous vectorized execution. The paper also proposes a simple yet effective baseline algorithm, REINFORCE with ReBN, which is shown to be well-suited for multi-turn settings with dense rewards.", "questions": "ReBN vs. Advantage Normalization: Could you please elaborate on the empirical or theoretical trade-offs between using ReBN without a critic versus using a learned critic with GAE and advantage normalization? Is the primary benefit of ReBN its implementation simplicity and reduced computational overhead?\n\nRoadmap: This is an excellent starting point for a valuable community resource. Could you share more about your future vision for GEM? Specifically:\n1. Do you plan to incorporate more open-ended, sandbox-style environments that involve multi-turn interaction with persistent state and feedback?\n2. Have you considered structuring the benchmarks into different difficulty tiers or with specific focuses to allow for more targeted evaluation?\n3. Regarding framework integration, beyond individual scripts, have you considered designing a more universal and simple interface or adapter to further lower the cost of connecting to new and existing RL frameworks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, this paper addresses a critical need for standardized infrastructure of agentic RL. It provides a foundational tool that can accelerate progress and improve the rigor of research in this area. \n\n1. Comprehensiveness of the GEM Framework: GEM with a clean API, support for asynchronous execution , a diverse suite of tasks (games, reasoning, code, QA), modular tool integration (Python, Search), and extensibility through wrappers. This represents a significant  engineering effort.\n2. Rigorous and Insightful Empirical Evaluation: The paper includes a thorough benchmarking of several key RL algorithms.\n3. Practicality and Usability: The authors have clearly prioritized usability by providing ready-to-use, single-file integration scripts for five popular LLM RL training frameworks. This significantly lowers the adoption barrier and demonstrates that GEM is agnostic to the training infrastructure.", "weaknesses": "Novelty of ReBN: The core algorithmic proposal, ReBN, is an effective application of a well-known technique (return/advantage normalization) in RL. The paper is transparent about this, but it's worth noting that the primary novelty of the paper lies in the framework (GEM) itself, rather than in a breakthrough RL algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vZ5abrqkKb", "reviewer": "Reviewer_rtWh", "summary": "The paper presents GEM, which is a unified and reproducible environment suite for reinforcement learning with LLMs. GEM offers an OpenAI/Gym style API with asynchronous vectorization, tool integration (Python, Search, MCP), and multiple task types (Math, Code, QA, Reasoning, Terminal).\nThe authors benchmark REINFORCE, PPO, and GRPO, showing their Return Batch Normalization (ReBN) variant improves stability and efficiency across multi-turn tasks. GEM integrates with major RL frameworks and provides evaluation suites like Terminal-Bench and MCPMark. The main strengths are its clean environment API and comprehensive benchmarking; the main drawbacks are no released code and limited technical comparison to existing APIs.", "questions": "- Why not releasing the code in an anonymized way? That would help to judge how easy (or not) it would be for the community to adopt it (as well as doc, testing etc)\n- Is the vision to make GEM a parallel ecosystem of environments, or build extension points and integrations with other frameworks (recent OpenEnv or extension for VeRL etc)? The ecosystem of RL post-training libraries is already quite crowded, so interoperability with similar frameworks is key.\n- Can the authors provide throughput or scaling benchmarks (e.g., vectorized rollout speed, GPU utilization) to substantiate the claim that GEM enables high-throughput training beyond local setups?\n- Does GEM does use Ray or other RPC actor services, or any distributed rollout pool? Distributing actors across a cluster is key in settings where the tool / simulator requires a lot of compute resources.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I love the well-structured, reproducible environment interface modeled after OpenAI Gym. Asynchronous vectorized execution is cool, but I would advise against auto-reset which in my experience provides a very marginal gain and introduces a lot of pain when dealing with last observations of a trajectory. The clear separation between environments and trainers fills a major reproducibility gap left by prior frameworks like Verl and OpenRLHF.\n- Enabling turn-level rewards and arbitrary γ is a nontrivial practical and conceptual upgrade, although other frameworks like torchrl already provide this.\n- The paper has good benchmarks and ablations. The major frameworks are compared which gives a good intuition of the strengths of the library.", "weaknesses": "- not being able to see the codebase makes me worried about accepting the paper. Reproducibility is the base of any scientific advance, and since the core claim is about standardization and benchmarking, lack of release undercuts the entire point.\n- The related work acknowledges VerlTool, Verl-Agent (GiGPO), OpenRLHF, and RL2, but only superficially. Since the paper is about a technical advance and a new standardization of environment APIs, a clear comparison should have be undertaken. It doesn’t discuss TorchRL’s EnvBase or OpenEnv at all — which now also propose standardized step/reset interfaces and multi-process async collection.\n- It's unclear if GEM scales beyond small local setups or integrates well with cluster-level runners like Ray / Monarch or others.\n- ReBN shouldn't be framed as a new algorithm but as a useful implementation tweak.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)