R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning
Authors: Yongchao Chen
Yueying Liu
Junwei Zhou
Yilun Hao
Jingquan Wang
Yang Zhang
Na Li
Chuchu Fan
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Code Interpreter
Reinforcement Learning
Curriculum Learning
Symbolic Reasoning
Tool Use
Primary area: foundation or frontier models, including LLMs
TL;DR: We present R1-Code-Interpreter, an LLM trained with supervised and multi-stage RL to integrate code execution with reasoning, achieving strong performance across 144 diverse tasks and outperforming GPT-4o with Code Interpreter.

Abstract: Practical guidance on training Large Language Models (LLMs) to leverage Code Interpreter across diverse tasks remains lacking. We present R1-Code-Interpreter, an extension of a text-only LLM trained via multi-turn supervised fine-tuning (SFT) and reinforcement learning (RL) to autonomously generate multiple code queries during step-by-step reasoning. Unlike prior RL + tool-use efforts focused on narrow domains such as math or retrieval, we curate 144 diverse reasoning and planning tasks and show that training a general-purpose Code Interpreter across them presents significant challenges due to task heterogeneity and scarcity of effective samples. To address this, we introduce a multi-stage curriculum learning approach that partitions training samples by measured improvement potential. The RL training prioritizes samples with higher potential and gradually shifts to lower-potential ones, increasing the average RL gains from merely +3.4\% to +9.3\% across Qwen-2.5 models (3/7/14B). Our final model, R1-CI-14B, improves average accuracy on the 37 test tasks from 44.1\% to 72.4\%, outperforming text-only GPT-4o (58.6\%) and GPT-4o with Code Interpreter (70.9\%). Notably, R1-CI-14B also exhibits emergent self-checking behavior through code generation. Datasets, Codes, and Models are available at https://github.com/yongchao98/R1-Code-Interpreter and https://huggingface.co/yongchao98.

Published as a conference paper at ICLR 2026
R RI-CODE-INTERPRETER: LLMs REASON WITH CODE VIA SU- PERVISED AND MULTI-STAGE REINFORCEMENT LEARNING
Yongchao Chen
Yueying Liu University of Illinois Urbana-Champaign 1.1368i1linois. edu
MIT' / Harvard
yongchaochen fas. waw.saaaarra harvard.edu
Junwei Zhou
Yilun Hao
Jingquan Wang
University of Michigan
MIT
University of Wisconsin-Madiin (aang2373wwisee edu
zhou jw? wwwchichhh...mmmcmmmmmmmmmm.cc.e edu
yilunhao@mit.ed
Yang Zhang
Na Li
Chuchu Fan MIT chuchu@mit. edu
MIT-IBM Watson Al Lab Yang. Zhang20 ibm. com
Harvard
nali@seas www.aaad..da.......
ABSTRACT
Practical guidance on training Large Language Models (LLMs) to leverage Code Interpreter across di- verse tasks remains lacking. We present R1 -C-Code- Mi--ood-Inteepreee Interpreter, an extension of text-only LLM trained via multi-turn supervised fine-tuning (SFT) and reinforcement learning (RL) to autonomously gener- ate multiple code queries during step-by-step reasoning. Unlike prior RL + tool-use efforts focused on narrow domains such math or retrieval, we curate 144 diverse reasoning and planning tasks and show that training a general-purpose purpose Code Interpreter across them presents significant challenges due to task heterogeneity and scarcity of effective samples. To address this, introduce multi-stage stage curriculum learning approach that partitions training samples by measured improvement potential. The RL training prioritizes samples with higher potential and gradually shifts to lower pewerr-ptential ones, increasing the average RL gains from merely +3.4% to +9.3% across Qwen-2.5 models (3/7/14B). Our final model, R1-CI-14B, improves average accuracy on the 37 test tasks from 44. % 72.4%, outperforming text-only GPT-4o (58. 6%) and GPT-4o with Code Interpreter (70.9%). Notably, RI- CI-14B also exhibits emergent self-checking behavior through code generation. Datasets, Codes, and Models are available at https: / com http:/yonnccc/ooeeeeee R1- Interpreter and
co hot..ccccoooogggge
1 INTRODUCTION
While reinforcement learning bRsed fine-tuning has significantly improved LLMs' reasoning and planning Wang et al. 2024): Guo et al. (2025): Jaech et al. (2024), models still struggle with seemingly simple tasks (Chen et al. 2025) and incur high token costs during inference-time search (Chen et al. (202a)). Textual reasoning excels at semantics and commonsense, but falls short in precise computation, symbolic manipulation, optimization, and algorithmic processing (Valmeekam et al. 2022). In contrast, symbolic code generation handles these rigorously and benefits from external tools (e.g., equation sol vers). Prompting LLMs to generate and execute code often outperforms pure textual
reasoning (Madaan al. 2022 Liang et al. 2022 Chen al. (022).
A key challenge is guiding LLMs to decide when to rely on textual reasoning versus programmatic solutions, given that most input questions lack explicit cues about which approach is best and the possible text/code solution space is large. OpenAl's GPT models address this by incorporating a Code Interpreter, allowing iterative code generation and reasoning outputs (Achiam et al. 2023). However, recent work Chen et al. (2024c) show that current Code Interpreter implementations struggle to effectively steer between text and code, underutilizing symbolic capabilities. Recent work such ToRL (Li et al. 2025b) and ReTool (Feng et al. 2025) investigates training reasoning models to integrate with Code Interpreters. However, their training and eval evaluation luation are limited to math problems, leaving a significant gap from real- world applications that demand effectiveness across broader benchmarks. ToolRL (Qian al.) 2025) instead focuses on teaching models to select among multiple tools, where the Code Interpreter is used only for generating relatively simple code. Currently, public research still lacks comprehensive understanding of how to fine-tune LLMs to integrate with Code Interpreter for robust, generalizable performance across hundreds of tasks.

---

Published as a conference paper at ICLR 2026
(a)
(b)
Training and Testing Score of GRPO with CL and IP
Train Thckks
80
37 Test Tasks
Stage
Stage 2
Stage 3
Stage
Score
60
Score
Training reward
Testirig score
Ohedee 148
ABl Tout Owet- 14B
Ohean 148
1488
14B
touns) ape4e
of GPt Chale
Ct.
Inteppicces
390
300
50
689
Steps
(c)
Code
Stage 1: high potential samples
Agent
Potential Ranking
Stage moderate potential samples
CodeSteer
107 Train Tasks
GRPO Finetuning with different stages
Stage 3: emerging potential samples
All text
Stage 4: low potential samples
All code
Curriculum Learning
Agent group calculate improvement potential
Figure 1: Training Code interpreter-augmentee reasoning models with multi-stage GRPO on 144 reasoning and planning tasks. (a) Our best model, R1-CI-14B, outperforms both GPT-4o (text-only) and GPT-4o with Code Interpreter. (b) Training reward and test scores improve steadily through the curriculum learning, then plateau stage 4 after adding low-potential samples. (c) To assess sample effectiveness, we estimate improvement potential by repeatedly sampling answers with different agent frameworks and analyzing the correct/wrong distribution. GRPO begins with high-potential samples and gradually incorporates lower-potential ones.
To tackle these challenges, we present RI E1-Codd-Interppeee source LLMs. We curate 144 reasoming and planning tasks and synthesize 6.5k multi-turn text/code trajectories for supervised fine-tuning (SFT), followed by Group Relative Policy Optimization (GRPO) (Shao et al. 2024). In contrast to prior work, which shows that RL training can yield substantial improvements on single or simple tasks, we find that applying traditional DeepSeek-style RL training to a general Code Interpreter across 144 challenging tasks yields only marginal gains. This difficulty arises from task heterogeneity and the scarcity of effective samples. To address this, we propose novel multi-stage curriculum learning approach guided by measured improvement potential. Specifically, after the SFT stage, the model is prompted with diverse single- and multi-tun multi-turn agent strategies to answer the same question. The accuracy discrepancies among these answers used to estimate improvement potential. RL training then proceeds in four stages, beginning with high-potential samples and moving to lower-potential ones.
framework for integrating Code Interpreter into open-
We finetune Qwen-2.5 (Qwe al. 2025) models (3/7/14B), achieving average success rate improvements of 33.7%
107 train tasks and 34.1 14.1% on 37 test tasks. As shown in Fig. [ our best model, R1-CI-144 11,, raises testing accuracy from 44.1% to 72.4%, outperforming much larger GPT-4o text-only (58.6%) and GPT-4o with its inherent Code Interpreter (70.9%). The resulting model effectively combines code execution with textual reasoning, solving tasks through iterative eexecute-and-explore interactions before producing final answers. The model gradually learns to generate code to verify its answers, an emergent behavior rarely observed before training. Our main contributions are:
1) To our best knowledge, this is the first published work train general Code Interpreter across multiple tasks and domains. Unlike prior work that focused on single task or tasks with simple reasoning, we curate 144 challenging reasoning and planning tasks, each with over 200 samples of varying difficulty. All tasks are standardized into a unified format to enable efficient rollout and automated correctness evaluation. They cover diverse reasoning skills, including mathematical, spatial, logical, ordinal, optimization, and rearch-based reasoning.
2) Analysis of RL limitations and proposal of an effective multi-stage curriculum learning framework guided by improvement potential. We synthesize 6.5k multi-turn trajectories with interleaved reasoning and code execution for SFT, followed by RL for further optimization. By tuning the number of training tasks, find that RL for
2

---

Published as a conference paper at ICLR 2026
general-purpose Code Interpreter is substantially more challenging due to task heterogeneity and scarcity of effective samples. Our proposed multi-stage curriculum learning with measured improvement potential effectively mitigates this bottleneck, raising RL performance gains from +3.4% to +9.3%.
3) Cost-efficient training by separating model gradient calculation from code execution. We find that time- consuming code execution significantly reduces GPU utilization and accounts for large portion of RL training time in the Code Interpreter, limiting the maximum batch size and hindering parallel efficiency. To address this, we design specialized Code Execution Sandbox on multiple CPU nodes, decoupling execution from GPU-based gradient computation. This approach reduces overall training time by 39%, decreasing from around 4500 to 1845 GPU hours.
4) Comparison of training strategies: 1) The multi-turn Code Interpreter framework proves more generalizable and effective than single-turn text or code generation frameworks. 2) Warm-starting with SFT significantly improves the training of Code Interpreter. 3) Qwen-2.5 models base are better than DeepSeek R -isitiidled reasoning models.
2 CHALLENGES IN 144 REASONING AND PLANNING TASKS
We compile 144 tasks from three major reasoning and planning benchmarks: 33 from SymBench (Chen al. 2025 27 from Big- Hig-Bencch-Hate (Suzgun et al. 2022). and 84 from Reasoning-Gymff After removing near-duplicatss, each task retains over 200 diverse samples. All tasks are standardized into unified format and evaluated using rule-based criteria (e.g., exact match or constraint checks) for efficient rollout and testing. The tasks cover diverse reasoning and planning challenges for LLM evaluation. Detailed task descriptions are Appendix Sec. [H] and their associated capability categories, logic, spatial, order, optimization, search, and math, are summarized in Appendix Table Recent work (Chen et al. (2024c) highlights key challenges steering LLMs to choose effectively between code and text: (1) The better mode (code vs. text) depends on task type, complexity, and model capabilities, which is often hard to predict; (2) LLM-generated code frequently degenerates into hard-coded text-like scripts, limiting its symbolic utility (Yang
ct al. 2024); (3) Many challenging tasks require models to iteratively explore and optimize the solutions in difficult multi-turn behavior. Appendix Fig. shows a representative example of GPT-5 failing to use the Code Interpreter.
Question: Blocksworld Task: Initial State: stackl: D stack2: 6 C A stack3: 8 I E Goal State: stack1: A F H stack2: C stack3: D Please provide series of to reach the geal state. Your
RI-CI-148: Let solve this Blocksworld prablen
by step. First, let create Python script help sinulate and find the solution. The script will: Represent the stacks VValidate moves (only top blocks can be noved] Find path fron initial to goal state Use search algorithn to find the solution. Use synbolic efficient search algorithn,
computing
implenent depth-first search (DFS) approach .sonitted textz..
"" python
def is_goal__tatteeeeaaaaaeeeeeesss_ statsssstts,,,,, goal_state) return state == goal_state
def pec_possible_noves(tt
noves - =!]
for stack in enunerate(state): ..onitted codes..
Code Execution Results: FimeoutExpired:: Command "run.py" timed out after 60 seconds.
The current approach us using depth-first search (DFS) is step in the right direction, but it seens to be inefficient, leading to tineout. To optinize search for solution in the Blocksworld task, can implement the As search algorithn. As search is more efficient than DFS .sooitteed text>.. "python
import heapo
def pet_posstble_noes//eee
moves !! in enunerate(state): -ooitttdd codes..
Code Execution Results: Move Ht from 3 to 1\nMovve E fron 3 to 1'nMtvve I from 3 to 2\nMave F fron 2 to 3. see the code has found valid solution. Let ne verify this solution by writing the checking code ..conitted texts.. ppytonn
def verify_blie wtw.yyyboocswwrlddooo soteridddoturiioonntttaa
goal_state, noves):
current_staie _sttttstteeee-e (stack: list(blacks) for stack, blocks antiai_state.iee for move in noves:
parts spiit))
sonitted codes..
Code Execution Results: torrect
The current solution is correct: ece Hove fron to 1\MMvv fron to TVnoove fron to 2NnMove fron 2., >>>
Figure 2: Example response of R1 Ro--erde-ntterpreter in Blocksworld task.
3

---

Published as a conference paper at ICLR 2026
3
R RI-CODE-INTERPRETEE
Fig. 2 shows representative example of generates code for execution, and refines its reasoning based on the results, continuing this process until the final answer is produced. The Code Interpreter is invoked only when deemed beneficial; otherwise, the model relies on pure textual reasoning. The system instruction directs the model to enclose code between the natural tokens python' and the Code Interpreter, then appends the result (prefixed with the special token 'Code Execut ion Results: to the ongoing generation. This loop continues until either (1) the maximum of 8 code calls is reached, or (2) the model emits final answer enclosed between '<<<' and >>>>
in action. The model iteratively reasons, optionally
when execution is needed. Upon detecting a code block, the system extracts and executes via
To train ---Code-Interpeetee we begin by designing a simple head prompt that guides the initial LLM to follow our predefined structure. As shown in Table I the prompt organizes the output into three iterative parts: reasoning, optional Code Interpreter invocation, and the final answer. We avoid imposing content- content-specific constraints (e.g., enforcing reflective reasoning or code calls) preserve the model's natural learning dynamics during RL.
Response format:
Unlike prior work that enforces section tags like '<think>', '<answer>', or '<search>' (Guo et al. 2025 Jin et al. 2025 Zhang et al. (025)), rely solely on the final answer marker "<<<answer content>>>>
for answer extraction. For code, we leverage the LLM's pretrained behavior of naturally starting code blocks with .*. python', which serves as implicit tagging. Our initial tests show this natural format performs better than forced tagging, as it aligns more closely with the model's original distribution.
Table 1: Head prompt for R1-Code-Interre Interpertrrrrr.r
The User asks question, and you solve it. You first generate the reasoning and thinking process and then provide the User with the final answer. During the thinking process, *Eyou can generate python code** for efficient searching, optimization, and computing with the format starting the python block with python. **A code query must involve only a single script that uses 'print' funct ion for the output . *x, Once the code script is complete, stop the generation. Then, the code interpreter platform will execute the code and return the execution output and error. Once you feel you ready for the final answer, directly return the answer with the format <<<answer cont ent >>> at the end of your response. Otherwise, you can continue your reasoning process and possibly generate more code query to solve the problem.
4 TRAINING SETTINGS
We fine-tune RI E1-Code-Interpretee using SFT and GRPO on a subset of 144 tasks. We randomly select 107 tasks for training: 26 from SymBench, 20 from Big- hig-Bench-Harde and 61 from Reasoning-Gr with the test set. The remaining 37 tasks are used for evaluation. To generate SFT supervision, we prompt GPT-4o to produce multiple reasoning/cxecution trajectories per task and retain only those yielding correct answers. To enhance diversity and adaptability, we use varied prompt formats: some allow free-form reasoning such as the prompt in Table while others enforce transitions between text and code. To encourage exploratory reasoning, we increase the proportion of answer trajectories with multi-turn generation, particularly those that adaptively adjust solving strategies (e.g., switching between code and text, or refining generated code). To balance the dataset, each task includes at most 70 valid trajectories, resulting in a final dataset of 6.5k high- high-quaiity samples for SFT. To avoid training collapse, the training samples in GRPO has no overlaps with SFT. We conduct experiments using three default base models: Qwenn..-114B-IIstmnee 1M, Dowen2.5-7B-Instteee MM, and Qwen2.5-3BB -3B-truuttttt (Qwen et al. 2025).
ensuring sample overlaps
SET is trained for 3 epochs to prevent overfitting. GRPO uses a learning rate of le-6 with 5 sampled responses per prompt and KL penalty of 0.001. Learning rates are tuned in early-stage experiments. Training and inference temperatures are set to 1.0 and 0.6, respectively. We use a batch size of 32 for SFT and 128 for GRPO. Both stages perform full-parameter fine-tuning on 16 H100 GPUs. Answers are evaluated using predefined rules, with GPT-4o assisting in format normalization when necessary. For methods that output code as the final answer, we extract and execute the code using predefined logic to obtain the final result.
4.1 GRPO WITH CODE INTERPRETER
We formulate our RL objective with a Code Interpreter C as:
max Eg y==)(() [ro(x. v)] B DEL [To(y) xx:)] || Feer(y) x:; xx:)]]
(4.1)
where To is the policy LLM, nee is the reference model, re the reward, and Dyk, is the KI. divergence (Shlens 2014). Unlike prior work Guo et al. 2025) that samples from me(- | æ), our policy Te(- 2 C) integrates external code
4

---

Published as a conference paper at ICLR 2026
execution, enabling hybrid reasoning. For each input z, GRPO samples group of responses {31, U2. reference policy Tref and optimizes the policy by maximizing:
yG} from the
G
To(Vitt
2, V.Viiii;C C)
(O) EE D, Vi.o Fou (1230)
min
Are,,
Taan Fite I., Vi,icii)))
(4.2)
Folyse 3r W, C)
clip
1 1+ 1+)) Astt)
/3 DKL(TO mme]
Trer( I. Ni,
where c and B are hyperparameters, and Are is the advantage, computed from the relative rewards of responses within each group. We mask code execution tokens and compute the policy gradient only over L.M--generate tokens. The total rule-based reward Ri is defined as weighted combination of correctness, format compliance, and efficiency. The agent receives +1.0 if the final outcome correct, +0.1 if all intermediate responses satisfy the format requirements (and -..1 otherwise), and -0.1 if the number of generation turns exceeds six. In factual reasoning tasks, final outcome correctness is judged by exact matching: in planning tasks, it checks whether all constraints and goals are satisfied.
Code Execution Sandbox: To save training and inference time, code execution allows up to 8 code calls and a 60-second timeout per script. However, as shown Appendix Fig. [2] code execution lowers GPU utilization during training. Multi-turn code execution can be time-intensive, making Code Interpreter training significantly slower. Meanwhile, executing code on the GPU increases the risk of memory overflow, which forces smaller batch sizes and reduces training efficiency. To address this, we decouple gradient computation from code execution by deploying specialized sandbox on five 64-core CPU nodes. Generated codes are executed directly in this sandbox in parallel during batch inference, reducing overall RL training time by about 39%.
4.2
BOTTLENECKS IN GRPO TRAINING OF CODE INTERPRETER
(a)
H80, GRPD un Curricsiun Learsing
(b) Testing Score, 14B, GRPO wo Curriculum Learning
(c)
One Task, Game24
Traknton Rewwrd
Snasiig Remand
60.1
62.2
Curriculum
(d)
Score Imprrvement Number
Foam Seprocmnt
30
155 50
107
Truining Steps
Training Steps
Numther Tasks
Figure 3: GRPO training without curriculum learning. (a) Training rewards increase slightly in the early steps, then plateau. (b) In the 14B setting, test scores across individual tasks (colored lines) show diverse trends, while the average score (bold black line) rises slightly before plateauing, mirroring (a). (c) Training curve on the single task Game 4. (d) Average score improvement vs. number of tasks for GRPO training.
Fig. 3-bb show the evolution of GRPO training rewards and test scores. Unlike prior work focusing on single task (Jin et al. 2025| Zhang al. 2025 Guan et al. 2025), where direct RL training effectively integrates tools with L.MSs and achieves notable improvement, we find that training a Code Interpreter on over one hundred diverse tasks is highly challenging. Direct SeepSeek-style GRPO yields little improvement: task heterogeneity dilutes the reward signal, preventing effective optimization. Moreover, as shown in Fig. 3p, many tasks remain at scores below 10 (often O), indicating that tasks are too difficult for LLMs solve and rewards are too sparse drive progress. In Fig. Bp. training on single task shows clear improvement, consistent with prior work on single-task RL (Li et al. 2025b Qian et al., 2025 Feng et al.) 2025). Fig. 33 further varies the number of training tasks while fixing 50 samples per task. We observe that the maximum average score improvement decreases with task count, from a 27. 47 lift in the single-task case only 3.3% with 107 tasks. These results highlight how severe task heterogeneity and sparse effective samples hinder GRPO training, limiting its effectiveness.
5

---

Published as conference paper at ICLR 2026
Why vanilla GRPO underperforms on mixed data? Let E {0,1} be the final reward for rollout in group of size G, with group mean F à -j= Broadcasting the sequence-level advantage tokens (and omitting clipping for clarity), the GRPO gradient splits into a policy term and KL regularizer:
VoJGRPO
Σ i=1
(n
v)
VaDDKL. Meef
Vo log no (V(. ct C).
(4.3)
If lid Bernoullipp), then
E (rr - [+)- = p p) 1 - )
(4.4)
Applying
to the policy term in equation 4.3 yields
G (r F)
S p(1 p(1- (1-
E
(
E
(4.5)
G
G
Hence the policy signal is maximized at and vanishes as + or 1. In batches dominated by too-easy or too-hard items, the update is therefore governed by the KL term, contracting To toward Tea. Hence, optimization cannot make headway. See Appendix Sec. for detailed proving and articulation.
4.3 MULTI-STAGE CURRICULUM LEARNING WITH POTENTIAL MEASUREMENT
Training and Testing Score of GRPO with CL and IP
Score Distribution of Methods
All Text + CoT
Stage
Stage
Stage
Stage
Code Interpreter tume
RI-Code-Inlerpreic GRPO
R1 Code Interpreter
Prequency
50
Training reward
20
Testing score
100
200
300
400
500
600
Steps
Scores
Figure 4: Multi-stage curriculum learning with the guid- ance of measured improvement potential for each sample.
Figure 5: Score distribution across 144 training and testing tasks for the four compared methods.
Based on above phenomenon and analysis, we propose a multi-stage curriculum learning method for GRPO training of general Code Interpreter. Unlike conventional curriculum learning, which progresses from easy to difficult samples, our approach orders samples by their improvement potential, i.c., their expected benefit to model optimization. Samples that are either trivially easy (almost always solved) or excessively difficult (almost never solved) provide limited training signal. In contrast, samples that the model solves correctly about half the time offer the strongest optimization opportunities: training can steer the model toward the correct solution.
LLMs, problem-solving can follow multiple strategies (e.g.. pure textual reasoning, code execution, or mixed multi-turn approaches), resulting to notable performance gap. To capture this variability, implement four agent variants (Table 2 each using distinct strategies solve the same problem. Starting from the SFT-initialized base model, each agent generates samples at different temperatures, yielding N 20 answers per question. Let di denote the j-th answer for sample 2;, and Vi.j E {0, its correctness label. The empirical correctness rate and the improvement potential score are defined as
Estimating improvement potential: For Code-
Cooee-Interpreee--aaggeeee
Pr(1 - pi) 4p; - Pt).
pi Σ
II
(4.6)
By construction, II; E [0, 1]. maximized when pt = 0.5 (mixed outcomes) and minimized when pt E [0,1) (uniformly correct or incorrect). This formalizes our intuition and analysis in equation 4.5 that greatest improvement arises from samples with balanced successes and failures.
6

---

Published as a conference paper at ICLR 2026
Table 2: Four pre-designed agents used in the measurement of improvement potential.
Full Name
Description (4 agents)
A11
Text
Prompting LLMs to reason using only text with Chain-of-Thought (CoT) (Wei et al. 2022).
All Code
Prompting LLMs to first reason with CoT, then produce code as the answer.
Code Agent
LLM determines the use of Code Interpreter during problem-solving with the prompt in Table
CodeSteer
Code Agent guided by another Steering Agent based on the same LLM (Chen et al. 2025).
Multi-stage curriculum:
To implement curriculum learning. we sort all training samples by II; and partition them into four equally sized groups (four group IP ranges are: [0.0, 0.32], [0.32, 0,48), [0.48, 0.64), and [0.64, 1.00), from highest lowest potential. The partition is sample-wise rather than task-wise, as samples within the same task can differ significantly in potential due to varying difficulty. GRPO training begins with the
group, then progressively incorporates lower-potential groups in subsequent stages. Each stage runs for 150 steps, gradually expanding the training distribution until the full dataset is included by stage 4. Because the usable gradient scales with p(1 p). ranking by II, prioritizes samples with maximal expected signal. Fig. 4 shows training rewards and test scores over steps. Contrary to training without curriculum learning (Fig. 3). both metrics rise apparently, particularly in the first two stages. We also observe sharp drops in reward when new samples are merged at each new stage, However, little improvement is gained in the final stage, suggesting that adding low-pote p---etttentll samples offers limited benefit. These observations align with our theoretical analysis.
5 EXPERIMENTS AND ANALYSIS
5.1 OVERALL BETTER PERFORMANCE
To evaluate the effectiveness of All Code. Code Agent, and CodeSteer (illustrated in T Note that Code Agent can also be regarded as a baseline, where the LLM is not fine-tuned with SFT or GRPO but is granted access to the Code Interpreter (CI wo Fine-tune). For broader comparison, we also compare RI-CI with GPT-4o All Text and GPT-4o + OpenAT Code Interpreter. To assess the effectiveness of our introduced multi-stage curriculum learning and improvement potential, we compare against the following RI-CI variants: (1) a purely SFT-trained model without GRPO (RI-CI wo GRPO); (2) standard GRPO training without curriculum learning (R1-CI wo CL); and (3) GRPO with curriculum learning where data are calibrated by question difficulty rather than improvement potential (R1-CI wo IP).
Baselines:
(RI-CI), we compare it against baselines: All Text,
Table Preesnts the experimental results of all compared methods across 107 training and 37 testing tasks. RI-CI significantly enhances the model's 's reasoning and planning abilities, improving the average success rate by 36.4% on training tasks and 31.5% on testing tasks across 3B, 7B, and 14B model sizes. Notably, R1-CI-14B achieves 72.4% success rate on testing tasks, outperforming the much larger GPT--0 with textual reasoning (58.6%) and GPT-4o with its trained Code Interpreter (70.9%). GPT-4o is utilized to synthesize SFT training data. After training, RI-CI-14B even outperforms GPT-4o. These consistent improvements across all model sizes highlight the method effectiveness and generalizability. Fig. 3 shows the score distribution across 144 tasks for the four compared methods. SFT and multi-stage curriculum learning of GRPO effectively reduce the number of tasks on which R1-CI models perform poorly. However, some tasks still yield low or even zero scores. This indicates that the inherent capabilities of the base LLM strongly affect overall performance, and training alone may not overcome limitations on tasks beyond the model's inherent reasoning or knowledge abilities. When comparing RI-CI with RI-CI wo CL. and R RI-CL wo IP, we find incorporating multi-stage curriculum learning and using improvement potential instead of question difficulty as the calibration factor consistently enhance training performance across all three model sizes.
(OOD) tasks. In Appendix Table 4 we evaluate the performance of R1-CI-7B and RI 1I-CCI-4B on unseen OOD tasks, including Graduate-Leve Google- Proof Q&& (GPQA, Diamond) (Rein et al. 2024) and the American Invitational Mathematics Examination (AIME 24&25). Both models significantly outperform their untrained counterparts. In Appendix Fig. [3 we further assess the generalizability of our training framework by training the 14B model with SymBench and Reasoning-Gym as training data, and evaluating it BBH as OOD benchmark. The number of training tasks remained 107 as before. The green curve in the figure shows the new model's 's performance on BBH, which is comparable the purple curve representing the original R1-CI-14B model. These results demonstrate the satisfying generalizability of RI-Code-Interpreter to unseen tasks from diverse sources.
Evaluation
5.2 RESPONSE CHARACTERISTICS
Emergent behavior of Geff-Checking: During GRPO training, we observe emergent behavior where the model integrates textual reasoning with code execution to improve solution verification. For example, in the final two reasoning
7

---

Published as a conference paper at ICLR 2026
Table 3: Scores of compared methods on 144 tasks across three benchmarks SymBench (SymB.), Big-Beenh--FR (BBH), and Reasoning-Gyn (Rea.G... Best result for each dataset is bold. We abbreviate RI-Code-Inte Inttre--ntrrrreter as RI-CI, Curriculum Learning CL, and Improvement Potential IP.
107 (26/20/61) Train Tasks
37 (7/7/23) Test Tasks
Method (Ace %)
SymB.
BBH
Rea.G.
SymB.
BBH
Rea.G.
Avg. Train
Avg. Test
GPT-4o
All Text
40.7
86.7
45.5
43.1
87.3
54.6
52.5
58.6
Code Interpreter
63.7
85.0
63.3
63.9
85,9
68.4
67.4
70.9
Qwen3-14B
All Text
65.3
85.5
52.9
57.4
73.8
60.5
62.0
62.4
CodeSteer
65.0
86.0
53.8
60.2
76.2
62.1
62.5
64.4
Qwen2.5- 10nn225--4B--nniee 14B-Instruct-IM
All Text
24.0
77.7
35.8
22.3
80.4
39.6
40.8
44.1
All Code
40.0
78.1
40.4
53.1
78.0
35.7
47.3
47.0
CodeStcer
36.7
75.6
37.8
45.3
79.0
40.5
44.6
48.7
Code Agent (CI wo Fine-tune)
27.9
64.6
30.0
33.7
70.4
39.7
35,9
44.4
RI-CI wo GRPO
48.3
56.0
77.3
56.1
60.6
60.1
RI-CI wo CL
71.3
87.9
52.8
59.4
79.7
60.0
63.9
63.6
RI-CI wo
72.2
88.6
54.4
61.1
81.4
63.5
65.1
66.4
RI-CI
74.4
91.9
58.9
65.6
86.5
70.1
68.8
72.4
Qwen2.5-7B- 17-15----B-ntt
All Text
19.5
66.0
25.7
14.6
69.1
26.3
31.7
32.2
All Code
27.5
70.2
25.7
44.1
63.1
34.0
34.5
41.5
CodeSteer
29.0
69.2
26.4
43.8
67.5
29.3
35.0
39.3
Code Agent (CI wo Fine-tune)
20.0
42.6
69.7
27.0
31.0
38.1
RI-CI wo GRPO
65.6
83.7
45.6
55.3
70.1
53.6
57.6
57.0
RI-CI wo
67.7
55.9
74.1
$7.9
60.4
60.6
RI-CI wo IP
68.5
86.5
50.8
57.3
76.6
60.1
61.8
62.7
R1-CI
72.0
89.2
53.6
60.8
80.0
64.3
64.7
66.6
All Text
12.0
55.1
11.2
12.0
49.7
10.7
19.6
18.3
All Code
18.3
60.4
5.0
33.9
52.3
8.9
18.6
21.8
CodeSteer
20.1
8.9
32.2
56.6
21.6
23.9
Code Agent (CI wo Fine-tune)
13.8
56.9
5.4
25.4
51.9
8.8
17.1
20.1
RI-CI wo GRPO
36.9
47.6
61.6
44.0
50.9
48.0
RI-CI wo CL
66.0
81.1
40.9
49.3
64.7
48.1
54.5
51.5
RI-CI wo IP
67.1
83.2
42.5
51.0
66.2
50.3
56.1
53.4
R1-CI
70.0
86.0
46.6
54.8
71.2
54.9
59.7
58.0
turns of Fig. the model generates code to test whether the proposed solution satisfies the constraints. Across test samples, the model leams to verify answers through either textual reasoning or code execution, exhibiting an emergent self-checking behavior that strengthens reasoning and planning. In Fig. 6h. report the proportion of answer trajectories containing code-based self-checking, identified by querying GPT-4o on both current and preceding turns to determine whether the generated code attempts validate earlier answers. After GRPO training, this proportion increases substantially, indicating that the model naturally acquires self-checking as strategy to improve performance.
Code usage turn number: Fig. 6p. we show the distribution of Code Interpreter interaction turns: indicates direct textual reasoming without code, corresponds to solving with single code execution, and values greater than denote multi-turn code generation and refinement. The model learns to employ multi-tur reasoning with code, yet most problems are solved within fewer than four code interactions, keeping the reasoning process not overly costly.
Response length study: Previous work (Guo al.) 2025 Jin et al., 2025) observed that LLM responses tend grow longer during RL training, the model learns to explore solutions through long-chain reasoning. Fig. 6C shows the average response length over training steps. In contrast to prior findings, we observe no significant length increase even though GRPO truly improves model performance. Possible reasons include: (1) The SFT stage already instills
8

---

Published as a conference paper at ICLR 2026
(8)
Self-Checking Code
(0)
Code Interpreter Usage Number
(c)
Response Length per
we Fize-tuee GRTO
Ri-01 R3-CI
3B
14
930
Respoare Leagth
0.15
0.20
830
500
0.10
730
0.45
760
0.00
Model Size
Code Usage Number
Training
Figure Response characteristics of RI-CI models. (a) Proportion of answer trajectories that include self-checking with code. (b) Distribution of Code Interpreter usage number per answer. (c) Evolution of average response length per generation turn during the training process.
long-chain reasoning; (2) The multi-turn interaction spreads reasoning across turn,, reducing per-turn response length; (3) Code-augmented reasoning reduces reliance on long CoT chains, as it does not require iterative textual search.
Training Reward
(b)
Testing Score
Testing Score over Training Steps
$1.9
78.8
76
Roward
17.0
Score
67.5
Stage
Stage
Stage 3
Stage
Original-Offline
Onlinn-Merge
Onling Unmerge
60.0
Reinforce+
Rem
150
300
450
Training Step (evaluation)
20
40
60
8D
120
120
140
20
40
60
80
100
120
140
Training Step
Training Step (evaluation)
Figure 8: Offline vs. online RL for multi- stage curriculum learning.
Figure 7: Comparison of GRPO, PPO, and Reinforce++ RL algorithms.
GRPO, round
(a)
Ablation on Multi-Turn Code Interpreter
Ablation on SET Dataset Design
40
Time Alse ser teat Xit 301
715
33
Score
Score
00
20
4:3
40
Poll
Wrong
wie Mulli-farn Emphasds
Prenopts
Training reward Testing
All codd
200
Figure 9: Ablation studies SFT dataset design and training frameworks. All experiments are trained from 7B models. (a) We compare SFT perfor- mance across three frameworks: multi-turn Code Interpreter (CI), single- turn All Text, and single-turn All Code. (b) We evaluate alternative SFT dataset designs: (1) including incorrect answers, (2) omitting prompt varia- tion during synthesis, and (3) removing the multi turn interaction emphasis.
Steps
Figure 10: Continued multi-stage curricu- lum learning. We conduct a second stage GRPO training, incorporating updated im- provement potentials and newly parti- tioned dataset.
5.3 ABLATION STUDY
Multi-turn Code Interpreter: To investigate the gains from the multi-turn framework, train the LLM with All Text and All Code using the same amount (6.5k) of GPT-40 sampled training data, as shown Fig. 99. Both All Text and All Code yield smaller improvements than CI, particularly on testing tasks, indicating that the multi-turn CI framework is more generalizable and effective, and that the gains are not primarily due to in-domain training.
SFT dataset design In Fig. Dp, we further evaluate alternative SFT dataset designs: (1) w/ Wrong Data: In addition to the existing 6.5k correct answer trajectories, we also keep the same-sized incorrect answers synthesized by GPT-4o. However. this setting degrades model performance and increases variance, indicating instability. (2) w/o Varied Prompts: We synthesize the SFT dataset using a single generic prompt without varying code-use strategies. Results show that diverse prompts and strategies are crucial for enhancing dataset diversity and overall performance. (3) w/o Multi-Tun Emphasis: We remove the emphasis multi-turn answer trajectories and adaptive solving strategies. This ablation
9

---

Published as a conference paper at ICLR 2026
leads to noticeable performance drops, demonstrating that diverse and high-quality multi-turn data play a key role in improving model capability.
Warm starts vs. cold starts: Appendix Fig. L5 compares GRPO training with and without the initial SFT stage. Unlike prior findings (Guo et al. [025] Jin et al. 2025) suggesting SFT is unnecessary or only marginally helpful, we observe that SFT is crucial for enabling the model to reason effectively with the Code Interpreter. Multi-stage GRPO without the initial SFT stage brings mere improvements. This is likely due to the limited availability of effective samples for GRPO training when the model lacks sufficient capability.
GRPO vs. PPO vs. Reinforce++: In Fig. 7 we evaluate the effectiveness of our framework with alternative RL algorithms. We compare GRPO (Shao al. 2024), PPO (Schulman et al., 2017), and Reinforce++ (Hul 2025 and observe that their performances on both training rewards and test scores are comparable. The overall upward performance trend indicates that our framework is adaptable across different RL algorithms.
Offiine vs. online RL training: In our multi-stage RI. training setup, the dataset is partitioned into four groups before the first stage and remains fixed throughout, corresponding an offline training setting. In Fig. 8] we compare this with two online variants: (1) Online-Merge: Before each stage, the dataset is repartitioned using the current model, and training proceeds on these updated groups. As in the offline case, later stages merge data from higher lower potential groups, so the final stage accesses all data. (2) Online-Unmerger Similar to (1), but each stage trains only on a single unmerged group, proceeding from high to low potential groups across stages. We find that offline and online training achieve comparable performance in later stages. Although the online variant converges faster, it incurs higher inference costs due to repeated repartitioning. The unmerged online setting performs worse overall. These results suggest that offline training is comparable to the relatively costly online training in our setting.
RL training round number: In Fig. 10 we extend the RL training process with a newly partitioned dataset, derived from the first-round model shown in Fig. 4 However, the testing score does not improve with continued training; in fact, during the first stage, the testing score decreases even as training rewards increase. We attribute this to overfitting on the limited data group used in stage 1, which is partially alleviated in later stages as more data are introduced. These results indicate that a single round of RL training is sufficient, as the model effectively adapts across the entire dataset.
Impact of sample improvement potential GRPO training: In Fig. 14 we evaluate the effect of training GRPO with datasets of equal size but different ranges of improvement potential. Training samples are drawn from Reasoning Gym and SymBench, while evaluation is conducted on BBH for fair comparison. Models trained on higher-potential samples [0.64, 1.00] exhibit a more pronounced increase in training rewards and achieve higher BBH test scores. These results support our theoretical analysis in Section 4.3 and validate the correctness of multi-stage curriculum learning.
Qwen-2.5 vs. SeepSeek-distillee models as base: Appendix Table 5 compares training performance using the general- purpose Qwen-2.5 models versus the same sized long-chain reasoning models distilled from DeepSeck RI (Guo) et al. 2025). Whether after SFT or using the raw models, Qwen consistently outperforms DeepSeek, particularly in code generation for solving tasks. Hence, choosing Qwen-2.5 series as base models for training is reasonable.
5.4 R1-CI TRAINING AND INFERENCE COST
GRPO training for the Code Interpreter is computationally expensive. For instance, training RI-CI-14B takes around 1845 GPU hours even though our specialized Code Execution Sandbox already saves around 39% training time. The cost arises mainly from two factors: (1) GRPO requires multiple sampled rollouts per answer turn to enable reward- based comparison, which is further intensified in our multi-turn generation setting; (2) the Code Interpreter introduces additional overhead due to costly code execution, especially for scripts involving search, iteration, or optimization. Although we build the specialized Code Execution Sandbox and cap the execution time at 60 seconds per script, it still remains major time sink. RI-CI requires multi-turn reasoning and code execution to reach the final answer, resulting in higher time and token costs compared to single-turn textual reasoning. In our evaluation, however, RI-CI solves most questions with fewer than four code executions and within two minutes.
6 CONCLUSION
We introduce framework for training Code learning. Standard GRPO training is limited by task diversity and the sparsity of effective samples. Our proposed multi- stage curriculum learning addresses this by partitioning training questions according to their measured improvement potential, and progressively training from high- to low-potential samples. Training costs are reduced by 39% by decoupling consuminn code execution from model optimization through specialized Code Execution Sandbox. The resulting model, RI R1-CI-14B, outperforms GPT-40 + OpenAl Code Interpreter. We further investigate related training strategies and identify emergent self-checking behaviors via code generation. To our best knowledge, this the first open-source, ceneral-purposs Code Interpreter trained with these methods.
LLMs using both supervised and reinforcement
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
This paper contributes to advancing Foundation Models by augmenting language models with a Code Interpreter, which has strong potential improve safety, performance, and alignment with human preferences. However, such capabilities are inherently dual-use, the same techniques that augment models toward harmless outputs can, with minor changes, misused generate harmful content. While misuse concern, believe the broader societal benefits outweigh the risks.
8 REPRODUCIBILITY STATEMENT
For better reproducibility, we include detailed descriptions of our curated 144 tasks in Appendix Sec. H and the synthesis of SFT and GRPO training datasets in Sec. [4] The theoretical analysis of multi-stage GRPO training and the measurement of improvement potential are included in Appendix Sec. D Sec. 4.2 and Sec. 4.3, The full code of training and dataset synthesis are attached Supplementary Material of the submission. Our code, model, and dataset will be made publicly available under open-source license following the acceptance of the paper.
9 LARGE LANGUAGE MODEL USAGE FOR WRITING
In this paper, we use LLMs specificcyl ChatGPT-as general-purpose writing aids. Draft text was provided to these models for grammatical correction and structural refinement, after which the output was verified and further edited when necessary. Their use was strictly limited text refinement; they were not employed to generate content references.
10 ACKNOWLEDGMENTS
This work was supported by ONR under Award solely reflects the opinions and conclusions of its authors.
and MIT-IBM Watson AI Lab. However, this article
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report, arXiv preprint 2rXiv:2303.08774 2023.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprini arXiv:2204.01699 2022.
Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B Ashok, and Shashank Shet. Codeplan: Repository-level coding using Ilms and planning. Proceedings of the ACM on Software Engineering, 1(FSE):675-6)) 2024.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 2022.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. arXiv preprint arXiv:2412.21187, 2024a.
Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan. Scalable multi-robot collaboration with large language models: Centralized or decentralized systems? In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4311-4317. IEBE, 2024b.
Yongchao Chen, Harsh Jhamtani, Srinagesh Sharma, Chuchu Fan, and Chi Wang. Steering large language models between code exccution and textual reasoning. arXiv preprint arXiv:2410.03524, 2024c.
Yongchao Chen, Yilun Hao, Yueying Liu, Yang Zhang, and Chuchu Fan. Codesteer: Symbolic-augmenteed language models via code/text guidance, 2025. URL hrt.sttsss. arxiv. http://wwwwwwwwppa/////////so 220./2502..333

---

Published as a conference paper at ICLR 2026
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504. 1/336, 2025.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uni Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aide rriee language models. In International Conference on Machine Learning, pp. 10764-10799, PMLR, 2023.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking. arXiv preprint wrxiv:2501.045// 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint 2rX1v:2501.12450 2025.
Yilun Hao, Yongchao Chen, Yang Zhang, and Chuchu Fan. Large language models can plan your travels rigorously with formal verification tools. arXiv preprint wrXiv:2404.11891 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search- r1: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint wwXv:225030095/6 2025.
Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model-augmented code emulator. arXiv preprint 1rXiv:2312.044777 2023.
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366, 2025a.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.233383, 2025b.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.0775.. 2022.
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. arXiv preprint 1rXiv:2210007128. 2022.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. A.Vv:2303.116511 17651, 2023.
Cheng Qian, Emre Can Acikgoz, Qi He. Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv: 139558, 2025.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL hhtpa:::: : : arwiv.iiv.. org/ abs/2412. 15115
12

---

Published as a conference paper at ICLR 2026
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms, 2017.
06377
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Jonathon Shlens. Notes on kullback-leibler divergence and likelihood, 2014. URL https://arxxvvoorg//ga 1404.20 2000
Mirac Suzgun and Adam Tauman Kalai. Meta-prompting: Enhancing language models with task-agnostic scaffolding. arXiv preprint arXiv:2401.12954 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't 't plan (a benchmark for Ilms on planning and reasoning about change). In NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022.
Haozhe Wang. Chao Qu, Zuming Huang. Wei Chu, Fangzhen Lin, and Wenhu Chen. VI-rethinker: Incentivizing self-reflection of vision- langugae models with reinforcement learning. arXiv preprint arXiv:2504.000 08837, 2025.
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406.00 04692, 2024.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le,, Denny Zhou, et al. Chain- of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2485 24837,2477722 2022.
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing IIm reasoning via reinforcement learning on open software evolution. arXiv preprint 184449 2025.
Nathaniel Weir, Muhammad Khalifa, Linlu Qiu, Orion Weller, and Peter Clark. Learning to reason via program generation, emulation, and search. arXiv preprint 24Xi::2405.1633: 2024.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations, 2022.
Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, et al. Crab: Cross-environment agent benchmark for multimodal language model agents. arXiv preprint arXiv:2407.00 015511 2024.
Kevin Yang, Yuandong Tian, Nanyun Peng. and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4399344799 2022.
Yuan Yang, Siheng Xiong, Ali Payani, Ehsan Shareghi, and Faramarz Fekri. Can Ilms reason in the wild with programs? arXiv preprint wxXiv22406.3364, 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. 12937, 2025.
13

---

Published as a conference paper at ICLR 2026
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. 2023a.
arXiv preprint arXiv:2308.0793
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307. 13854, 2023b.
14

---

Published as conference paper at ICLR 2026
APPEND--RL-COD--IVERRRETRR LLMS REASON WITH CODE VIA SUPERVISED AND MULTI-STAGE REINFORCEMENT LEARNING
Related Work
16
B Example of GPT-5 failure in code/text decision
16
C Sparse GPU utilization due time-consuming code execution
D Why vanilla GRPO fails in mixed difficulty, and why potential-guided curriculum works
17
Preliminaries and notation
D.2 Á variance identity for relatve Bernoulli rewards
17
D.3 A bound on the policy-gradient magnitude
D.4
Rebuttal of "given enough steps, GRPO will succeed anyway"
18
D.5
Descent lemma view: expected improvement per step
D.6
Potential proxy for learning signal
18
D.7
Why stage curriculum helps
D.8
Conclusions
19
E
Performance of RI-Code-Interpreter out-of-distribution (OOD) tasks
Impact of sample improvement potential on GRPO training
20
G Ablation studies on warm start vs. cold start, Qwen vs. DeepSeck-distilled reasoning models as base
models
21
Description of reasoning and planning tasks
21
15

---

Published as conference paper at ICLR 2026
A RELATED WORK
Code generation and symbolic computing LLM tasks
LLMs are widely used in agent tasks such as software/web
interaction (Zhou et al. 2023b; Hao al. 2024 Xu et al. 2024), robot planning (Chen et al. 2024b Ahn al., (2022).
and logical inference (Suzgun et al. 2022), Many benchmark tasks can in fact be solved directly through code (Suzgun
& Kalai 2024 Gao al. 2023), and recent work extends coding reasoning and semantic analysis (Li et al. 2023
Weir et al. (024).. Most prior approaches use either text (Yao et al.) 2024 Ahn et al. 2022) or code (Liang et al. 2022 Bairi et al. 2024 Zhou et al.
2023a) exclusively output. Recent work Chen et al. (024c) emphasizes the need to dynamically switch between modalities, proposing CodeSteer (Chen et al. 2025) as guidance model. Recent work ToRL (Li al., 2025b) and ReTool (Feng al. (025) explore training reasoning models integrated with Code Interpreters, but their training and evaluation are restricted to math problems, leaving a gap from real-world applications that require broader effectiveness.
2025) focuses on tool selection, where the Code Interpreter is used only for generating relatively simple code, and the evaluation tasks demand limited reasoning capabilities. Training a general-purpose Code Interpreter remains largely unexplored.
contrast, ToolR (Qian et al.
LLM long-chain reasoning L.M self-exploration, reflection, and evaluation can enhance task performance across domains (Yang et al. 2022 Welleck et al. 2022, Madaan al., 2023). Models like OpenAl (Jaech al.) 2024) and DeepSeek RI (Guo al., 2025) showcase agentic behavior via long-chain reasoning and self-reflection, which is learned by RL-based training with rule-based outcome rewards (Shao et al. 2024 Wei et al. 2025). Extensions with vision modality (Wang et al., 2025), retrieval (Jin et al. 2025 Li et al. 2025a), and tool use (Qian al. 2025 further
improve reasoning, but lack symbolic computing and code generation, limiting performance on complex symbolic tasks and incurring high token and time costs (Chen al. 2024a).
B EXAMPLE OF GPT-5 FAILURE IN CODE/TEXT DECISION
(a)
ChatGPT
bb)
ChatGPT 5
@
(c)
ChatGPT
Solve:
Solve:
Solve: 5.9 = x 5.11
5.11
5.9 5.11
with code generation
with code generation. No
We want solve:
textual reasoning
Thought for 4 seconds
X
X
5.9 = x 5.11
-0.21
0.79
Subtract 5.11 from both sides:
aalso generated code
programmatically (using Pythan's Decimal 1 for exact subtraction) and ran just now-the calculation confirms the result. (+)
Analysis
5.9 5.11
Pyther
Ieport esmpy
Now compute carefully:
perire veziible
5.90 5.11 0.79
de
Analysis
Equations
aquacdor ap 4p.6olo.7,
Since 5.90 < 5.11, the result is negative:
Pyrhon Codk
Sclie www.oloetezaatiin.
*sow decimal Ingort Decinal, getcants
Treeseee precuasze junt cas (1)
1.21
Salto Fur 1.11
Reaul
7900000000000000
Decivall (8.29"1
Figure 11: Example of GPT-5 failure code/text decision. In this case, the question is incorrectly solved with textual reasoning (a) but can be easily addressed through code generation (c). However, GPT-5 remains overconfident in textual reasoning, relying on even when prompted to use code, despite the generated code already yielding the correct solution (b).
C SPARSE GPU UTILIZATION DUE TO TIME-CONSUMING CODE EXECUTION
16

---

Published as conference paper at ICLR 2026
GPU Utilization Ratio
100
GPU Utilization (%)
80
60
40
20
0
1
40
80
120
160
200
Time (hours)
Figure 12: GPU utilization during GRPO training without the Code Execution Sandbox, where execution runs purely on GPU. Utilization remains low and fluctuates with frequent idle periods, as training must wait for time-consuming code execution that is not GPU-accelerated.
D WHY VANILLA GRPO FAILS IN MIXED DIFFICULTY, AND WHY POTENTIAL-GUIDED CURRICULUM WORKS
D.I PRELIMINARIES AND NOTATION
Fix prompt z and a code execution context In a GRPO update we draw group G rollouts y, = (M1, Vi, wil).
receive terminal reward r; E {0, 1}, and define the within-group mean t/ Zj=1 T'.. Broadcasting the sequence- level advantage (r: - F) to all tokens (and ignoring clipping for analysis), the per-group policy gradient and KL regularizer are
G = Σ (r. F)
Ivdl
1
VoJGRrO(O)
3 VoDKL Trot).
Vo log (olii x, Vi,see; C). (D.1)
G 1=1
regularizer
policy term
All expectations below over the rollout sampling procedure (and, when stated, over data x).
D.2 VARIANCE IDENTITY FOR GROUP-RELATIVE BERNOULLI REWARDS
Lemma D.1 (Within-group Bernoulli variance). ffri....,re
~ Bernoullip)) a 1 & Eim rj. then
B((r=-F) = Var(r; f) =)==)) (1-
(D.2)
Proof. Var(r;) = p - p). Var(F) = p(1 G)/G. and Cov(r,, = Var(r;) = p(1 - p)/G. Thus Var(r; - F)
Var(r:) + Var(F) - Cov(ri, p(1 - (1 t)
D.3 BOUND ON THE POLICY-GRADIENT MAGNITUDE
Proposition 1 (Policy term is controlled by p(1 -p). Let Ipol = 1. DE (P:=))) v, be the policy term in equation D.I
Then
|101p1111 5 p(1 - p) 1 (1-
1 E
||
(D.3)
17

---

Published as a conference paper at ICLR 2026
Proof. By Cauchy-Schwarz
Schwarz for mixed scalar/vector sumss
a
Σ || 11ai1
With a, r; and the prefactor 1/G, get |lop 2 きº Exir, -)))
Ex
Taking expectation and
using Lemma D.1 E (r r) Gp (1 à), yielding equation D.3
Corollary D.1 (Vanishing signal the extremes). The upper bound in equation D.3 vanishes or p 1, and is maximized at p
D.4 REBUTTAL OF "GIVEN ENOUGH STEPS, GRPO WILL SUCCEED ANYWAY"
In a heterogeneous batch dominated by too-easy or too-hard items, p(1 - p) ze 0 for most samples, so equation D.3 implies
E [llgpa ||
(D.4)
Consequently, the update in equation D.1 is governed by the regularizer - V (Tol(Tl||Tee) which contracts Te back toward Tyef. No amount of additional steps recovers signal from items with identically zero Bernoulli variance.
D.5 DESCENT LEMMA VIEW: EXPECTED IMPROVEMENT PER STEP
Assume JGRPO is L-smooth and consider a stochastic gradient step 0+ = 0 + n7 where is an unbiased estimator of VaJGRPO. The descent lemma yields
Ln7² [[
(G - [O)] 2
E
(D.5)
2
Potential-guided sampling (defined below) increases || and decreases the fraction of near- sea--eerovvarian items, thereby increasing the first term in equation D.5 at fixed batch size. In contrast, vanilla GRPO on mixed difficulty often drives [polll] toward zero by equation
D.3
making the KL term dominate the dynamics.
D.6 POTENTIAL AS PROXY FOR LEARNING SIGNAL
For each training item Iú, let (0) = Protr; = }} be the success probability under the (tool-augmented) policy. Estimate it by sampling N answers from a mixture of agent frameworks and define
1 N P = N
Σ (answer is correct),
11: == 4 (1 (1- - (
(D.6)
Lemma D.2 (Concentration of the potential estimator). For any > 0, Pr(p: Pil concentrates around 4p, - px) as N grows.
s 22ex((-2Ne Hence II,
Proposition 2 (Potential aligns with gradient strength). Let vs be defined as in equation shows factor |[]|,,]| II, the bound in equation D.3 shows that IE [[lgpoal]] is maximized when p. Ps) is maximized, i.e. near
D.I
Up to the slowly-varying
P: 1. Therefore 11, serves as proxy for per-item learning signal.
D.7 WHY MULTI-STAGE CURRICULUM HELPS
Let B denote the distribution over batch items. Define (B) =- (ea-eeBB(pa111 From equation D.3 larger (BB increases the policy term's expected squared norm. A potential-ranker curriculum selects items with large 11, in early stages and gradually lowers the threshold, so that previously too-hard items (with small p:) enter the curriculum when training has moved them into the vigh-variance region. Thus the curriculum maintains batches with large (B) throughout training, improving the expected gain equation D.5
Remark D.1 (Clipping and KL). Our unulysis ignores clipping in equation D.1 including it only reduces the policy term's 's magnitude, strengthening the conclusion that mixed-diffieue batches yield weak signal. The KI. regularizer is essential for stability but dominates updates exactly when the policy signal is small.
18

---

Published as conference paper at ICLR 2026
D.8 CONCLUSIONS
Vanishing-signal regime. On too-easy or too-hard items, Bernoulli variance p(1 p) collapses and the policy gradient is provably tiny (Prop. J.
Why vanilla GRPO underperforms. Mixed batches over-weight near-zeno-variance items; the KI. term then governs the update and contracts toward Tref-
Why potential-guided curriculum works. Selecting by 11, 4p; - (4) targets the high- variance region
where learning signal is largest (Prop. 2 and maintains across stages, improving the expected per-step gain (Eq. equation D.5
E PERFORMANCE OF -I- RI--ODE-INNEEPPETER IN OUT-OF-DISTRIBUTION (OOD) TASKS
Table 4: Performance of RI-CI-14B and RI-CI-7B OOD tasks: Graduate-Level Google-Proof Q&A (GPQA,
Diamond) RRein al. 2024). and American Invitational Mathematics Examination (AIME 24&25).
Each slot is the average of three repeated runs
Model
Qwen-2.5-7B All Text
Qwen-2.5-7B CodeSteer
RI-C1-7B
Qwen-2.5-14B All Text
Qwen-2.5-14B CodeSteer
R1-CI-14B
GPQA
31.2
32.9
39.0
41.2
AIME 2024 & 2025
8.33
8.33
15.0
30.0
33.3
42.0
Effects on OOD Tasks
90
85
Training reward CeamB.++Re.GG.
Testing score (R1-CI-14B)
Testing score (BBH, 0OD)
Score
75
65
55
0
50
100
150
200
250
300
Steps
Figure 13: Evaluation on out-of-distribution (OOD) tasks. We trained the 14B model using two-stage GRPO with an OOD) benchmark.
training data, and evaluated it on Hagg-Bench-Hard(BBBB) The number of training tasks remained 107. The blue curve shows the steadily increasing reward during two-stage GRPO training. Comparing the BBH test results of the new model (green curve) with the original RI-CI-14B model (purple curve) shows similar performance, indicating that our training framework generalizes effectively to unseen task distributions from varied sources.
SymBench and Reasoning-
19

---

Published as conference paper at ICLR 2026
F IMPACT OF SAMPLE IMPROVEMENT POTENTIAL ON GRPO TRAINING
Training Reward (Rea.Gym+SymB.)
BBH Testing Score
87.5
87.4
Group [0.64,
69.2
Group (IP [0.48. 0.64))
87.0
Group (IP (0.32. 0.48))
66
86.5
Reward
61.7
Score
60
86.0
54
15..6
10.64, 1.0))
Group [0.48, 0.64))
85.0
84.9
Group (0.32, 0.48))
48
84.5
125
25
150
Training Step
Training Step (evaluation)
Figure 14: GRPO training using datasets grouped by improvement potential: Group 1 [0.64-1.00) Group (0.48-0.64)) and Group 3 [0.32-0.48) Each group contains the same number of samples drawn from the same Reasoning Gym and SymBench tasks, but with different improvement potential ranges. Models were evaluated BBH for fair comparison. Models trained on Sigher-potential samples show consistently rising training rewards and achieve higher BBH test scores.
20

---

Published as conference paper at ICLR 2026
G ABLATION STUDIES ON WARM START VS. COLD START, QWEN vs. DEEPSEEK-DISTILLED REASONING MODELS AS BASE MODELS
Warm vs Cold Start
80
75
mmm Juil ha pww.
M Mawnmanhhhhhaaaooommm / M
70
65
Score
60
55
50
45
Warm ştart
Cold Start
40
0
50
100
150
200
250
Steps
Figure 15: Warm- vs. cold-start. With GRPO, warm start (preceded by SFT) outperforms cold start for Qwen-14B model as the base model. The model without prior SFT process gets barely performance lifting during training, even though the training data also has been calibrated by improvement potential for multi-stage training.
Table 5: Ablation studies on using DeepSeck-dissee distiilled reasoning models as the base model.
Tested on 37 Test Tasks
Model
7B, SFT
7B, All Text
7B, All Code
14B, All Text
14B, All Code
DeepSeck
53.1
27.9 32.2
28.7 41.5
40.1 44.1
43.4 47.0
Qwen-2.5
57.0
H DESCRIPTION OF REASONING AND PLANNING TASKS
Here describe the 144 training and testing tasks. They require strong symbolic, mathematical, logical, geometrical, scientific, and commonsense reasoning capabilities. The Tl to T33 tasks originate from SymBench (Chen et al, 2025), T34 to T60 tasks originate from Big- hig-Bench-Hard (Suzgun et al. 2022). and the last T61 to T144 tasks are from Reasoning- G We select questions with diverse difficulty and standardize them as a unified format to support fast rollout and testing.
T1 2048 Similarly to the 2048 game, in a grid, numbers representing powers of can move in any direction, combining when they encounter matching number form the next power of 2. Given starting position and sequence of movements, the goal is to determine the resulting grid after executing the moves.
T2- Blocksworld In Blocksworld, the objective is to stack
of blocks (brown) according to specific order. The robot can perform four actions: (1) pick up a block, (2) unstack a block from the top of another block, (3) put down a block, (4) stack a block on top of another block. A robot only pick up, unstack, stack block if clear, that is, the block has no other blocks on top and is not currently being held.
21

---

Published as a conference paper at ICLR 2026
T3- BoxLiift This task involves coordinating robots of various types to lift boxes of different sizes and weights. Each robot has a specific lifting capacity and can collaborate with others to lift single box. box can only be lifted if the combined lifting capacity of the robots exceeds the box's weight. The objective is to lift all the boxes in the minimum number of time steps.
T4- BoxNet This task involves coordinating robot arms to move colored boxes (squares) into corresponding colored goal locations (circles) in the fewest time steps. Each robot arm is assigned and restricted to a cell indicated by the dotted lines. The arms have two possible actions: (1) move box within their cell to neighboring cell, or (2) move box within their cell to a goal location within the same cell. The objective is to ensure all boxes are placed in their matching goal locations efficiently.
T5 Combinatoral Calculation Given a set of integers, the goal is to use arithmetic operations (addition, subtraction, multiplication, division) and parentheses to arrange the numbers in such way that the final result matches a specified target value. Each number must be used exactly once, and the order of the numbers cannot be changed.
T6 Constrained Linear Arrangement In two-player card game, the task is to deduce your opponent's 's moves based on the game's rules, your played cards, and the announced results of each round. Each card can only be used once, and the game follows specific interaction rules between different card types, where certain cards can defeat, be defeated by, or draw with others according to predefined relationships.
T7 Cryptanalysis In this task, you are provided with a combination lock consisting of numbers and letters, where neither the numbers nor the letters repeat. Using a series of guesses and feedback, the goal is to deduce the correct password based on the given conditions.
T8- Eight Queens Given grid with some queens already placed, the task is to place the remaining queens such that no two queens share the same row, column, or diagonal, while avoiding positions with obstacles in the grid.
T9- Game 24 This task involves querying LLMs to use a given set of integers to generate an equation that evaluates to 24.
T10 Gridworld This task involves querying LLMs to plan the robot actions in a grid world, reaching all goals in any order while avoiding obstacles.
T11 - GSM (Gao et al. 2023) This is the more challenging version of GSM8K (Cobbe et al. 2021 math reasoning dataset, where the numbers in the original questions of GSM8K are replaced with larger, less common values.
T12 - Letter Logic Diagram The task is to complete an incomplete grid by selecting from a list of letters, where each row and column must contain each letter exactly once, and all cells on the minor diagonal (top-right to bottom-left) must contain the same letter. Some cells are already filled in as constraints.
T13 Letters This task involves querying LLMs to count the total number of specific letters in a long word and specify their positions. An example question can be 'How many r's in the word strawberry and what are their positions?". This task has recently gained significant attention because current L.MSs struggle to perform it effectively and accurately.
T14- Light Puzzles In this task, you are given an n X n grid representing network of lights, where a lit light is represented by "1" and an unlit light by "O". Several buttons control the state of these lights by turning them on or off in certain positions. The state of each light can be affected by multiple buttons. The task is to follow series of button presses and determine the final state of the grid.
T15 Logical Puzzle The task involves querying LLMs to select specified number of different values from a grid of numbers, ensuring that certain mathematical constraints (sum or product) are satisfied for selected numbers for each row and column.
T16 - Logical Equation The task is to assign a specific numeric value to each letter from a given set, using a predefined range of numbers and a set of inequalities. Each letter corresponds to a unique number, and the relationships between the letters are defined by mathematical equations or constraints.
T17 Mahjong Given initial set of letter cards, in each round, new card is added and one card is removed. Some effects may happen when specific combinations of the cards appear after introducing the new card, A result is determined based on these specific conditions. The goal is to determine result based on a series of rounds.
T18- MATH-Count& Probability This is the math reasoning dataset from MATH dataset (Hendrycks et al. 202 with specific focus on counting and probability questions.
T19 Meom-Gy This is the math reasoning dataset from MATH dataset (Hendrycks et al. 2021 with specific focus on geometry questions.
22

---

Published as a conference paper at ICLR 2026
T20 Matrix Transformation
Rotate a given matrix of characters based on given instruction (e.g., 90 degrees clockwise), preserving each character's position relative to others the transformed output. The input matrix can be of any size and contain any character.
T21 New Operator
This task introduces custom mathematical operations involving two numbers, defined with unique formulas. The goal is to use the given definitions of these operations to compute the result of specific expression.
T22 - Number Multiplying
This task involves querying LLMs to compute the product among integers. It represents classic problem that LLMS are not able to solve through pure textual reasoning.
T23 Pattern Recognition The task involves querying LLMs to find all squares in character matrix where each square consists of identical characters and has a side length of at least 3.
T24- Permutation and Combination Given set of objects with specific positioning constraints, the task is to determine the correct arrangement of the objects on a shelf. Each object must be placed in a position according to the rules provided, ensuring that the conditions adjacency, order, and specific positions are met. For example, rule about adjacency could be 'Book A must be adjacent to book I'.
T25 Pooling This task involves applying a pooling operation a numerical N x N grid. The pooling operation uses an n x n sliding window (n < N) that moves across the grid from left to right and top to bottom. The results from each window then arranged based their positions create new output matrix.
T26 Reversi In this game similar to Reversi, players take turns placing pieces on an n x n grid. After placing a piece, any of the opponent's pieces located between two of the player's pieces (in the same row, column, or diagonal) will be flipped. The task is to determine the state of the board after rounds, starting from a given configuration.
T Standard Sudoku Given partially filled Sudoku grid, the task is fill the remaining empty cells with numbers between I and 9, ensuring that no number repeats in the same row, column, or 3 x 3 subgrid.
T28- Statistical Counting Calculate the total score of a string by scanning it from left to right, where consecutive identical letters earn points (for example, two or more consecutive A's add 1 point, B's add 2 points, etc.). The task is to start with score of 0 and return the final summing value.
T29 String Deletion and Modification The task is to transform a string by repeatedly applying set of ordered string manipulation rules until no more changes are possible, where each rule modifies the string based on specific patterns or conditions present in the current string state. For example, a modification rule can be "If the string ends with 'ba', replace it with aab..
T30- String Insertion The task is to transform string by scanning it from left to right and inserting specific characters after certain character patterns (e.g., each pattern WXYZ requires inserting W immediately after occurs). All operations are performed simultaneously on the original string.
T31 String Splitting dismantling engineer has old machines and can obtain machine parts through a set of predefined methods. By continuously cycling through these methods in a specific order, the engineer dismantles machines or combines parts to create new components, and the task is to determine the total number of parts and remaining machines after all possible cycles.
T32 - String Synthesis Given an initial set of blocks and a set of synthesis rules that combine different types of blocks, the task is to determine the final block(s) after repeatedly applying these rules in order until no more combinations are possible.
T33 Synthesis Decomposition A farmer grows various crops and can exchange them for agricultural products. Using a set of methods, he can trade specific combinations of crops for products, following a cyclic pattern until no further exchanges are possible. The goal is to determine the synthesis result for each round.
T34- - Boolean Expressions This task determines whether a randomly generated Boolean expression built from the constants True and False and the operators and, or, and not evaluates to true false.
T35 Causal Judgment This task involves querying LLMs to read a brief scenario and predict the answer an average person would give causal question about it, including moral, intentional, counterfactual aspects.
T36 - Date Understanding This task involves querying LLMs to interpret a few sentences that reference dates and answer related question (e.g., compute and return a specific date in /DDD/YYYY format).
23

---

Published as a conference paper at ICLR 2026
T37 - Disambiguation QA For a sentence containing a potentially ambiguous pronoun, the task is to decide whether its reference genuinely unclear; if it clear, identify the noun to which the pronoun refers.
T38 Dyck Languages The task aims to complete a Dyck-4 string by providing the missing closing parentheses that properly balance the given prefix.
- Formal Fallacies The task examines set of statements and judges whether the informal argument that follows is deductively valid or commits a formal fallacy, with particular attention to negations.
T40- Geometric Shapes The task aim to analyze full SVG path description and identify the geometric figure that would be drawn.
T41 Hyperbaton Given two sentences, choose which of two sentences follows the natural English ordering of adjectives.
T42-T44- - Logical Deduction 3/5/7 Objects Use spatial clues to determine the correct ordering of a set of objects (3/5/7 objects).
T45 - Movie Recommendation From four candidate films, select the one that best matches the preferences implied by the movies a user has already enjoyed.
T46 Multi-Step Arithmetic Perform multi-step calculations involving addition, subtraction, multiplication, and division to obtain the correct result.
T47 Navigate Follow a sequence of movement instructions and state whether the agent finishes at its starting point.
T48- Object Counting Given a list of items and their quantities, count how many belong a specified category.
T49 Penguins in a Table Refer to a table of individual penguins and their attributes (possibly with extra information) to answer a question about one of those attributes.
T50 Reasoning about Colored Objects Using the provided context, state the color of particular object on a surface.
T51 Ruin Names Make single-character change an artist, band, or movie name to create humorous new meaning.
T52 Salient Translation Error Detection Examine a German sentence and its English translation, and classify the main translation error present.
T53 Snarks From pair of nearly identical sentences, identify the one that is sarcastic.
T54- Sports Understanding Judge whether a fabricated sports-related statement is plausible.
T55 Temporal Sequences Given timeline of a person's 's daily activities, identify a time slot when they could have performed another specified task.
T56-T58- Tracking Shuffled 3/5/7 Objects Trace a set of objects (3/5/7 objects) through series of pairwise swaps to determine their final positions.
T59- Web of Lies Decide whether Boolean function described in word problem evaluates to true or false.
T60 Word Sorting Arrange the provided words standard alphabetical order.
T61 AB Rewrite an A: token string by exhaustively applying neighbor collision rules and return the final sequence.
T62- Acre From example Blicket detector outcomes, decide whether new object turns the detector "on", "off", is "undetermined".
T63 Advanced Geometry Solve analytic geometry questions (e.g. angle, orthocentre, in-circle radius) given vertex coordinates.
T64- AIW Answer small Aalicein--Wondellane social reasoning problems about siblings, friends, colleagues.
T65 ARC_1D Infer the mapping rule that maps example ID grids to output grids and apply to a test grid.
T66 ARC_AGG Same as ARC__D but with rotations, mirrors and permutations on 2-D grids.
T67 - Base Conversion The task is to convert integers between arbitrary bases.
T68 Basic Arithmetic The task evaluate the value of basic arithmetic expressions.
T69 BF Based on outputs of example BF codes, the task is to output the contents of a new BF program.
24

---

Published as a conference paper at ICLR 2026
T70 - Binary Alternation The task is to produce the minimum number of character swaps to make binary string to be alternating, that is, no two adjacent characters are equal.
T71 - Binary Matrix Given binary matrices, the task is to find the distance to the nearest O for each cell in the matrix.
T72 Bitwise Arithmetic The task is compute results of expressions with mixed bitwise and arithmetic operators.
T73 Caesar Cipher The task is to decrypt a Caesar cipher text.
T74- Calendar Arithmetic Given a description of the calendar, answer a question by conducting arithmetic calculations such as adding or subtracting days / months/ / years or computing weekday differences.
T75 Chain Sum The task is to calculate simple arithmetic problem and output the answer.
T76 Circuit Logic Given a logic circuit with logical operators, the goal is to evaluate the output of given inputs.
T77 Codeio The task is to read and reason about task description and pseudocode programs and report outputs of the
program given inputs.
T78- Color Cube Rotation After rotating 3D colored cube, the task to name the color on queried face.
T79- - Complex Arithmetic The task is to perform arithmetic with complex numbers and report answers.
T80 Count Bits Given large number, the goal is count the number of occurrence of bits in the binary representation of this number.
- Primes The task count number of primes numbers within interval.
T82 Countdown The task is to write an expression that can reach a target integer using given numbers and the four operations.
T83 Course Schedule Given a list of courses need to be taken and their prerequisites, the task is to determine if all courses be finished.
T84 - Decimal Arithmetic The task to evaluate decimal expressions with given precision.
T85 Dice The task is compute probabilities of rolling results in fair-dice experiments with various dices with different number of sides.
T86 Emoji Mystery The task is deduce hidden sentences expressed with emoji symbols.
T87 - Family Relationships The task is to answer kinship queries in family trees.
T88 Figlet Font The task is read FIGlet banners and output the content as strings.
T89 Fraction Simplification The task is to simplify fractions to the lowest terms.
T90 Futoshiki The task is to fill in values empty spaces of Futoshiki puzzles that have inequalities.
T91 Game of Life The task to simulate Conway's Game-of-Life for k steps.
T92- Game of Life Halting The task is to decide whether Game-of-Life configuration halts within k steps, that is,
there are no cells alive.
T93 GCD The task is to compute greatest common divisors of numbers.
T94- Graph Color The task is to provide a coloring for this graph such that every vertex is not connected to a vertex of the same color.
T95 Group Anagrams Given a list of words, the task is to cluster words that are anagrams.
T96 Intermediate Integration Given an expression, the task is to calculate the indefinite integral.
T97 Isomorphic Strings The task is to decide if two strings can be isomorphic the characters in one string can replaced to get the second string.
T98 Jugs Given empty jugs with different sizes, the task is to give a plan of how to fill any of the available jugs with the target amount of water by filling, emptying, or pouring from jug to another.
T99 Knight Swap The task swap two knights on chessboard the fewest moves.
T100 Knights Knaves The task is to determine who is knight (truth-teller) or knave from their statements.
25

---

Published as a conference paper at ICLR 2026
T101 - Largest Island The task is to find max connected component size in a binary grid.
T102 LCM The task find the Least Common Multiple (LCM) of numbers.
T103 - Leg Counting The task is to count how many legs there are in total when given a list of animals.
T104 Letter Jumble For each word in sentence, the letter may have been randomly shuffled. The task is reconstruct original words from jumbled letters.
T105 List Functions Given examples of how imputs are mapped outputs, reason and use the same mapping to generate output given an input.
T106 Manipulate Matrix Apply a sequence of matrix transformations a matrix and output the result.
T107 Maze Compute the shortest path length from start to goal in an maze.
T108 Modulo Grid Identify the mathematical pattern which defines a grid, then that pattern to fill in the question
marks in this grid.
T109 Needle Haystack The task is to locate short pattern inside a longer string.
T110 Number Filtering Given a list of numbers and a requirement, remove numbers not satisfying this requirement.
T111 Number Format The task is pick the largest/smallest number out of several options.
T112 - Number Sequence Predict the next term of integer sequences based on previous patterns.
T113 Number Sorting The task is sort number lists based required order.
T114 Palindrome Generation The task is, given a list of letters, to form a valid palindrome.
T115 Palindrome Partitioning Given string, the task is find all ways to partition it such that every substring is palindrome.
T116 Polynomial Equations The task is find real values of a variable in polynomial equation.
T117 Polynomial Multiplication The task is to calculate the result multiplying two polynomials.
T118 Pool Matrix The task is to perform max- average-pooling on numeric matrices.
T119 Products The task is to compute multiplications of numbers.
T120 Propositional Logic Given list of premises, the task is to infer correct conclusion from the premise.
T121- - Quantum Lock There are some buttons, a light, and number. The light will toggle between red and green whenever you press button. Each button performs mathematical operation to the number, but the operation may depend on the state of the light. You must press the shortest correct sequence of buttons to reach the target value.
T122 Ransom Note Given two strings representing ransom note and magazine, determine if the ransom note can be constructed using the letters in the magazine
T123 Reare Find the common rule that maps input grids to output grids and apply the rule to predict corresponding output of a test input grid.
T124 Rectangle Count The task is to count how many rectangles are present in an ASCII grid.
T125 - Rotate Matrix Given a square matrix, the task to rotate it and output the rotated matrix.
T126 Rotten Oranges You are given an grid where each cell can be empty cell, contain fresh orange, or contain a rotten orange. Every minute, any fresh orange that 4-directionally adjacent to rotten orange becomes rotten. Your task is determine the minimum number of minutes that must elapse until no cell has fresh orange.
T127 - Rubiks Cube Given a Rubik's 's cube, the task is to provide a solution to solve this cube using Singmaster notation.
T128 - Rush Hour Given a rush hour parking grid, the task is to give a plan movements cars to achieve required positions.
T129 Self Reference The task is to evaluate self-referenttia arithmetic expressions to output the number of possible solutions.
T130 Shortest Path The task is to find the length of the shortest path in a grid.
26

---

Published as conference paper at ICLR 2026
T131 -Simple Equations The task is to solve equations with one variable.
T132 Simple Geometry Given polygon with different number of sides and all interior angles but one angle, the task is to calculate the remaining interior angle.
T133 Simple Integration The task is to find solution indefinite integral problems with one variable.
T134 Sokoban The task is to find list of actions to solve a Sokoban level.
T135 Spell Backward The task is to reverse input strings.
T136 SSpirl Matrix Given a matrix, the task is to generate list of elements in spiral order, starting from the top-left element.
T137 - String Manipulation The task is to repeatedly transform string according to a set of rules until no further transformations can be performed, or state repeated.
T138
Syllogism Given some statements, answer the provided question by retrieving information from the statements.
T139- Time Intervals The task compute durations between two times dates with various formats and complexities.
T140 - Tower of Hanoi Output an optimal (or specified) move list to transfer disks between pegs to solve tower of Hanoi problem.
T141 Tsumego Choose the single correct Go move to capture or save stones.
T142 Word Ladder Transform word another by single-letter changes using dictionary words.
T143 - Word Sequence Reversal Given a list of words, the task to reverse order of words.
T144- Zebra Puzzles Given some statements, solve the logic puzzle by gathering information from statements and deduce the answer of the question.
Table 6: The evaluated capabilities of all 144 tasks, classified as Execution, Planning, and Reasoning tasks. The classification is based on human experts' knowledge and also the classification in original datasets it exists.
Task
Math
Spatial
Logical
Order
Optimization
Search
2048
Execution
Light Puzzles
X
Mahjong
X
X
X
Matrix Transform.
X
New operator
X
X
X
X
Number Multiplying
X
X
X
X
Pattern Recognition
X
Pooling
X
Reversi
X
Statistical Counting
X
X
String Del. &Modi.
String Insertion
String Splitting
String Synthesis
Synthesis Decomp.
Dyck Languages
X
Multi-Step Arithmetic
Navigate
Object Counting
X
Ruin Names
Tracking Shuffled Obj-
Word Sorting
AB
ARC_ ID
ARC AGI
Base Conversion
Basic Arithmetic
X
Continued on next page
27

---

Published as a conference paper at ICLR 2026
Table 6 (continued from previous page)
Task
Math
Spatial
Logical
Order
Optimization
Search
Binary Alternation
X
X
Binary Matrix
X
Bitwise Arithmetic
XXXX
X
X
Caesar Cipher
X
Chain Sum
X
x
Codeio
X
X
Color Cube Rotation
Complex Arithmetic
X
Count Bits
X
Count Primes
Decimal Arithmetic
Dice
Fraction Simplification
GCD
Group Anagrams
Intermediate Integration
Isomorphic Strings
Largest Island
L.M
Leg Counting
Letter Jumble
List Functions
Manipulate Matrix
Number Filtering
Number Format
Number Sorting
Palindrome Generation
Palindrome Partitioning
Poly. Equations
Poly. Multiplication
Pool Matrix
Products
Rectangle Count
Rotate Matrix
Rotten Oranges
Simple Equations
X
Simple Geometry
Simple Integration
Spell Backward
Spiral Matrix
String Manipulation
Time Intervals
Word Seq. Reversal
Blocksworld
Planning
BoxLift
BoxNet
Combinatorial Calc.
X
Const. Linear Arrange.
Cryptanalysis
Eight Queens
Game 24
Gridworld
Letter Logic Diagram
Letters
Logic Puzzle
Continued on next page
28

---

Published as conference paper at ICLR 2026
Table 6 (continued from previous page)
Task
Math
Spatial
Logical
Order
Optimization
Search
Permut, and Combina.
Standard Sudoku
XX
Movie Recommendation
X
XXX
Temporal Sequences
X
Countdown
X
X
Course Schedule
Futoshiki
Graph Color
Jugs
Knight Swap
Maze
Modulo Grid
Quantum Lock
Rubiks Cube
Rush Hour
Shortest Path
Sokoban
Tower of Hanoi
Tsumego
Word Ladder
Logical Deduction
Reasoning
GSM
MATH-Count&Probb
MATH-Geone MATH-Geometry
Hyperbaton
Logical Deduction
X
Penguins in Table
Reasoning Colored Obj.
Salient Trans. Err. Detect.
Snarks
Sports Understanding
Web of Lies
Acre
Advanced Geometry
AIW
Calendar Arithmetic
Circuit Logic
Emoji Mystery
Family Relationships
Figlet Font
Game of Life
Game of Life Halting
Knights Knaves
Needle Haystack
Number Sequence
Propositional Logic
Ransom Note
X
Reare
Self Reference
Syllogism
Zebra Puzzles
29


Reviews:
[{"review_id": "bHeVeigu3q", "reviewer": "Reviewer_Rp2m", "summary": "This paper introduces R1-Code-Interpreter, a framework to train text-only Large Language Models (LLMs) to effectively utilize a Code Interpreter by using multi-turn supervised fine-tuning (SFT) and reinforcement learning (RL). Addressing the challenges of task heterogeneity and a scarcity of effective samples when training across 144 diverse reasoning tasks, the authors propose a novel multi-stage curriculum learning approach. This curriculum prioritizes training samples based on their measured \"improvement potential\" (IP)—focusing on samples where the model's success is mixed rather than trivially easy or excessively difficult—which boosts RL gains from +3.4% to +9.3%. The final model, R1-CI-14B, significantly improves average accuracy on test tasks to 72.4%, outperforming both text-only GPT-4o (58.6%) and GPT-4o with its Code Interpreter (70.9%), while also exhibiting an emergent self-checking behavior through code generation.", "questions": "See weakness and follow questions：\n\n1. Why is it assumed that sample rewards follow a Bernoulli distribution? Given the diversity of tasks and the complexity of model parameters, the reward distribution of samples may be complex and difficult to represent explicitly. I am uncertain whether modeling it as a Bernoulli distribution is justified.\n\n2. What is the criterion for incorporating new lower-potential groups at each stage of curriculum learning? Specifically, at each stage, what improvement potential score threshold determines which samples are included? What is the data volume at each stage? This aspect is not clearly explained in the paper.\n\nWhile the method demonstrates good generalizability and methodological soundness, and I believe it would be effective, I have concerns regarding the experimental rigor and the support for some claims. If these concerns are adequately addressed, I would consider raising my score accordingly.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper extends the task of combining symbolic code generation with text with reasoning to broader benchmarks, providing a thorough investigation of the generalizability of this paradigm.\n\n2. This paper demonstrates innovation by proposing a curriculum learning method based on Improvement Potential, successfully extending TIR training from single-task settings to multi-task scenarios.\n\n3. Compared to traditional curriculum learning approaches, the proposed method is designed based on improvement potential, enabling effective adaptation to TIR tasks.\n\n4. Comprehensive evaluation across multiple benchmarks demonstrates the effectiveness of the proposed training method.", "weaknesses": "1. The improvement potential score defined in this paper is modeled as a function symmetric about p=½. In this case, even if two samples have the same improvement potential score, their empirical correctness rates may differ significantly. Therefore, in curriculum learning, simply incorporating samples with low improvement potential scores may overlook the training contribution differences brought by samples with different empirical correctness rates. For example, training samples with low empirical correctness rates focuses on enhancing the model's ability to handle complex problems, while training samples with high empirical correctness rates focuses on improving the model's ability to handle simple problems.\n\n2. Curriculum learning is the core contribution of this paper, yet the paper lacks detailed ablation studies on curriculum learning rounds, only showing a single RL validation curve.\n\n3. The paper lacks exploration of the method's adaptability to other algorithms beyond GRPO, such as Reinforce++, ARPO, CISPO, etc.\nIn Figure 3(b), the distinguishability between different curves is poor. I suggest trying more discriminative visualization methods (e.g., using logarithmic scale).\n\n4. In curriculum learning, the model used to compute the improvement potential score of samples is the initial SFT model, while the model being trained is not the SFT model. Therefore, this curriculum learning method is an offline reinforcement learning approach, and I believe its training effectiveness may differ from online reinforcement learning methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XGPowdRB6l", "reviewer": "Reviewer_Eyn2", "summary": "The authors introduce R1-Code-Interpreter, training a single LLM to orchestrate natural-language reasoning with Python execution across 144 heterogeneous task families. The training pipeline couples 6.5k multi-turn SFT traces with GRPO-based RL, and its key ingredient is an improvement-potential curriculum that prioritizes items the model currently solves ~50% of the time, maximizing learning signal. Code is executed in a CPU sandbox (with an 8-call cap) to keep GPUs saturated while enabling multi-turn tool use. On 37 held-out tasks, the 14B model improves from 44.1% to 72.4%, narrowly surpassing GPT-4o + Code Interpreter. The authors conduct ablations, showing that vanilla RL plateaus on batches that are either too easy or too hard, while potential-guided scheduling sustains informative variance. The authors also characterize code-call counts, response lengths and report emergent self-checking where the model writes code to verify its own outputs. The authors provide this as evidence for a multi-stage, code interpreter-centric training paradigm in LLM reasoning research.", "questions": "Your reward seems largely binary (correct/incorrect) with small format/turn terms. Did you fine-graded or curriculum-aware reward shaping (unit-test pass fraction, etc)? \n\nHow sensitive are learning curves and final accuracy to reward design, and did you observe reward-hacking (optimizing for format/turns over task progress)?\n\nSmall Comments:\n- Paper mentioned (15 agents), but I'm unsure? Did the authors mean N=20?\n- Typo: polynimial equation -> polynomial", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Comprehensiveness:**\nThe authors provide a broad, carefully controlled experimental program, spanning diverse task families, staged training with curriculum learning, warm-start ablation, strong baselines, and behavioural system measurements (emergent self-checking, code-usage, verbosity), that provides compelling evidence of the paper’s general-purpose claims.\n\n**Unlocking Code Interpreter Potential:**\nThis work offers a concrete, scalable recipe that elevates code execution from a math-only crutch to a general-purpose reasoning tool, yielding sizable gains across heterogeneous tasks and setting a practical foundation for the research community.\n\n**Technical & Artifact Contribution:**\nThe authors commit to open-sourcing the code, model checkpoints, and datasets, and they document clearly the 144-task suite and the SFT/GRPO dataset-synthesis pipeline, materially strengthening reproducibility and the research community further studies.", "weaknesses": "**Single Scope Language:**\nThe paper aims for a general code interpreter for code generation and reasoning across tasks and domains, but trains and evaluates only with a Python executor; transfer to other languages/runtimes is untested. Identifying details on different languages would materially strengthen the general code-generation claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WwwR8jCGfl", "reviewer": "Reviewer_HYQj", "summary": "This paper introduces R1-Code-Interpreter, a novel framework for training Large Language Models (LLMs) to autonomously leverage a code interpreter across a diverse set of reasoning and planning tasks. The authors first identify a critical challenge: standard reinforcement learning (RL) methods like GRPO fail to yield significant improvements when applied to a heterogeneous set of tasks due to task heterogeneity and the scarcity of effective samples.\nTo address this, the core contribution of the paper is a multi-stage curriculum learning (CL) approach guided by \"Improvement Potential.\" This method first uses an \"Agent group\" to estimate the empirical correctness rate $p_i$ for each training sample, then calculates its \"Improvement Potential\" $\\Pi_i = 4 p_i(1-p_i)$. This metric is maximized when $p_i \\approx 0.5$, i.e., when the model is \"on the fence\" about the sample. The authors theoretically justify this by linking it to the variance of the GRPO gradient, $p(1-p)$, where samples with $p_i \\approx 0.5$ provide the strongest learning signal. The RL training curriculum proceeds in stages, starting with high-potential samples and gradually incorporating lower-potential ones.\nAdditionally, the paper presents a \"Code Execution Sandbox\" that decouples code execution from GPU gradient computation, significantly improving training efficiency (39% reduction in training time). Experimental results show that their R1-CI-14B model (based on Qwen-2.5-14B) achieves a 72.4% average accuracy on 37 test tasks, outperforming GPT-4o (58.6%) and GPT-4o with Code Interpreter (70.9%). The paper also reports an \"emergent self-checking behavior,\" where the model learns to generate code to verify its previous reasoning steps.", "questions": "1. Training uses 107 tasks and testing 37 tasks from the same three suites. Can you provide leave-one-benchmark results (train on two suites, test on the held-out suite) and/or results on entirely new, unseen tasks to demonstrate out-of-distribution generalization beyond the same-source split?\n2. Your SFT is generated by GPT-4o with multiple trajectories per task, retaining only correct ones and varying prompt formats (including enforced text↔code transitions). How much does each design choice contribute? Please include ablations toggling (i) keep-only-correct vs. keep-all, (ii) prompt-format diversity on/off, and (iii) any multi-turn emphasis, and report the impact on accuracy, variance, and training stability.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Practical, replicable CI protocol: simple python code blocks, a clear final-answer marker, and tight caps on tool use (e.g., max code calls and per-call timeout) that help reproducibility.\n2. Tangible engineering win: decoupling code execution into a CPU sandbox reduces RL training wall-clock and avoids GPU stalls.\n3. Within-scope performance & diagnostics: decent gains on their own benchmark plus ablations and behavior analysis (e.g., code-based self-checking, typical call counts) that clarify where improvements come from.", "weaknesses": "1. Improvement Potential and curriculum learning: This looks novel at first glance, but in RL it is increasingly standard to focus on data where the model neither gets everything right nor everything wrong as the main RL signal. The paper largely describes this under a Bernoulli correctness assumption (Pi = 4p(1 - p)), which—at least to me—does not amount to a significant new idea or contribution beyond formalizing that intuition.\n2. Limited evaluation breadth; same-source testing; augmented SFT without ablations: The paper evaluates on targeted splits from the same source suites and uses data augmentation for SFT. For example: “To generate SFT supervision, we prompt GPT-4o to produce multiple reasoning/execution trajectories per task and retain only those yielding correct answers. To enhance diversity and adaptability, we use varied prompt formats: some allow free-form reasoning such as the prompt in Table 1, while others enforce transitions between text and code.” However, there is no experiment isolating how much this augmentation/design choice contributes. As far as I know, data quality is a very important factor in LLM training, so the lack of targeted ablations weakens the evidence.\n3. Missing comparisons to other open-source SOTA models at similar scale: The paper does not include head-to-head comparisons with strong contemporary open-source peers of comparable size (e.g., Qwen3-8B, etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)