ReVeal: Self-Evolving Code Agents via Reliable Self-Verification
Authors: Yiyang Jin
Kunzhao Xu
Hang Li
Xueting Han
Yanmin Zhou
Cheng Li
Jing Bai
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Reinforcement Learning
Code LLM
multi-turn RL
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement learning with verifiable rewards (RLVR) has advanced the reasoning capabilities of large language models. Howerer, existing methods rely solely on outcome rewards, without explicitly optimizing verification or leveraging reliable signals from realistic environments, leading to unreliable self-verification and limited test-time scaling. To address this, we widen the verification–generation asymmetry by explicitly optimizing self-verification, making it a reliable driver of deeper test-time scaling. We introduce ReVeal, a multi-turn Reinforcement learning framework that evolves code generation through self-Verification and tool-based evaluation. ReVeal structures long-horizon reasoning as iterative generation–verification turns and incorporates TAPO for turn-level credit assignment, fostering the co-evolution of code and test generation. At inference, this strengthened self-verification enables the model to use self-constructed tests and tool feedback to continuously evolve code for 20+ turns on LiveCodeBench despite training on only three. It also significantly improves Pass@k, indicating stronger exploration that expands the reasoning boundaries of the base model. These findings highlight the promise of ReVeal as a scalable paradigm for RL training and test-time scaling, paving the way for more robust and autonomous AI agents. Code is available at [https://ReVeal.github.io/](https://shimly-2.github.io/ReVeal.github.io/).

Published as a conference paper at ICLR 2026
REVEAL: SELF-EVOLVING CODE AGENTS VIA RELIABLE
Yiyang Jin
Kunzhao Xu Cheng Lis
Hang Li' Jing Bai"
Xueting Han
Yanmin Zhou²
Microsoff Research Asia
Tongji University University of Science and Technology of China
ABSTRACT
Reinforcement learning with verifiable rewards (RLVR) has advanced the rea- soning capabiliiiess of large language models. However, existing methods rely solely on outcome rewards, without explicitly optimizing verification or leveraging reliable signals from realistic environments, leading unreliable self-verification and limited test-time scaling. To address this, we widen the verification gentriatoonn asymmetry by explicitly optimizing self-verification, making it reliable driver of deeper test-time scaling. We introduce Re Veal, a multi-turn Reinforcement learning framework that evolves code generation through self- Verification and tool-based evaluation. Re Veal structures long-horizon reasoning as iterative gener- vtron-verification turns and incorporates TAPO for turn-level credit assignment, fostering the co-evolution of code and test generation. At inference, this strength- ened self-verification enables the model use self-construeted tests and tool feedback to continuously evolve code for 20+ turns LiveCodeBench despite training on only three. It also significantly improves Pass @ k, indicating stronger exploration that expands the reasoning boundaries of the base model. These find- ings highlight the promise Re Veal as a scalable paradigm for RL training and test-time scaling, paving the way for more robust and autonomous Al agents. Code is available at https://Re Vet..github.io/
(a) Multi-Turn Inference
(b)
LiveCodeBench (2025.02 2025.05) Pass@k
0.40
0.55
38.7
0.39
0.50
Code Passel
0.38
0.45
Pass@k
0.37
0.40
0.35
0.35
0.30
ReVeal (DAPO-Qwen2.5-328) MAPO-QWen2.5-33
0.34
ReVeal (DAPO-Qwen2.5-328)
0.25
Single-turn RL w/ outcome reward
0.33
1
5 7 9 11 13 15 17 19 21 23 Inference Turns
25
2
B 16 32
64
128
(number samples)
Figure 1: Performance of ReVeal on LiveCodeBench V6. (a) ReVeal enables effective test-time scaling, with Pass@1 accuracy improving from 34.8% at turn to 38.7% at turn 25. (b) ReVeal (max_turn=10) consistently outperforms both the base model and the RI. baseline in @ass@k, expanding the base model's reasoning boundaries, which the RL baseline fails to achieve.
INTRODUCTION
Reinforcement learning with verifiable rewards (RLVR) has recently shown strong potential to enhance the reasoning abilities of large language models (LLMs) (DeepSeck-AI et al. 2025 OpenAl).
"qqual contribution. Work was done during the internship at MSRA. Project leader; Correspondence Xueting Han <chrihan @microsoft.com..

---

Published as a conference paper at ICLR 2026
A key factor behind this success is the emergence of reflection and seff-verification, which allow models to iteratively refine their reasoning. Recent analyses identify the verification generation - asymmetry (i.e., easier to verify than to solve) the underlying mechanism for these improvements and key driver of test-time scaling (Wei, 2025 Setlur et al. (025). However, current RIVR methods rely solely outcome rewards without explicitly optimizing verification. This leads to unreliable self-verification, where models often produce verbose, uninformative reflections or random guessing on hard problems, and limits the effectiveness of test-time scaling: prior studies show that reasoning performance plateaus once test-time compute exceeds the training horizon (Setlur et al. 2025
Complex problem-solvingg such as competitive programming, typically requires multiple iterations of verification and revision rather than being solved in a single attempt, making accurate feedback essential to guide refinement. This highlights the need for verification-driver multi-turn reasoning. Prior work has attempted this either by training separate critic model to assess each withmpt-without leveraging tool feedback and at the cost of added inference-time complexity (Xie et al., (025) or by relying execution feedback against pre-existing public tests, which are rarely available in real-world scenarios (Gehring et al. 2025). As a result, these methods provide limited and non- non-generalizable verification, leaving self verffcctrtin unreliable and limiting sustained improvement.
To address these limitations, we propose Re- Veal, multi-turn RL framework that explic- itly optimizes self-verification, thereby widen- ing the cnafiiaation-genrratiinn (V-G) asymme- try and fostering co-evolution of both capabil- ities during training. This enables models at inference to obtain reliable verification signals from realistic environments and iteratively re- fine their solutions, without needing to rely on pre-existing tests. The widened V-G gap allows verification to drive sustained improvements in generation, ultimately enabling deeper test-time scaling (Figure 2).
ReVeal Widens V-G Asymmetry Drive Test-time Scaling
Aeauaa (9))
85
Wicened V-8 gap vie explicit verficatio optimization
Generation
verification-Frr scaling
35
Vertficatlon Accuracy Generation
15
ReVeal Training
Test-time Scaling
Figure 2: Ve expands the V-G gap.
Concretely, Veal structures long-horizon reasoning into iterative generation and verification turns. At each turn, the model generates candidate code and self-verifies its correctness by constructing test cases and invoking external tools (e.g., a Python interpreter) for execution. This closed loop yields actionable verification signals and fine-grained feedback, allowing the model identify emrrs, revise strategies, and progressively refine its output across turns. For training, we attach dense, turn-level rewards that directly supervise both code quality and verification accuracy. To ensure robustness, Veal employs a Turn-Aware Policy Optimization (TAPO) tailored for the generation-verificatio interplay, assigning credit the turn granularity and preventing reward gaming (e.g., generating trivial code to hack verification rewards). Unlike outcome-only RL methods, Re makes verification itself an optimization target, turning verification signals into reliable drivers of improvement.
We evaluate ReVeal on the challenging LiveCodeBench benchmark (Jin et al., 2024). Notably, despite being trained on only three reasoning turns, ReVeal sustains continuous refinement for over 20+ inference turns, showing robust extrapolation beyond its training horizon and tackling problems previously unsolved. Furthermore, Re'eaa significantly outperforms the base model in Pass @k by leveraging verification signals and tool feedback to guide more effective exploration, achieving an expansion of the underlying model's reasoning boundaries that standard RL methods fail to reach. These results validate Re Veal as not only a practical framework for self-evolving code agents, but also as a general RL paradigm for tasks with verification -entrrrtnngeettioonn asymmetry, where explicitly optimizing verification unlocks reliable long-horizon reasoning.
2
METHODS
2.1
REVEAL FRAMEWORK
2.1.1
ITERATIVE GENERATION- CENNERATION-VERIFICAATI VERIFICATION LOOP
ReVeal organizes long-horizon reasoning into an interleaved generation-verrffcattion loop with tool execution feedback, where verification itself is explicitly optimized to provide reliable signals for
2

---

Published as a conference paper at ICLR 2026
multi-turn refinement. As illustrated in Figure B verification to reduce system complexity and cost and to enable cross crossbcapaiility transfer, so that solutions and their verification strategies co-evolve under shared training scheme. In the code- generation setting, generation produces candidate code, whereas verification synthesizes and executes tests assess correctness. Fine-grained feedback from tool execution (e.g., Python interpreter) is appended to the rollout and conditions the next turn. The loop continues until a valid solution is found turn budget K is reached, enabling progressive refinement without external crities or predefined test cases.
single policy for both generation and
Input Prompt Question
Rollout
<Verification
Memory
Memorize
<Generation> Code
<Tool> Feedback
<Venficcaion> Test Cases
Tumn
<Tool> Feedback
Correct
Python Interpreter
Policy LLM
<Genaration> Code
Generationn> Code
eVenfication> Test Cases
Tum N
Veriiication Test Cases
<Tool> Feedback
(a) Denneratonn-eeriiiatte Loop with Tool Feedback
Outcome Reward
R Rastcome
Prompt
Generation,
Verification,
Tooly [Masked]
Generation,
Verifications
Tooly (Maskedd
Tooly [Masked]
Turn-Level Rewards
Rier
Rowr
8
ge
Rignn
R
(b) TAPO with Joint Verifiable Rewards
Figure 3: Illustration of ReVeal. (a) Iterative generation-verrficatoo loop with tool feedback. (b)
TAPO with joint verifiable rewards: outcome, generation, and verification rewards.
Table illustrates multi-turn rollout under ReVeal's structured prompting, which decouples gener- ation, verification, and tool feedback into distinct segments. At each turn, the policy first reasons thoroughly and explores diverse reasoning patterns freely, then emits structured outputs: executable code in <generat i on answer> and executable tests in <veri f caatttti ion answer>, As shown the case study, after producing candidate code the model begins verification: hypothesizes potential failure modes and edge conditions to propose diverse test cases. The stool-feedbal section then records execution results, including runtime errors, invalid test cases, as well as expected output, actual output, and pass/fail judgment for each valid test case. Based on this feedback, the model interprets traces and error messages, diagnoses underlying causes, and adjusts both its candidate code and its verification plan in the next turn. Full prompting and feedback templates are provided in Tables 5 land 6
2.1.2
VERIFICATION
The interaction with external tools provides reliable, fine-grained supervisory signals that condition subsequent reasoning and enable systematic refinement of both code and verification strategies across turns. More importantly, tool interaction broadens exploration during reinforcement learning by revealing concrete failure modes, steering the policy into promising regions of the search space beyond a single attempt and helping escape local optima. Empirically (see $3.3) this yields consistently higher @ass@k than the base model.
During RL training, the Stool-feedbacio section excluded from the loss and used only as contextual input, which stabilizes optimization while preserving coherent multi-turn rollouts. To ensure feedback quality during training, we adopt filtering mechanism: model-generated test cases executed candidate code only if they verified against golden solution. This guarantees that execution traces provide legitimate supervision, thereby improving feedback precision and guiding exploration toward correct solutions. At test time, golden reference is available; all generated

---

Published as a conference paper at ICLR 2026
test are executed, making verification fully autonomous. This places strong demand the model'' s ability to generate high-quality tests. To meet this demand, Re Veal adopts a novel RL algorithm that incentivizes diverse and reliable test construction.
2.2 TURN-AWARE RL FOR THE GENERATION- VERIFICATION PARADIGM
Prior RLVR methods rely on outcome-only signals to optimize an entire long reasoning trace, but this provides imprecise credit intermediate verification and often degenerates into blind reflection. Yet one may ask: can the current paradigm sustain reliable verification and deeper test-time scaling? Verification, however, is non-trivial task: with well-designed verifiable rewards, task can often be solved effectively. This motivates ReVeal to explicitly optimize verification with hard-to-hack rewards, which widen the perification-generaae gap. At test time, this asymmetry becomes an asset: easier and more reliable verification signals can effectively guide the harder generation process to evolve over many turns.
2.2. JOINT VERIFIABLE REWARDS
To jointly train generation and verification, ReVeal decomposes the reward into three complementary components (Fig. 3D): an oulcome reward supervising the final solution, a generation reward capturing improvements across generation turns, and a verification reward evaluating the quality of generated tests. This design naturally links the two roles in a co-evolutionary loop.
Outcome reward. The outcome reward shapes the entire reasoning chain by the final solution quality:
I Toutcome Fformat paassaate
where the format reward Fformat ensures that the model produces well-formed generation and verifi- cation blocks,
1, if the output format is correct,
Tformat --1,
otherwise,
and "passrate passrate measures final code accuracy with passrale [0, 1), giving Toutcome []1, The format reward ensures that the output follows the prescribed generation-tur and verification-turn tags, so that we can reliably identify each turn and assign the correct turn-level reward, and that the code and test-case blocks satisfy the required format, allowing us to reliably extract code and tests for tool execution.
Generation reward. For each generation turn i (odd), we compute the pass rate p produced and define:
passsrate of the code
y passrate:
h = 1, A: 3 3,
k gen
abss
pok passsrate + tmp (1 ("passrate
passsaate
where abs and imp weight absolute accuracy and iterative improvement. We set abs 0, imp = so that the reward encourages real improvements in code accuracy across turns.
Verification reward. For each verification turn & (even), we reward the proportion generated tests that succeed when executed golden code:
# [test cases in turn & that pass)
# test cases generated in turn k.]
2.2.2 TURN-AWARE POLICY OPTIMIZATION
Preliminaries. Our algorithm builds on the Proximal Policy Optimization (PPO) framework (Schul, man et al. (017), an on-policy actor-critic method that optimizes clipped surrogate objective for stable updates. PPO typically estimates token-level advantages using Generalized Advantage Estimation (GAE) (Schulman et al. 2018):
AG (A
Σ (7)) (rett + +VV4441 - V2+1), 1=0
where 7 E [0, 1] is the discount factor and À E [0, 1] controls the bias-variance trade-off.
4

---

Published as a conference paper at ICLR 2026
Turn-Aware Policy Optimization. Building structured reward design, we introduce Turn- Aware Policy Optimization (TAPO), which preserves the PPO actor-critic framework but only modifies the advantage estimator: instead of the standard BAE-based advantages, TAPO uses turn-aware return to construct the advantage estimates. TAPO leverages the critic to efficiently bootstrap from both token-level Monte Carlo returns and turn-level returns, enabling stable learning across these two reward granularities.
Token-level return. We À 1 and 7 1 (pure Monte Carlo). For a response of length T, define the token-level rewards as rT = Toutcome and Tz for all / < T. For token step l:
T-t
R Σ
Tet = + Re+1, RT+1 = 0.
(5)
1=0
Turn-level return. To mitigate adversarial reward gaming (e.g., generating trivial code that hacks the verification reward), we introduce a turn-level return tailored to the interplay. Specifically, (i) each generation reward is assigned both its own generation turn and to the immediately preceding verification turn, and (ii) each verification reward is confined strictly to its own verification turn. This design prevents reward hacking by ensuring that generation turns are rewarded solely based code quality, rather than verification success. Let {ti, ...... .....t, denote the token indices at which each turn ends (alternating generation and verification), and define:
if turn is generation,
Rturn (tt)
gen
Ru (tx+)t++)+++ = 0.
(6)
vee + Ruurn (fx+1), if turn & is verification, For token let r(t) min ts 1) and define
R (r(t)), 7() exists, otherwise.
R 0,
Turn-aware return. The final return combines the two levels:
R. R₂ + R A = R, Vi, where V/2 is the critic model's s estimate at step These advantages A, then replace the standard GAE estimates in the PPO objective, completing the TAPO update.
(8)
Discussion. TAPO provides sharper supervision than outcome-only methods by explicitly assigning credit at both token and turn levels. It integrates outcome rewards, which keep the process aligned with final correctness, and turn-level signals, which provide dense supervision for progressive refinement. This structure establishes feedback loop: stronger tests expose errors that drive code improvements, which are then reinforced by the generation reward, while improved code raises the bar for verification, pushing the model to generate richer and more challenging tests. By design, TAPO prevents reward gaming and turns this loop into stable co-evolution of code and tests. Crucially, TAPO is general credit-assip assigmmentt algorithm, applicable to any reasoning task with verifiable rewards for both generation and verification.
EXPERIMENTS
3.1 SETTINGS
Dataset We construct our training dataset from TACO Li et al. 2023). large-scale corpus comprising 26,443 algorithmic programming problems sourced from competitive programming platforms such as LeetCode (LL.C 2015) and Codeforces (Codeforces] (025).. Each problem consists of natural language description, golden solutions, and multiple test cases.
To address noise in the raw dataset, we first filter out problems containing unsupported content types, specifically those tagged with interactive or image elements. To ensure testability and correctness, we process two types of test case format, function-based tests and standard input/output tests, into unified structure compatible with our code execution environment. We then execute each test case against the first available golden solution in execution environment. Problems where the golden code fails to pass all associated test cases are discarded. After preprocessing, we retain high-quality dataset of 11,151 problems for training and 509 problems for testing.
5

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison of ReVeal with baseline methods LiveCodeBench V6 and CodeContests. Pass @ indicates the success rate; A and A represent the percentages of incorrect solutions corrected and correct solutions degraded after revision, respectively.
Model
LiveCodeBench V6
CodeContests
@ 1
A
Pass@1
A
A
Existing Baselines
Swen2.5-32B-Instrue
24.8
13.3
DAPO-Qwen-32B
31.1
18.5
Owen2.5-Cooer:322Bnnnete
29.5
14.6
w/ criticx5 Qwen2.5-Coder
29.6
2.14
3.04
w/ c GPT-4o
32.9
4.82
2.50
w/ criticx5 CTRL
33.4
3.75
0.89
RL based on DAPO- PPPPO-Qwen-2BB
Single-turn RL
32.8
21.0
Re ReVeal x25
38.7
7.50
0.0
33.6
15.69
0.0
Ablation Study: TAPO with Joint Verifiable Rewards
ReVeal x w/ outcome reward ReVeal x w/ TAPO with joint rewards
36.1
4.69
1.32
27.4
9.24
2.36
37.7
5.62
0.0
30.4
12.30
0.0
Models and Training Details We adopt DAPO-Qwen-32BB (Yu et al., 2025) as our base model, which is reinforced with mathematical data, and continue RL training code datasets adapt its reasoning capabilities to coding tasks. Our models are trained using Verl (Sheng et al.) (2024) framework 8/16 AMD Mi300x GPUs. The RL training process follows the hyperparameter settings listed in Table 2
We set the maximum number of turns to 3 during RL training.
Evaluation We evaluate Veal on two code-generation benchmarks: LiveCodeBench (LCB) V6
20025.02-0255.02 (Jain et al. 2024 and CodeContests (Li et al., 2022). The evaluation process follows the hyperparameter configuration specified in Table Although training is performed with maximum of 3 turns, we evaluate the model under extended turn settings (8 and 25 turns) to assess its generalization longer reasoning horizons and test-time scaling performance.
We use Pass to measure the success rate of the model's final code solutions. To evaluate the model's verification and self-correction capabilities, introduce two additional metrics: A denotes the fraction of initially incorrect solutions that become correct after revision, and A₁ denotes the fraction of initially correct solutions that become incorrect after revision. In line with recent work (Yue et al. 2025). we use @ass@k k up to & = 128 to assess whether Re Veal can push the reasoning boundaries beyond the base model, with at most 10 generation vnteration-viiiiaatton turns per example.
Memory Mechanism for Context Management To improve inference efficiency under extended multi-turn rollouts, context, which prevents excessive context growth without hurting accuracy (detail in Appendix C
short-term memory mechanism that retains only the last three turns as
We use Code Judge' as our code execution environment. Code Judge supports both function-based and standard input-output test case formats through consistent interface. Designed for scalability and robustness, it enables efficient long-batch execution through multi- processing and provides reliable code evaluation.
Code Execution Tool
Baselines We compare ReVeal against following baselines: (1) Base: base models without code-specific RL training: (2) CTRL (Xie et al. 2025) + Owen2.5-Coder-B33 Co five-turn critic-revision with a dedicated critic model; results cited from the original paper (evaluated on LCB 24.08-24.11): (3) Single-turn RL with outcome reward: RL with outcome-only rewards under standard <think> prompting template without any external tool calls.
https::/ thub. wttp./coo/.w..//oooeeeee.. judge

---

Published as a conference paper at ICLR 2026
3.2 MAIN RESULTS
Table T shows that single-turn RL (outcome-only, no explicit optimization of self-verification or tool use) improves @ass@ I over the base models. ReVeal goes further by explicitly optimizing verification and enabling deeper inference, it surpasses the single-turn RL baseline by a wide margin. Beyond deeper-turn gains, Re also achieves higher Pass@ I at turn (34.8%) than the single-turn RL baseline under equal inference budget on LCB V6, indicating that multi-turn training (3 turns) transfers exploration benefits into a stronger policy and that increasing training depth may further amplify gains.
ReVeal significantly outperforms critic-based methods such as CTRL. While critic models tailored for code tasks can be paired with policy models for multi-turn critique and revision, ReVell employs a single policy model that self-verifies and iteratively refines its own outputs, yet achieves superior results, highlighting the benefit of jointly optimizing generation and verification. Specifically, Re Veal attains larger correction rates with near-zero degradation, demonstrating highly robust and reliable capabilities in seff-verification, critique, and revision. (CTRL numbers cited from earlier LCB version; see Table 7 for a V5 comparison on Bren2.5532BBInsstuuce
Ablation studies confirm the benefit of TAPO with joint verifiable rewards: the same turn budget it yields higher Pass@) @1, I, increases 4,, and suppresses A, compared to outcome-only training. In contrast, outcome-only rewards exhibit higher A₁, indicating that insufficiently optimized verification can drive incorrect revisions.
Extended Evaluations and Analyses. To validate the effectiveness and scalability across models, evaluate Veal another base model, Qwen2.5-33BBInsstnee
The detailed results are in Appendix [ (see Table 7 and Figure 5). Re outperforms the single-turn RL baseline by 4.1%, which demonstrates the effectiveness of Veal. To demonstrate that the performance improvements are statistically significant, we repeat the experiment 8 times and report the mean + std in the Appendix Table Across models of varying capability, Re remains effective. With the stronger DAPO-Qwen-32BB backbone, Veal unlocks greater headroom: accuracy continues to improve with deeper inference turns and surpasses outcome-only RL by a wider margin. This underscores Re Veal's potential on stronger backbones.
To further validate the generalizability of ReVeal and the specific contributions of its components, we provide extensive additional evaluations and analyses in Appendices to K Specifically, evaluations on Dwen3-4B-Instrrue demonstrate Re effectiveness on smaller model scales and analyze the impact of varying training horizons (Appendix E. H. We also provide ablations decouple the gains from our multi-ture turn framework versus RL traning (Appendix(F) (F) and analyze the TAPO reward components (Appendix G). Beyond the main results, Veal's robust performance is confirmed on additional benchmarks like HumanEval+ and MBPP+ (Appendix DD. Furthermore, a gradient- incentive analysis for TAPO is provided in Appendix P Finally, discuss the current limitations and future directions in Appendix K
3.3 ANALYSIS
Re Enables Test-time Scaling into Deeper Inference Regimes. As shown in Figure I (a). Veal enables effective test-time scaling through iterative generation and verification. Although the model is trained with maximum of three reasoning turns, it continues to improve its solutions when more turns are allowed at inference time, leading to progressively higher code accuracy. For instance, Pass@ increases from 34.8% at turn I to 36.7% 7% turn 3, and further rises to 38.7% by turn 25 for LiveCodeBench. This compellingly demonstrates how reliable self-verification and iterative environment feedback enable compute scaling into deeper inference regimes, allowing Veal to solve previously intractable problems and evolve novel solutions. As a result, Veal supports self- improvement beyond the training horizon, enabling strong generalization in long- morizooroon reasoning during inference. Furthermore, these newly discovered solutions can be distilled back into the code L.LM to further enhance its reasoning capabilities through continued training.
ReVeal Pushes Beyond the Reasoning Boundaries of the Base Model. We compare DAPO- Qwen-32B and single-turn RL baseline with Re Veal using Pass metrics on LiveCodeBench. As shown in Figure (b), the RL baseline outperforms the base model when k 32, but its performance
7

---

Published as a conference paper at ICLR 2026
First Turn Code Accuracy
(b)
Accuracy
0.700
0.85
0.675
Code Pess Rate
0..00
0.650
Prss
0.625
0.70
0.600
0.375
0.03
0.350
ReVeal reward
0.50
ReVeal w/ outcome reward
ReVeal w/ TAPO joint rewards Single-turn outcome
0.525
ReVeal TAPO joint rewards
0.500
20
80
0.50
20
40
60
80
100
Train steps
Train steps
5750
(c) Response Length Meen
Expected Output Pass Rate
1.0
(d)
Last Turn Test Cases Accuracy
5500
Rasponde Langth
5250
0.0
4750
0.6
4500
4250
4000
0.2
3750
ReVeal w/ TAPO joint rewards
ReVeal w/ TAPO joint rewards
3500
40
0.0
80
100
Train steps
Train steps
Figure 4: Training curves for (a) first-turn code accuracy, (b) last-turn code accuracy, (c) mean response length, and (d) last-turn test-case accuracy across three training methods. Note on (b): the dip before step 40 is due to expanded evaluation coverage: as format score reaches 0.9 around step 40, more problems enter the evaluation set, temporarily lowering accuracy.
gain gradually diminishes as k increases. In contrast, Veal consistently outperforms both the base model and the RL baseline across all values from 128, demonstrating its ability to surpass the reasoning boundaries beyond the base model. We attribute this improvement to ReVeal's verification- driven exploration: tool-assisted verification provides targeted, execution based feedback and precise judgments that guide the model to explore better solutions more effectively. With this enhanced exploration capability, the model continually self-evolves and grows beyond its initial reasoning capability during RL training. We believe this approach offers promising path towards developing self-evolving agents with stronger reasoning capabilities.
ReVeal Co-evolves the Model's Generation and Verification Capabilities. Figure (b, d) il- lustrates the co-evolution of the model's code and test case generation capabilities. As shown in Figure (b) once the format is learned, final code accuracy stcadily improves during training and significantly surpasses the single-turn RL bascline. Moreover, comparing Figure and (b) reveals that final solutions consistently outperform those generated at Turn 1, with the performance gap widening time. This trend indicates that the model's verification ability strengthens, multi-turn refinement enables the exploration of better solutions, progressively enhancing its capacity to generate and refine code. After the format learned, test-case accuracy rises substantially from about 50% at step 40 to nearly 88%, as shown in Figure (d). Additionally, for correctly generated test cases, the model achieves over 85% accuracy in judging code correctness. This demonstrates that during inference, the model can reliably generate valid test cases and effectively leverage the tool to produce accurate verification signals, which are critical for continuous improvements in code quality. These results provide strong evidence that Real jointly and effectively optimizes both generation and verification, enabling the model to evolve its reasoning capabilities throughout training.
The Effectiveness of TAPO with Joint Verifiable Rewards. As shown in Table TAPO with joint rewards further enhances multi-turn performance compared to relying solely on outcome rewards. The training curves in Figure 4 b.b) show TAPO with joint rewards achieves more stable and consistent per-turn code gains, and Figure d) shows it achieves higher test-case accuracy, indicating that explicitly optimizing verification yields higher-quality tests and more effective reasoning in code generation task. These benefits amplify in longer-sequence and harder verification scenarios. On the stronger DAPO-Qwen-32B backbone with longer chains, dense turn-level supervision yields larger gains than Dwve2.5-33B-Insttee because outcome-od only signals are too coarse for extremely long chains, providing imprecise credit to intermediate verification steps. Furthermore, in more challenging verification scenarios, such
with much shorter chains (see Table 7 and Figure (5). This is
8

---

Published as a conference paper at ICLR 2026
fine fine-grained supervision becomes increasingly essential, offering richer learning signals to enhance the model's verification capabilities.
4
RELATED WORK
4.1 TOOL-AUGMENTED REASONING
Recent advances in post-training for LLMs have led to substantial progress in improving reasoning capabilities (Chen et al. 2025h a). Among these efforts, tool-augmented approaches enables large
language models (L..Ms) leverage external tools, such search engines code interpreters, to overcome inherent limitations in domain knowledge and mathematical operations. Early approaches
demonstrated the benefits of tool integration via prompt engineering (Yao et al. 2023 Chen et al.
2023) Shinn et al. (023) and supervised fine-tuning (Gou et al., (024). ReAct (Yao et al. (023) and
Reflexion (Shinn al. 2023) interleave reasoning with acting or verbal self-critique to iteratively
refine solutions under tool access. These approaches highlight the value of interactive signals, but
they typically rely on prompt heuristies. More recently, multi-turn RL has been adopted to further
enhance this capability on various reasoning tasks (Jin al. 2025 Feng al. [025 Li ct al. 2025). For example, Search-R1 (Jin et al. 2025) incorporates multi-turn interactions with a search engine to retrieve relevant contextual information during RL training. ReTool (Feng et al. 2025) and ToRL (Li et al. 2025) enable multi-turn code execution to support mathematical reasoning. Building on the promising potential of tool-integrated RL, Agent-R1 (Ouyang al. 2025) introduces open-source RL framework capable of supporting multi-turn, customization tool invocations.
Despite their effectiveness, most tool-augmented RL methods are predominantly outcome-driven: they rely on task success or failure as the sole training signal and do not explicitly optimize verification or assign credit across turns. Likewise, prompt-only agents lack turn-level, verifiable supervision, which can make self-verificatio unreliable on harder problems and limit sustained test-time improvement. Unlike prior tool-augmented works, ReVaal treats verification itself as a first-class optimization target alongside generation, and introduces Turn-Aware Policy Optimization rewards (TAPO) to provide fine tre-graided credit to both generation and verification turns. Our approach and prior tool-augmented methods are orthogonal and complementary, and can be naturally combined to further enhance reasoning capability.
4.2 SELF-VERIHICATIN OF LLMS
Enabling LLMs iteratively refine their outputs is critical for enhancing their reasoning capabilities. However, LLMs typically lack reliable self-judgment (Huang et al. 2024). One common solution to introduce separate critic model to verify the output of the policy model (Zhang al. 2025 Xie et al.] 2025). For example, CTRL (Xie et al., 2025) uses RL to train a critic model for code completion tasks. Although effective, these approaches incur the cost and complexity of maintaining and coordinating two distinct models.
An alternative strategy is enable one single model to generate outputs and velf-verify them. In mathematical reasoning, (Xiong et al., (025)) synthesizes long chains of thought that incorporate "self-reward" and "self-correction" signals as seed data for supervised fine- fin-ttning, and then further enhances this ability via RL. In the code domain, execution feedback effectively verifies code correctness and provides useful information for fixing errors. RLEF (Gehring et al. (025) performs multi-turn code generation and verification with an integrated code execution tool; however, it depends on publicly available test cases, limiting its applicability.
In contrast, Re Veal advances self- seff-verificatio by having the model generate its own high-quality test cases on the fly. By explicitly crafting and executing these tests, Re eliminates the dependency on pre- pre-existing test suites and improves applicability to real-world software systems.
5 CONCLUSION
We presented REVEAL, multi-turn reinforcement learning (RL) framework that makes verification first-class optimization target alongside generation and organizes long reasoning chains into iterative inoeration-verifiaaaiin turns with tool feedback. Using TAPO with joint verifiable rewards, RE REVAL

---

Published as a conference paper at ICLR 2026
equips LLMs with strong verification capabilities and demonstrates the surprising power of enabling code LLMs to self-evolve- bothve during RL training. where pushes boundaries beyond the base model, and test time, where multi-turn generation and verification continually refine outputs, up to 20+ inference turns. This compellingly demonstrates that REVEAL can enable compute scaling into deeper inference regimes, allowing it to solve previously intractable problems and evolve novel solutions. Furthermore, these newly discovered solutions can be distilled back into the code LLM to further enhance its reasoning capabilities through continued training.
Although we demonstrate VEVEAL on code tasks, its general concept of generation-verriicttion TAPO, and turn-level reward design can be applied to any domain with verifiable rewards for both generation and verification and that exhibits verification asymmetry, offering promising blueprint for future advances in self-iimroving, more robust, and autonomous Al agents.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. Beyond two-stage training: Cooperative sft and rl for Ilm reasoning, 2025a. URL htttps://axxi// www.wworggeeee www.abs/g///ooooooo//....o...ara /209
06948
Liang Chen, Xueting Han, Qizhou Wang, Bo Han, Jing Bai, Hinrich Schutze, and Kam-Fai Wong. Eepo: Exploration-enhaneed policy optimization via hample-then-foreee 2025b. URL https
//aaxivvorr/abbaa...aaa 22510 05837
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting:
Disentangling computation from reasoning for numerical reasoning tasks, 2023. URL. https
7/arxiv wwtg/bba//ii 12588
Codeforces. Codeforces competitive programming platform, 2025.
URL
https:/
codeforces. com/
Accessed: 2025-03-18.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang. Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li. H. Zhang, Han Bao. Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L.. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang. Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. J. Chen, R. L. Jin, Ruyi Chen. Shanghao Lu, Shangyan Zhou Shanbuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. I.. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li. Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xic, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
URL
wttp:////gbbbbbb/boobb/////. 12948
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategie tool use in 1lms, 2025.
URL httpts::/aaaaiiii org. att.////s/0.. 210.2100111110011111111111111110 11536
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code llms in execution feedback with reinforcement learning, 2025. BRL.pttss:///xxiii wwwp:iiv.org//abaa abs 2210 02089
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving, 2024.
URL httptttts///aatttte org/ abs /tt./////............ 7452
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot celf-correct reasoning yet. 2024. URL https
7/arxiva wttpivv..rrgaaa///233/
11

---

Published as a conference paper at ICLR 2026
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. http::/: pttttpps./aaxxv.org///aa 07974
Bowen Jin. Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Iims to reason and leverage search engines with reinforcement learning. 2025. URL htt.t:ttt////tt //axps org/ abs/ 2503.00000000000000 2503309516
Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset, 2023. httptttts///axl...oooo
abs 20s//232.1444 14852
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl, 2025. URL https: 23383
//axxiv.org/abs////0
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago. Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume. Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-leve code generation with alphacode. arXiv preprint arXiv:2203.072 2022.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 01211558-21722 2023.
LeetCode LLC. Leetcode: Online coding platform for technical interview preparation, 2015. URL https://leetcoost com/ Accessed: 2025-05-16.
OpenAL. Learning reason with llms.
httppp/s//oppaaaa.ccc//////////s to-reaoon- So-rrason-with-llnns- Accessed: 15 March 2025.
Jie Ouyang, Ruiran Yan, Yucong Luo, Mingyue Cheng, Qi Liu, Zirui Liu, Shuo Yu, and Daoyu Wang. Training powerful Ilm agents with end-to-end reinforcement learning, 2025. URL https
//github. com wwwusussseeeee www.ssswwstttseeneeeeeeeeee
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https: /ttp///axvvvv//aa///aa org. www.aa//........
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018. URL URRlattss///axiit
3bbs/1506.om 02438
Amrith Setlur, Matthew Y. R. Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-time compute for Ilms, 2025. URL https: /tt.sttsss///////.ooooo httpp/////////00.. 020022266000000666
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint arXiv:2409. 19256, 2024.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.
Jason Wei. Asymmetry of verification and the verifier's law. https: www. jasonwei.net. bl blog/asymmnt flog/asymmetry-oss -otry-o----iir----e ficat i veri weerriiirrsaa lawerslwwwww 2025.
Zhihui Xie, Jie chen, Liyu Chen, Weichao Mao, Jingjing Xu, and Lingpeng Kong. Teaching language models to critique via reinforcement learning, 2025. URL http:::/t////c //aarr//...ia.a/ wwwpsiiirrrr////. aagg/bbs/ abs 2502 034
12

---

Published as a conference paper at ICLR 2026
Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self- rewarding correction for mathematical reasoning, 2025. URL bttps://arxiv/: www. abss/
2962.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang. Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL httpt:://aaatttt
org/ www.as/ss.../ 09388
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023, URL httpp:://axxt
Skg/abs/2210 03629
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong. Chi Zhang. Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song. Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025. URL /ttps::/arxiv... http://awxii..ccaaa////2/// 14476
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model?, 2025. URL /ttttt:://atiiiiiiis
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction, 2025, Utthttps: pitetts://axx./ //arxiv org/abs/2408. 15240
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
In preparing this manuscript, we used a large language model (LLM) solely for polishing the writing style and improving the clarity of the manuscript. The LLM was not used for generating research ideas, designing experiments, conducting analyses, or deriving results. All scientific contributions, including the conceptualization, methodology, experiments, and conclusions, were developed entirely by the authors.
B IMPLEMENTATION DETAILS
B.1 HYPERPARAMETERS
Table 2 and Table show the detailed hyperparameters we use during training and evaluation.
Table 2: RL Training Hyperparameters for ReVeal-Qwen2.5-3233 and Re ReVeal-DAPO-Owen-
32B
Parameter
Qwen2..5-2B-Insttrue
DAPO-Qwen-32B
Max Turn
3
Training Batch Size
128
1024
Mini-Batch Size
16
256
Learning Rate (Actor)
5 X 10
1 X 10
Learning Rate (Critic)
x 10
1 x 10
KL Coefficient
0.0
0.0
Maximum Prompt Length
4,096
4,096
Maximum Response Length
8,192
12,288
Maximum Tool Response Length
4,096
4,096
Temperature
1.0
1.0
Training Epochs
10
Table 3: RL Evaluation Hyperparameters for Owwen2.--32B-Instruue and DAPO-Qwen-32B
Parameter
Qwen2..533BBIInstrue
DAPO-Qwen-32B
Maximum Prompt Length
4,096
4,096
Maximum Response Length
20,480
32,768
Maximum Tool Response Length
4,096
4,096
Do Sample
True
True
Temperature for Pass @
0.6
0.6
Top for Pass@1
0.9
0.9
Rollout for Pass@1
10
10
Temperature for Pass @k
0.6
0.6
Top_p for Pass
0.9
0.9
Rollout for Pass@k
128
128
B.2 CASE STUDY OF REVEAL
Table showw detailed case of ReVeal.
B.3 PROMPT TEMPLATES
Table shows the comparison between the commonly used Think-Answer prompt and our Generation- Verification Prompt. Our prompt guides the model to continuously alternate between generation and verification until the correct answer is obtained. Additionally, enable the extraction of code generated by the model for providing accurate training rewards, we instruct the model to enclose the code within python blocks.
14

---

Published as a conference paper at ICLR 2026
Table 4: Case Study of ReVeal Performing Iterative Generation and Verification
Question:
The avo-dimensional kingdom has regular army n people... vro-dimensional kingdom has m vests at disposal, the j-th vest's s size equals & Help mobilize the Two-dimensional kingdom's 's army: equip with vests as many soldiers as possible.
Response:
sgeneration-thins
The problem can be solved using dual circulation.
Here's step-by-step approach to the solution:
Wait, I can use a greedy approach to solve this problem. Let me try the greedy algorithm by following steps:
ceneerationthhnks
sgeneration-answer>
python
def equip_ m. b):
dggeneration-asswer>
<verification-think
Now, we will verify the code with the provided test cases.
-Example Input should result in matching 2 pairs: (Soldier index with Vest index and (Soldier index with Vest index 2). There no possible matching remaining soldiers with the available vests within the given range.
Wait, I could add an extra test case for verification; more vests than soldiers...
<lverification-thing
everification-answer
IIppu::
..11 35
Expected Output:
..332"
<'verfiiation-answer>
<tool-feedback>
Actual Output:
"'31
JJdgemeet:
Failed
</eool-feedback
<generation-thinl
The initial code provided did not successfully handle the actual output as expected. There is discrepancy where the vest index tracked the loop not correctly being Alternatively, should create list that keeps track which vests have already been used and adjust code logic to ensure each vest is used only once. Here a revised version of the code:
updated...
<lverification-answerr
<tool-feedback>
Actual Output:
..32
Judgement: Passed
<tooo-feedbackk
15

---

Published as a conference paper at ICLR 2026
Table 5: Comparison Between Think-Answer Prompt and Re Veal Prompt
Think-Answer Prompt
ReVeal Prompt
system
system
You Qwen, created by Alibaba Cloud. helpful ussistant. user
Qwen, created by Alibaba Cloud. You helpful assistant.
user [question]
[question]
First think through reasoning process and write Python
First think about the reasoning, process the mind and then provides the user with the answer. The reasoning process answer enclosed within <think> </think> and <answer>> </answer> lags, respectively, i.c, <thinks reasoning process <ithink> <answer> answer here <lanswer>. Enclose your code within delimiters as follows.
code solve the problem, encloose your reasoning process in <gemeration-think <igensration-thinss and present the code in python
Your code
within <generation-answeer <igenoratiion-nnswer After that, verify your code by generratin test cases:
python
YOUR CODE HERE
1. Extract sample test cases if the problem description includes them. When peces- sary, generate u small the generated code.
additional lest validate correctness of
assistant
Enclose your reasoning process in everifiation--hinks c/verification and enclose the final test cases and your verification conclusion within everification- answer> </verification-anne wnswer lags and wrap each test case using the following format:
tags
IIput:
testcase input ...
Expected Output:
expected output
Based questions, testcase input should function call format, fn_ fn_name(12,, (12, 12, 12).
assistant
B.4 TEMPLATES USED FOR TOOL FEEDBACK
Table shoows the mapping between execution results and hint templates: (1) for test cases that are verified successful, verified as failed, we give a [Failed] signal in the judgement area; (3) for test cases that are verified as wrong, we give clear feedback of [Wrong test case] for individual failures, or [No correct test cases generated] if all test cases are invalid; (4) for format error, we will give the feedback of formatting instructions guide correct generation.
give [Passed] signal in the judgement area; (2) for test cases that
C THE EFFECTIVENESS OF SHORT-TERM MEMORY
To prevent the interaction context from growing unbounded and to keep inference efficient, use a short-term memory mechanism. The system keeps rolling window of the last three genera- tion-verification turns and discards older interactions. For those retained turns, preserves the full content of code, test cases, and tool feedback. This design allows the model to reuse information from recent history and build on previous attempts, which helps reduce redundant exploration and speed up convergence.
We compare models with and without memory integration. The baseline model without memory provides complete historical information from all previous turns directly to the model, maintaining full contextual details throughout the interaction sequence. We evaluated ReVeal's impact on LiveCodeBench Pass performance with and without memory. Test results indicate that introducing short-term memory does not cause a decline Pass I (w/o memory 38.2% vs. w/ memory 38.3% at turn 15), and may yield slight performance boost. This sustained improvement capability highlights the memory mechanism's effectiveness in enabling continuous learning and adaptation within computational constraints.
16

---

Published as a conference paper at ICLR 2026
Table 6: Tool Feedback Templates for Different Execution Result Types.
Execution Results
Feedback
Inppt:
[input]
Expected Output: (expected output)
Success Test cases
Actual Output: (actual output)
Judgement
Passed
Input:
(input)
Expected Output: (expected output]
Actual Output: (actual output)
Failed Test cases
Judgement Failed
Failed Reason (failed reason)
Input:
[input)
Expected Output: (expected output)
Actual Output: (actual output)
Wrong Test Cases
Judgement
Wrong test case.
No correct test cases generated.
No valid code because of the incorrect format. Write Python code again, present the code in python
Your code
within <generation answer> edgeneration-answeeeee answer> tags. After that, verify your code by generating test cases:
1. Extract sample test cases if the problem description includes them...
Error Format
Wrap each Input:
using the following format:
testcase input
Expected Output:
expected testcase output
17

---

Published as a conference paper at ICLR 2026
D MORE EXPERIMENTS ON
Main Results. To further validate the effectiveness and scalability of ReVeal across different base models, we additionally conduct experiments Qwen2.5-32B-Insttue achieves a 4.1% improvement in Pass@ I over the single-turn RL baseline. Moreover, using TAPO with joint verifiable rewards yields higher Pass I compared ReVeal with outcome-only reward, while maintaining higher A and near-zero A J. These results demonstrate that Re Veal remains effective and stable across backbones of varying capability.
As shown in Table 7 Veal
Training Curves. Figure Showws concurrent improvements in both code accuracy and test-case accuracy when using TAPO with joint rewards, indicating that explicitly optimizing self-verification leads more reliable verification and drives stronger multi-turn refinement, echoing main findings on DAPO-Qwen-32B.
Significance. To demonstrate that the performance improvements are statistically significant, repeated the experiment 8 times and reported the mean = std in Table confirming the significance of the gains.
Table 7: Performance comparison of Veae D0wen2.5-32B-Insiie with baseline methods LiveCodeBench. Pass@) indicates the success rate; A, and A represent the percentages of incorrect solutions corrected and correct solutions degraded after revision, respectively.
Model
LiveCodeBench V5
Pass @1
A
A
Existing Baselines
Qwen2.5-322 Dwent.5-32B-Instruet Instruct
26.6
DAPO-Qwen-3215 Swen2.5-Coder-322B-Iie
29.6
30.5
w/ criticx5 Qwen2.5-Coder
29.6
2.14
3,04
w/ 5 GPT-4o
32.9
4.82
2.50
w/ criticx5 CTRL
33.4
3.75
0.89
RL based on Sueen.55-32BInstteee
Single-turn RL Re 6
33.9
38.0
3.41
0.0
Ablation Study: TAPO with Joint Verifiable Rewards
ReVeal x6 w/ outcome reward ReVeal x6 w/ TAPO with joint rewards
37.1
2.98
0.0
38.0
3.41
0.0
Table Significance test ReVeal (Qwen2.5 2-..2-33-2B--nnttte on LiveCodeBench V5. mean + std indicates the average code Pass from 8 repeated experiments.
Model
LiveCodeBencc V5
mean + std
Re x turn w/ outcome reward Re Veal x turn w/ TAPO with joint rewards
37.09 + 0.31
38.02 0.33
E EXPERIMENTS ON -WWNN--BBINNSTRUC
To cover models of different scales, additionally evaluated ReVeal smaller model Qwen3-4B- instruct YYang et al., 2025). The results on gains, far exceeding other baseline methods and enables deeper test-time scaling. This provides further evidence that Re Veal is not tied to a single seale (32B) and can generalize across different models.
that ReVeal brings substantial performance
18

---

Published as a conference paper at ICLR 2026
Accuracy
Accuracy
0.70
Code Pess Rete
0.70
0.60
0.60
0.55
0.50
ReVeal w/ outcome reward
ReVeal w/ outcoome reward
ReVeal w/ TAPO joint rewards
0.45
0.40
0.40
50
100
150
200
250
50
100
150
200
250
300
Train steps
Train steps
Response Length
(d)
Accuracy
2400
Response Length
2200
Expected Output Pass
0.9
2000
0.8
1800
1.00
0.6
1.000
1200
ReVeal w/ outcooee reward
0.5
ReVeal w/ outcome reward
1000
0.4
50
100
150
200
250
50
100
150
200
250
Train steps
Train stops
Figure 5: Comparison of code accuracy, test case accuracy, and response length across training for with turn-level rewards, ReVeal with outcome-only rewards, and
ReVeal Bren2.5-32B-Inae single-turn RL without tool integration.
Table 9: Performance comparison of ReVeal trained based on OWen3-4BBIInsruun with different training turns. Pass@1 I indicates the success rate. Model
LiveCodeBench V6
CodeContests
Pass
Pass @
Owen3-4B-Instruct
33.1
24.3
Single-turn RL Re (Max train turn=l)
39.0
26.9
x turn
37.3
26.6
x5 turn
41.5
30.6
x8 turn
41.7
30.7
Re Veal (Max train turn=3)
x turn
40.6
28.5
x5 turn
44.1
33.6
x8 turn
44.5
33.9
Re Veal (Max train turn=5)
x1 turn
38.6
28.4
x5 turn
43.1
33.2
x8 turn
44.0
33.5
F ABLATIONS: SEPARATING FRAMEWORK-LEVE AND RL TRAINING GAINS
To disentangle the effect of the ReVeal multi-turn framework from the effect of ReVeal RL training, we evaluate several variants that progressively add components of ReVeal.
We first apply the Re Veal multi-turn framework Qwen2.5-32B-Ing using exactly the same ReVeal prompt format and code-execution tool. As shown in Table [0 this variant produces only modest gains and fails to sustain effective deep multi-turn refinement: performance improves slightly from 1 to turns and then saturates simply changing the prompting scheme and enabling multi-ture tool feedback not sufficient.
without any RL training,
drops. This indicates that
We then apply the same Veal multi turn framework with code execution to stronger baselines such as DAPO-Que HAPO-Qwen-322 and Single-turn RL at test time, again without additional RL under the Re Veal objective, these multi-turn variants either exhibit gains in the first few turns and then begin to degrade,
19

---

Published as a conference paper at ICLR 2026
(a)
(b)
0.80
Last Turn Code Accuracy (Train) Owen2.5-328-insttee
0.40
Last Turn Code Accuracy (Eval) on Qwen2.5-32B-Bn 3-3--nttuuut
0.75
0.338
0.70
0.36
Pacs
0.65
LiveCodeflerde
0.34
0.60
0.32
0.55
imp=0 abs=1 Imp=1
abs=1 imp-0
0.50
abs-1 imp-1
abs=0 Imp=1
abs-0 imp-1
0.45
250
Train staps
Train
Figure 6: Effect of generation-reward coefficients (abs, imp) on Re (Qwen2.5-32B -2nntusttucc). We plot (a) training code accuracy and (b) LiveCodeBench v5 accuracy across training for three settings of: (1, (1, 1), and (0, 1).
or show almost no improvement at all (Table ITp. While this confirms that most of Re Veal's s gains cannot be explained by multi-turn framework alone.
Finally, Table IT reports two stronger ablations, ReVeal (no-gen RL) and ReVeal (no-ver RL), where we still use the Veal multi-turn framework but RL-train only one role: we pair Re verifier with an untrained base generator (no- RL), or a Re Veal-trained generator with an untrained base verifier (no-ver RL). Veal (no-ver RL) yields gains in the first few turns but quickly saturates, whereas the full Veal model continues to improve with more turns and achieves higher A and lower A This indicates that Veal''ss explicit optimization of verification yields clear performance gains and enables deeper test-time scaling. Veal (no-gen RL) continues to improve with more turns even when paired with a relatively weak generator, suggesting that Re explicit optimization of verification is key to enabling deeper test-time scaling. Taken together, these ablations indicate that while the multi-turn framework and tool feedback are helpful, the main gains come from explicitly training by Re Veal RL objective.
Table 10: Performance of Qwen2.5-32BB --32BBInnttuctt under the Re Veal multi-turn framework, with and without RL training. Pass@1 indicates the success rate; A and A represent the percentages of incorrect solutions corrected and correct solutions degraded after revision, respectively.
Model
LiveCodeBench V5
Pass@1
A
4
Re Multi-turn Framework w/o Training
x 1 turn
26.3
x3 turn
27.5
x6 turn
27.4
2.40
1.31
Re Veal Multi-tur Framework w/ Training
x1 turn
35.7
x3 turn
37.5
x6 turn
38.0
3.41
0.0
G TAPO REWARD ABLATIONS
TAPO Reward Ablations Besides the reward-signal ablation comparing Re (outcome-only variant with the full joint-reward in Table we additionally include two ReVeal variants top of the outcome-only baseline: Re (outcome+gen) and Veal (outcome+ver). As shown in Table 12 both variants outperform the outcome-only version, and the full Veal achieves the overall best results. This indicates that both Tgea and Fver make useful contributions beyond the outcome reward: T Tgen supervises both generation and verification by encouraging exposing more informative failure modes and larger subsequent code improvements, while Iver supervises test case accuracy directly.
Generation-Rewe Ablation Our generation reward decomposes into an absolute term (abs, the code accuracy at the current turn) and an improvement term (imp, how much the current turn's
20

---

Published as a conference paper at ICLR 2026
Table 11: Performance LiveCodeBench V6 for configurations designed to disentangle the gains from the ReVeal multi-turn framework and from Re Veal RL training, We compare: (i) applying the multi-turn framework existing baselines without ReVeal RL, (ii) ReVeal variants where only the generator only the verifier is RL-trained (no-gen RL/ no-ver RL), and (iii) full ReVeal with RL on both roles.
Configuration
LiveCodeRench V6
Pass@1
A
A
Existing Baselines + Veal Multi-turn Framework
Base Model Ve multi-turn framework
x turn
28.4
x5 turn
32.8
4.53
0.75
x8 turn
31.8
4.15
1.70
Single-turn RL model + Re Veal multi-turn framework
x1 turn
33.4
x5 turn
32,8
1.53
1.36
x8 turn
33.0
2.05
1.53
ReVeal with RL on one role
Re Veal (no-gen RL): Base model Generator + Re Veal as Verifier
x1 turn
29.0
x5 turn
33.6
6.16
0.91
x8 turn
33.8
6.39
0.91
Re Veal (no-ver RL): Veal as Generator Base model as Verifier
x1 turn
34.6
x5 turn
36.9
2.94
0.53
x8 turn
36.8
4.25
1,26
Re Veal with full RI. training
Re Veal
x1 turn
34.8
x5 turn
37.2
3.71
0.0
x8 turn
37.7
5.62
0.0
code accuracy improves over the previous tum). In our default setting, we use abs = 0 and imp that is, we only use the change in code pass rate reward the quality the generated tests and the current-turn correction. In early experiments on weightings (abs/imp 0/1, 1/1, 1/0) and observed 0/1 > 1/0 as shown in Figure suggesting that emphasizing improvement over absolute accuracy leads to better multi-turn performance.
we compared several
H TRAINING-TURN COUNT ABLATION
We vary the training horizon to 1, 3, and 5 turns while keeping other settings fixed. As shown in Table all Veal variants outperform the single-turn RL baseline: training with turns yields much larger gain than 1-turn training, while increasing the horizon further to turns brings no additional improvement, suggesting that the benefit saturates around 3-5 turns on our current (mostly solvable) training set. On this training set, mean rewards can reach 4.5, which means most problems already solved within generation-verificalee verification turns, so the additional 2 training turns provide limited extra signal small fraction of very difficult questions. We expect longer training horizons to be more beneficial on more challenging training data.
Importantly, for all training horizons (1, 3, and 5), the corresponding models continue improve when we allow more inference turns than they were trained on, indicating that extrapolation beyond the training horizon is effective when the model is trained with only single turn.
21

---

Published as a conference paper at ICLR 2026
Table 12: Performance comparison of Re Veal trained based DAPO-Qwen-32B with different verifiable rewards. Pass@) indicates the success rate. Model
LiveCodeBench V6
CodeContests
Pass 1
Pass@1
ReVeal (outcome only)
x turn
32.7
20.0
x5 turn
35.9
26.1
x8 turn
36.1
27.4
Re Veal (outcome+ver)
x x turn
32.3
17.3
x5 turn
36.6
26.5
x8 turn
36.8
27.8
Re Veal (outcome+gen)
x1 turn
30.7
17.2
x5 turn
37.1
26.6
x8 turn
38.0
28.2
Re Veal
Full (outcome+gen+ver)
x1 turn
34.8
22.3
x5 turn
37.2
28.1
x8 turn
37.7
30.4
Table 13: Performance comparison of ReVeal (DAPO-Qua) )BAPO-Qwen-32B) with baseline methods on Hu- manEval+ and MBPP+. Pass@ indicates the success rate.
Model
HumanEval+
MBPP+
Pass@1
Pass@1
Existing Baselines
Qwen2.5-332B-Instruce DAPO-Qwen-32B
85.4
75.4
86.0
73.8
RL based on DAPO-Qwen-32B
Single-turn RL
86.0
76.2
Re Ve x
x1 turn x3 turn
85.2
77.9
88.6
78.9
x4 turn
88.4
78.9
I EVALUATION ON ADDITIONAL BENCHMARKS
We evaluate Veal two additional code-generation benchmarks: HumanEval+ (Liu al. 2023) and MBPP+ (Liu et al. 2023). The evaluation process follows the hyperparameter configuration specified in Table We Pass@ measure the success rate of the model's final code solutions. Although these benchmarks differ from our training data, Veae still yields larger gains over the baselines as shown in T33
J GRADIENT-BASED INCENTIVE ANALYSIS FOR TAPO
ReVeal's multi-turn framework explicitly elicits verification turns in the trajectory, allowing veri- fication tokens to be meaningfully optimized. Within this multi-turn framework, we analyze how TAPO modifies the underlying PPO objective and gradients relative to an outcome-only baseline. The analysis below shows that TAPO's credit assignment strengthens the optimization signal for veri- fication, which helps explain why ReVeal widens the verification -eateratinn (V-G) asymmetry. We then contrast TAPO with naive token-level implementation that lacks turn-aware credit assignment,
22

---

Published as a conference paper at ICLR 2026
illustrating how such scheme suffer from misattributed credit and reward gamiin, and how TAPO's s design avoids these issues and promotes co-evolution of code and test quality.
Outcome-only PPO baseline. Consider an outcome-only PPO baseline that uses only the scalar outcome reward Youteome from Eq. with the same KL and entropy regularizers as ReVeal. Ignoring these shared regularizers, its objective is
Jour (0) Erowo [Toutcomo(r]] (T)]
With the Monte Carlo return in Eq. and our choice 7 = 1, A = 1, the token-level return for all tokens before the end-of-sequence
R
Toutcome! (T).
(10)
Let Viail be the critic's estimate trained to regress lit and define the corresponding ad advantages
(11)
Denote by Tgm and Teer the token indices belonging to generation and verification turns respectively. Up to the usual PPO clipping and scaling factors, the actor gradients for the two roles can be written
Ugen
X
Σ
A
log Ta(a st)
(12)
TET
Iver
x
E
Σ
A log (T(a: st)
(13)
TE
Thus, the outcome-only baseline both generation and verification tokens are trained only through the same outcome-based advantage.
TAPO-augmented objective and gradients. generation and verification rewards:
TAPO augments the outcome objective with the
JTAPO (0) BBT loutcome (7) + Σ gen (T)
Σ r (r)
(14)
where y. gen is the pass-rate based reward at generation turn and r is the test-quality reward at verification turn k. TAPO routes these rewards tokens through the turn-level return in Eq. and the combined return in Eq. 8 Following Eq. g we write the TAPO return and advantages as
R = R. + R.mm
A
TAPO = R V TAPO
(15)
where VTAPO; is the critic trained to regress R.. Using Eq. 6 together with the fact that Rs Toutcome for all tokens before EOS, we obtain explicit forms for the TAPO return. For a token belonging to generation turn k
RG Trancome + gen"
(16)
gens
For a token belonging to verification turn k whose successor turn + is generation, Eq. 6 gives
tur
+
1++1
R Toutcome + p +
R++1
(17)
gen
gen
The corresponding actor gradients under TAPO are
ggen
TAPO (8)
Σ
A.
ATAPO
V log To(a, (8))
(18)
TE
Iver
TAPO (0)
Σ
A TAPO log To(a:) 18))
(19)
23

---

Published as a conference paper at ICLR 2026
Incremental gradients and widened V-G asymmetry. baseline explicit, we define the incremental gradients
To make the difference the outcome-only
Agen (0) x IE
Σ
(ATAPO-
AQt Vo log (T( | |
(20)
LETpm
Aver (0) x
(4 A Vo log TT $1)
(21)
1E
so that
Ogen
TAPO (0) =
hase Ugen
(0) +
Agen (0),
Over
TAPO (0)
= Iver
hase
(0) + (vee
(22)
Using the definitions above we can write
ATAPO
A = (Re V,JAPO) (124
Viout
R (V,TAPO
(23)
In the actor update, the critic outputs V are treated as baselines and are not differentiated with respect to 9. Thus the additional actor gradients Ageen and A are driven by the turn-level returns R, that TAPO introduces.
For generation tokens, R. = r depends only on the per-turn code pass-rate signal and does not involve the verification reward. Relative to outcome-only PPO, TAPO therefore provides finer, outcome-aligned signals for generation by decomposing changes in pass rate into turn-level contri- butions; this can be viewed as process-level reward shaping rather than introducing new objec- tive beyond code quality. For verification tokens, in contrast, 1 = r + r-++1 combines two environment-groundee
signals: test quality on the golden code and the effect of these tests on the next generation turn. The outcome-only baseline optimizes verification tokens only through signals mediated by final code quality, whereas TAPO adds verification-specdie gradient components that directly reward constructing high-quality, informative tests.
Thus, on the one hand, Veal''s 's multi-turn framework explicitly elicits verification turns in the trajectory, allowing verification tokens to be meaningfully optimized, which is not guaranteed by other baseline methods. On the other hand, relative to outcome-only PPO, TAPO strictly enriches the task-aligned reward signals: it explicitly optimizes the verification objective while leaving the generation objective essentially unchanged up to finer credit assignment. This structural asymmetry in the incremental gradients provides gradient- levell explanation for our empirical finding that TAPO substantially increases verification accuracy and widens the V-G gap compared to other baselines.
Accurate turn-aware credit assignment and robustness to reward gaming. A naive token- level implementation for multi-tum generation-verrficatiin would simply attach each reward r or r.k to the last token of the corresponding turn and then use standard Monte Carlo return for all earlier tokens in the trajectory, without the turn-aware routing in Eq. 0 This would propagate the verification reward r back through all preceding generation tokens and introduce gradient terms of the form r Ve log Te (a) pen st for tokens that do causally inffuence r since r is evaluated on the golden code. Code tokens would then be updated by signal they do not influence (e.g.. wrong code but strong tests still giving high verification reward to the code, or correct code but weak tests inducing low verification reward), and the policy may learn spurious correlations between code patterns and high test accuracy rather than improving true code quality.
By contrast, TAPO's reward design enforces accurate turn-level credit assignment, which is crucial for stable and effective RL training. Its turn-level routing ensures that generation tokens are rewarded only via outcome and pass-rate improvements (Fauteome and r.k r and that verification tokens are rewarded only when their tests are both strong on the golden code y and helpful for subsequent code refinement pk++ and Vnutcome- Together with the shared policy for both roles, this structure encourages co-evolution of code and tests: better tests expose more informative failure modes and increase gen and Toutcome: while better code raises the bar for verification and encourages tests that achieve higher k
The same with task-specific verification rewards, while remaining robust and compatible across different tasks. For example, if verification turn were rewarded for judging correctness, the naive implementation
pattern naturally extends to more general cnteraaionn-eificaatti designs
24

---

Published as a conference paper at ICLR 2026
mentioned above could lead to collusion between generation and verification: generation might collapse to trivially wrong solutions that are extremely easy for verification to flag as incorrect, allowing verification to obtain high verification reward while the solution quality remains poor. TAPO's separation of roles prevents such self-referential reward exploitation and make the Re Veal training paradigm robust across different verification reward designs.
K
LIMITATIONS
Our current instantiation of ReVeal is primarily optimized for functional correctness: solutions that produce correct outputs on the available golden tests are treated as correct, and violations of time/space complexity are only penalized when they cause failures under these tests (e.g.. timeouts within the execution budget). As a result, brute-force or computationally inefficient solutions that still pass all golden tests may not be distinguished from truly efficient ones. natural direction for future work is to add stress tests with very large inputs; these test cases can then be used to compute the outcome and generation rewards, directly encouraging solutions that remain correct under stricter time/space constraints. Another complementary direction is to enrich the tool interface with runtime and resource statistics (c.g., execution time, memory usage) and incorporate these signals into the outcome, generation, and verification rewards, encouraging both efficient code and stress test cases that reliably expose inefficient solutions.
25


Reviews:
[{"review_id": "zNEIYSxQyU", "reviewer": "Reviewer_utHn", "summary": "Current Reinforcement Learning with Verifiable Rewards (RLVR) only optimizes \"generation capability\" relatively well, while \"verification capability\" is barely explicitly optimized—leading to a significant gap between generation and verification—this paper proposes ReVeal, a multi-turn reinforcement learning (RL) framework for code agents. Within a single trajectory, ReVeal alternates between \"generation → verification → tool feedback\" and, for the first time, places \"self-verification\" and \"generation\" under the same optimization target. In this process, the model not only generates code but also automatically produces executable test cases and leverages feedback from external tools for self-correction. To ensure the balanced improvement of generation and verification capabilities, ReVeal introduces Turn-Aware Policy Optimization (TAPO), which assigns explicit reward signals to both generation and verification actions at each turn. This enables fine-grained credit assignment, thereby simultaneously enhancing these two capabilities during the training phase and narrowing the performance gap between generation and verification. Unlike traditional sparse reward methods that rely solely on the final pass rate, ReVeal incorporates tool-based feedback from interpreters/code judges at each turn. This allows the model to continuously optimize code quality during the inference phase, even when performing a large number of inference turns. Experimental results demonstrate that ReVeal achieves excellent performance on LiveCodeBench, validating the effectiveness of optimizing verification capability on par with generation capability.", "questions": "1.\tThe turn-level return (Eq. 6 in the manuscript) assigns each generation reward to both the generation turn and the preceding verification turn with weight 1.0. This tightly couples the verification reward to the next-turn generation quality, which can increase variance when the next turn underperforms (e.g., due to context truncation or large code edits). Have the authors tried attenuated backflow or exponential decay across later turns to test whether verification still resists reward hacking under weaker coupling? Even a small ablation would clarify whether the current choice of weight 1 is essential or merely convenient.\n2.\tSince the policy is shared between generation and verification, it is hard to disentangle whether the gain comes from (i) better code generation under multi-turn tool feedback, or (ii) explicit optimization of the verification subtask. A simple ablation where the verification head (or the verification part of the loss) is frozen/removed, while keeping the rest of the setup identical, would clarify the contribution of “training verification as a first-class task”. Without such an ablation, the current evidence could still be explained by “multi-turn generation with tool context”.\n3.\tA key claim is that ReVeal, by explicitly optimizing verification, can extrapolate beyond the training horizon (3 turns → 25 turns at inference). This is compelling, but currently only one training horizon is reported. Could the authors report results with a longer training horizon (e.g., 5 turns) and a shorter one (e.g., 1–2 turns) to show (i) whether the benefit saturates, and (ii) whether the ability to extrapolate is truly due to the proposed TAPO design rather than a coincidental choice of 3 turns? This is especially relevant since context is truncated to the last few turns during inference.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe ReVeal framework innovatively introduces explicit verification turns during the training process and uses the gold solution to pre-verify the generated test cases. This method enables the model to explicitly learn self-verification capabilities during training, thereby enhancing the reliability of self-verification.\n2.\tReVeal introduces a Turn-level return mechanism: once the generation is correct, the test generator of the previous round can also receive rewards. This design effectively ties the generation and verification processes together, preventing undesirable behaviors such as \"writing useless tests\" and other reward gaming phenomena, thus promoting the co-optimization of the generation and verification processes.\n3.\tExperimental results show that on LiveCodeBench V6, as the number of inference turns increases, the model's code generation capability and performance continue to improve. This verifies the phenomenon that even with short training turns, the model can still perform continuous multi-turn reasoning during inference. The model indeed learns effective self-verification strategies during training, enabling it to continuously improve its performance in the inference phase.", "weaknesses": "1.\tThe paper motivates ReVeal as a way to make self-verification reliable “in realistic environments where public tests are unavailable”. During training, however, the model-generated tests are filtered against a golden solution to guarantee high-quality feedback. At evaluation time (e.g., LiveCodeBench) the final correctness is judged by the benchmark’s own test suites, i.e., still under a setting with reliable canonical tests. Thus, the experiments mainly demonstrate that optimizing verification helps on code benchmarks that already provide trustworthy execution oracles, rather than showing robustness in the fully autonomous, low-test-coverage setting highlighted in the introduction. This is not fatal — code is exactly the domain where such oracles exist — but the paper’s claim about “realistic environments without pre-existing tests” would be stronger with a no-filter / noisy-test ablation.\n2.\tCurrent experiments are only conducted on large 32B models, and do not involve the performance of small-scale models such as 7B or 14B, nor sparse models. Considering that small-scale models may face greater challenges in generating test cases, the lack of verification on small-scale/sparse models limits the generalization ability of the paper's conclusions regarding model scale. It is suggested that the authors expand the experimental scope in future research to cover models of different scales, so as to evaluate the applicability of the ReVeal framework on various models.\n3.\tThe paper provides a relatively brief description of the reward mechanism, and does not elaborate on the specific calculation methods and weight settings of format rewards, final pass rewards, turn-level rewards, etc. This may require researchers to adjust these hyperparameters by themselves when reproducing the experiments. It is suggested that the authors provide more detailed reward mechanism design and hyperparameter settings or released the open-source implementation of the ReVeal framework in future research to improve the transparency and reproducibility of the study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QrzOJIDj8k", "reviewer": "Reviewer_jPh8", "summary": "This paper introduces a novel reinforcement learning framework named ReVeal, which aims to enhance the long-horizon reasoning and self-correction capabilities of Large Language Models (LLMs) in code generation tasks. The core contribution of this work lies in decomposing complex programming competition tasks into an iterative, multi-turn process of \"code generation\" and \"test case generation.\" For this process, the authors designed a joint reward mechanism that includes Turn-Aware Policy Optimization (TAPO) to collaboratively optimize the model's code generation and self-verification abilities, thereby enabling deeper test-time expansion.", "questions": "To further enhance the paper's persuasiveness and rigor, I suggest the authors consider the following points for revision and supplementation:\n1. Recommendation to Expand Datasets to Verify Generalization\nTo more comprehensively validate the effectiveness and generalizability of the ReVeal framework, I strongly recommend that the authors supplement their experiments on more diverse datasets. For example: Foundational code generation datasets like HumanEval+ and MBPP+. Datasets that are closer to real-world development scenarios, such as CodeUJB or SWE-bench. Experiments on these datasets would demonstrate that ReVeal is not only applicable to algorithmic problems but can also generalize to complex, real-world software engineering scenarios, which would significantly enhance the value of this work.\n2. Recommendation to Systematize Experimental Setup for Fair Comparison\nRegarding the systematicity and fairness of the experimental setup, I have two main concerns:\nUnify evaluation metrics for direct comparison: In the main results table (e.g., Table 1), the evaluation turns should be unified for all multi-turn methods being compared (e.g., reporting Pass@1 at turns 1, 5, and 8 for all methods). This would allow readers to more intuitively see the performance differences between methods at equivalent inference costs.\nConfigure a fair reasoning mechanism for key baselines: Baseline models like DAPO-Qwen2.5-32B and Single-turn RL should also employ a multi-turn feedback mechanism for test-time generation, similar to ReVeal. This would ensure a fair evaluation and help clearly distinguish whether the performance gains come from the RL training itself or from the reflection mechanism.\n3. Report and Align Training Costs\nThe paper lacks a discussion of training costs. I recommend supplementing the paper with data on training overhead, such as the GPU hours for both ReVeal and Single-turn RL. This would not only help readers assess the method's cost-effectiveness but also allow for a fairer measurement of the ReVeal framework's true value and efficiency when compared under a similar training budget.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "• Utilizing reinforcement learning to enhance a model's ability to solve complex problems, self-reflect, and correct code is a crucial research direction.\n\n• The paper is well-structured with a clear and logical flow.", "weaknesses": "• The experimental dataset is somewhat limited: The main experiments are based on programming competition-style problems. While this effectively tests the model's algorithmic capabilities, it raises questions about the method's generalizability to broader, more realistic real-world development scenarios.\n\n• Some experimental setups are not sufficiently clear or systematic: The experimental comparison section has issues that could affect the reliability of the conclusions, such as inconsistent evaluation turns and a lack of fairness in baseline comparisons. This makes it difficult for readers to accurately determine the source of the performance improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qusXUBjWGD", "reviewer": "Reviewer_beFR", "summary": "This paper introduces ReVeal, a multi-turn reinforcement learning framework for code generation that explicitly optimizes self-verification alongside code generation. The key innovation is treating verification as an optimization target rather than relying solely on outcome rewards. ReVeal structures long-horizon reasoning as iterative generation-verification loops, where the model generates code, constructs test cases, executes them via external tools (Python interpreter), and uses the feedback to refine solutions across multiple turns. The authors propose Turn-Aware Policy Optimization (TAPO), which assigns credit at both token and turn granularity using joint verifiable rewards (outcome, generation, and verification). Experiments on LiveCodeBench and CodeContests show that ReVeal enables test-time scaling beyond the training horizon and achieves higher Pass@k than baseline methods, suggesting expanded reasoning boundaries.", "questions": "Can you provide complete ablations isolating: (a) generation reward only, (b) verification reward only, (c) outcome + generation, (d) outcome + verification? This would clarify each component's contribution. Additionally, what happens with different reward hyperparameters (the factor of 5, abs/imp coefficients)?\n\nWhat percentage of model-generated tests are filtered during training because they fail on the golden solution? How does this evolve over training? Since filtered tests provide no learning signal, how does the model learn to avoid generating bad tests?\n\nCan you provide theoretical or empirical analysis of TAPO's convergence properties? How does the combination of token-level and turn-level returns (Equation 8) affect optimization stability compared to standard GAE? Have you observed any instabilities?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a genuinely novel perspective on multi-turn code generation by explicitly optimizing verification as a co-equal objective with generation. While prior work has explored critic models or execution feedback, ReVeal's approach of jointly training generation and verification within a single model through structured turn-level rewards is innovative. The TAPO algorithm, though building on PPO, introduces a sensible credit assignment mechanism tailored to the generation-verification paradigm.\n\nThe paper is generally well-structured with clear motivation. Figure 3 effectively illustrates the generation-verification loop and TAPO's reward structure. The case study provides intuitive understanding of how the iterative refinement process works in practice. The distinction between training (with golden solution filtering) and inference (fully autonomous) is clearly articulated.", "weaknesses": "The ablation analysis is concerningly limited for a paper making multiple methodological contributions. Table 1 only compares \"outcome reward\" versus \"TAPO with joint rewards\" as a monolithic change, without isolating the contribution of individual components. Critical missing ablations include: (1) What happens with only generation rewards or only verification rewards? (2) How does the specific turn-level return formulation in the equation compare to simpler alternatives? (3) What is the effect of different reward hyperparameters (abs/imp coefficients, the factor of 5 in passrate)? (4) Why is the ablation only shown at turn 8 when main results extend to turn 25? These gaps make it difficult to assess which design choices truly matter.\n\nThe paper claims to \"widen the verification-generation asymmetry\" (Figure 2) but provides only empirical observation without theoretical grounding. Why does TAPO's specific formulation in Equations 6-8 prevent reward hacking? The intuitive explanation that \"generation turns are rewarded solely based on code quality\" is not rigorously proven. What are TAPO's convergence properties compared to standard PPO? The paper would benefit from formal analysis showing that TAPO's credit assignment scheme provably incentivizes the desired co-evolution behavior rather than reward exploitation.\n\nDespite claiming the method is \"applicable to any reasoning task with verifiable rewards\" and \"verification asymmetry,\" experiments are confined entirely to code generation on two similar benchmarks (LiveCodeBench, CodeContests). No evidence supports the broader applicability claim. Even within code generation, evaluation is limited to competitive programming problems—what about software engineering tasks like debugging, refactoring, or API usage? The paper tests only two base models from the same model family (Qwen). Testing on diverse model architectures and problem types would strengthen generalization claims significantly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7UViXbZqh4", "reviewer": "Reviewer_6pm9", "summary": "The paper proposes ReVeal, a multi-turn RL method for code generation explicitly optimizing for self-verification, i.e., generating unit test cases in this context, for which the authors adapt PPO into TAPO (Turn-Aware Policy Optimization) through deliberate reward shaping for the \"generator\" turn and the \"verifier\" turn. The authors train DAPO-Qwen-32B on a processed subset of TACO and eval on LiveCodeBench (LCB) v6 and CodeContest. The reported Pass@1 = 38.7% on LCB v6 at 25 turns. They also show higher pass@k compared to single-turn RL.", "questions": "Please see above for major questions. Also minor question:\n- The number on CodeContests is on validation set or test set? I didn't find such information.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "About the originality, the problem of getting more signal than existing public test cases is crucial to multi-turn code generation in the competitive programming problems domain. The paper proposes to use model to self-generate test cases, and explicitly optimizes for this in the training time using Reinforcement Learning.\n\nTo train a single model end-to-end with the ability of both generating code solutions and unit test cases, the paper details the design of different reward for generation turn and verification turn, which the authors show that it leads to better performance empirically, and its effectiveness is shown via the ablation by comparing to RL training with outcome only reward.\n\nI appreciate the authors' effort on getting results with mean and std deviation with running multiple experiments to reduce the noise of the numbers. Besides reporting the pass rate, the reported metrics of corrections and degradations are also interesting.", "weaknesses": "Despite the problem the manuscript tries to tackle and the approach it takes are interesting: the generation-verification gap and the method to train a single model trained for both verification and generation, there're several points that I feel less convincing and the manuscript could be of better form and positioned if these points are properly addressed:\n\n1. Novelty & Contextualization\n\n1.1 I see the core idea of the manuscript as explicitly optimizing self-verification ability, and better turn-level credit assignment, going beyond existing multi-turn RL work RLEF. Therefore, the algorithmic novelty sits largely in the reward shaping design and the TAPO construction, which is consistent with standard PPO variants and MC returns with custom shaping at turn level. \n\nSince the design of TAPO involves multiple component. Ablation of outcome only reward and single-turn RL seems not sufficient for me. These reward shaping necessites better motivation and comparison. The manuscript would benefit from more empirical evidence of the choice of the specific design of reward beyond motivation for justification. To name a few, for example:\n- the format reward and the pass rate reward: is the format reward shaping essential and removing it could lead to different performance.\n- generation reward: did the author try settings other than `abs = 0` and `imp = 1`?\n\n1.2 Apart from the concrete algorithmic design. The baseline the paper compares to, ReAct is a prompt-only baseline, CTRL is another approach where the verifier is trained against a fixed generator. If the claimed novelty is to explicitly optimize for self-verification, comparing to stronger baseline or alternative design provides stronger evidence of the proposed framework: For example, it's still not clear the gain of the proposed framework comes from the framework design or the RL loss design, which is not totally decoupled. \n\nThis is my biggest concern when reading the manuscript: it tries to center around these 2 novelties but provides under-substantial ablation to decouple and justify each of the both. For example, for the framework design thing, the manuscript could compare (under the same classical RL loss such as PPO, or turn-level PPO as shown in the previous work RLEF) the following alternative designs that gradually leads to the current design:\n\n- the verification step is coming from a fixed model, similar to https://arxiv.org/abs/2306.09896 or the mirroring setting of CTRL, either from the same starting policy or from another model, while the generation step is normally RL trained.\n- starting from the same starting policy but maintained 2 set of weights separately for generation and verification.\n- starting from the same policy, but does 2 separate RL training update for generation and verification where the loss is masked for one another.\n\nFor another thing, there're quite a few components deviated from the standard PPO/GRPO practice in the reward design, please see above. \n\n2. Limitation\n\nThe propose framework aims at providing more unit test cases to serve as signal in the multi-turn code generation in the competitive programming framework. Wrong code solutions that leads to wrong answer for hidden test cases could benefit from the proposed framework if a verifier produces new test cases. However, another big set of wrong code solutions, which are functionally correct but do not have the desired time/space complexity, could fall out of the catch of the framework: think of a brute force solution where it produces expected output given input. To correct such solution, the verifier needs to have the ability either to produce signal for stats in runtime, or to produce large test cases to hopefully trigger timeout (which might exceed the context length for example). Discussion on whether the framework can properly address such chanllenge or even discussion in a dedicated limitation section is appreciated.  \n\n3. Writing and clarity\n\n- It seems that the \"tool feedback\" refers to solely python execution environment in this paper, which would be good if this could be confirmed.\n- The short-term memeory is only briefly mentioned. I thought it means in the multi turns, it only kept recent history (like most recent 3 turns), but L699 says \"Critical information such as successful patterns, error types, and effective test structures are preserved in complete formats.\" Further explanation or example would be appreciated.\n- The reward design part necessite more clarity. For example, I'm confused since Verification reward is \"For each verification turn k (even), we reward the proportion of generated tests that succeed when executed on a golden code\", which therefore, doesn't depend on the generated code but only on the golden code. Which should make it unhackable, but the paper later says in L222 \"To mitigate adversarial reward gaming (e.g., generating trivial code that hacks the verification reward) ...\". I do not understand how this could happen.\n- Eq (5) seems not quite useful as the manuscript never present the token level reward r_t. The whole reward design makes it look like there're token-level return and turn-level return. If I understand correctly, in the end the return is the same for all tokens in the same turn? Is there case where the return is different within one turn for different token? My understanding for the current moment is that the whole design for return is actually at the turn level, this return is \"densified\" by broadcasting to each token, and the advantage for each token is  this return substract a value baseline.\n- L214 says \"replaces GAE-based advantages with a turn-aware return.\" I do not think the author mean it or otherwise the authors use the term \"advantage\" and \"return\" interchangeably (which I found confusing), since there're still the advantage estimation in Eq(8), where the V I presume is coming from a value model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)