Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Authors: Zhuohao Yu
Weizheng Gu
Yidong Wang
Xingru Jiang
Zhengran Zeng
Jindong Wang
Wei Ye
Shikun Zhang
Venue: ICML 2025 poster (2025)
Keywords: Code Generation
Process Supervision
Reasoning
Reward Models
Inference-Time Scaling
Large Language Models
Primary area: deep_learning->large_language_models

Abstract: Large Language Models excel at code generation yet struggle with complex programming tasks that demand sophisticated reasoning. 
To bridge this gap, traditional process supervision relies on learned reward models requiring costly training data and suffering from reward misalignment, while outcome supervision fails for complex tasks needing coordinated intermediate steps.
We introduce **O**utcome **R**efining **P**rocess **S**upervision, which unifies process and outcome supervision by leveraging executable verification: a tree-structured search framework generates strategic alternatives, profiles execution metrics, and scores candidates via self-critique mechanisms that integrate runtime feedback with reasoning.
Experiments across 5 models and 3 benchmarks show consistent gains, with **26.9%** higher correctness and **42.2%** improved code efficiency. 
The results demonstrate that ORPS enables LLMs to overcome local optima in code generation, suggesting a promising direction for combining verifiable outcomes with structured reasoning to tackle complex challenges.

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Zhuohao Yu"
Weizheng Gu Yidong Wang'
Xingru Jiang'
Zhengran Zeng'
Jindong Wang?
Wei Ye
Shikun Zhang'
Abstract
Outcome Supervision
Optimize generated supervising outcomes
Large Language Models excel at code genera- tion yet struggle with complex programming tasks that demand sophisticated reasoning. To bridge this gap, traditional process supervision relies on learned reward models requiring costly train- ing data and suffering from reward misalignment, while outcome supervision fails for complex tasks needing coordinated intermediate steps. We in- troduce Outcome Refining Process Supervision. which unifies process and outcome supervision by leveraging executable verification: a tree- structured search framework generates strategic alternatives, profiles execution metrics, and scores candidates via self-critique mechanisms that in- tegrate runtime feedback with reasoning. Exper- iments across 5 models and benchmarks show consistent gains, with 26.9 % higher correctness and 42.2% % improved code efficiency. The results demonstrate that ORPS enables I.LMs to over- come local optima in code generation, suggest- ing a promising direction for combining verifi- able outcomes with structured reasoning to tackle complex challenges. We open- at: https: / //github. com/ zhw.hhoyoooooRR
Generated Code
</> FAIL
Outcomes
SFT
PASS
</>
Annotaeed data
Policy Model
Code Excoutor
Process Supervision
Step t: Frine numbors do not have other flctors,. [Stap Beore:4) 4]
Reasoning Steps
Step 2: Sinceve we group each number by its prime
twches [Step Sicone: 3)
Policy Modial
Step & Now wo can fry mans each node by coemcee than Searc the
[Step Score: 6)
SFT
Procese Rewards
Jutignn Rensooing Procusa
Optimina gener generatied code by supervising reasoning process
Rawarrd Labels
Outcome Outcome-Refining
Refining Process Supervision (Ours)
Reesoning Steps
No trsining Aeeruree
Solution
Step 2: The fu function previousty talll generated
test case #2, as the Union- Find not cornecly handling ergiing subsets- New Attempt def man_ pure)ts((re) Thinking on Outcomes, Now: time complexity is City 22 dus therested thiis not ellizint.. Step
Bel nevemmaemmmmee Guocare
Policy
Model
Exemoe Braw From Search
Outcome Rellection
Process Rewards
Optimize generated code by supervsiig reasoning and outcomes
Signlicndly Improve Compler Coding A
Figure 1: Comparison of outcome and process supervision.
even in state-of-ther models (OpenAI, 2023; Touvron et al., 2023a), revealing critical gap in current supervision paradigms.
As shown in Figure traditional approaches follow two main paradigms: outcome supervision, which evaluates only final outcome quality (Chen al., 2021b), and pro- supervision, which guides intermediate steps using learned Process Reward Models (PRMs) with search algo- rithms (Lightman et al., 2023). While PRMs have shown success in mathematical reasoning (Wang et al., 2024c: Chen et al., 2024a), their application to code generation faces fundamental challenges: (1) PRMs require expensive human annotations distillations of other models on in- termediate steps to train (Wang al., 2024b); (2) Learned rewards can suffer from hallucination (misjudging invalid steps as correct) (Huang et al., 2023; Stechly et al., 2024) and reward hacking (exploiting superficial patterns to maxi- mize scores) (Skalse et al., 2022); (3) Code-specific PRMs are scarce, and math-focused PRMs may not suit program-
1. Introduction
Large Language Models (LLMs) have revolutionized code generation through their ability synthesize programs from natural language specifications (Brown et al., 2020; Guo et al., 2024). However, complex programming tasks requir- ing multi-step algorithmic reasoning- such as implement- ing dynamic programming solutions or optimizing paral- lel computation patterns-rent remainn challenging (Jiang al., 2024b; Jimenez al., 2023). These limitations persist
'Peking University, Beijing, China "William & Mary, VA, USA. Correspondence to: Wei Ye <wye& oku.edu.cn>.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
ming's structured logic. More fundamentally, our work questions the necessity of separately trained PRMs when LLMs' intrinsic reasoning capabilities can be effectively guided by verifiable outcomes.
efficiency by exploring superior algorithmic strategies rather than just performing local fixes.
Our key contributions include:
Code generation offers a unique advantage through concrete, verifiable signals: code can be executed throughout develop- ment, providing objective feedback on correctness and per- formance (Zhang al., 2023; Shinn al., 2024). However, existing -eeeubion- methods like Reflexion (Shinn et al., 2024), LDB (Zhong et al., 2024), Self- Repair (Olaus- son et al., 2023), and REx (Tang et al., 2024a) primarily focus on local code repair, missing opportunities to explore fundamentally different algorithmic strategies.
We propose ORPS: a novel inference-only framework that unifies outcome and process supervision by guiding LLMs to reason about solution strategies at a higher abstraction level, moving beyond local code repair.
We demonstrate that combining verifiable execution feed- back with self-critique effectively eliminates the need for specially trained PRMs, outperforming them while reducing overhead.
We propose Outtiie-RRfining Process Supervision. which treats the reasoning about outcome refinement itself as the process to be supervised. Unlike repair-fouuse meth- ods that incrementally fix code blocks, ORPS operates at higher abstraction level, guiding LLMs to reason about over- all solution strategies. Through tree-structured exploration, our framework maintains multiple reasoning trajectories simultaneously, enabling models to explore different algo- rithmic approaches when initial attempts prove suboptimal (e.g., switching from brute-force to wwvide-and-oongeee ap- proaches), rather than being trapped in local optima (e.g.. brute-force solution that passes the test but is inefficient).
ORPS achieves substamtial improvements across diverse benchmarks: an average @ass@1 improvement of 26.9% across datasets and models, while reducing running time by 42.2% on average.
2. Related Work
2.1. Outcome Supervision vs Process Supervision
Outcome Supervision in language models traditionally evaluates and optimizes LLM outputs through three primary paradigms: (1) Open-ended generation tasks (e.g. instruc- tion following) assessed via text similarity metrics with reference answers (Zhang et al., 2019) or LLa--u--- scoring (Zheng et al., 2023; Wang et al., 2023c; Yu et al., 2024a); (2) Constrained-ouutpul tasks (multiple-choice, math problems) judged by exact answer matching (Brown et al., 2020; Hendrycks et al., 2020) of answer keys or solutions; and (3) Code generation where correctness depends on test case execution with generated programs (Chen et al., 2021b; Austin et al., 2021). These outcomes are then used to cu- rate the data (e.g., selecting higher-quality conversations or code solutions) for further training of the model (Liu et al., 2023b). Such approaches share critical limitation: they ignore the reasoning process that produced the output (Ue- sato et al., 2022). This proves particularly problematic for complex tasks where optimal solutions require coordinated intermediate steps (Lightman et al., 2023).
Our key insight is that LLMs' intrinsic reasoning and self- critique capabilities, when anchored by verifiable execution feedback, can generate high-quality process rewards, elimi- nating the need for specially trained PRMs. Execution out- comes and performance metrics provide objective ground- ing, while self-critique offers strategic guidance. This cre- ates a powerful, inference-only feedback loop where reason- ing chains and code implementations co-evolve: execution failures prompt strategic re-analysis, while refined algorith- mic insights lead to better implementations.
ORPS is the first inference-only process supervision frame- work for code generation that systematically explores al- gorithmic strategies without PRM training. Experiments across 5 models and 3 benchmarks reveal:
Strategic Reasoning Over Model Scale: Providing suffi- cient reasoning space for algorithmic exploration is more crucial than model size; ORPS enables smaller models to achieve high success rates, often outperforming larger counterparts using conventional methods.
Process Supervision addresses this gap by optimizing in- lermediate reasoning trajectories through specially trained Process Reward Models (PRMs) that score each intermedi- ate step (Uesato et al., 2022: Lightmann et al., 2023). PRMs have proven particularly effective in domains requiring com- plex reasoning, such as math problems, where they guide search or sampling algorithms toward better reasoning tra- jectories and solutions (Wang et al., 2024c; Chen et al., 2024a; Wang et al., 2024b). While outcome supervision gives Routcome (y) 1 [correct), process supervision gives step-wise reward signals Rpmocess = ΣE
Combining execution feed- back with self-critique creates superior verification mech- anism compared to learned PRMs, eliminating costly training while leveraging LLMs' inherent capabilities.
Effective PRM Elimination:
Algorithmic Improvement Beyond Local Optima: ORPS consistently improves correctness and solution
2

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Termina Trensition
Non- Terminal Trarettion
State Expansion
Previous States
Legends St.
I-th state 1-th layor search
Critic Thoughts
Pruvious Rezsoning Chain
has SCD preater than Heaever, there are fem texses and fntrovements that can be rade:
You are given an array of int which needs be split into subsets such that the greatest common divisor within each subset is greater than 1. Retum the minimum number of subsets required to create such a split.
Problem:
Step Reward: &
Steps with best scores are extended in the tree.
Candidate Generation Potentiil Stotes For Next: Facesoning Stop
Reasoning Tree
Self-Critic & Process Rewarding
LLM
S1.3
S1e
Thoughts
The prozlom requires aplitting such algeritts,
Motrics 1
S2.1
S2.2
$2.3
S2.4
Sa.5
S2.8
azray and and eech suases their ocD is greater stan ... su Atseept
existing
Static Analysis:
Overall Scare: B.74
Code Length: cogulitiio Coplicitty
det mtn_g.c. (xceens
$
Peased Texts:
Sa1
Sa.2
Ss.3
Sa.4
Sa.5
Sa.8
Continue Sealching with new states
Execution & Profiling
Menary muge 30.204
Terminal States
Figure 2: nutcome-Refining Process Supervision framework overview. A language model serves as both programmer and critic in step-by-step reasoning process. Through beam search, the framework maintains multiple solution trajectories, where each state contains reasoning chains, code implementations, and step reward.
This approach has been predominantly used in math rea- soning tasks(Luo et al., 2024; Jiang et al., 2024a), but with limitations: 1) The requirement for dense human annota- tions to train reliable PRMs makes the approach expensive and consuming (Lightman et al., 2023). 2) The gener- alization capability of PRMs is often limited, as reasoning patterns can vary significantly across different tasks and domains. 3) When serving judges, LLMs may produce unreliable evaluations due to hallucination (Hu et al., 2024; Li al., 2024). particularly for complex tasks (Thakur et al., 2024). Recent studies show that LLMs cannot reliably self- correct (Huang et al., 2023) or self-validate without external verification (Stechly et al., 2024). However, emerging work suggests that specially trained PRMs may be unnecessary when LLMs' intrinsic reasoning capabilities are effectively guided by verifiable outcomes (Chakraborty et al., 2025). These limitations motivate our approach of grounding pro- cess supervision in concrete, verifiable signals rather than learned judgments.
Repf-R (Olausson et al., 2023) demonstrated that rea- soning quality and self-repair capabilities correlate with code generation outcomes, establishing the importance of structured reasoning through repair trees. REx (Tang et al., 2024a) further explored code repair as an exploration- exploitation tradeoff. However, these execution-guided ap- proaches primarily focus on local code debuggin specific functions fixing syntax errors-missing opportu- nities to explore fundamentally different strategies.
Although these execution- guided approaches show promise, our experiments indicate they are insufficient for complex programming tasks that require deeper reasoning. While execution feedback is easy to measure, it alone provides little guidance how to improve solutions that fail or how to make working solutions more efficient. More importantly, it offers no feedback during the intermediate stages of de- velopment, when early course corrections could prevent cascading errors.
Consider implementing an efficient sorting algorithm: a model might write code that passes all test cases but uses O(n²) approach. Existing outcome supervision meth- ods would mark this as success, missing the opportunity to guide the model toward an optimal O(n log n) solution. Similarly, if the code fails, a sparse "fail" signal provides no insight into whether the error lies in the algorithmic ap- proach, the implementation details, or edge case handling. These limitations highlight the need rethink how to su- pervise the development of complex programs, where both theoretical understanding and practical implementation must evolve together- at higher abstraction level than typical recair-focused methods.
2.2. Execution-Driven Code Generation
Code generation is typically formulated as a sequence-to- sequence problem: given input specification x (including natural language description and test cases), generate a pro- gram y that correctly implements the required functional- ity (Jiang et al., 2024b). While most existing approaches treat this as single-step generation process (Chen et al., 2021a), recent work has explored using execution feedback to guide code generation (Zhong et al., 2024; Zhang et al., 2023) or use CoT prompting to improve correctness (Shinn al., 2024).
3

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
3. Methodology
Algorithm Outcome- Refining Process Supervision
Problem at Unit Tests U. Model M. Beam size K, Steps T, Candidates N. Process Reward Weight Outcome Reward Weight / Initiaiize beamg t- {(x, 0) (Start with problem description)
Input:
We propose a framework that unifies process supervision with outcome supervision by combing reasoning, code im- plementation, and execution verification into single tree- structured search process.
step I to do
paths + 0 (Initialize reasoning paths} state beam,-1
chain t- s.raasening.cht chain {Copy current reasoning chain) candf dates e- Mesu (z, chatn, N) (Generate N pairs)
When tackling coding tasks, particularly complex ones, it is challenging for a model to generate fully correct so- lution on the first attempt. Instead, LLMs often produce imperfect yet heuristically valuable code, requiring iterative self-correction to eventually arrive a correct implementa- tion. Each iteration of code refinement can be considered step in the problem-solving process.
for candidate candidates do
reasoning code <- Extract from cand newchain +- chain @ reasoning code feecback, outcome.rew +- peccue..nd.protllccoo U) newchain +- newchsin @ feedback @ out zex crit, process_res < Mate (newcha n, code, feedback) newchain 4- newchain @ crit @ process sew step.acore <- procesa.rew f out.ocmo.rey paths +- patha U ([newchaing step.atee.... score)] end for end for baam, <- Select Top paths highest st ap.ccer if Any path in neam, is complete then
As formalized in Algorithm 1 and illustrated in Figure 2, at step 1, the node in a given search beam represents a state 8, = (R,, C22 Ft, Wt, Ke, Pt). where R2 denotes the current reasoning chain, Ce the code implementation, F; the execution feedback, w/ the outcome reward score, K₁ the self-critic reasoning and pr the process reward score.
break end if end for
Return: reasoning chain from beamy
The search progresses through three phases:
ensures algorithmic
competing ap-
proaches until empirical feedback resolves ambiguities.
1. Candidate Generation. The LLM is leveraged to do rea- soning how to refine or alternative strategies and then attempt to generate corresponding code implementations.
3.2. Execution & Profiling Outcome Rewards
We execute the model-generated code (ct)) y predefined Unit tests U and execute them to obtain results. U may be either generated by the LLM itself based on the problem or provided by the datasct. We will discuss the results under different scenarios in subsection 4.1.
(c))),} on a set of
2. Execution & Profiling. Each candidate code is executed and profiled on unit tests, to measure key performance metrics, such as correctness, efficiency, and code quality.
3. Self-Critic & Process Rewarding, The LLM is prompted to generate a texual self-critic considering rea- soning chain and execution metrics, then gives a numeri- cal process reward score which used to guide searching.
The outcome reward w²(") combines dynamic analysis (Ball, 1999) and static analysis (Nielson et al., 1999) metrics into unified score, grounding solution quality in both runtime behavior and structural properties. We compute:
Unlike linear CoT approaches that commit to a single tra- jectory, this tree structure enables parallel exploration of divergent strategies instance, maintaining both greedy and dynamic programming approaches for optimization problems until empirical feedback identifies the superior solution.
Bx normalize(m (2),
We have correctness, execution time, CPU instruction count, page faults as dynamic analysis metrics and code length, AST node count, cyclomatic complexity and cognitive com- plexity as static analysis metries. Due to space limitations, put detailed introduction to each metric in Table 6.
3.1. Candidate Generation
The candidate generation phase expands the search tree by producing diverse refinements. For each node $t-1 = (RR-1, Fr-1, w1-1, 1t-1, the L.M generates
This framework rewards solutions that balance functional correctness with comprehensive aspects-a brute-force im- plementation passing all tests would score highly in cor- rectness but poorly in complexity metrics, incentivizing refinement toward optimal algorithms.
N successor candidates:
M(RL-I, ( Fi-1),
where 7!!) represents incremental reasoning updates (c.g.,
3.3. Self-Critic & Process Rewarding
'Adjust loop termination to prevent off-by-one errors') or
strategic pivots (e.g.. 'Replace recursion with iteration to avoid stack overflow"). This generatne mechanism
The same model M serves dual roles: generating reasoning along with solution candidates, and judging their viability.
4

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
After executing of to obtain p.D), the model produces a texual critique k,)) and a numerical process reward p
dataset manually curated by human experts with competi- tive programming experience. HumanEval and MBPP are popular code generation benchmarks but could be trivial for current LLMs (Matton et al.. 2024). Moreover, a significant proportion of the data is leaked in multiple pre-training cor- pora (Riddell et al., 2024). To ensure reproducibility, we report our detailed hyperparameters in Appendix A, we also open-source all our code and scripts.
(w0) ((0) M(Re- r C 3 00 p(D).
The process reward grounds subjective evaluation of the reasoning process by combining objective execution met- rics. This hybrid scoring prevents reward hacking- model cannot inflate rewards without corresponding improvements in verifiable exccution outcomes. Candidates with high but low w signal outcome- processs mismatches, which is a critical failure mode conventional process supervision.
Unit Tests. Our framework utilizes unit tests to verify and profile code solutions. We use LLM to generate these unit tests, which are then employed in computing the outcome re- ward w!!") However, self-generated unit tests do not always effectively assess code quality. Consequently, some prior works (Zhong et al., 2024) directly utilize pataset-provider unit tests. To facilitate a fair comparison with related ap- proaches, we also consider using unit tests provided by datasets. In Table 2, methods using unit tests from datasets are denoted by (w/ (w/T).
3.4. Unifying Process and Outcome Supervision
The beam search mechanism proceeds with Top-K succes- sor states using a weighted step score: gt = ap; + Sui.. Where a + B = 1 governs the trade-off between theoret- ical soundness (pt) and empirical effectiveness (wy). The framework balances functional correctness while preserving promising reasoning trajectories.
Baselines. We compare several strong baselines for code generation. For outcome supervision, Reflexion (Shinn et al., 2024) is a recent self-improvement approach that utilizes execution results to refine generated code. LDB (Zhong et al., 2024) extends this by incorporating de- bugger outputs, and intermediate variable values for iterative solution refinement. For test-time scaling, we implement Best-of-N sampling, which generates multiple solutions and selects the best one based on test outcomes. Since exist- ing process supervision methods have been designed specifi- cally for code generation, we adapt a similar approach from mathematical reasoning (Luo et al., 2024) in comparison, which we include our ablation studies.
This formulation generalizes conventional supervision paradigms. When 3 = O, the framework reduces to pure process supervision akin to mathematical reasoning ap- proaches that prioritize stepwise correctness over answers during reasoning. Conversely, a- 0 recovers outcome supervision's 's focus final code quality, similar to Best-of- N sampling but on tree. Our unified perspective reveals these as endpoints on a continuum of supervision strate- gies, with the proposed a/ / balance enabling simultaneous optimization of reasoning and outcome solution quality.
The synergy emerges through bidirectional feedback: exe- cution outcomes ground reasoning process by identifying discrepancies between intended and actual behavior (c.g., test failures revealing flawed base cases in recursive algo- rithms), while process rewards guide exploration toward algorithmically superior implementations (e.g., recogniz- ing that memoization could transform an O(2") brute-force solution into an (n) dynamic programming approach).
4.2. Main Results
Table 2 shows the comparative results of our method and baselines, Figure provides detailed multi-dimensional profiling of the performance of generated solutions with different methods.
4. Experiments
Table 1: Dataset Statistics. Characteristics of the program- ming benchmarks used in evaluation.
Our experimental evaluation aims to address three key ques-
tions: (1) How effective is our framework compared to
existing approaches? (2) How does each component of
LBPP (Mation all, 2014)
HvaanEval (Chan ei al. 2021b9
MBPP (Auelin etall 2021)
our framework contribute to the overall performance? (3) What insights can we gain about the relationship between reasoning quality and code generation?
# Test Prohlems # Unit Tests
162 5.1
164 6.5 169 / 622
257"
3,0
Solution Length
627 / 627/3039
/
Contamination
New Dutaset
18.9%*
20.8%
Competitive Programming Algorithms
Basic
Basic
4.1. Experimental Setup
Difficulty
Functions
Functions
Task Type
Func. Completion
Basic Prog.
Datasets. We evaluate on 3 programming benchmarks as shown in Table 1. LBPP is recent complex programming
From canit t red vinions "Centmiiaation results reported from Riddell (2024): AAeragghmussmmm
ende.
5

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Table 2: Main Results on Code Generation Benchmarks. Pass @ I: solutions passing all test cases. Tests: average test cases passed. Valid: solutions that compile and execute. Time: relative execution time, compared to the standard solution. Best results are in bold and second-bes. are underlined, every metric in percentage.
LBPP (2024)
HumanEval (2021b)
MBPP (2021)
Model/Method
Pass@/1
Testsf
Valid
Time
@
Tests
Validf
Time
Pass@
Testst
Valid?
Time
Llama-3.1-8P Instruct
(2024)
CoT
30.9
44.3
63.0
176.8
50.0
68.4
82.9
98.1
58.0
64.9
72.4
91.9
Reflexion
34.0
49.3
67.3
148.5
54.9
83.5
107.5
58.8
65.0
71.2
88.6
LDB (w/T)
25.9
39.8
58.0
252.2
54.3
62.3
66.5
127.1
43.6
47.1
49.4
170.7
BoN
46.9
64.7
84.6
107.6
71.3
84.7
93.3
77.3
73.5
79.9
86,4
72.1
ORPS
4.59
66.9
88.5
99.1
70.3
87.5
96.2
65.8
71.8
78.2
84.3
84.5
ORPS (w/ T)
67.1
81.4
93.7
89.4
91.4
95.7
98.1
63.6
90.4
93.1
95.6
59.1
(2024)
CoT
32.7
45.9
67.3
160.1
65.9
78.2
85.4
86.9
69.3
75.0
80.9
77.7
Reflexion
25.9
41.9
63.0
153.0
63.4
77.1
86.6
101.0
68.9
74.4
80.2
74.2
LDB (w/ T)
31.5
45.7
61.7
206.2
74,4
80.0
81.7
85.6
61.1
64.0
66.1
98.3
BoN
49.4
63.9
80.2
123.4
73.8
88.1
94.5
64.1
74.3
80.2
86,8
68.9
ORPS
56.3
71.1
88.0
89.4
76.2
90.0
96.3
40.6
73.2
80.3
87.5
46.8
ORPS (w/ T)
80.8
74.4
95.7
98.0
99.4
93.0
96.1
(2024)
CoT
55.3
72.2
118.6
72.6
79.0
82.3
79.2
79.0
83.3
88.3
67.3
Reflexion
37.7
57.1
78.4
111.2
75.6
81.1
84.1
73.6
79.0
84.0
88.7
63.5
LDB (w/ T)
35.8
49.9
65.4
187.8
87.8
90.3
91.5
76.1
66.9
69.4
72.0
96.8
BoN
53.1
68.8
85.8
117.9
77.4
85.1
87.8
66.8
82.9
87.2
91.8
62.6
ORPS
75.7
84.1
79.9
91.6
96.3
76.7
82.4
88.3
68.0
ORPS (w/ T)
77.8
87.9
96.9
82,4
96.3
98.0
98.8
43.9
94.9
97.3
Owen=2.5-Cooder-44B---iee
(2024)
CoT
53.7
63.9
77.2
119.2
82.9
88.5
90.2
76.6
84.0
87.4
91.1
67.5
Reflexion
60.5
70.5
82.1
113.3
83.5
89.9
92.7
68.8
83.3
87.2
91.1
66.0
LDB (w/ T)
51.9
62.9
75.3
225.2
89.6
92.0
92.7
140.5
72.4
74.6
76.3
149.7
BoN
61.7
74.9
90.7
115.6
87.8
93.9
95.7
58.8
81.7
86.4
91.1
58.4
ORPS
77.4
84.8
81.7
96.3
76.3
87.9
58.8
ORPS (w/ T)
85.8
90.7
95.7
64.2
97.0
98.5
99.4
43.8
95.3
96.9
98.1
41.0
GPT-4o-Mini (2024)
CoT
50.0
65.9
80.2
124.5
79.9
87.5
90.9
80.5
78.6
83.5
87.9
70.3
Reflexion
62.3
73.9
87.7
93.2
75.0
83.6
87.2
75.1
79.4
84.0
88.3
67.6
LDB (w/
54.9
67.8
82.7
220.1
88.4
92.2
93.9
133.4
72.8
75.5
77.8
157.9
BoN
78.6
88.9
82.9
92.7
80.5
85.5
89.9
ORPS
67.9
81.2
94.4
81.5
84.8
92.7
96.3
57.5
80.2
86.0
91.8
64.7
ORPS (w/ T)
88.9
94.3
98.1
61.6
97.6
98.7
99.4
46.2
95.7
97.3
98.4
51.4
Our results indicate significant improvements in both cor- rectness and code quality metrics, especially harder benchmarks. Even smaller model (Qwen 7B), when paired with our method, could surpass its larger variant (Qwen 14B) without our method, suggesting that providing sufficient reasoning space can be more effective than solety scaling model parameters which is significantly more computa- tionally expensive. This finding has important implications for practical applications where computational resources are limited.
fundamental difference in approach: while these outcome- based methods focus primarily local information like resolving execution errors and reasoning in chain struc- ture, our method provides LLMs with broader reasoning space to reflect on higher-level aspects such as algorithm selection and problem properties by using process reward guided search. For instance, LDB achieves 35.8% Pass@ LBPP with Qwen-7B with test case access, while method reaches 77.8% under the same conditions.
Particularly noteworthy is the performance boost when mod- els have access to gold unit tests from test datasets (with- out access to solutions). All models show drastic improve- ments on all metrics in this setting. For instance, Qwen-7B achieves 77.8% Pass @ on LBPP and 96.3% on HumanEval
When compared to other execution-feedhact and outcome reward based methods like Reflexion and LDB, our ap- proach consistently demonstrates superior performance re- gardless of case access. This improvement stems from
6

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Code Length
Table 4: Analysis of Process Reward Model. Granularity
1.50x
BcN Reftexion
refers to the level of detail in the reward signal (line-level or outcome-level). Train indicates whether the process reward model requires training.
Page Faults
AST Nodes
CCT
dand Solutton
Methods
3,50x
Pass
Testst
Valid
Time
Granularity
Train
Outcome
37.0
48.3
65.4
153.8
Branch
Cyclomatic Complexity
Line
/
32.1
43.9
59.3
153.4
Mispredictions
Outcome
X
59.9
75.7
92.6
89.1
Line
X
38.3
52.8
70.4
123.7
1. Execution Outcomes. Studies whether the model can ac- cess execution results to guide solution refinement. With- out execution feedback, the model must rely solely on internal reasoning assess correctness.
CPU
Cognitive Complexity
Instructions
Execution Speed
2. Reasoning, Investigates whether the model should ex- plicitly perform reasoning before generating new code at each step. Without reasoning, the model directly gen- erates new code based only past code and execution results.
Figure 3: Multi-dimennional Performance Analysis. Met- rics are normalized against the LBPP standard solutions (1.0x) and averaged across all backbone models. Higher values indicate better performance.
Table 3: Ablation Study Results. Execution: Remove execution from our framework. Reasoning: Remove rea- soning process. Every metric is in percentage.
The results are presented in Table 3. When the model is unable to access execution outcomes during searching, Pass@1 decreases by 16.1% This highlights the critical role of environment fexironment in guiding the model to gen- erate correct solutions. Since LLMs struggle to accurately predict execution outcomes for given piece of code (Jain et al., 2024), incorporating execution results ensures that the model benefits from concrete feedback.
Method
Pass@1 1
Tests
Valid
Time
ORPS
59.9
75.7
92.0
84.1
Execution
43.8
56.4
72.8
200.5
Reasoning
55.6
74.5
94.4
124.5
Omitting reasoning during searching results in 4.3% de- crease in Pass@1 1. Reasoning enables the model to itera- tively refine its approuch based on feedback, addressing issues that may not be resolved through execution feedback.
with test case access, compared to 59.9% and 79.9% with- out. This suggests that while our self-generated test cases may be relatively weak, given feedback for higher quality test cases, models can effectively guide themselves through the reasoning process generate significantly better code.
4.4. Analysis of Process Reward Model
Figure 3 further supports these findings through detailed profiling results, showing consistent improvements over baselines across all models terms of code efficiency and quality metrics. Guided by our framework, models are capa- ble of refining themselves generate faster, more coherent code. However, we do observe slight disadvantage on MBPP, particularly when comparing with Best-of-N sam- pling. This is less concerning given that, as shown in Table 1, MBPP consists of relatively simple problems with short so- lutions, and a significant portion (20.8%) of its test data already exists in publicly available pre- training datasets.
Our framework challenges the necessity of training Process Reward Models (PRMs) by combining outcome rewards with reasoning work process rewards. While most prior work process supervision (Luo al., 2024) assumes that specially trained PRMs are required to guide reasoning dur- ing inference, we demonstrate that execution feedback as verifiable rewards, combined with existing L.MMs' reason- ing capabilities, can generate high-quality process rewards. This motivates us to investigate a fundamental question: Is training PRMs necessary for effective code generation, or can outcomes alone provide sufficient supervision signals?
Experimental Design. Table 4 presents a comparison across two dimensions: granularity of supervision (line- level vs. outcome-level) and training approach (trained vs. inference-only). For line-level methods, the model gener- ates step-by-step reasoning with numerical rewards assigned
4.3. Ablation Study
We conducted experiments on the challenging LBPP dataset using the Qwen-70 7B model to investigate the importance of two key components in the exploration process:

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
each step, following mathematical reasoning approaches. For outcome-level methods, outcome rewards are computed after complete code generation and execution.
Table 5: Computational Cost Analysis. Pass perfor-
mance on LBPP with Qwen-7B controlling LLM calls.
Method
20 Calls
50 Calls
100 Calls
To determine whether the popular method of training line- level PRMs with higher-quality data can more effectively bridge the performance gap with our inference-only ap- proach, we trained two PRMs with different data quality: PRM-GPT using a larger GPT-4o labeled dataset (containing 13644 synthetic labels for each line of reasoning text), and PRM-Human using human- human-filtered data where three authors spent 12 hours each to validate reasoning steps, achieving an inter-annotator agreement of 0.44 (Cohen's Kappa) and extracting 836 high-quality steps for training. This exper- iment thus directly tests if providing PRMs with optimal, high-quality training data enables them match or surpass the performance of our method.
Refiexion (2024)
37.0
40.7
39.5
LDB (2024)
37.0
36.4
37.0
REx (2024a)
43.2
53.7
54.3
PRM-GPT
44.4
37.0
35.8
PRM-Human
40.7
38.3
42.0
ORPS (Ours)
48.4
55.6
64.2
Results. Table 5 presents results on LBPP with Qwen-7B under controlled computational budgets. ORPS consistently outperforms all baselines across different call limits, demon- strating superior computational efficiency. Notably, repair- focused methods like REx (Tang al., 2024a) show dimin- ishing returns compute increases, while ORPS scales effectively. These results confirm that ORPS achieves better performance through more effective utilization of computa- tional resources, not brute-force computation.
Results and Analysis. insights. First, outcome-level reward signals prove more effective than line-level signals, as line-level feedback can only evaluate incomplete thought processes with limited information. Second, and more importantly, our inference- only approach substantially outperforms all trained PRMs given the same calls, even when PRMs are trained with high-quality human annotations.
The results confirm several key
4.6. Scaling Analysis
While the previous analysis controlled LLM call budgets, we now examine performance scaling given the same num- ber of generated code solutions. We compare ORPS against Best-of-N (BoN) sampling by varying candidate solutions on LBPP. The results in Figure 4 reveal distinct scaling behaviors: ORPS demonstrates superior scaling potential, with performance improving rapidly as more candidates become available. This suggests that the structured reason- ing in our framework more effectively identifies and refines promising solutions from the candidate pool. In contrast, BoN exhibits slower improvements, indicating that simple selection from larger pools provides diminishing returns without strategic guidance.
When combined with the computational cost analysis (Ta- ble 5), these results indicate that while training data quality does impact PRM performance, trained PRMs still under- perform our hybrid approach of verifiable execution rewards combined with LLM self-critique. This suggests that out- comes alone, when properly integrated with LLM reasoning capabilities, provide superior supervision signals compared to learned reward models. The effectiveness stems from grounding supervision in concrete, verifiable execution feed- back rather than learned approximations.
4.5. Computational Cost Analysis
A critical concern raised by reviewers is whether ORPS's improvements come at the cost of increased computational overhead. To address this, we conduct controlled experi- ments limiting the total number of LLM calls, which is the bottleneck of inference-only methods, ensuring fair compar- ison under identical computational budgets.
0.6
Pass@1
0.5
Qwen7B-Curs
0.4
Qwen7BBBBo BoN
Complexity Analysis. We analyzed the number of LLM calls by examining the source code of baseline methods. ORPS requires 2 x x (K x T + 1) calls, where N is candidate samples, K beam size, and T is reasoning steps. Reflexion (Shinn et al., 2024) requires 2 x T calls, while LDB (Zhong et al., 2024) requires 1 + P x T (2 B N) calls, where P is pass at k, B is code blocks, and N is max trials. For trained PRMs, we replace self-critic LLM calls with PRM calls, maintaining identical call counts.
DeepSeek7B-Ours
DeepSeek78-Bok
0.3
5
15
20
Inference Budget
Figure 4: Performance vs. Inference Budget. The y-axis represents Pass@1 scores on LBPP. The x-axis represents the number of candidates generated during inference.
8

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
5. Conclusion
40
COT Baseline
Solutions
Our improvement
We introduce Mutcome-Refining Process Supervision (ORPS), a unified framework that fundamentally challenges the necessity of specially trained Process Reward Models in code generation. By combining execution-driver feedback with LLMs' intrinsic reasoning capabilities, ORPS demon- strates that verifiable outcomes can effectively guide process supervision without costly reward model training. Our com- prehensive analysis across different data qualities from GPT-4 labels to human aanotations-showe that even high- quality trained PRMs underperform our inference-only ap- proach, suggesting paradigm shift in how approach process supervision for complex reasoning tasks.
Unsolved
20
0
list math string graph
dietionary bit pulationn backtrackip
dynamic programmin
kina
loop tree
numpy
date binary tree hashing linked flst regex
Figure 5: Performance by Problem Class. Top-20 problem classes in LBPP showing success rates and unsolved cases for ORPS vs baseline.
ORPS reveals several key insights: (1) Strategic reasoning over local repair: Unlike methods that incrementally fix code blocks, ORPS operates at a higher abstraction level, enabling exploration of fundamentally different algorithmic approaches rather than being trapped in local optima. (2) Reasoning space over model scale: Smaller models with ORPS often outperform larger variants using conventional methods, highlighting that structured reasoning frameworks can be more effective than pure parameter scaling. (3) Com- putational efficiency: ORPS achieves superior performance through strategic resource utilization rather than brute-force computation, scaling effectively while baselines show di- minishing returns.
4.7. Case Studies
We also analyzed the improvements of ORPS across dif- ferent problem categories. As shown in Figure 5, on the competitive programming dataset LBPP, our method shows significant improvements over the CoT Baseline, especially in more difficult categories. For instance, in complex al- gorithmic tasks such dynamic programming, loops, and graphs, our method correctly solves nearly twice as many problems as COT. This further confirms that high-quality intrinsic reasoning can help models avoid logical pitfalls when tackling difficult coding tasks.
These findings collectively suggest that LLMs' intrinsic capabilities, when properly guided by verifiable signals, can eliminate the need for costly reward model training. Future work could explore extending this framework to other domains requiring rigorous reasoning and verification, opening new avenues for developing more efficient and practical reasoning frameworks in large language models.
Through detailed case studies, we demonstrate how our framework enhances code generation by improving reason- ing. As shown in Appendix F, the response generated by the traditional CoT method for the Minimum Greatest Com- mon Divisor problem in LBPP demonstrates that while the model provides a detailed thought process during solution generation, the complexity of the task results in an imper- feet code implementation. For instance, in CoT's approach, the reliance nested loops and pairwise GCD calculations introduces inefficiencies and fails to address scalability for larger datasets. Similarly, our method's 's initial implemen- tation demonstrates a lack of robustness in handling edge cases and unnecessary redundancies in subset formation.
Impact Statement
This work advances code generation capabilities in large lan- guage models through more efficient reasoning processes. While our primary focus is methodological improving algorithmic problem-solving without costly reward mod- els els-we we acknowledge broader implications common to code generation systems. Enhanced programming assistants could democratize software development but may also lower barriers for generating malicious code. The framework's s emphasis on code efficiency could reduce computational overhead in generated programs, though its environmental impact depends deployment contexts.
However, ORPS achieves more accurate solution through finer reasoning. The code initially generated by our model contains redundancies and erroneous logic. Nevertheless, with the feedback from the critic on the execution outcomes, the programmer successfully refines the code to reach a correct implementation. This iterative process not only elim- inates logical errors but also optimizes performance, demon- strating the advantage of integrating structured feedback into code generation.
These considerations reflect well-established societal trade- offs in Al Al-powered code tools rather than novel risks intro- duced by our approach. We encourage responsible deploy- ment with standard safeguards against misuse, consistent with ethical practices for generative AI systems.

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
References
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman,
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E.,, Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021b.
Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05/226 2023.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang. E., Cai, C., Terry, M., Le. Q.. et al. Program synthesis with large language models. arXiv preprint XxXiv:2108.07732, 2021.
Chen, Z., Zhao, Z.,, Zhu, Z., Zhang, R., Li, X., Raj. B., and Yao, H. Autoprm: Automating procedural supervi- sion for multi-step reasoning via controllable question decomposition. arXiv preprint arXiv:2402.11452. 2024c.
Ball, T. The concept of dynamic analysis. ACM SIGSOFT Software Engineering Notes, 24(6):216-234, 1999.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L.., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open- spen-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuaa Imsys. org (accessed 14 April 2023), 2023.
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, 7., Yu, T., Chung, W., et al. A multi- task, multilingual, multimodal evaluation of chatgpt on reasoning, ballucination, and interactivity. arXiv preprint arXiv:2302.04022 2023.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:/802. 003505557 2018
Bao, G., Zhang, H., Wang, C., Yang, L., and Zhang, Y. How likely do Ilms with cot mimic human reasoning? arXiv preprint ArXiv:2402. 16048, 2024.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1416. 2021.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Dai, N., Wu, Z.,, Zheng, R., Wei, Z., Shi, W., Jin, X., Liu, G., Dun, C., Huang, L., and Yan, L. Process supervision- guided policy optimization for code generation. arXiv preprint arXiv:2410. 01.410.076611/ /7621, 2024.
Campbell, G. A. Cognitive complexity-a new way of mea- suring understandabillty. SonarSource SA, pp. 10, 2018.
Chakraborty, S., Pourreza, M., Sun, R., Song, Y., Scherrer, N., Huang, F., Bedi, A. S., Beirami, A., Gu, J., Palangi, H., et al. Review, refine, repeat: Understanding itera- tive decoding of ai agents with dynamic evaluation and selection. arXiv preprint arXiv:2504.01931, 2025.
Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., and Smith, N. Fine-tuning pretrained language mod- els: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06330 2020.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
Chang, Y., Wang, X., Wang, J., Wu, Y., Zhu, K., Chen, H., Yang. L., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models, arXiv preprint arXiv:2307.03109 2023.
A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., al. The llama 3 herd of models. arXiv preprint 2024.
urXiv:2407.22.
Evans, J. S. B.
In two minds: dual-process accounts of reasoning. Trends cognitive sciences, 7(10):454 2003.
Chen, C.-C., Huang, H.-H., and Chen, H.-H. Evaluating the rationales of amateur investors. Proceedings of the Web Conference 2021, pp. 3987-3998, 2021a.
Freitag, M. and Al-Onaizan, Y. Beam search strate- gies for neural machine translation. arXiv preprint 170.v:170..01800 2017.
Chen, G., Liao, M., Li, C., and Fan, K. Alphamath almost zero: process supervision without process. arXiv preprint 2.Xiv24405.03553 2024a.
Fu, J., Ng. S.-K., Jiang, Z., and Liu, P. Gptscore: Evaluate you desire. arXiv preprint 200. 2023.
Chen, G. H., Chen, S., Liu, Z., Jiang, F., and Wang, B. Humans or 1lms as the judge? study on judgement biases. arXiv preprint arXiv:2402. 10669, 2024b.
Gehring, J., Zheng, K., Copet, J., Mella, V., Carbonneaux, Q., Cohen, T., and Synnaeve, G. Rlef: Grounding code
10

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Ilms in execution feedback with reinforcement learning. arXiv preprint arXiv:2410.02000 2024.
Hu, X., Gao, M., Hu, S., Zhang, Y., Chen, Y., Xu, T., and Wan, X. Are Ilm-based evaluators confusing nlg quality criteria? arXiv preprint arXiv:2402. 240.:.2200222255555. 2024.
Godbole, V., Dahl, G. E., Gilmer, J., Shallue, C. J., and Nado, Z. Deep learning tuning playbook, 2023. /oLLhtt:://ittee hitbbb.. com/ gaggg/ollllleeaaeeeeee Buuning_plabboe Version 1.0.
Huang, A., Block, A., Foster, D. J., Rohatgi, D., Zhang, C., Simchowitz, M., Ash, J. T., and Krishnamurthy, A. Self-improvement in language models: The sharpening mechanism, arXiv preprint arXiv:2412.01990 2024.
Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y. Deep learning, volume 1. MIT Press, 2016.
Huang, A., Block, A., Liu, Q.. Jiang, N., Krishnamurthy, A., and Foster, D. J. Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment. arXiv preprint arXiv:2503.21877 2025.
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang. Y., Duan, N., and Chen, W. Critic: Large language models can self- correct with tool-interactive critiquing. arXiv preprint arXivv2305..1138, 2023.
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, W., Song, X., and Zhou, Large language mod- els cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023.
Guo, D., Zhu, Q. Yang. D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deepseek-coder: When the large language model meets programming- the rise of code intelligence. arXiv preprint arXiv:2401.1419 141966 2024.
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J.. Yu, B., Lu, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. /2/86, 2024.
He, M., Shen, Y., Zhang, W. Tan, Z., and Lu, W. Ad- vancing process verification for large language mod- els via tree-based preference learning, arXiv preprint arXiv:2407.003:3 2024a.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21220 2024.
He, T., Li, H., Chen, J., Liu, R., Cao, Y., Liao, L., Zheng, Z., Chu, Z.., Liang, J., Liu, M., al. A survey on complex reasoning of large language models through the lens of self-evolution, 2025.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint arXiv:2403.0799 2024.
He, Y., Wang, H., Jiang, Z., Papangelis, A., and Zhao, H. Semi-supervised reward modeling via iterative self- training, arXiv preprint arXiv:2409.0690 2024b.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
Hendrycks, D., Burns, C.. Basart, S., Zou, A., Mazeika, M.. Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Jiang, J., Chen, Z., Min, Y., Chen, J., Cheng, X., Wang. J., Tang, Y., Sun, H., Deng, J.. Zhao, W. X., et al. Technical report: Enhancing Ilm reasoning with reward-guided tree search. arXiv preprint arXiv:2411. 2024a.
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C.,, Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint arxiv: 1470, 2020.
Jiang, J., Wang, F., Shen, J., Kim, S., and Kim, S. A survey on large language models for code generation. arXiv preprint arXiv:2406.00511 2024b.
Hinton, G. E., Osindero, S., and Teh, Y. W. A fast learning algorithm for deep belief nets. Neural Computation, 18: 1527-1554, 2006.
Jiang, J., Yan, Y., Liu, Y., Jin, Y., Peng, S., Zhang, M., Cai, X., Cao, Y., Gao, L., and Tang, Z. Logicpro: Improving complex logical reasoning via program -uidedd learning. arXiv preprint arXiv:2409.122 12929, 2024c.
Hirschman, L. and Gaizauskas, R. Natural language ques- tion answering: the view from here. natural language engineering, 7(4):275-300, 2001.
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In International Conference on Learn- ing Representations, 2021.
Jiao, F., Qin, C., Liu, Z., Chen, N. F., and Joty, S. Learning planning- Jlanningbbased reasoning by trajectories col- lection and process reward synthesizing. arXiv preprint arXiv:2402.0m 00658, 2024.

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Jimenez, C. E., Yang. J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-hench: Can language mod- els resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
Liu, Y., Singh, A., Freeman, C. D., Co-Reyes, J. D., and Liu, P. J. Improving large language model fine-tuning for solving math problems, arXiv preprint arXiv:2310.100477 2023b.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint Ntt::20001088361 2020.
Luo, L.,, Liu, Y., Liu, R.,, Phatale, S., Lara, H.,, Li, Y., Shu, L., Zhu, Y., Meng, L.,, Sun, J., et al. Improve mathemati- cal reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.0633
2024.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners, Ad- vances in neural information processing systems, 35: 22199-22213, 2022.
Ma, Q. Zhou, H.,, Liu, T., Yuan, J., Liu, P., You, Y., and Yang, H. Let's reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint arXiv:2310. 10080, 2023.
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C.,, Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: benchmark for ques- tion answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Matton, A., Sherborne, T., Aumiller, D., Tommasone, E., Alizadch, M., He, J., Ma, R., Voisin, M., Gilsenan- McMahon, E., and Gallé, M. On leakage of code generation evaluation datasets. arXiv preprint urXiv:2407.07565 2024.
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Sole Solo, T., et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Muñoz Barón, M., Wyrich, M., and Wagner, S. An empir- ical validation of cognitive complexity as measure of source code understandability. In Proceedings of the 14th ACM/IEEE international symposium on empirical soft- ware engineering and measurement (ESEM), pp. 1-12, 2020.
Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z.,
Nielson, F., Nielson, H. R., and Hankin, C. Principles of program analysis. 1999.
Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., al. From generation to judgment: Opportunities and challenges of judge. arXiv preprint arXiv:2411.16544 2024.
Olausson, T. X., Inala, J. P.. Wang, C., Gao, J., and Solar- Lezama, A. Is self-repair silver bullet for code genera- tion? arXiv preprinl arXiv:2306.08 09896, 2023.
Li, D., Cao, S., Cao, C., Li, X., Tan, S., Keutzer, K., Xing, J., Gonzalez, J. E., and Stoica, I. S*: Test time scaling for code generation. arXiv preprint a50iv:2502. 14382, 2025.
OpenAl. Gpt-4 technical report, 2023.
Li, J., Tang, T., Gong, Z., Yang, L., Yu, Z. Chen, Z., Wang, J., Zhao, W. X., and Wen, J. Eliteplm: an empirical study on general language ability evaluation of pretrained language models. arXiv preprint arXiv:2205.01523, 2022a.
OpenAL. ol system card. https: /ddn openai. com/ o1 wt--ssstte-crrdd.pp. September 2024. Ac- cessed: Dec 9, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Li, Y. An open source data contamination report for llama series models. arXiv preprint arXiv:2310.17588 2023.
Li, Y. Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eocles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. Competition--eeeel code generation with alpha- code. Science, (7922)622866622 1097, 2022b.
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruc- tion tuning with gpt-4. arXiv preprint arXiv:2304.03327,, 2023.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20050. 2023.
Peng, K., Nisbett, R. E., and Wong, N. Y. Validity problems comparing values across cultures and possible solutions. Psychological methods, 2(4):329, 1997.
Qin, Y., Li, X., Zou, H., Liu, Y., Xia, S., Huang, Z., Ye, Y., Yuan, W., Liu, H., Li, Y., et al. O1 replication jour- ney: A strategic progress report-pur 1. arXiv preprint arXiv:2410. 18982, 2024.
Liu, Y., Chen, L., Wang, J., Mei, Q., and Xie, X. Meta se- mantic template for evaluation of large language models. arXiv preprint arXiv:2310.01444 2023a.
12

---

and Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training, 2018.
Sun, C., Qiu, X., Xu, Y., and Huang, X. How to fune-tune bert for text classification? In Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kumming. China, October 18-20, 2019, Proceedings 18, pp. 194-206. Springer, 2019.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Per- formance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Swiechowski, M., Godlewski, K., Sawicki, B., and Mandziuk, J. Monte carlo tree search: A review of recent modifications and applications. Artificial Intelligence Review, 56(3) 2023.
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings the Inter- national Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-14, 2021.
Tang, H., Hu, K., Zhou, J., Zhong, S. C., Zheng, W.- L., Si, X., and Ellis, K. Code repair with Ilms gives an exploration-explootation tradeoff. Advances in Neu- ral Information Processing Systems, 17:117944-177996 2024a.
Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deep- speed: System optimizations enable training deep learn- ing models with over 100 billion parameters. In Proceed- ings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Tang, X., Zhang, F., Zhang, S., Liu, Y., He, B., He, B., Du, X., and Du, X. Enabling adaptive sampling for intra- window join: Simultaneously optimizing quantity and quality. Proceedings of the ACM on Management of Data, 2(4):1-31, 2024b.
Thakur, A. s., Choudhary, K., Ramayapally, V. S., Vaidyanathan, S., and Hupkes, D. Judging the judges: Evaluating alignment and vulnerabilities in Ilms-as- judges. arXiv preprint 2rXiv:2406.1206 2024.
Riddell, M., Ni, A., and Cohan, A. Quantifying contamina- tion evaluating code generation capabilities of language models. arXiv preprint arXiv:2403.048/ 2024.
Sainz, O., Campos, A., Garefa-Ferrero I., Etxaniz, J., de Lacalle, O. L., and Agirre, E. Nlp evaluation in trouble: On the need to measure Ilm data contamination for each benchmark. arXiv preprint arXiv:2310.18012 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302 13971, 2023a.
Setlur, A., Nagpal, C., Fisch, A., Geng, X., Eisenstein, J., Agarwal, R., Agarwal, A., Berant, J., and Kumar, A. Re- warding progress: Scaling automated process verifiers for Ilm reasoning. arXiv preprint arXiv:2410.08146 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi. A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.092888 2023b.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Tunstall, L., Von Werra, L., and Wolf, T. Natural language processing with transformers. "'Reeill Media, Inc.", 2022.
Uesato, J., Kushman, N., Kumar, R., Song, H. F., Siegel, N. Y., Wang, L., Creswell, A., Irving, G., and Higgins, I. Solving math word problems with outcome-based feedback. 2022.
Skalse, J., Howe, N., Krasheninnikov, D., and Krueger, D. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460-999 9471,, 2022.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit. J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.03344 2024.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and anal- ysis platform for natural language understanding. In International Conference on Learning Representations, 2018.
Stechly, K., Valmeekam, K., and Kambhampati, S. On the self-verification limitations of large language mod- els on reasoning and planning tasks. arXiv preprint arXiv:2402.08/ 08 15, 2024.
13

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Wang, A., Song, L., Tian, Y., Peng, B., Yu, D., Mi, H., Su, J., and Yu, D. Litesearch: Efficacious tree search for Ilm. 2024a.
Xie, R., Zeng, Z., Yu, Z., Gao, C., Zhang. S., and Ye, W. Codeshell technical report. arXiv preprint arXiv:2403.15741 2024.
arXiv preprint arXiv:2407.00322
Wang, C., Liu, X., Yue, Y., Tang, X., Zhang, T., Jiayang, C., Yao, Y., Gao, W. Hu, X., Qi, Z., et al. Survey on factuality large language models: Knowledge, retrieval and domain-speen speciiiiiic arXiv preprint arXi:2331000522, 2023a.
Xiong, K., Ding, X., Liu, T., Qin, B., Xu, D., Yang, Q., Liu, H.,, and Cao,, Y. Meaningful learning: Enhancing ab- stract reasoning in large language models via generic fact guidance. Advances in Neural Information Processing Systems, 37:120501-12055 2024a.
Wang. J., Fang, M., Wan, Z., Wen, M., Zhu, J.. Liu, A., Gong, 7.,, Song, Y., Chen, L.,, Ni, L.. M., et al. Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671, 2024b.
Xiong, W., Song, Y., Zhao, X., Wu, W., Wang, X., Wang, K., Li, C., Peng, W., and Li, S. Watch every step! Ilm agent learning via iterative step-level process refinement. arXiv preprint arXiv:2406. / 176, 2024b.
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q.. Liu, T., and Sui, Z. Large language models are not fair evaluators. arXiv preprint arXiv:2330.. 17926, 2023b.
Yang, L., Zhang, S., Qin, L., Li, Y., Wang, Y., Liu, H., Wang. J., Xie, X., and Zhang, Y. Glue-x: Evaluating natural language understanding models from out-of- distribution generalization perspective. arXiv preprint arXiv:2211.08073. 2022.
Wang, P., Li, L.., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math- Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9426-9439, 2024c.
Yang, L., Zhang. S., Yu, Z., Bao, G., Wang, Y., Wang, J., Xu, R., Ye, W., Xie, X., Chen, W., et al. Supervised knowledge makes large language models better in-context learners. arXiv preprint arXiv:2312. 15918, 2023.
Yao, W., Wang, Y., Yu, Z., Xie, R., Zhang, S., and Ye, W. Pure: Aligning IIm via pluggable query reformulation for enhanced helpfulness. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 8721- 8744, 2024.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruet: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10553
2022.
Wang, Y., Yu, Z., Zeng, Z., Yang, L., Wang. C., Chen, H., Jiang, C., Xie, R.. Wang, J., Xie, X., et al. Pandalm: An automatic evaluation benchmark for IIm instruction tuning optimization. arXiv preprint arXiv:2306.05087, 2023c.
Yu, Z., Gao, C., Yao, W., Wang, Y., Ye, W., Wang, Xie, X., Zhang, Y., and Zhang, S. Kieval: A knowledge-groundee interactive evaluation framework for large language mod- els. arXiv preprint arXiv:2402. 1504:333..1043, 2024a.
Wang, Y.. Yu, Z., Wang, J., Heng, Q.. Chen, H., Ye, W., Xie, R., Xie, X., and Zhang, S. Exploring lisnua-language models for imbalanced learning, International Journal of Computer Vision, 132(1):22 (2(1))2244237 2024d.
Yu, Z., Gao, C., Yao, W., Wang, Y., Zeng, Z., Ye, W., Wang, J., Zhang, Y., and Zhang, S. Freeeval: A modular framework for trustworthy and efficient evaluation of large language models. arXiv preprint arXiv:2404.0600 2024b.
Wang. 7., Li, Y., Wu, Y., Luo, I., Hou, L.., Yu, H., and Shang, J. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervi- sion. arXiv preprint arXiv:2402.02655. 2024e.
Yu, Z., Zeng. J.. Gu, W., Wang. Y., Wang, J., Meng, F., Zhou, J., Zhang, Y., Zhang, S., and Ye, W. Rewardanything: Generalizable seinccppe--ollowwi reward models. arXiv preprint arXiv:2506.03637. 2025.
Wei, J., Wang, X., Schuurmans, D.,, Bosma, M., Xia, F., Chi, prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
E., Le, Q. V., Zhou, D., al.
Zhang, D.,, Zhoubian, S., Hu, Z., Yue, Y., Dong, Y., and
Tang, J. Rest-mets*: Llm self-training via process reward guided tree search. arXiv preprint 2024a.
Xie, Q. Li, Q. Yu, Z., Zhang, Y., Zhang, Y., and Yang, L. An empirical analysis of uncertainty in large language model evaluations. arXiv preprint 107009, 2025.
Zhang, K., Li, Ji, Li, G., and Jin Z. Self-edit: Fault- aware code editor for code generation. arXiv preprint wrX0v:2305.04087, 2023.
14

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint MeXivvI904.00000 2019.
Zhang, Z., Wang, C., Xiao, X., Zhang, Y., and Wang, D. Nash cot: Multi-path inference with preference equilib- rium. arXiv preprint (rXiv:2407.07099, 2024b.
Zheng, K., Decugis, J., Gehring, J.. Cohen, T., Negrevergne, B., and Synnaeve, G. What makes large language models reason in (multi-turn) code generation? arXiv preprint 2rX.v:2410.08105 2024a.
Zheng, L.,, Chiang, W.-L., Sheng. Y.. Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z., Feng, Z., and Ma. Y. Llamafactory: Unified efficient fine- tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024b. Association for Computa- tional Linguistics. URL 2403.133 13372.
abs/
Zhong, L., Wang, Z., and Shang, J. Ldb: A large language model debugger via verifying runtime execution step-by- step. arXiv preprint arXiv:2402. /6906, 2024.
Zhu, M., Weng, Y., Yang. I.., and Zhang, Y. Deepreview: Improving llm-based paper review with human-like deep thinking process. arXiv preprint 2rXhv:2503.08569,200 2025.
15

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
A. Experimental Setup and Hyperparameter Details
This appendix provides a comprehensive description the experimental setup, encompassing the hyperparameters, software, and hardware configurations employed in this study.
A.1. Search Algorithm Hyperparameters (ORPS)
The following hyperparameters were used for the search algorithm in ORPS:
Search Depth (num. rounds): 5, This parameter defines the maximum depth of the search tree, representing the number of iterative steps in the search process.
Beam Width (top_k): 3. This parameter specifies the number of highest-scoring candidate solutions (traces) retained at each step of the beam search.
Expansion Factor (num_s sampless) 20. This represents the number of new states (candidate solutions) explored from each state during the search process.
Process Reward Weight (a): 0.5. This metric determines the proportion the Process Reward in the step reward.
Outcome Reward Weight (3): 0.5. This metric determines the proportion of the Outcome Reward in the step reward.
Table Performance Metrics Description. Our evaluation framework uses both dynamic execution profiling and static code analysis metries to comprehensively assess code quality and efficiency.
Category
Metric
Description
Dynamic Execution Profiling
Time Enabled
Total CPU time spent executing the code, measured in nanoseconds. Lower values indicate efficient execution and better algorithmic optimization.
Instruction Count
Number of CPU instructions executed during runtime. Reflects computa- tional efficiency, with lower counts suggesting more optimized code paths and better algorithm implementation.
Branch Misses
Frequency of incorrect branch predictions during execution. Lower values indicate better code predictability and CPU pipeline efficiency, resulting in faster execution times.
Page Faults
Number of times the program needs to access virtual memory. Fewer page faults suggest better memory management and more efficient memory access patterns.
Static Analysis
Code Length
Total number of lines in source code. Generally, shorter code length indicates more concise solutions while maintaining readability and function- ality.
AST Node Count
Number nodes the Abstract Syntax Tree. Measures structural complex- ity of the code, with fewer nodes suggesting simpler and more maintainable implementation.
Cyclomatic Complexity
Quantifies the number of linearly independent paths through the code. Lower values indicate mesier-to-maintain and test code, reducing potential bug sources.
Cognitive Complexity
Measures how difficult the code is understand. based control flow structures and nesting. Lower scores suggest more readable and maintain- able code that is easier to debug.
16

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
A.2. Inference Configuration
All inference experiments were conducted single machine using the FreeEval (Yu al., 2024b) codebase, integrated with Hugging Face's sestt-gneraaiin-nneeeeeee toolkit for efficient model serving. The following inference settings were applied:
Maximum Context Length (max_ : 18,000 tokens. This parameter defines the maximum number of tokens allowed in the input sequence to the model.
Generated Tokens per Round: 1,500 tokens. This specifies the number of new tokens generated by the model in each round of inference.
A.3. Execution Constraints
To ensure consistent and reproducible results, the following execution constraints were enforced during inference:
Timeout per Test Case: seconds. This limits the maximum execution time allowed for each test case.
Memory Limit: 512 MB. This constraint restricts the maximum memory allocation permitted for each test case.
Maximum Test Cases per Problem: 15. This sets upper bound the number of test cases evaluated for each problem.
A.4. Model Training Configuration
This section outlines the hyperparameters and settings used during the training phase of the model, which pertinent to the analysis experiments (subsection 4.4). While ORPS itself does not require training, these details provided for completeness and reproducibility.
Training Framework: 11 llamafactory (Zheng et al., 2024b)
DeepSpeed ZeRO3 (Rajbhandari et al., 2020) (Zero Redundancy Optimizer Stage 3). This enables efficient training of large models by partitioning optimizer states, gradients, and model parameters across data parallel processes.
Optimization Framework:
qwen-2. 5-coder-7b-insteeeeeeee training was conducted.
Base Model:
This is the pre-trained language model upon which further
Batch Size per Device: This defines the number of training examples processed on each GPU before a gradient update step.
This allows simulating a larger effective batch size by accumulating gradients over multiple forward and backward passes before updating model weights. The effective batch size is therefore 8 (2 per device * 4 steps).
Gradient Accumulation Steps: 4.
Learning Rate: 10 5, This parameter controls the step size taken during gradient- based optimization.
Learning Rate Scheduler: Cosine decay. This gradually reduces the learning rate over the course of training, following a cosine function.
Number of Training Epochs: 2.0. This specifies the number of complete passes through the entire training dataset.
Maximum Sequence Length: 16,384 tokens. This defines the maximum length of the input sequences during training.
Mixed Precision Training: Enabled with bfl5 6 (Brain Floating Point 16-bit format), This accelerates training by performing computations with reduced precision while maintaining model accuracy.
A.5. Hardware Environment
All experiments were performed on NVIDIA A800 GPUs, each equipped with 80GB of GPU memory.
17

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
B. Additional Results CodeContests
During rebuttal, reviewers were interested in an additional competitive programming contest benchmark, CodeContests (Li al., 2022b). Due page limitation, we include our new results here Table 7.
Table 7: Pass@
performance on CodeContests with Qwen-7B controlling LLM calls at 100.
Method
Pass@1 1
Tests %
Valid
Time (ms)
Reflexion
8.48
14.53
27.27
52318
LDB
7.88
11.79
21.21
3350
REx
13.33
20.17
32.12
27791
PRM-GPT
4.94
8.64
17.28
3084
PRM-Human
9.88
15.04
23.46
1985
ORPS (Ours)
20.61
28.36
40.61
36824
C. AI Usage in Code Development
During the development of ORPS and the design of its experiments, LLMs were employed to assist with coding. All Al-assisted code were reviewed and refined by the authors ensure correctness and alignment with the research goals.
D. Impact of Optimizing Different Metrics on Code Quality
In ORPS, we integrate multiple evaluation metrics to guide the model's reasoning and code generation. These metrics include both static analysis (e.g., AST nodes, cyclomatic complexity) and dynamic execution profiling (e.g., execution speed, CPU instruction count, branch mispredictions). To better understand their individual contributions, we conduct an ablation study where each metric is optimized in isolation.
D.1. Experimental Setup
To isolate the effect of each metric, we conduct a series of ablation experiments. In each experiment, the model receives rewards only from a single metric, while all other evaluation criteria remain unchanged. Specifically, we consider the following setups:
+AST Nodes: Encourages structurally simpler code by minimizing the number of AST nodes.
+Cyclomatic Complexity: Penalizes excessive branching and loop structures improve maintainability.
Cognitive Complexity: Rewards code that easier to understand based on nested structures and control flow.
+Execution Speed: Optimizes for faster execution while maintaining correctness.
+CPU Instructions: Minimizes the number of CPU instructions executed.
+Branch Mispredictions: Encourages predictability to improve processor efficiency.
+Page Faults:
Reduces memory access overhead for better performance.
+All Metrics:
Incorporates all the above metrics into single optimization objective.
D.2. Results and Discussion
As shown in Figure 6, optimizing for single metric significantly improves performance in that specific aspect, yet comes at severe cost to other dimensions of code quality. This phenomenon suggests that the model falls into form of local metric optimality, where it overfits to the given reward signal while neglecting other critical properties of high-quality code.
More specifically, we observe that optimizing for static analysis metrics (c.g., AST nodes, cyclomatic complexity) often leads to a sharp decline dynamic execution metrics (e.g., execution speed, CPU instructions). For instance, minimizing
18

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Static Analysis Improvements
Dynamic Analysis Improvements
No Metrics (Baseline)
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
+Code
Length
+85.2%
+32.5%
-11.9%
+160.1%
-113.5%
-54.5%
-57.4%
+6.3%
+AST Nodes
+43.4%
+19.3%
3.1%
+146.6%
-217.2%
-134.3%
-186.5%
-161.8%
Cyclooattcc Complexity
+43.1%
+43.9%
+178.6%
+146.6%
-22.1%
+48.7%
+27.1%
-18.8%
+Cognitive Complexity
+62.8%
+40.8%
+126.3%
+203.3%
-67.7%
+13.9%
-14.3%
-24.6%
+Execution Speed
+3.1%
-11.5%
+23.7%
+33.2%
-23.6%
+46.0%
+16.9%
+13.6%
+CPU
-43.8%
-61.0%
-49.2%
-25.3%
-49.9%
+22.5%
-12.9%
-5.0%
Instructions
+Branch Mispredictions
-24.9%
-52.6%
-31.0%
-8.8%
-30.9%
+34.0%
+7.2%
0.0%
+Page Faults
+54.0%
+49.4%
+119.7%
+130.7%
+10.7%
+78.7%
+58.2%
+71.4%
+All Metrics
+44.0%
+44.4%
+96.8%
+82.4%
+48.9%
+86.6%
+-1.5%%
+35.7%
Code Length
AST Nodes
Cyclomatic Complexity
Cognitive Complexity
Execution Speed
CPU
Branch Mispredictions
Page Faults
Instructions
Figure 6: Impact of single-metric optimization on code quality, Each value represents the difference from the baseline. Optimizing for a single metric significantly improves performance that dimension but leads to severe degradation in others.
eyclomatic complexity encourages structurally simpler code, yet it may suppress efficient algorithmic choices that involve loops and conditionals. Conversely, optimizing for execution speed often results in obfuscated redundant code, the model prioritizes raw performance over maintainability.
These observations highlight an inherent challenge in code optimization: static structure and dynamic efficiency often conflict when optimized in isolation. This suggests that achieving well-balanced code quality requires a multi-objective optimization strategy rather than single-metric reinforcement. Traditional reward models struggle in such scenarios, as they often assume reward signals are aligned across different dimensions. However, our results indicate that code generation requires more nuanced supervision- one that dynamically balances trade-offs between readability, maintainability, and execution efficiency.
Furthermore, the steep performance drop in non-optimized metrics suggests that reward sparsity is critical issue in single-metric training. Since the model receives no information about other quality dimensions, it fails to generalize improvements beyond the specific reward it observes. This contrasts with human programming intuition, where engineers naturally balance multiple objectives, such as runtime efficiency, readability, and modularity. Future work could explore techniques like adaptive reward scaling, where the model dynamically adjusts its focus based real-time trade-offis rather than rigid metric-specifir optimization.
19

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Comparison of Good and Bad Reasoning
Reasoning
Good Ressoning
10-
Value
0.7-
06
05
pass, 1
avg time
avg. insstructttinss
avg. brancha.nhh...... misses brassh
avg. page_paeeeppp paultffults faults
avg. ast www.eoneeecom node_count count
avg wyw.occlommticc.oommm catpommattcccooeeiityy.. complexity
avg. avg_cognltive_comm www.coonnntieeeccccc htwp__onntveeecmmeeee leaigntive_coppexity complexity
Figure 7: Impact of reasoning quality on code quality. Each metric is normalized, where higher values indicate better performance. The red bars represent the code quality when generated from higher-quality reasoning, while the blue bars correspond to lower-quality reasoning. The results suggest that better reasoning significantly improves execution-related metrics while having a limited effect on structural complexity.
E. Correlation of Reasoning Quality and Code Quality
Our method is based on core hypothesis: higher-quality reasoning leads higher-quality code. To validate this assumption, conducted simple motivation experiment.
E.1. Experimental Setup
We first prompted GPT-4o to generate two different reasoning chains for each problem in the LBPP dataset, explicitly ensuring clear quality difference between them. To verify that the generated reasoning indeed exhibited significant quality differences, we used GPT-4o again to reassess and confirm their relative quality.
Next, we concatenated each reasoning chain with the original problem description and fed them separately into GPT-4o to generate code solutions. Finally, we evaluated the quality of the generated code across multiple metrics.
E.2. Results and Analysis
As shown in Figure 7, all metrics are normalized, where higher values indicate better performance. The red bars represent the quality of code generated from higher-quality reasoning, while the blue bars correspond to the lower-quality reasoning.
From the results, we observe that better reasoning generally leads to better overall code quality, particularly in dynamic execution metries. This suggests that high-quality reasoning not only improves correctness but also enhances execution efficiency, possibly by guiding the model generate more optimal algorithmic structures.
Interestingly, the complexity of the generated code does not increase significantly. One possible explanation is that more detailed and higher-quality reasoning naturally guides the model toward generating more sophisticated solutions that handle a wider range of scenarios, rather than simply making the code structurally more complex.
These findings reinforce the importance of reasoning supervision in code generation. A strong reasoning framework helps models not only arrive at correct solutions but also optimize execution efficiency without unnecessarily inflating complexity.
20

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
F. Example Model Outputs
To illustrate effectiveness of our approach, we present a representative example from LBPP dataset (Problem #101) that demonstrates how our method handles complex logical reasoning tasks. The original prompt this problem is:
You are given an array of integers, each of which is greater than 1. This array needs to be split into subsets such that the greatest common divisor within each subset is greater than 1. Write Python program to return the minimum number subsets required to create such split.
CoT Output
Prob: Given array of integers, split it into the minimum number of subsets such that the GCD of each subset is greater than 1.
Reasoning Process:
First, we need to understand what makes numbers group-able: Numbers can be same subset their GCD
? We want minimize the number of subsets
Key observations: GCD(a.b) I and GCD(b,c) à a,b,c can potentially in the same group This suggests graph-like structure where numbers connected if their GCD &
3. Solution approach: Treat numbers nodes in graph Connect numbers with edges if their GCD Find
connected components in this graph Each component represents one subset
4. Implementation strategy: Use Union-Find data structure for efficient component tracking Calculate GCD using
Euclidean algorithm Final answer is the number connected components Generated Solution:
def god (a, b) while a, return
b, a
def minGroups (nums)
len (nums)
parent list (n))
10 def find (x):
if parent [x] != x:
parent find (parent [x]) return parent [x]
13
14
def union (x, y):
parent find
find
17
for in range (n) 18
19
for in range (i + 1, n): if ged (nums [i], nums [i]) 1: union j)
23
return len (set (i) for in range (n) )
21

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
ORPS Programmer Output
=== Programmer Thoughts == # The problem requires splitting an array into subsets such that the greatest common divisor (GCD) within each subset is greater than To achieve this, we can use a greedy algorithm. The idea to iterate through the array and add each element the existing subset if their GCD is greater than the GCD is we start new subset.
# We need implement function that calculates the GCD of numbers. Then, we can this function determine if element can be added to existing subset or if need start subset.
# We also need ensure that our solution is efficient. The time complexity of the GCD calculation can be optimized using the Euclidean algorithm, which has a time complexity of O(logmmin(a, b))). Solution
import math
def god (a, b)
Calculate the greatest common divisor of two numbers using the Euclidean algorithm. m..
while
a, b b, a a
return
def min miing__a dubbsett (nums):
12
mun
13
Split the array into subsets such that the greatest common divisor within
each subset is greater than 1. Return the minimum number of subsets required.
16
subsets []
17
for num in nums: added = False
18
for subset in subsets:
if ged (num, [ubset[-1]]
21
aubset.aapennd
22
added True break if not added: subsets. append [Inm]] return len (subsets)
23
25
26
ORPS Critic Feedback
== Critic Thoughts == The code provided attempts to solve the problem by iterating through each number and adding it to an existing subset if the GCD with the last element of the subset is greater than 1. no such subset exists, a new subset is created. This approach ensures that each subset has GCD greater than However, there are few issues and improvements that can be made: *eTest Case Failures**; The test cases for indices 5, 8, and 10 fail. This indicates that the current implementation does not correctly handle certain scenarios where numbers with common divisor greater than are present. 2. **Performance** The code has higher cognitive complexity, which means it is harder to understand and maintain. The use of nested loops and the nested if-else structure can be optimized.
**ffficienccy*** The current approach has a time complexity O(n²) due to the nested loops. This can be optimized Onn n) by sorting the numbers first and then using a single pass to form subsets.
**Code Clarity**; The code can be made more readable by breaking down the logic into smaller functions and adding comments.
5. **Handling Edge Ca The code does handle edge cases such empty list or list with single element.
= Score === $$3$S
22

---

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
ORPS Programmer Improvement
=== Programmer Thoughts == The feedback indicates that the current implementation has several issues, including failing test cases and high cognitive complexity. We need to address these issues by improving the logic and ensuring that all edge cases are handled correctly.
* **Key Improvements**: 1. ** **Hnndling Edge Cases** Ensure that the function handles an empty list or a list with single element. 2. **Ffficiency*** Optimize the approach achieve time complexity of O log n) by sorting the numbers first and then using single pass to form subsets. **Code Clarity#*: Break down the logic into smaller functions and add comments for better readability.
# *pproach** 1. **Sort the Array*# Sorting the array ensures that numbers with a common divisor are adjacent, making it easier to form subsets. 2. **Single Pass to Form Subsets"*: Iterate through the sorted array and add each number to the current subset if the GCD with last element of the subset is greater than not, start new subset. #dge Case Handling** the list is empty, return O. the list has single element, return 1.
Solution
import math
def god b):
"""Caicuiate the greatest common divisor of two numbers using the Kuclideas a algorithm. *** while b: a, b b, a $
return
10 def min g subsets (nums):
11
Split the array into subsets such that the greatest common divisor within each subset greater than 1. Return the minimum number of subsets required.
13
14
15
:param noms: List of integers greater than
17
:return: Minimum number of subsets required
18
19
if not nums:
return 0
if len (nums) -- 1:
22
return
23
24
Sort the array to ensure numbers with common divisor are adjacent nums nums.sort ()
27
subsets
28
for num in nums:
added = False
for subset in subsets:
if ged (num, aubset[-1]] aubset.appendd (num)
32
33
added True break
if not added:
36
subsets. append [num]]
37
38
return len (subsets)
23


Reviews:
[{"review_id": "xFB1qa8lej", "reviewer": "Reviewer_DEgP", "summary": "This paper introduces ORPS, a novel framework that unifies outcome and process supervision to address complex code problems. Notably, this approach does not require training PRMs. ORPS demonstrates significant improvements when utilizing ground truth test cases.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n## Answers to Questions\n\n**Q1**: Currently, there's an assumption in process supervision and test-time scaling research (e.g. OmegaPRM, Math-Shepherd, Let's Verify Step by Step, Deepseek-Math, etc.) that a specially trained Process Reward Model is required to guide reasoning (during training with RL or inference with search algorithms like MCTS). **ORPS fundamentally challenges this assumption and study necessity of training PRMs, by combining execution feedback as verifiable rewards with existing LLMs to generate high quality process rewards to guide reasoning.** We compare ORPS with trained PRMs by using different methods to guide LLM reasoning during inference (Table 4) and we now add 2 new experiments (in rebuttal for 4j6r) to extensively validate our claims.  Results indicates ORPS produce better rewards than trained PRMs to guide reasoning and consistently performs better given the same compute budget.\n\nRegarding S*[1], this work was made public one month after the ICML submission deadline which further validates our approach.\n\n**Q2**: We agree that the certain phrasing suggest a focus on efficiency over correctness. To clarify: our framework prioritizes optimizing correctness as the primary objective, while considering comprehensive metrics involving efficiency, code quality, complexity, etc. In all our experiments we report correctness metrics along with efficiency for comprehensive comparison. **Our motive here is that a good solution should be correct, efficient (evaluated with dynamic analysis), easy to read and maintain (static analysis). Furthermore, we study how dynamic and static analysis metrics affects overall performance(Figure 6 in Appendix D) and this should also address Weakness 2.**\n\n## Unit Test Cases\n\nThank you for raising the concern on the scarcity of ground truth test cases in practical scenarios. We agree quality of test cases might significantly affect the performance of all approaches that utilize execution feedback to debug or improve the code. Our baseline Reflexion points out \"they rely upon ground truth test cases that invalidate pass@1.\" **However, we have explicitly addressed this concern, and the test cases used in ORPS are not the ground truth from test sets.** Instead, we follow Reflexion to prompt the LLM given the problem and example tests and filter invalid cases. These self-generated, weak tests are then used during reasoning, and only one final solution will be tested against the actual ground truth cases. To ensure fairness and reproducibility, we cached the cases generated by Qwen 7B and used the same cases in all our experiments. **While the quality of generated cases are suboptimal compared to ground truths, ORPS performs better, even compared to the baselines using ground truth cases.** Notably, we also evaluate ORPS with ground truth test cases (ORPS (w/T) in Table 1) in comparison. We will be including all system prompts in revised Appendix.\n\n## Computational Cost\n\nThe bottleneck of computational cost of ORPS and baselines is LLM Inference Calls instead of dynamic and static analysis, as this require GPU computing and introduce the majority of latency. Although we compared ORPS with BoN in Figure 4 by allocating the same number of samples per problem, we now include a new experiment for other baselines given the same number of LLM calls.\n\nTo be precise, we analyze the worst case cost for each method: ORPS requires $2\\times N \\times (K \\times T + 1)$ Calls, where N = Candidate Samples, K = Beam Size, T = Reasoning Steps. Reflexion requires $2\\times T$ calls, where T = Steps. LDB requires $1 + P×T×(2+B×N)$ where P = pass_at_k, T = max_iterations, B = number of blocks in code control flow graph, N = max_trials. For trained PRMs, we swap the self-critic LLM call with the PRM and thus the calls are identical. Please refer to rebuttal for 4j6r for PRM training details.\n\n|20 Calls|Pass@1|Tests%|Valid%|Time%|\n|-|-|-|-|-|\n|Reflexion|37.0|51.7|71.6|119.5|\n|LDB|37.0|50.8|66.0|274.7|\n|PRM-GPT|44.4|58.1|77.8|**100.1**|\n|PRM-Human|40.7|53.1|69.1|124.7|\n|ORPS|**48.4**|**64.8**|**84.5**|105.6|\n\n|50 Calls|Pass@1|Tests%|Valid%|Time%|\n|-|-|-|-|-|\n|Reflexion|40.7|57.6|79.6|130.3|\n|LDB|36.4|50.4|66.0|272.6|\n|PRM-GPT|37.0|52.9|71.6|**112.7**|\n|PRM-Human|38.3|53.5|69.1|137.7|\n|ORPS|**55.6**|**72.1**|**89.5**|116.8|\n\n|100 Calls|Pass@1|Tests%|Valid%|Time%|\n|-|-|-|-|-|\n|Reflexion|39.5|54.4|72.8|113.8|\n|LDB|37.0|51.0|66.7|275.7|\n|PRM-GPT|35.8|51.1|70.4|127.3|\n|PRM-Human|42.0|56.4|76.5|106.6|\n|ORPS|**64.2**|**75.4**|**88.9**|**91.0**|\n\nResults indicate that **even we use weaker, self-generated test cases, ORPS consistently outperform baselines that use ground truth test cases given the same compute budget and scales up consistently.**\n\nWe sincerely appreciate your time and constructive feedback, which has helped us strengthen our work. **Given these these clarifications and new results, we respectfully request reconsideration of your score.**", "claims_and_evidence": "Overall, most of the claims are well-claimed. Please check my other comments below.", "methods_and_evaluation_criteria": "The method lacks persuasiveness because the salient improvement assumes the availability of ground truth test cases, which is uncommon in real-world code completion scenarios. Without these ground truth test cases, the performance advantage is not prominent.", "theoretical_claims": "N.A", "experimental_designs_or_analyses": "This paper should include a more detailed cost analysis of both the dynamic and static analysis processes.", "relation_to_broader_scientific_literature": "The primary contribution of this paper is the integration of dynamic and static analysis to construct the reward.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "The main weakness of the paper is that the performance improvement heavily depends on the availability of ground truth test cases, which is uncommon in real-world scenarios.\n\nAdditionally, the rationale for combining dynamic and static analysis is unclear. For instance, the necessity of static analysis is questionable when dynamic analysis has already been performed. If static analysis is indeed necessary, there should be more consideration regarding the weighting of scores between static and dynamic analysis.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "1. It's somewhat puzzling that the paper begins with Process Reward Models. I initially assumed the designed reward would be used to train the model, but it's not addressed in this paper. Instead, it seems more like an inference-time strategy, similar in spirit to [1]. It's important to note that a minor resemblance to [1] does not affect the paper's rating. My question is why the narrative starts with Process Reward Models rather than an inference-time strategy when the paper doesn’t involve training. Please note that this affects the understanding of contribution 2 to eliminate specially trained PRMs\n\n2. Additionally, while I understand that a tree-structured exploration space could enhance the pass rate of generation, I'm curious why the instances mentioned in this paper focus solely on efficiency, which is a non-functional requirement, rather than on functional aspects measured by the pass rate. This is evident in the instances mentioned in lines 72-73 and 161-163. \n\n\n\n[1] S∗: Test Time Scaling for Code Generation", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BGrHjZYLtN", "reviewer": "Reviewer_Kbe7", "summary": "- The paper proposes an LLM-based algorithm (outcome-refining process supervision, ORPS) for code generation. The primary contributions are algorithmic and empirical. ORPS leverages LLMs abilities significantly more than prior work in this area. Specifically, this includes self-reflection, self-critique and process (per-step) rewards. The key difference from prior LLM-based codegen models is that ORPS eliminates the need for a separately trained process reward model (PRM), which provides dense execution feedback at each step of the iterative reasoning chain. Combined with tree search, this leads to significant performance improvement on modern codegen benchmark datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for your thoughtful review and constructive feedback. We deeply appreciate your recognition of our work and we address your questions and concerns point by point:\n\n> **Q1:** What initial LLM inputs (system prompts) used in each call to the LLM, if any. If system prompts exist, is it possible to include them in the appendix.\n\nWe sincerely appreciate your emphasis on reproducibility. **Reproducibility is our first priority** and thus we have anonymously open-sourced all implementation details, including code, prompts, setup instructions, commands to run and reproduce our experiments in our anonymous repository. The repository contains all prompts used in ORPS. We attach our prompts at the end of this response. However, due to space constraints of rebuttal, we cannot paste everything here, but **we will make sure to include all prompts in our revised appendix**. For the new experiments and data involved in rebuttal, we will also add them to our repository. \n\n> **Q2:** What's the current SOTA for LBPP? Is it ORPS?\n\nTo the best of our knowledge, ORPS currently achieves SOTA performance on the LBPP benchmark. Moreover, this improvement requires no additional model training and can be applied to any sufficiently capable base LLM. As demonstrated in our experiments, ORPS consistently outperforms other baselines across multiple models given the same compute budget.\n\n> **W2:** The paper can be somewhat handwavy with terminology and some claims. A description or definition of concepts and terms before their usage would be preferable. Examples include \"reasoning\", \"reasoning space\", \"finer reasoning\", \"intrinsic reasoning\", and so on.\n\nThank you for pointing out our presentation issues. We acknowledge the need for clearer definitions and will revise the paper to explicitly clarify key terms and provide more explanation to our notations. We will ensure all technical terms are clearly defined before their first use.\n\nWe are very grateful for your expertise and time in evaluating our work. Thank you again for your contributions to improving this work.\n\nThe system prompt of model for reasoning and generating codes:\n\n````\nYou are a Python programmer in a code interview. You're solving a coding problem while sharing your thoughts and discussing with an critic. Always follow the format below for each round:\n1. First, share your thoughts as comments:\n   - Your current approach (or an analysis of the problem and your plan if you haven't written any code yet)\n   - What you learned from previous feedback (if there is any previous feedback, otherwise think about your plan, what might be missing from your previous plan)\n   - Why you chose this approach (or how you plan to tackle the problem), do you need to shift your approach?\n   - Be clear, concise, detailed and pay attention to the comments given by the critic, no chit-chat, no small talk.\n   - Always use first person, e.g. I, we, our, etc.\n2. Then write your solution:\n   - Clean, efficient Python code that follows requirements exactly\n   - No test cases in code, just the solution\n   - Your code will then be tested by the critic, so do not include any test cases in your code, this is very important\nFormat your response as:\n# === BEGIN PROGRAMMER THOUGHTS ===\n# [Your response to previous feedback]\n# === END PROGRAMMER THOUGHTS ===\n# === BEGIN SOLUTION ===\n```python\n[Your code]\n```\n# === END SOLUTION ===\nOutput strictly as shown above.\n````\n\nThe system prompt for the self-critic role:\n\n```\nYou are a technical critic reviewing a programmer's Python code. Provide clear, constructive feedback and suggest improvements or propose a new approach. The programmer's thoughts, code, execution analysis will be provided to you each round and you need to give constructive feedback to the programmer.\nHere are some rules you need to follow:\n1. First, share your thoughts:\n   - How did the code perform in the execution, did any test cases fail?\n        - If any case failed, why it failed? If all passed, think about performance improvement.\n   - Key observations on the code, e.g. what's good or bad, what's the most important thing to improve, etc.\n        - Potential improvements: time / space complexity, a different algorithm, strategy, etc.\n   - Think about the programmer's thoughts, propose a new direction if you have any better idea, or give some advice to the plan, or give additional analysis of the problem\n   - Your thought process should be clear and concise, no chit-chat, no small talk. Guiding the programmer to improve their code is your main goal, you do not write any code.\n  - Always use first person, e.g. I, we, our, etc.\n\n...\n<omitted due to character limitation of rebuttal, please refer to our open-source repository for details>\n```", "claims_and_evidence": "- Yes.", "methods_and_evaluation_criteria": "- Yes.", "theoretical_claims": "- Not applicable.", "experimental_designs_or_analyses": "- Not applicable.", "relation_to_broader_scientific_literature": "- The paper extends the state-of-the-art (to my knowledge) of using LLMs for code generation. It builds on very recent prior work (LDB, Reflexion). It leverages the LLM significantly more. That said, LLMs-for-code-generation is a very fast moving area so I can't be certain that I haven't missed relevant prior work or baselines.", "essential_references_not_discussed": "- No (but I'm not certain).", "other_strengths_and_weaknesses": "Strengths\n  - The paper tackles an important and relevant problem. Improvement here is likely to have a large impact and be of large interest to the community.\n  - The paper is well structured and mostly clearly written. The main ideas are easy to follow and nicely illustrated.\n  - The experiments show significant performance improvement compared to strong baselines.\n\n\nWeaknesses\n  - Overall, the paper is quite good. I wasn't able to spot any major technical issues. My biggest concern is around reproducibility. I didn't see the full prompt or prompts used as inputs to the LLM listed in any of the appendices. The format of the LLM responses \"Programmer Thoughts\", \"Critic Thoughts\", and numerical rewards suggest some prompt engineering was likely involved. If yes, they should be included in the appendices to aid reproducibility. (Please note that my current rating of the paper assumes a clear listing of the system prompts, if any, used to interact with the LLM will be included in the author response.)\n\n  - The paper can be somewhat handwavy with terminology and some claims. A description or definition of concepts and terms before their usage would be preferable. Examples include \"reasoning\", \"reasoning space\", \"finer reasoning\", \"intrinsic reasoning\", and so on.", "other_comments_or_suggestions": "- No additional coments.", "questions_for_authors": "- (Q1) What initial LLM inputs (system prompts) used in each call to the LLM, if any. If system prompts exist, is it possible to include them in the appendix.\n\n- (Q2) What's the current SOTA for LBPP? Is it ORPS?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NKQ7iLveYu", "reviewer": "Reviewer_xFyb", "summary": "The paper proposes outcome refining process supervision (ORPS), a unified framework to bridge the gap between process supervision and outcome supervision through structured reasoning and execution-based feedback. The experiment results show that concrete feedback signals are pivotal for solving complex programming tasks. Furthermore, the work shows that eliminating trained process reward models with hybrid process rewards significantly boosts model performances.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and recognition of our work's effectiveness. We appreciate your thoughtful questions and address them below.\n\n>  **Q1:** Does the proposed approach composed of components which already exists in the literature? If yes, how do the authors see the novelty of the work?\n\nRegarding novelty, while we integrate some established concepts, we introduce several key innovations rather than merely combining existing components:\n\n1. Prior process supervision approaches (OmegaPRM, Math-Shepherd, etc.) assume specially trained Process Reward Models are required for effective reasoning guidance. **ORPS challenges this assumption and studies necessity of training PRMs, by combining execution feedback as verifiable rewards with existing LLMs to guide reasoning.** Results show that ORPS, as a pure inference-time method, produces better reasoning guidance than trained PRMs.\n2. Unlike repair-based approaches (e.g., Self-Repair, LDB) that incrementally fix specific code blocks, ORPS reasons at a higher abstraction level about solution strategies. This enables exploring different algorithms rather than being trapped in local optima when the initial approach is suboptimal. Moreover, unlike previous works, we also aim to generate code solutions that are correct, efficient, easy to understand and maintain, instead of focusing solely on correctness.\n3. ORPS effectively operates without access to ground truth test cases from benchmark testsets. ORPS utilizes self-generated unit tests during reasoning - a practical advantage in real-world scenarios where test cases are unavailable. \n\n>  **Q2:** I am a bit confused but wanted to confirm, during inference, does the proposed approach iterative generation? I mean does it generate and then refine the generation?\n\nYes, ORPS is an iterative generation approach. It maintains multiple solution trajectories simultaneously through beam search, where each iteration involves reasoning, code generation, execution, and self-critique. This iterative process allows for strategic pivots when initial approaches prove suboptimal.\n\n> **Q3:** If I understand correct, the proposed method does not require any training, right? It is an inference-only approach?\n\nCorrect, ORPS requires no training whatsoever. It's a fully inference-time framework that leverages an LLM's existing capabilities for reasoning, code generation, and self-critique. Our experiments in Table 4 and additional analyses (in our response to Reviewer 4j6r) confirm that this inference-only approach outperforms methods requiring PRM training, especially as compute budget increases.\n\nThank you again for your supportive review. We believe ORPS represents a significant step forward in unifying process and outcome supervision for complex code generation tasks.", "claims_and_evidence": "Yes, the paper provided evidences to support the claims made in the work. For example, the paper establishes: code generation offers a unique opportunity through concrete, verifiable signals. The paper further shows that eliminating the need for specially trained process reward models help in improving reasoning quality.", "methods_and_evaluation_criteria": "The proposed method is composed of three steps, namely (1) candidate code generation, (2) executing candidate codes and run profiling on unit tests, and (3) self-critic and process rewarding. All these 3 steps make sense for the code generation. The evaluation is performed on three popular code generation benchmarks.", "theoretical_claims": "The paper does not have any proofs or theoretical claims.", "experimental_designs_or_analyses": "The paper performed experiments on three popular programming problem solving benchmarks. The evaluation setup is well detailed and looks sound. Quite a few strong models are evaluated with the proposed approach.", "relation_to_broader_scientific_literature": "It seems like the contributions made in this paper is not highly novel, if we compare to the existing literature. For example, self criticism is not a new idea. Getting execution feedback to get an accurate criticism is not new as well. However, if we judge the proposed method as a whole, it seems quite interesting and effective.", "essential_references_not_discussed": "I didn't carefully check but there seems to be enough paper in the reference.", "other_strengths_and_weaknesses": "The paper is overall well written. The paper has done rigorous experiments which is the strength of the work.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. Does the proposed approach composed of components which already exists in the literature? If yes, how do the authors see the novelty of the work?\n2. I am a bit confused but wanted to confirm, during inference, does the proposed approach iterative generation? I mean does it generate and then refine the generation?\n3. If I understand correct, the proposed method does not require any training, right? It is an inference-only approach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "dEmgQTOkOz", "reviewer": "Reviewer_4j6r", "summary": "This paper introduces Outcome-Refining Process Supervision (ORPS), a method unifying outcome supervision and process supervision in large language models (LLMs) for code generation tasks. The authors propose a tree-structured, inference-only, search framework using a combination of execution-based feedback, self-generated unit tests, and self-critiques to iteratively refine solutions. Empirical evaluations across three code-generation benchmarks (LBPP, HumanEval, MBPP) using different LLM models demonstrate improvements in correctness (Accuracy, or the Pass@1) and code efficiency metrics. The authors compare against LDB and Reflexion and show that the proposed ORPS achieves higher accuracy and more efficient code implementation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We address your concerns point by point:\n## Metrics\nUnlike methods that report Pass@k which attempts k solutions on testset (e.g. LDB and Self-Repair use ground-truth test cases which can be considered contamination), **our Pass@1 measures attempting only 1 solution on testset**. We only generate and execute multiple codes during reasoning on self-generated test cases. The final code (one per problem) is evaluated on the actual test set (unless marked \"w/ T\"). In fact, although baselines like LDB report Pass@k where k solutions are tested against testset cases, they sample much more code and debug on test cases from testset to obtain each solution. Thus it's still a fair comparison if we control the attempts on the full testset cases for all methods. Furthermore, we report metrics beyond Pass@1 as shown in Table 5.\n\n## PRM Training\n\nWe report training details in Appendix A.4. While we used 81 problems for PRM training to avoid contamination, we collected a larger dataset by beam search with 20 reasoning steps each. This generated 5414 steps with GPT-4o labels for training. **To further address your concern on label quality, we trained 2 new line-level PRMs:**\n\n1. PRM-GPT: A larger GPT-4o labeled dataset (13644 steps).\n2. PRM-Human: To obtain better reward labels, we sampled 400 reasoning chains with GPT labels and ask 3 authors to spend 12hrs each, to classify if the chains and rewards are valid. We got an average inter-annotator agreement of 0.44(Cohen's Kappa,AB=0.27,AC=0.61,BC=0.44). We kept 209 valid chains from 261 chains that annotators agree on each other and extracted 836steps for PRM training.\n\n## Cost\nWe add controlled experiments on LBPP with Qwen7B limiting LLM calls(Details in Rebuttal DEgP) with 2 new PRMs and REx[4].\n\n|20 Calls|Pass@1|Tests%|Valid%|Time%|\n|-|-|-|-|-|\n|Reflexion|37.0|51.7|71.6|119.5|\n|LDB|37.0|50.8|66.0|274.7|\n|REx|43.2|57.4|72.2|268.4|\n|PRM-GPT|44.4|58.1|77.8|**100.1**|\n|PRM-Human|40.7|53.1|69.1|124.7|\n|ORPS|**48.4**|**64.8**|**84.5**|105.6|\n\n|50 Calls|Pass@1|Tests%|Valid%|Time%|\n|-|-|-|-|-|\n|Reflexion|40.7|57.6|79.6|130.3|\n|LDB|36.4|50.4|66.0|272.6|\n|REx|53.7|66.6|84.0|199.9|\n|PRM-GPT|37.0|52.9|71.6|**112.7**|\n|PRM-Human|38.3|53.5|69.1|137.7|\n|ORPS|**55.6**|**72.1**|**89.5**|116.8|\n\n|100 Calls|Pass@1|Tests%|Valid%|Time%|\n|-|-|-|-|-|\n|Reflexion|39.5|54.4|72.8|113.8|\n|LDB|37.0|51.0|66.7|275.7|\n|REx|54.3|65.6|79.0|218.4|\n|PRM-GPT|35.8|51.1|70.4|127.3|\n|PRM-Human|42.0|56.4|76.5|106.6|\n|ORPS|**64.2**|**75.4**|**88.9**|**91.0**|\n\nResults show repair-based methods (LDB, Reflexion) fail to scale effectively. LDB focuses on fixing code blocks in a single solution, which works for simple bugs but fails when the algorithmic approach is suboptimal, while ORPS improves significantly. REx may appear advantaged as it generates code every call while others generate code every 2 calls, attempting twice as many solutions given the same calls. **Given the same compute budget, ORPS consistently generates better solutions without training or test cases from testset.**\n\n## Novelty and Contributions\nORPS fundamentally challenges the assumption in process supervision(OmegaPRM,MathShepherd) that specially trained PRMs are necessary for reasoning guidance. Compared to trained PRMs, although increasing training data quality(which is expensive) could lead to improvements, **combining verifiable rewards with existing LLMs without training outperforms trained PRMs as compute increases**. This is significant conceptual advancement for process supervision that eliminates the overhead of PRM training while improving results.\n\nUnlike Self-Repair, REx, LDB that focus on repairing code or specific blocks, ORPS reasons about solution strategies at a higher level to avoid local optima. Instead of incrementally fixing specific solutions to pass more tests, ORPS explores different algorithms with extensive reasoning, even on solutions that pass all generated cases for improvement. \n\nCompared to REx which focus on selecting best nodes to explore by solving arm-acquiring bandit problem from exec outcomes, we propose **LLMs are capable of directly generating high quality process rewards given some feedback to select and expand certain reasoning chains**. In fact REx might be used in conjunction with ORPS process rewards since REx focus on selecting better nodes while we try to produce better rewards but is out of the scope of this work.\n\nBesides, previous methods require testset cases, which is practically hard to obtain(as pointed out by DEgP and LiveCodeBench), while ORPS relies on self-generated cases(**for unit-test generation, please refer to Rebuttal DEgP**).\n\nFor notation issues, we will fix them in revised version. We will also cite, discuss and compare with works you listed that we missed previously,like REx and Self-Repair.\n\n**Given these clarifications and new results demonstrating ORPS's superior performance, we respectfully and strongly request reconsideration of your score.**", "claims_and_evidence": "The paper makes several claims, notably:\n\n1. Structured reasoning guided by execution outcomes significantly improves code generation. Essentially, better textual reasoning leads to better code generation output. \n2. A tree-structured search provides substantial performance gain due to multiple reasoning trajectories being maintained simultaneously.\n3. Combining execution feedback with self-critique mechanisms creates a more reliable verification system than learned reward models. Notably, it showcases that a general model without PRM training is better than a trained PRM model.\n\nClaims 1, 2 are well supported by experiments. Claim 3 is not convincing enough IMHO, despite empirical evidence given in Table 4 for the following reasons:\n- The PRM training procedure is not clearly presented (from L366-370): It's not totally clear whether the line-level reward is collected by pure prompting GPT-4 to give line-level reward or it requires Monte-Carlo rollout, e.g. existing works such as [1] for math reasoning or [2] for code generation, and train another PRM with an modified value head.\n- The PRM training is done on half of the LBPP dataset only, which contains 162 / 2 = 81 problem entries. It seems problematic to me to claim, solely from experiments on such small-scale training, that learned PRM is worse.\n\nMoreover, Claims 1 and 2 are already undermined by existing literature:\n\n1. The central claim that better reasoning improves outcomes has already been well-established by Olausson et al. [3], who demonstrated explicitly that reasoning quality and self-repair capabilities of LLMs correlate strongly with better outcomes in code generation. Notably, they also maintain a tree structure, called repair tree and show that the better quality of textual reasoning on previous failing code, such as coming from a stronger model or even human written, further increase the subsequent code generation performance. \n\n2. Tree-structured search in LLM-based code refinement has been previously explored by Tang et al. [4] through a probabilistic exploration-exploitation approach for iterative code refinement, in which selecting which node in the tree to expand is heavily discussed.\n\nAnother minor claim that worth revisiting is the L161-163 and L68-73 saying that outcom-supervision could lead to inefficient solution such as brute-force solution: the correctness in these benchmarks is determined by unit tests of IO pairs w/ time and memory constraint; and could be avoided by adding in large IO pairs with tight timeout.\n\n[1] OmegaPRM https://arxiv.org/abs/2406.06592\n\n[2] Process Supervision-Guided Policy Optimization for Code Generation https://arxiv.org/abs/2410.17621\n\n[3] Is Self-Repair a Silver Bullet for Code Generation? https://arxiv.org/abs/2306.09896\n\n[4] Code Repair with LLMs gives an Exploration-Exploitation Tradeoff https://arxiv.org/abs/2405.17503", "methods_and_evaluation_criteria": "Benchmark:\n\n- The authors report performance on 3 coding benchmarks, including LBPP, HumanEval, and MBPP. This is a legitimate choice, given the comparison is done against LDB https://arxiv.org/abs/2402.16906 primarily, which reports performance on HumanEval, MBPP. Still, it remains a question about whether the claimed evidence extends to competitive programming benchmarks such as CodeContests and LiveCodeBench. Also, the result could be more solid if the results are reported in an enhanced version benchmark, due to the false positive rate in the original ones, for example HumanEval+ or MBPP+.\n\nMetrics:\n\n- The main metrics reported are solely pass@1. Other code-specific metrics are also reported in Figure 3, which looks interesting. However, the pass@1 in the manuscript is closer to what is defined as \"accuracy\", due to the fact that it overlooks the number of code sampled. This has been discussed in [1] and [2], where both argue that pass@k should consider the number of code sampled in iterative refinement code generation process, or furthermore, a fairer metrics could be pass n@k used in [2, 3, 4] (for example a code generated followed by 2 self-repair attempts should be reported as pass 1@3 instead of pass@1 since 3 code snippets are generated) or pass@tokens in [1].\n\n[1] Is Self-Repair a Silver Bullet for Code Generation? https://arxiv.org/abs/2306.09896\n\n[2] What Makes Large Language Models Reason in (Multi-Turn) Code Generation? https://arxiv.org/abs/2410.08105\n\n[3] Competition-Level Code Generation with AlphaCode https://arxiv.org/abs/2203.07814\n\n[4] RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning https://arxiv.org/abs/2410.02089", "theoretical_claims": "There's no theoretical contribution in the manuscript, as it's a paper with empirical findings.", "experimental_designs_or_analyses": "The overall experiments look valid and sound, in particular for the main claim that ORPS is better than other variants such a LDB and Reflexion.\n\nAs I mentioned before, results in 4.4 are not convincing enough due to the lack of details of PRM and the small scale of PRM training being done. The presented Process Rewarding in Sec 3.3 and in L192 is not the same as Process Reward Model in Table 4, which creates confusion. The Process Rewarding proposed by the authors still operates on a full code snippet.\n\nAlso, the authors mentioned using generated unit tests in L246-247. There's no sufficient amount of details about how the authors generate the unit tests, the amount of the unit tests generated, and also a lack of reference for unit tests generation, which contains a rich literature either it being LLM generated or mutation methods.", "relation_to_broader_scientific_literature": "Please see above.", "essential_references_not_discussed": "Please see above.", "other_strengths_and_weaknesses": "Strengths:\n\n- A clear and practical inference-only framework that integrates execution-based feedback with LLM-driven critiques.\n- Empirical results indicating meaningful improvements in both correctness and efficiency. The including of metrics other than pass rate looks interesting.\n\nWeaknesses:\n\n- There're non-negligible flaws in the claims and the methods part (Please see above). More analysis would be appreciated; for example, the number of LLM calls, or the number of tokens generated, or the number of invocations of code evaluation of ORPS compared to LDB & Reflexion for a fairer comparison rather than just \"Accuracy\". \n- Limited conceptual novelty, as the core ideas (reasoning improvement from an LLM for self-repair, tree search) were previously established. Otherwise clarification from the authors is highly appreciated.\n- Presentation and writing could have been improved. Some important details are missing such as unit tests generations.", "other_comments_or_suggestions": "Notations are not clear, some symbols are never defined:\n\n- What is $j$ in L213?\n- What is $m_k^{j}$ in the equation in L177?\n- How is the weighted step score $q_t$ (L212) used? The beam size is only presented in Algo 1 and seems to be not mentioned in the main text.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents an inference-time methodology for improved accuracy in code generation. Like a lot of work in this space it assumes access to test cases. It achieves state of the art results across different base models. Reviewers are broadly positive, but the one weak objector takes issue with the requirement of ground truth test cases. While this is an important limitation, it does not a limitation that is specific do this paper but one which is specific to this entire family of works.

Therefore I recommend accepting the paper, but I suggest that the authors revise to discuss all of the related work that the reviewers raised: This is a crowded and fast moving area, so I don't fault the authors not discussing everything, but they should spruce up the related work as described by the reviewers.
Decision: Accept (poster)