ICML 2025

Reasoning Through Execution: Unifying Process and
Outcome Rewards for Code Generation
Zhuohao Yu 1 * Weizheng Gu 1 * Yidong Wang 1 Xingru Jiang 1 Zhengran Zeng 1
Jindong Wang 2 Wei Ye 1 Shikun Zhang 1
Abstract
Large Language Models excel at code genera-
tion yet struggle with complex programming tasks
that demand sophisticated reasoning. To bridge
this gap, traditional process supervision relies
on learned reward models requiring costly train-
ing data and suffering from reward misalignment,
while outcome supervision fails for complex tasks
needing coordinated intermediate steps. We in-
troduce Outcome Refining Process Supervision,
which unifies process and outcome supervision
by leveraging executable verification: a tree-
structured search framework generates strategic
alternatives, profiles execution metrics, and scores
candidates via self-critique mechanisms that in-
tegrate runtime feedback with reasoning. Exper-
iments across 5 models and 3 benchmarks show
consistent gains, with 26.9% higher correctness
and 42.2% improved code efficiency. The results
demonstrate that ORPS enables LLMs to over-
come local optima in code generation, suggest-
ing a promising direction for combining verifi-
able outcomes with structured reasoning to tackle
complex challenges. We open-source at: https:
//github.com/zhuohaoyu/ORPS
1. Introduction
Large Language Models (LLMs) have revolutionized code
generation through their ability to synthesize programs from
natural language specifications (Brown et al., 2020; Guo
et al., 2024). However, complex programming tasks requir-
ing multi-step algorithmic reasoning—such as implement-
ing dynamic programming solutions or optimizing paral-
lel computation patterns—remain challenging (Jiang et al.,
2024b; Jimenez et al., 2023). These limitations persist
1Peking University, Beijing, China 2William & Mary, VA, USA.
Correspondence to: Wei Ye <wye@pku.edu.cn>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada...