Multi-Turn Code Generation Through Single-Step Rewards
Authors: name: Arnav Kumar Jain; name: Gonzalo Gonzalez-Pumariega; name: Wayne Chen; name: Alexander M Rush; name: Wenting Zhao; name: Sanjiban Choudhury
Published: 2025-02-27 18:55:05

Abstract: We address the problem of code generation from multi-turn execution feedback.
Existing methods either generate code without feedback or use complex,
hierarchical reinforcement learning to optimize multi-turn rewards. We propose
a simple yet scalable approach, muCode, that solves multi-turn code
generation using only single-step rewards. Our key insight is that code
generation is a one-step recoverable MDP, where the correct code can be
recovered from any intermediate code state in a single turn. muCode
iteratively trains both a generator to provide code solutions conditioned on
multi-turn execution feedback and a verifier to score the newly generated code.
Experimental evaluations show that our approach achieves significant
improvements over the state-of-the-art baselines. We provide analysis of the
design choices of the reward models and policy, and show the efficacy of
muCode at utilizing the execution feedback. Our code is available at
https://github.com/portal-cornell/muCode.