StepCoder: Improve Code Generation with Reinforcement Learning from
  Compiler Feedback
Authors: name: Shihan Dou; name: Yan Liu; name: Haoxiang Jia; name: Limao Xiong; name: Enyu Zhou; name: Junjie Shan; name: Caishuang Huang; name: Wei Shen; name: Xiaoran Fan; name: Zhiheng Xi; name: Yuhao Zhou; name: Tao Ji; name: Rui Zheng; name: Qi Zhang; name: Xuanjing Huang; name: Tao Gui
Published: 2024-02-02 13:14:31

Abstract: The advancement of large language models (LLMs) has significantly propelled
the field of code generation. Previous work integrated reinforcement learning
(RL) with compiler feedback for exploring the output space of LLMs to enhance
code generation quality. However, the lengthy code generated by LLMs in
response to complex human requirements makes RL exploration a challenge. Also,
since the unit tests may not cover the complicated code, optimizing LLMs by
using these unexecuted code snippets is ineffective. To tackle these
challenges, we introduce StepCoder, a novel RL framework for code generation,
consisting of two main components: CCCS addresses the exploration challenge by
breaking the long sequences code generation task into a Curriculum of Code
Completion Subtasks, while FGO only optimizes the model by masking the
unexecuted code segments to provide Fine-Grained Optimization. In addition, we
furthermore construct the APPS+ dataset for RL training, which is manually
verified to ensure the correctness of unit tests. Experimental results show
that our method improves the ability to explore the output space and
outperforms state-of-the-art approaches in corresponding benchmarks.