StepCoder: Improving Code Generation with Reinforcement Learning from Compiler Feedback (2024)

Abstract: The advancement of large language models (LLMs) has significantly propelled the field of code generation. Previous work integrated reinforcement learning (RL) with compiler feedback for exploring the output space of LLMs to enhance code generation quality. However, the lengthy code generated by LLMs in response to complex human requirements makes RL exploration a challenge. Also, since the unit tests may not cover the complicated code, optimizing LLMs by using these unexecuted code snippets is ineffective. To tackle these challenges, we introduce StepCoder, a novel RL framework for code generation, consisting of two main components: CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks, while FGO only optimizes the model by masking the unexecuted code segments to provide Fine-Grained Optimization. In addition, we furthermore construct the APPS+ dataset for RL training, which is manually verified to ensure the correctness of unit tests. Experimental results show that our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks1.

heading: 1 Introduction
text: Code generation or program synthesis aims to automatically generate source code that adheres to a specified programming requirement, which is typically described in natural language (Svyatkovskiy et al., 2020; Gulwani et al., 2017). Recently, with the development of large language models (LLMs), techniques based on LLM (Li et al., 2023a; Luo et al., 2023b) have demonstrated impressive ability
* Equal contributions. † Corresponding authors 1 https://github.com/Ablustrund/APPS_Plus
in code generation. However, challenges persist in aligning these models with complex human requirements (Hendrycks et al., 2021; Roziere et al., 2023), indicating a gap that still exists in fully meeting user expectations.
In this context, learning from compiler feedback exhibits impressive potential to improve the comprehension of complicated human requirements and the quality of generated codes (Le et al., 2022). This feedback from compilation and execution results is instrumental in directly ascertaining the functional correctness of programs (Wang et al., 2022a; Li et al., 2022). Researchers (Liu et al., 2023; Shojaee et al., 2023) introduce reinforcement learning (RL) and leverage compiler feedback from unit tests as a reward metric to guide the exploration of the output space of LLMs. The intention is for the policy model to favor actions that yield higher rewards increasingly. Nevertheless, the optimization of LLMs for code generation via RL presents several hurdles. First, the increasing complexity of human requirements often results in the generation of longer code sequences, which makes exploration struggle (Hao et al., 2023; Ladosz et al., 2022). Second, in cases where a single unit test fails to cover the complex code, unexecuted code snippets may emerge that are not relevant to the reward. Rendering optimization based on the entire code sequence is potentially imprecise. Additionally, our analysis reveals quality limitations in existing datasets like APPS (Hendrycks et al., 2021) for RL training, which impedes accurate learning from compiler feedback through RL.
To tackle these challenges, we first introduce StepCoder, an innovative framework developed for enhancing code generation through reinforcement learning. StepCoder integrates two key components: Curriculum of Code Completion Subtasks
4571
(CCCS) and Fine-Grained Optimization (FGO). CCCS is designed to alleviate the complexities associated with exploration in code generation, while FGO is designed to provide more precise and effective optimization strategies. Specifically, CCCS employs a step-by-step strategy to break down complex exploration problems (i.e., code generation) into a curriculum of easier sub-tasks (i.e., code completion). As the training progresses, the difficulty of code completion tasks rises by increasing the portion of code that needs to be completed. Eventually, the aim is for the model to evolve to a stage where it can effectively generate code solely from human requirements, thus fulfilling the original training goal of code generation. On the other hand, the key insight of FGO is that code snippets that are not executed in a unit test do not contribute to the final reward calculation. Therefore, FGO uses a dynamic masking technique to mask unexecuted snippets from unit test evaluations, ensuring that the model is optimized utilizing only the relevant code segments.
Subsequently, our endeavor involves the development of APPS+, a dataset of superior quality specifically curated for code generation. APPS+ is meticulously designed to exclude code segments that exhibit syntax errors, are irrelevant to the stipulated problem, or fail to produce any output. Additionally, we have taken measures to standardize the format of inputs and outputs in unit tests to guarantee deterministic output comparisons.
We evaluate the effectiveness of popular LLMs on APPS+. The results reveal that although LLMs show progressive improvements, they face difficulties with complex human requirements. We further evaluate our method on several extensively used benchmarks including MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021). The experimental results show that StepCoder effectively eases the exploration difficulty in code generation, outperforming other reinforcement learning-based methods in effectiveness. The main contributions of our paper are as follows:
• We introduce StepCoder, a novelty training method via RL, including CCCS and FGO. CCCS makes exploration easier by breaking down the complicated goals into subobjectives curriculum. FGO provides finegrained optimization by only utilizing the executed code in unit tests.
• We constructed APPS+, a high-quality dataset
designed for code generation. APPS+ provides a more rigorous evaluation of LLMs’ capabilities and a foundation to introduce reinforcement learning in the training phase.
• Experiments show that StepCoder can improve the exploration efficiency and effectiveness and outperform other methods.

heading: 2 Motivation
text: In this section, we clearly illustrate the challenges faced by reinforcement learning in code generation using a simplified example from APPS (Hendrycks et al., 2021), which was widely used for RL training in code generation.
Exploration problems of RL in code generation. In tackling complicated sequence but sparse reward challenges, exploration methods play a crucial role (Yang et al., 2021; Ladosz et al., 2022). When a policy model explores a trajectory with high returns, it undergoes optimization, making it inclined to take similar actions in the future (Williams, 1992; Salimans and Chen, 2018).
Consider the code shown in Figure 1, aimed at fulfilling a given human requirement. We first collect the conditional statements (CS) that are indicated by the dashed box by analyzing its abstract syntax tree. Conditional statement introduces new independent paths, increasing the complexity of the program (Shepperd, 1988). Suppose Pθ(CSi) denotes the probability that the policy model with parameter θ completes the i-th conditional statement. The probability that the policy model correctly generates this code according to human requirements can be expressed as follows:
P ∝ Po 3∏
i=1
Pθ(CSi), (1)
where Po is the probability of other code snippets except the code labeled in the figure. Typically, we initialize the policy model with the SFT model in sequence generation tasks to facilitate easier exploration (Ouyang et al., 2022; Zheng et al., 2023; Wang et al., 2024; Dou et al., 2024). However, the limited performance of the SFT model in code generation still leads to the probability Pθ(CSi) at low values (Shojaee et al., 2023; Roziere et al., 2023). The increasing complexity of human requirements in code generation tasks often leads to a corresponding rise in the number of conditional statements. This escalation can result in a substantial decrease in the probability Pθ(CSi), potentially leading P to an exponential reduction. Such a scenario exacerbates the challenges associated with exploration in large language models. An alternative approach to facilitate exploration is through reward shaping, a technique where designers artificially introduce rewards more frequently (Ladosz et al., 2022). However, in unit test feedback, rewards can only be obtained after the execution of the completely generated code. Consequently, the exploration of high-return trajectories in tasks with complex sequences and sparse rewards poses a significant challenge in optimizing the policy model.
Optimization problems of RL in code generation. We first introduce the RL fine-tuning process in code generation. Formally, for a learned policy model πθ with parameter θ, we treat the prediction of each token as an action a taken by πθ according to the history token sequences. The history token sequences can be viewed as the state s. Given a human requirement x, we denote the solution code y generated by πθ as an episode, and r(x, y) is the reward function from the compiler based on compilation and execution. Updating the parameters
of πθ by using gradient policy algorithm (Sutton et al., 1999) can be represented as follows:
max θ E(x,y)∼Dπθ [ ∑
t
Atπ log(yt|y1:t−1, x; θ)] (2)
where Aπ is the advantage computed by the Generalized Advantage Estimator (GAE) (Schulman et al., 2015) from reward r, to reduce the variability of predictions.
In code generation, rewards are contingent upon the correctness of the unit test sample, which is only relevant to the code snippet being executed, as shown in Figure 1. It indicates that some actions in the code are irrelevant to the reward, which leads to inaccurate advantage. Therefore, optimizing the policy model πθ with all actions is ineffective by using Equation 2.

heading: 3 Method
text: In this section, we elaborate on the methodological details of StepCoder, which provide an easier exploration and fine-grained optimization for RL in code generation, respectively, as shown in Figure 2.

heading: 3.1 Priliminaries
text: Suppose D = {(xi, yi, ui, ei)}Ni=0 is the training dataset for code generation, which x, y, u denotes the human requirement (i.e., the task description), the canonical solution and the unit test samples, respectively. ei = {stj , enj}Eij=0 is a list of conditional statements by automatically analyzing the abstract syntax tree of the canonical solution yi, which st and en represent the start position and the end position of the statements, respectively. e is sorted in ascending order based on the start position st. For a human requirement x, its canonical solution y can be represented as {at}Tt=0. In code generation, given a human requirement x, the final states are the set of codes passing the unit tests u.

heading: 3.2 StepCoder
text: StepCoder integrates two key components: CCCS and FGO. CCCS is designed to break the code generation tasks into a curriculum of the code completion subtasks. It can alleviate the exploration challenge in RL. FGO is specifically designed for code generation tasks to provide fine-grained optimization by computing only the loss of executed code snippets.
CCCS. In code generation, the solution to a complicated human requirement usually involves a long
action sequence taken by the policy model. Meanwhile, the feedback from the compiler is delayed and sparse, i.e., the policy model only receives the reward after generating the entire code. In this scenario, exploring is difficult. The core of our method is to break down such a long sequence of exploration problems into a curriculum of short, easily explorable sub-tasks. We simplify code generation to code completion sub-tasks. These sub-tasks are automatically constructed from the canonical solution in the training dataset.
Consider a human requirement x, early in the training phase of CCCS, the starting point s∗ of exploration is the states near the final states. Specifically, we provide the human requirement x and the front part of the canonical solution xp = {ai}s∗i=0, and the policy model is trained to complete the code based on x ′ = (x, xp). Let ŷ be the combined sequence of xp and the output trajectory τ , i.e. ŷ = (xp, τ ). The reward model provides the reward r according to the correctness of the code snippet τ with ŷ as input, where we use the same setting as previous approaches (Le et al., 2022;
Shojaee et al., 2023) as follows:
r(x, ŷ) =    + 1, if ŷ passed all unit tests −0.3, if ŷ failed any unit test −0.6, if ŷ happened runtime error − 1, if ŷ happened compile error.
(3)
We use the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017) to optimize the policy model πθ by utilizing the reward r and the trajectory τ . In the optimization phase, the canonical solution’s code segment xp used for providing prompts is masked, such that it does not contribute to the gradient for the policy model πθ update. CCCS optimizes the policy model πθ by maximizing the objection function as follows:
Objective(θ) = E(x′ ,τ)∼Dπθ [r(x, ŷ)
− β log(πθ(τ |x ′ ))/πref(τ |x′)] (4)
where πref is the reference model in PPO, which is initialized by the SFT model.
As the training progresses, the starting point s∗
of exploration gradually moves towards the beginning of the canonical solution. Specifically, we set
a threshold ρ for each training sample. Each time the cumulative correct proportion of code segments generated by πθ is greater than ρ, we move the starting point toward the beginning. In the later stages of training, the exploration of our method is equivalent to the exploration process of original reinforcement learning, i.e., s∗ = 0, where the policy model generates code using only human requirements as input.
The starting point s∗ is sampled at the beginning position of the conditional statements to complete the remaining unwritten code segments. Specifically, a program with a greater number of conditional statements results in increased independent paths, leading to a higher logical complexity (Shepperd, 1988). This complexity necessitates more frequent sampling to improve the quality of training, while programs with fewer conditional statements need less frequent sampling. This sampling method allows for a balanced and representative sampling of code structures, catering to both complex and simple semantic constructs in the training dataset. To accelerate the training phase, we set the i-th sample’s number of curricula equal to ⌈√Ei⌉, where Ei is its number of conditional statements. The i-th sample’s stride of the training curriculum is ⌈ Ei⌈√Ei⌉⌉ instead of one.
The key insight of CCCS can be summarized as follows: 1) It is easy to explore from the states near the goal (i.e., final states). 2) Exploring starting from the states distant from the goal is challenging, but it becomes easier when can leverage states that have already learned how to reach the goal.
FGO. The relationship between reward and action in code generation differs from other reinforcement learning tasks such as Atari (Mnih et al., 2015; Lillicrap et al., 2015). In code generation, we can exclude a set of actions irrelevant to computing the rewards in generated code. Specifically, as mentioned in Section 2, for a unit test, the feedback from the compiler relates only to the code snippets being executed. However, in vanilla RL optimization objectives, as shown in Equation 4, all actions of the trajectory are engaged in the computation of the gradient used in the policy update, which is imprecise.
To improve the precision of optimization, we mask actions (i.e., tokens) that are not executed in unit tests when computing the loss for updating the policy model. The full algorithm of CCCS and FGO is detailed in Algorithm 1.

heading: 4 Experiments
text: In this section, we first introduce APPS+, a highquality dataset for code generation by manually verifying based on the APPS dataset. Then, we elaborate on the experiment details and the experimental results.

heading: 4.1 Dataset Preprocessing
text: Reinforcement learning requires an amount of highquality training data. During our investigation, we found that among the currently available opensource datasets, only APPS meets this requirement. However, we found there are incorrect instances, such as missing input, output, or canonical solution, canonical solutions that were uncompileable or unexecutable, and discrepancies in execution output.
To refine the APPS dataset, we excluded instances lacking input, output, or canonical solutions. Then, we standardized the formats of input and output to facilitate the execution and comparison of unit tests. We conducted unit tests and manual analysis for each instance, eliminating those with incomplete or irrelevant code, syntax errors, API misuse, or missing library dependencies. For discrepancies in output, we manually reviewed the problem description, correcting the expected output or eliminating the instance.
Finally, we construct the APPS+ dataset, containing 7,413 instances. Each instance includes a programming problem description, a canonical solution, a function name, unit tests (i.e., inputs and outputs), and starter code (i.e., the beginning part of the canonical solution). Appendix A illustrates an example from APPS+. The top section of the figure shows the problem description, and the section below presents the canonical solution, unit tests, and metadata. Further details of APPS+ are discussed in Appendix B.1.

heading: 4.2 Experiment Details
text: Benchmarks. In our study, we initially evaluated our method and baselines on our pre-processed APPS+ dataset and further assessed it on several widely-used benchmarks in code generation, i.e., MBPP (Mostly Basic Programming Problems) (Austin et al., 2021) and HumanEval (Chen et al., 2021). We evaluate the MBPP and HumanEval benchmark in a zero-shot learning setting which is the same as previous approaches (Le et al., 2022; Shojaee et al., 2023). In this setting, we fine-tune
the models only on the APPS+ dataset and evaluate the code generation performance on MBPP and HumanEval. The detailed description of benchmarks can be found in the Appendix B.1.
Baselines. To verify the effectiveness of StepCoder and evaluate the performance of LLMs on our APPS+ dataset, we consider a wide range of baselines, including StarCoder (Li et al., 2023a), WizardCoder (Luo et al., 2023b), DeepSeek-Coder (Guo et al., 2024), and three versions of CodeLlama (Base, Python, Instruct) (Roziere et al., 2023). Moreover, we also consider vanilla PPO and two state-of-the-art RL-based approaches, including PPOCoder (Shojaee et al., 2023) and RLTF (Liu et al., 2023). We carried out experiments applying these methods utilizing the same backbone (i.e., DeepSeek-Coder-Instruct (Guo et al., 2024)) on the APPS+ dataset to ensure a fair comparison. In addition to demonstrating the necessity and effectiveness of our method, we also supervised fine-tuning DeepSeek-Coder-Instruct (Guo et al., 2024) on the APPS+ dataset to exclude the effect of training data. The detailed description of these baselines is discussed in Appendix B.2.
Implementation Details. During the SFT phase, we adopt a learning rate set at 2e−5, conduct training for three epochs, and employ a warm-up period of 0.3 epochs, with a linear decay to zero. The fine-
tuning process was conducted on a device with eight NVIDIA A100 80G GPUs, with the global batch size set to 64. In the PPO training phase, we employ a learning rate of 5e−7 for the policy model and 1.5e−6 for the critic model. For each example, we collect a 16 roll-out code using nucleus sampling. The sampling temperature is set to 0.8, top-p is set to 0.9, and the maximum output token length is set to 1024. The token-level KL penalty coefficient β is set to 0.05, with a clip value of 0.8. In the decoding phase, the temperature and top-p are set to 0.2 and 0.95, respectively.
Evaluation & Metric. Our experiments and reward collection for reinforcement learning (RL) methods are conducted using Python3.x. Following prior studies (Roziere et al., 2023; Luo et al., 2023b; Le et al., 2022), we use Pass@k (Chen et al., 2021) metric to evaluate all the models. Pass@k quantifies the proportion of instances in which at least one of the k-generated code solutions per human requirement successfully passes all unit tests. Code generation prompts are detailed in Appendix D.

heading: 4.3 Experimental Results on APPS+
text: To assess the performance of widely used LLMs and our StepCoder on code generation, we conduct experiments on the APPS+ dataset that we constructed. APPS+ was categorized into three
difficulty levels: Introductory, Interview, and Competition. The experimental results are illustrated in Table 1. The results indicate that RL-based models outperform both base models and SFT models. It is reasonable to infer that reinforcement learning can further enhance the quality of code generation by more effectively navigating the model’s output space, guided by compiler feedback.
Furthermore, our StepCoder surpasses all baseline models including other RL-based approaches, achieving the highest score. Specifically, our approach obtains 59.7%, 23.5%, and 8.6% in the ‘Introductory’, ‘Interview’, and ‘Competition’, respectively. Our approach excels in exploring the output space compared to other RL-based methods, achieved by simplifying complex code generation tasks to code completion sub-tasks. We also found that the performance of StepCoder is better than LLM which supervised fine-tuning on the APPS+ dataset based on the same backbone. The latter did little to improve the pass rate of the generated code compared with the backbone. This also directly demonstrates that the method of using compiler feedback to optimize the model improves the quality of the generated code better than next-token prediction in code generation.

heading: 4.4 Ablation Studies
text: To investigate the impact of individual components in StepCoder, we conducted ablation experiments with two variations of our approach, including StepCoder only with CCCS and only with FGO, as shown in Table 1. Experimental results demonstrate that both components of our approach improve the quality of the generated code compared to vanilla PPO. CCCS can enhance its performance in addressing Competition-level problems. This improvement is logical, considering that CCCS effectively simplifies the exploration of more complex
human requirements. Simultaneously, FGO boosts the pass rate of unit tests by integrating compiler feedback with the relevant executed code snippet.

heading: 4.5 Results on MBPP and HumanEval
text: To further demonstrate the effectiveness of our method, we conducted comparative analyses of StepCoder against various approaches using the well-recognized benchmarks MBPP and HumanEval. These models are trained on APPS+ and then evaluated on MBPP and HumanEval. The experimental results are illustrated in Table 2 which shows that StepCoder is superior over all other models on both benchmarks.
However, there are concerns regarding potential overlaps in the training data between APPS+ and the two benchmarks, which might contribute to an improvement in performance. To address these concerns, we analyze the difference between APPS+ and the benchmarks by calculating the code line overlap ratio of two corresponding canonical solutions following previous work (Austin et al., 2021; Le et al., 2022). The findings are presented in Figure 3. This evidence underscores our approach’s effectiveness in enhancing the quality of generated code and its capability across a broad spectrum of code generation tasks, primarily by improving the exploration problem in reinforcement learning.
Meanwhile, our findings revealed a significant degradation in the performance of the SFT model on both MBPP and HumanEval benchmarks. Further analysis of the error cases showed that a minority were related to function name errors, while the majority were associated with program correctness errors. This also indicated that SFT on a single
dataset may impair the ability to follow instructions and the ability to generalize, thus affecting the performance of code generation on other tasks. In contrast, RL-based methods can improve the performance for unseen tasks of code generation.

heading: 4.6 Analysis by Unit Test Results
text: We further analyzed the results of cases that did not pass all unit tests, as shown in Figure 4. The results show that our method can effectively reduce the likelihood of compilation errors, which is particularly evident in Interview-level and Competitionlevel programming problems. However, it was also observed that all LLMs are more prone to runtime errors and failures as compared to compilation errors, albeit StepCoder shows a comparatively lower rate of runtime errors and failures. These results demonstrate that StepCoder is less prone to compilation errors, but still suffers from runtime errors and failure. These findings suggest that future research should further focus on minimizing runtime errors to improve code quality and pass rates.

heading: 5 Related Work
text: Large Language Models for Code Generation. Recently, pre-trained language models have shown remarkable ability in understanding natural language and code generation by training on large text corpora containing code data (Christopoulou et al., 2022; Li et al., 2023b). In addition, SFT models achieve more competitive performance such as StarCoder (Li et al., 2023a), WizardCoder (Luo et al., 2023b), Code Llama Instruct (Roziere et al., 2023), and DeepSeek-Coder (Guo et al., 2024).
Reinforcement Learning is a method of learning the optimal policy by exploring the environment and obtaining rewards (Williams, 1992; Sutton et al., 1998). Recently, some researchers have introduced RL to LLMs and improved the quality of the generated code by utilizing the unit test feedback to explore the output space of the pol-
icy model. For instance, CodeRL (Le et al., 2022) leverages unit test signals for rewards and employs actor-critic methods (Konda and Tsitsiklis, 1999; Sutton et al., 1999) to enhance models on code generation. PPOCoder (Shojaee et al., 2023) refines CodeRL by employing the PPO algorithm (Schulman et al., 2017) and RLTF (Liu et al., 2023) provides fine-grained rewards through the error locations, but the reward space is still sparse. However, the exploration of complex tasks in an environment characterized by a sparse reward is challenging, limiting the effectiveness of RL in boosting code generation model performance
Exploration in Reinforcement Learning. Exploration is crucial in addressing long sequences and sparse reward problems (Hao et al., 2023; Ladosz et al., 2022). In the sequence generation task, researchers improved exploration by initializing the policy model using the SFT model (Ouyang et al., 2022; Shen et al., 2023). Our proposed approach incorporates similar methods, but additional methods are necessary to ensure effective exploration, especially when tackling complex human-driven requirements, where the limited quality of generated code makes exploration still challenging.
Other notable methods introduce the ProcessSupervised Reward Model to provide step-by-step rewards for complex sequence generation tasks such as mathematical reasoning and code generation (Uesato et al., 2022; Lightman et al., 2023; Luo et al., 2023a; Ma et al., 2023). However, these methods require labelling a large preference dataset to train the reward model. Similar to our approach, some methods construct a learning curriculum by initiating each episode from a sequence of progressively more challenging starting states (Salimans and Chen, 2018; Florensa et al., 2017). In contrast to our approach, these methods are designed to address the problem of exploration in other fields, such as gaming and robotic manipulation. Meanwhile, our approach combines software engineering features to dynamically determine the starting states through conditional statements and introduces FGO to enhance fine-grained optimization with the coverage information.

heading: 6 Conclusion
text: In this paper, we introduce StepCoder, a novel training framework via Reinforcement Learning (RL). StepCoder breaks down complicated exploration problems to reduce the difficulty of exploring en-
vironments with sparse rewards while providing fine-grained optimization. In addition, we also construct a high-quality dataset APPS+, specifically for code generation. Experiments indicate that our method can effectively improve the quality of generated code via RL compared to other approaches.

heading: 7 Limitations
text: In this section, we discuss the potential limitations of the APPS+ dataset and our proposed method StepCoder. Firstly, while the APPS+ dataset we developed stands as a vital resource for code generation tasks, we only provided three manually verified unit tests for each instance (i.e., the same unit test number with MBPP) due to time and manpower constraints. We plan to increase the number of unit tests in the future, aiming for an average of over 10 unit tests per instance. Secondly, despite our method’s outstanding performance by breaking down complicated goals into sub-objectives curriculum and fine-grained optimization, it requires more training time compared to the traditional PPO algorithm. We expect a more time-efficient method while generating the higher-quality code. We leave these problems to future work.

heading: 8 Acknowledgements
text: The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.62441602,62206057,62076069), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (23ZR1403500).

heading: B Experiments Setup in Detail
text: In this section, we elaborate in detail on the baselines we compare and the implementation details of our method.
B.1 Benchmarks APPS+. We construct the new benchmark APPS+ by refining the popular benchmark APPS (Hendrycks et al., 2021). The APPS dataset consists of problems collected from different openaccess coding websites such as Codeforces, Kattis, and more. Codeforces, Kattis, and more APPS+ was categorized into three difficulty levels: Introductory (2,850), Interview (4,020), and Competition (586). The mean length of each problem is 255.3 words, and that of the code is 21.9 lines. On average, each instance is accompanied by three unit tests and includes a ‘conditional statement’ attribute representing the start and end position of the statement in the canonical solution. We randomly selected about 25% instances (700 Introductory, 1,000 Interview, and 140 Competition) for the validation dataset and another 25% instances for the test dataset.
MBPP. MBPP (Austin et al., 2021) is a smaller but common Python code generation benchmark. It contains 974 instances created by crowd-sourcing to an internal pool of crowd workers with basic Python knowledge. The difficulty level of the problems in this dataset is introductory. Most problems are often conveyed in a single sentence of natural language, and each problem consists of a task description, code solution, and three automated test cases. We evaluate LLMs in a zero-shot learning setting which is the same as previous studies (Le et al., 2022; Shojaee et al., 2023). In this setting, we fine-tune models only based on the APPS+ dataset and evaluate them on MBPP.
HumanEval. HumanEval (Chen et al., 2021) is another extensively used benchmark for evaluating the ability of code generation. It comprises 164 hand-written Python problems that test language comprehension, algorithmic thinking, and basic mathematics. The complexity of these problems is akin to that of simple software interview questions. We also evaluate models on the HumanEval benchmark in a zero-shot learning setting.
B.2 Baselines StarCoder. StarCoder (Li et al., 2023a) is a 15.5B parameter model trained on 80+ programming languages sourced from GitHub, encompassing one trillion tokens. It undergoes fine-tuning specifically for 35 billion Python tokens, enabling its proficiency across a diverse set of coding tasks. With an extended context length of 8K, StarCoder excels particularly in infilling capabilities.
CodeLlama. CodeLlama (Roziere et al., 2023) is a collection of pre-trained and fine-tuned generative text models ranging in scale from 7B to 34B parameters. CodeLlama comes in three variants: CodeLlama: base models designed for general code synthesis and understanding; CodeLlamaPython: designed specifically to handle the Python programming language; CodeLlama-Instruct: for instruction following and safer deployment.
WizardCoder. WizardCoder (Luo et al., 2023b) is fine-tuned by using a complicated dataset which is constructed by adapting the Evol-Instruct (Xu et al., 2023) on code-related tasks, which is a further improvement of self-instruct method (Wang et al., 2022b). It has proven to be highly effective in code generation by fine-tuning more complex instruction data.
DeepSeek-Coder. DeepSeek-Coder (Guo et al., 2024) demonstrates state-of-the-art performance
among open-source code models across various programming languages. It encompasses a collection of code language models from 1B to 33B trained from scratch. The training corpus for these models comprises an impressive 2 trillion tokens which is the combination of code and natural languages. Each model is trained to utilize a window size of 16K, and a fill-in-the-blank task is incorporated into the training process, which enhances the models’ capacity to facilitate code completion and infilling tasks.
PPOCoder. PPOCoder (Shojaee et al., 2023) initially employs the Proximal Policy Optimization algorithm (Schulman et al., 2017) for code generations. In addition, it integrates discrete compiler feedback with syntax and semantics matching scores between generated code and executable objectives which reduces the sparsity of the reward function, thereby providing better guidance for generating code that aligns more closely with the correct objectives.
RLTF. RLTF (Liu et al., 2023) features real-time data generation during the training process and multi-granularity unit test feedback. Except for the discrete compiler feedback, it penalizes specific sections in the code where errors occur through the error locations from the feedback of unit tests.

heading: C The algorithm of CCCS and FGO
text: The full algorithm of StepCoder is detailed in Algorithm 1.

heading: D The prompts used for code generation
text: For DeepSeek-Coder-Instruct (Guo et al., 2024), we use the same prompt as the previous paper. Moreover, DeepSeek-Coder-Instruct serves as the backbone model for PPOCoder (Shojaee et al., 2023), RLTF (Liu et al., 2023), and our proposed StepCoder. Consequently, we align the prompts for these RL-based approaches with the prompt of DeepSeek-Coder-Instruct to maintain consistency.
Algorithm 1 StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback Require: the train dataset D = {(xi, yi, ui, ei), 1 ≤ i ≤ n}, the threshold value ρt for curriculum
training. Require: the policy model πθ
1: Initialize the stride of curriculum s = ⌈ Ei⌈√Ei⌉⌉ for each sample 2: Initialize the current curriculum c = ⌈√Ei⌉ − 1 for each training sample 3: Initialize the pass rate ρ = 0 for each training sample 4: while TRUE do 5: Initialize mini-batch Ds = {} 6: Get latest policy model πθ 7: Sample a mini-batch of size M from D 8: for i in 0, · · · , M − 1 do ▷ Begin to sample the trajectories 9: Calculate the start position pos = si ∗ ci ▷ CCCS
10: Reorganize the given context x ′ i = xi + yi [: pos] 11: Sample trajectory ŷi ← πθ(.|x ′ i) 12: Compute reward ri using Equation 3 13: Calculate unexecuted snippets’ mask matrix mij = [1 if ŷ j i is executed else 0] ▷ FGO 14: Add {x′i, ŷi, ui, ri, si, ci,mi} to mini-batch Ds 15: end for 16: θ ← A(θ,Ds) ▷ Update the policy model by PPO algorithm 17: for i in 0, · · · , M − 1 do 18: if ri = 1 then ▷ Update pass rate using moving average 19: ρi = α+ (1− α) ∗ ρi 20: else 21: ρi = (1− α) ∗ ρi 22: end if 23: if ρi > ρt then ▷ Meet the update conditions, proceed to the next stage 24: ρi = 0 25: ci = min(ci − 1, 0) 26: end if 27: end for 28: end while
The prompt used for other models such as CodeLlama, WizardCoder and StarCoder is the same as in previous studies (Contributors, 2023; Luo et al., 2023b; Li et al., 2023a; Roziere et al., 2023).
The prompt used for DeepSeek-Coder-Instruct and LLMs based on it is as follows: You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer. ### Instruction: write an algorithm in python: {Task description} ### Response:
E Instances
Here we showcase an instance from the test dataset to demonstrate the effectiveness of our proposed method. Initially, the original model (i.e., DeepSeek-Coder-Instruct-6.7B (Guo et al., 2024)) failed to generate the code that satisfies the human requirement. On the contrary, after applying the StepCoder optimization, the model successfully generates the code according to the human requirement. The instance from the test dataset is shown as Figure 6:
In this instance, the human requirement is that, given an integer sequence of length n and an integer d, find the length of the shortest non-empty contiguous subsequence with a sum of elements at least d. If there is no such subsequence, return −1. The solution given by the original model has the following issues:
• Although the second loop attempts to find whether there is a subsequence that satisfies the condition, the logic is flawed. For the first unit test case, despite the existence of a qualifying subsequence, the model erroneously returns −1 instead of the correct output, which is 2.
• The final nested “for” loop aims to identify all possible subsequence lengths, but this method does not ensure finding the shortest subsequence. This arises because the inner loop does not check the length of the subsequence and exits as soon as a satisfying subsequence is found. As a result, in the second unit test
case, the model fails to output the correct answer (i.e., the output is 5 but the correct answer is 1).
In contrast, the model optimized by Stepcoder gives the correct output:
• It correctly handles edge cases, reliably outputting −1 when no subsequence satisfies the condition. The underlying logic is clearer and more concise.
• It checks the subsequence’s length to ensure finding the shortest one and uses binary search to achieve a quick search.

heading: Future Work
text: We plan to increase the number of unit tests in the future, aiming for an average of over 10 unit tests per instance. Secondly, despite our method’s outstanding performance by breaking down complicated goals into sub-objectives curriculum and fine-grained optimization, it requires more training time compared to the traditional PPO algorithm. We expect a more time-efficient method while generating the higher-quality code. We leave these problems to future work