ACECODER: Acing Coder RL via Automated Test-Case Synthesis
Authors: Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, Wenhu Chen
Accepted at: acl25

ACECODER: Acing Coder RL via Automated Test-Case Synthesis

Huaye Zeng†*, Dongfu Jiang†*, Haozhe Wang‡, Ping Nie§, Xiaotong Chen¶, Wenhui Chen†
†University of Waterloo, ‡HKUST, §Independent Researcher, ¶Netmind.AI
{w33zeng,dongfu.jiang,wenhuchen}@uwaterloo.ca

https://tiger-ai-lab.github.io/AceCoder

Abstract

Most progress in recent coder models has been driven by supervised fine-tuning (SFT), while the potential of reinforcement learning (RL) remains largely unexplored, primarily due to the lack of reliable reward data/model in the code domain. In this paper, we address this challenge by leveraging automated large-scale test-case synthesis to enhance code model training. Specifically, we design a pipeline that generates extensive (question, test-cases) pairs from existing code data. Using these test cases, we construct preference pairs based on pass rates over sampled programs to train reward models with Bradley-Terry loss. It shows an average of 10-point improvement for Llama-3.1-8B-Ins and 5-point improvement for Qwen2.5-Coder-7B-Ins through best-of-32 sampling, making the 7B model on par with 236B DeepSeek-V2.5. Furthermore, we conduct reinforcement learning with both reward models and test-case pass rewards, leading to consistent improvements across HumanEval, MBPP, BigCodeBench, and LiveCodeBench (V4). Notably, we follow the RL-style training to start from Qwen2.5-Coder-base directly and show that our RL training can improve model on HumanEval-plus by over 25% and MBPP-plus by 6% for merely 80 optimization steps. We believe our results highlight the huge potential of reinforcement learning in coder models.

1 Introduction

In recent years, code generation models have advanced significantly with compute scaling (Kaplan et al., 2020) and training data quality improvement (Huang et al., 2024; Lozhkov et al., 2024; Guo et al., 2024b). The state-of-the-art coder models, including Code-Llama (Roziere et al., 2023), Qwen2.5-Coder (Hui et al., 2024a), DeepSeek-Coder (Guo et al., 2024a) and so on, have shown unprecedented performance across a wide range of coding tasks like program synthesis (Chen et al., 2021), program repair (Zheng et al., 2024a), optimization (Shypula et al., 2023), test generation (Steenhoek et al., 2023), SQL (Yu et al., 2018), and issue fix (Jimenez et al., 2024). These models are all pre-trained and further supervised fine-tuned (SFT) on large-scale coding data from web resources like Common Crawl or Github.

Though strong performance has been achieved through SFT (Luo et al., 2023; Wei et al., 2024), very few models have explored the potential of reinforcement learning (RL) (Ouyang et al., 2022a), which has proven effective in other domains such as mathematical reasoning like DeepSeek-R1 (Shao et al., 2024). We argue that this absence of RL-

<table>
  <tr>
    <th>Seed Code Dataset</th>
    <th>Prompt:</th>
    <th>Program:</th>
  </tr>
  <tr>
    <td></td>
    <td>i've got this python code from an ocr tool but I could not find the best ...</td>
    <td>import collections<br>class Solution:<br>...</td>
  </tr>
  <tr>
    <th rowspan="2">Rewrite</th>
    <th>Leetcode-like Prompt:</th>
    <th>Tests:</th>
  </tr>
  <tr>
    <td>You are given a grid represented as a 2D integer...</td>
    <td><span style="color:red;">assert</span> findShortestPath((1, 2), (3, 4)) = 8<br>findShortestPath((2, 2), (4, 8)) = 12</td>
  </tr>
  <tr>
    <th rowspan="2">Filter Test Cases</th>
    <th>Leetcode-like Prompt:</th>
    <th>Tests:</th>
  </tr>
  <tr>
    <td>You are given a grid represented as a 2D integer...</td>
    <td><span style="color:red;">assert</span> findShortestPath((1, 2), (3, 4)) = 8<br>findShortestPath((2, 2), (4, 8)) = 12</td>
  </tr>
  <tr>
    <th colspan="3">Test Case Pass Rate:</th>
  </tr>
  <tr>
    <td>Pairs Generation</td>
    <td>pos 1.00</td>
    <td>neg 0.80 0.2</td>
  </tr>
</table>

Figure 1: Overall Workflow of our model: we start from the seed code dataset to create well-formatted questions and corresponding test cases. Then we adopt strong models to filter the noisy test cases. Finally, we adopt these test cases to harvest positive and negative program pairs for reward model training and RL.
based training in coder models is primarily due to two key challenges:

(1) Lack of reliable reward signals for code generation. In tasks such as mathematical problem-solving, rewards can be easily derived from rule-based string matches with reference answers (Guo et al., 2025) or large-scale human annotations (Ouyang et al., 2022b). In contrast, evaluating code quality typically requires executing test cases to measure the pass rate, making reward signal design more complex. This also explains why existing reward models like Skywork (Liu et al., 2024a) can hardly generalize to the coding domain (see subsection 3.4).

(2) Scarcity of large-scale coding datasets with reliable test cases. Most existing coding datasets like APPS (Hendrycks et al., 2021; Chen et al., 2021) heavily rely on costly human expert annotations for test cases, which limits their scalability for training purposes. The largest dataset is TACO (Li et al., 2023), containing 25K examples collected from popular coding competition websites, sources that have already been extensively utilized during the pre-training phase.

Therefore, we curate ACECODE-87K, on which we trained our reward models: ACECODE-RM-7B and ACECODE-RM-32B. Comprehensive experiments of best-of-N sampling show that ACECODE-RM can significantly boost existing LLM’s performance on coding benchmarks. For example, ACECODE-RM-7B can improve the performance of Llama-3.1-8B-Instruct by an average of 8.4 points across the 4 coding benchmarks, i.e. HumanEval (Liu et al., 2023), MBPP (Liu et al., 2023), BigCodeBench (Zhuo et al., 2024) and LiveCodeBench (Jain et al., 2024). Even for the stronger coder model Qwen2.5-Coder-7B-Instruct, our “7B+7B” combination still gets an average of 2.6 improvements. ACECODE-RM-32B is even more powerful, which pushes the former two numbers to 10.7 and 4.7 respectively, showcasing the effectiveness of ACECODE-RM.

Additionally, both ACECODE-RM-7B and ACECODE-RM-32B models demonstrate strong performance on the RM Bench (Liu et al., 2024c), outperforming existing state-of-the-art reward models across various categories. Specifically, ACECODE-RM-32B achieves the highest average score of 76.1, leading in key categories such as Coding, Chat, Normal, and Hard tasks. Even the smaller ACECODE-RM-7B model shows notable competitiveness, excelling particularly in Coding and Hard categories.

Furthermore, we adopt ACECODE-RM-7B and test case pass rate separately to do reinforcement learning with reinforce++ (Hu, 2025) over coder models. Experiments show 2.1 and 0.7 points of average improvement when starting from Qwen2.5-7B-Ins and the Qwen2.5-Coder-7B-Ins respectively, making the latter even more powerful than GPT-4-Turbo on benchmarks like MBPP. Inspired by the recent DeepSeek-R1 (Guo et al., 2025), we also perform RL training directly from the Qwen2.5-Coder-7B-base model and saw a surprising 25% improvement on HumanEval-plus and 6% improvement on MBPP-plus (Liu et al., 2023) with merely 80 optimization steps (48 H100 GPU hours). These improvements are also generalizable to other more difficult benchmarks.

To our knowledge, this is the first work to perform reward model training and reinforcement learning for code generation using a fully automated pipeline that synthesizes large-scale reliable tests. We believe our ACECODE-87K will unlock the potential of RL training for code generation models and help the community to further push the boundaries of LLM’s coding abilities.

2 Methodology

In this section, we will introduce the overall methodology of ACECODER. We begin with formulations of the problems we are investigating, including reward model training and reinforcement learning for LLMs. Finally, we elaborate on how we synthesize the test cases and construct the ACECODE-87K.

2.1 Problem Formulation

Reward Model Training Let \( x \) denote the coding question and \( y = \{y_1, \cdots, y_t\} \) denote the program solution, where \( y_i \) represents the \( i \)-th token of the program solution and \( (x, y) \in D \). Assuming \( \theta \) represents the parameters of the model, then \( n \) responses (\( y^1, ..., y^n \)) will be sampled from the model \( \pi_\theta \) given the input \( x \). Let \( (s_1, ..., s_n) \) be the target rewards, i.e. the test case pass rates in our scenario, then we define the Bradley-Terry loss (Bradley and Terry, 1952) for every pair of responses \( y^i \) and \( y^j \) with scores of \( s_i \) and \( s_j \) when we are training a reward model \( R_\phi \) as follows:

\[
\mathcal{L}_\phi(x, s_i, s_j)
= \mathbb{1}[s_i > s_j] \log \sigma(R_\phi(x, y^i) - R_\phi(x, y^j))
\]
where \( \mathbb{1}[\cdot] = 1 \) if the expression inside the brackets is true, otherwise, it’s 0. The final loss function for the reward training is:

\[
\mathcal{L}(\phi) = -\frac{1}{n(n-1)} \sum_{i=1}^n \sum_{j=1}^n \mathcal{L}_\phi(x, s_i, s_j)
\]

That means the reward model is trained to assign higher values to preferred responses and lower values to non-preferred ones, maximizing the difference between these ratings.

Best-of-N Sampling  After we get the trained reward model \( R_\phi \), one way to quickly test the performance of the reward model is Best-of-N sampling, which is usually used as a test-time scaling approach. We will simply select the highest-scored response according to the predicted value of \( R_\phi \). That is \( y^* = \arg\max_{y' \in y^1, ..., y^N} R_\phi(x, y') \).

Reinforcement Learning  We can finally conduct reinforcement learning for the original policy model \( \pi_\theta \) after we get a well-trained reward model \( R_\phi \). Proximal Policy Optimization (PPO) is an actor-critic RL algorithm that is widely used for LLM’s RL process. Let \( \pi_{\theta_{old}} \) be the reference model and \( \pi_\theta \) be the current policy model that is iteratively updated during the RL training. We denote \( r_t(\theta) \) as the probability ratio of the current policy model over the old policy model on the \( t \)-th generated token:

\[
r_t(\theta) = \frac{\pi_\theta(y_t|x, y_{<t})}{\pi_{\theta_{old}}(y_t|x, y_{<t})}
\]

Then the PPO algorithms optimize the LLM by the following surrogate objective:

\[
\begin{align*}
&\mathcal{L}_{PPO}(\theta) = \\
&-\frac{1}{|\mathcal{Y}|} \sum_{t=1}^{|\mathcal{Y}|} \min \left[ r_t(\theta) A_t, \operatorname{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t \right]
\end{align*}
\]

where \( y \sim \pi_{\theta_{old}}(\cdot|x) \), and \( A_t \) is the advantage computed through the Generalized Advantage Estimation (GAE) (Schulman et al., 2015) via the rewards generated by \( R_\phi \) and the learned value function \( V_\psi \). The PPO training objective will force the policy model \( \pi \) to increase the probability of generating tokens with higher \( A_t \) and decrease the probability ratio of generating tokens with lower \( A_t \) until the clipped bounds \( 1 + \epsilon \) and \( 1 - \epsilon \) are reached, respectively.

However, PPO usually requires training an additional value model \( V_\psi \) and thus makes the training inefficient. Recently, there have been some other works like Reinforcement++ (Hu, 2025) that eliminate the need for a value model but instead compute advantage only using the rewards generated by \( R_\phi \) and the KL-divergence of the tokens after the \( t \)-th token. This makes the RL process more efficient and has also proved to be more stable.

2.2 AceCode-87K

To be able to train a reward model specifically designed for code generation, the first step is to synthesize reliable test cases for each coding problem and use them as training signals. In this section, we explain the whole procedure of constructing AceCode-87K step by step. We show the overall statistics in Table 1.

Test Case Synthesis from Seed Dataset  We start from existing coding datasets with provided question \( x \) and corresponding program \( y \). Specifically, we combine Magicoder-Evol-Instruct\( ^1 \), Magicoder-OSS-Instruct-75K\( ^2 \), and StackPyFunction\( ^3 \) as our seed dataset. We only keep the questions written in Python that contain either a function or a class, resulting in a total of 124K entries. We find that these datasets contain highly noisy questions that cannot be easily evaluated using test cases. Therefore, we feed every question-solution pair (x, y) into a GPT-4o-mini (Hurst et al., 2024) to propose a refined LeetCode-style question \( x_r \) with highly structured instructions. Meanwhile, we also prompt it to ‘imagine’ around 20 test cases (\( t_1, ..., t_m \)) for each refined coding question \( x_r \) based on its understanding of the expected behavior of the desired program. See prompt template used in subsection A.3. Please note that we do not use the program solution \( y \) from the existing datasets at all in our final curated AceCode-87K. These datasets are purely used as seeds to help LLM formulate well-structured coding problems.

Test Case Filtering  These ‘imagined’ test cases generated from the LLM contain severe hallucinations. To filter out those hallucinated test cases, we facilitated a stronger coder model Qwen2.5-Coder-32B-Instruct (Hui et al., 2024a) as a proxy to perform quality control. Specifically, we prompt it for each \( x_r \) to generate a program \( y' \) and then run these programs over the test cases to approximate their quality. We removed all test cases \( t_i \) where the

1 ise-uiuc/Magicoder-Evol-Instruct-110K
2 ise-uiuc/Magicoder-OSS-Instruct-75K
3 bigcode/stack-dedup-python-fns
generated solution program \( y' \) could not pass. Furthermore, we removed questions with fewer than 5 tests after filtering, as these questions might be overly ambiguous. With the above filtering, we constructed the ACECODE-87K with 87.1K distinct coding questions and 1.38M cleaned test cases, as represented by \( (\mathbf{x}_r, (t_1, ..., t_{m_c})) \), where \( m_c \) represents the number of test cases after filtering.

<table>
  <tr>
    <th>Subset</th>
    <th>Evol</th>
    <th>OSS</th>
    <th>Stack Python</th>
    <th>Overall</th>
  </tr>
  <tr>
    <th colspan="5">Before Filtering</th>
  </tr>
  <tr>
    <td># Examples</td>
    <td>36,256</td>
    <td>37,750</td>
    <td>50,000</td>
    <td>124,006</td>
  </tr>
  <tr>
    <td># Avg Test Cases</td>
    <td>19.33</td>
    <td>17.21</td>
    <td>18.27</td>
    <td>18.26</td>
  </tr>
  <tr>
    <th colspan="5">After Filtering</th>
  </tr>
  <tr>
    <td># Examples</td>
    <td>26,920</td>
    <td>25,862</td>
    <td>34,367</td>
    <td>87,149</td>
  </tr>
  <tr>
    <td># Avg Test Cases</td>
    <td>15.14</td>
    <td>16.33</td>
    <td>16.08</td>
    <td>15.87</td>
  </tr>
  <tr>
    <td># Pairs</td>
    <td>89,089</td>
    <td>91,636</td>
    <td>126,784</td>
    <td>307,509</td>
  </tr>
</table>

Table 1: Dataset statistics of ACECODE-87K before and after test-case filtering.

Human Case Study Finally, as a last verification step, we conducted a human case study by randomly selecting 40 questions, each with 5 randomly sampled test cases, resulting in 200 manually annotated test cases. Encouragingly, only 3 out of these 200 test cases were found invalid upon review, indicating that our filtering methodology effectively mitigates hallucinations and preserves test-case quality. Moreover, Table 2 presents various pass-rate statistics for ACECODE-87K, illustrating the overall difficulty and performance trends.

<table>
  <tr>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>Pass @ 1</td>
    <td>34.90%</td>
  </tr>
  <tr>
    <td>Pass @ 4</td>
    <td>38.50%</td>
  </tr>
  <tr>
    <td>Pass @ 8</td>
    <td>39.20%</td>
  </tr>
  <tr>
    <td>Pass @ 16</td>
    <td>39.60%</td>
  </tr>
  <tr>
    <td>Avg Test Case Pass %</td>
    <td>70.30%</td>
  </tr>
  <tr>
    <td>% Question Where All 16 Inferences Pass All Test Cases</td>
    <td>22.80%</td>
  </tr>
</table>

Table 2: Dataset pass-rate statistics of ACECODE-87K after test-case filtering using Qwen2.5-Coder-7B-Instruct (16 responses per query with temperature=1.0)

Preference Pairs Construction We propose to use the Bradley-Terry loss to train the reward model as defined in Equation 1. Therefore, we need to construct (question, [positive program, negative program]) data from ACECODE-87K. Specifically, we sample programs (\( y^1, ..., y^n \)) from existing models (e.g. Llama-3.1 (Grattafiori et al., 2024)) w.r.t \( \mathbf{x}_r \) and utilize the test-case pass rate to distinguish positive and negative programs.\footnote{Appendix A.2 contains a detailed model breakdown for how different models are used throughout the training and evaluation process of ACECODE-RM for more clarifications.} Since the pass rate \( s_i \) for the sampled program \( y^i \) can be any number between [0, 1], a minor difference in pass rate may not represent that one program is more accurate than another. Therefore, instead of using \( \mathbb{I}[s_i > s_j] \) to select the preference pairs, we have thus modified the selection rules to be:

\[
\mathbb{I}[s_i > s_j + 0.4, s_i > 0.8, s_j > 0]
\]

This is to ensure the preferred program has at least a 0.8 pass rate to make sure it represents a more correct program. Also, we find that many sampled programs with 0 pass rates can be caused by some small syntax errors or some Python packaging errors during evaluation. We chose not to include them as the preference pair to make sure our constructed datasets represent only the preference based on the valid pass rate. We also ensure that the sampled programs all come from the backbone of \( R_\phi \), so the reward model is trained in an on-policy way. After that, we train our reward model \( R_\phi \) by fully fine-tuning an instruction coding model. Specifically, we extract the last token’s final hidden representation and pass it through a linear model head that generates a single scalar output, which is optimized via the loss function defined in Equation 1.

3 Experiments

3.1 Reward Model Training Setup

We mainly use Qwen2.5-Coder-7B-Instruct\footnote{Qwen/Qwen2.5-Coder-7B-Instruct} as the backbone of the reward model and sample 16 responses from it for each question in ACECODE-87K. Finally, following the rule defined in Equation 3, around 300K preference pairs were created out of 46,618 distinct questions (37.34% of the total questions) that have at least one pair satisfying the condition, and other questions are not used.

Our reward model is trained using LlamaFactory (Zheng et al., 2024b). We apply full fine-tuning with DeepSpeed stage 3. We train for 1 epoch using a cosine learning rate schedule, starting at 1e-5 with a warmup ratio of 0.1 to gradually increase the learning rate in the initial training phase. The training batch size is set to 128. We enable BF16 precision to reduce memory overhead without compromising model fidelity. The training takes 24 hours on 8 x A100 GPUs.
<table>
  <tr>
    <th rowspan="2">Mehod</th>
    <th rowspan="2"># N</th>
    <th colspan="2">HumanEval</th>
    <th colspan="2">MBPP</th>
    <th colspan="2">BigCodeBench-C</th>
    <th colspan="2">BigCodeBench-I</th>
    <th colspan="2">LiveCodeBench V4</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>Plus</th><th>Plus</th><th>Plus</th><th>Plus</th><th>Full</th><th>Hard</th><th>Full</th><th>Hard</th><th>Full</th><th>Hard</th>
  </tr>
  <tr>
    <td>GPT-4o (0806)</td><td>1</td><td>92.7</td><td>87.2</td><td>87.6</td><td>72.2</td><td>58.9</td><td>36.5</td><td>48.0</td><td>25.0</td><td>43.6</td><td>61.3</td><td></td>
  </tr>
  <tr>
    <td>DeepSeek-V2.5</td><td>1</td><td>90.2</td><td>83.5</td><td>87.6</td><td>74.1</td><td>53.2</td><td>29.1</td><td>48.9</td><td>27.0</td><td>41.8</td><td>59.5</td><td></td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td><td>1</td><td>91.5</td><td>86.6</td><td>87.6</td><td>73.0</td><td>62.2</td><td>39.9</td><td>50.0</td><td>27.7</td><td>63.5</td><td>64.6</td><td></td>
  </tr>
  <tr>
    <td>Qwen2.5-Coder-32B</td><td>1</td><td>92.1</td><td>87.2</td><td>90.5</td><td>77.0</td><td>58.0</td><td>33.8</td><td>49.0</td><td>27.7</td><td>48.3</td><td>62.6</td><td></td>
  </tr>
</table>

Inference Model = Mistral-7B-Instruct-VO.3

<table>
  <tr>
    <th></th>
    <th>Greedy</th>
    <th>Average</th>
    <th>Oracle</th>
  </tr>
  <tr>
    <td></td>
    <td>1</td><td>36.6</td><td>31.1</td><td>49.5</td><td>41.3</td><td>25.9</td><td>6.1</td><td>20.1</td><td>5.4</td><td>7.3</td><td>24.8</td>
  </tr>
  <tr>
    <td></td>
    <td>64</td><td>37.1</td><td>30.8</td><td>45.1</td><td>38.0</td><td>21.7</td><td>4.2</td><td>17.6</td><td>3.0</td><td>4.0</td><td>22.4</td>
  </tr>
  <tr>
    <td></td>
    <td>64</td><td>87.2</td><td>78.0</td><td>83.9</td><td>73.5</td><td>68.4</td><td>37.8</td><td>58.5</td><td>31.1</td><td>24.3</td><td>60.3</td>
  </tr>
  <tr>
    <td>AceCodeRM-7B</td>
    <td>16</td><td>65.9</td><td>56.7</td><td>59.3</td><td>52.4</td><td>35.1</td><td>10.1</td><td>29.3</td><td>8.8</td><td>11.9</td><td>36.6</td>
  </tr>
  <tr>
    <td></td>
    <td>32</td><td>68.3</td><td>58.5</td><td>59.8</td><td>51.6</td><td>37.4</td><td>8.8</td><td>30.7</td><td>10.8</td><td>14.6</td><td>37.8</td>
  </tr>
  <tr>
    <td></td>
    <td>64</td><td>71.3</td><td>61.6</td><td>59.8</td><td>51.6</td><td>39.4</td><td>6.8</td><td>31.8</td><td>9.5</td><td>15.4</td><td>38.6</td>
  </tr>
  <tr>
    <td>Δ (RM-greedy)</td>
    <td>-</td><td>+34.8</td><td>+30.5</td><td>+10.3</td><td>+11.1</td><td>+13.5</td><td>+4.1</td><td>+11.7</td><td>+5.4</td><td>+8.1</td><td>+13.8</td>
  </tr>
  <tr>
    <td>AceCodeRM-32B</td>
    <td>16</td><td>68.3</td><td>61.0</td><td>58.7</td><td>49.5</td><td>37.7</td><td>11.5</td><td>30.9</td><td>10.1</td><td>12.9</td><td>37.8</td>
  </tr>
  <tr>
    <td></td>
    <td>32</td><td>72.6</td><td>65.9</td><td>61.6</td><td>51.6</td><td>40.5</td><td>9.5</td><td>33.9</td><td>13.5</td><td>16.1</td><td>40.6</td>
  </tr>
  <tr>
    <td></td>
    <td>64</td><td>75.0</td><td>64.6</td><td>60.6</td><td>50.0</td><td>42.7</td><td>15.5</td><td>35.6</td><td>13.5</td><td>17.4</td><td>41.7</td>
  </tr>
  <tr>
    <td>Δ (RM-greedy)</td>
    <td>-</td><td>+38.4</td><td>+34.8</td><td>+12.2</td><td>+11.1</td><td>+16.8</td><td>+9.5</td><td>+15.5</td><td>+8.1</td><td>+10.1</td><td>+16.9</td>
  </tr>
</table>

Inference Model = Llama-3.1-8B-Instruct

<table>
  <tr>
    <th></th>
    <th>Greedy</th>
    <th>Average</th>
    <th>Oracle</th>
  </tr>
  <tr>
    <td></td>
    <td>1</td><td>68.9</td><td>62.2</td><td>67.2</td><td>54.8</td><td>38.5</td><td>12.8</td><td>31.8</td><td>13.5</td><td>18.0</td><td>40.9</td>
  </tr>
  <tr>
    <td></td>
    <td>64</td><td>61.7</td><td>54.9</td><td>64.5</td><td>54.5</td><td>32.8</td><td>10.1</td><td>26.6</td><td>9.0</td><td>13.8</td><td>36.4</td>
  </tr>
  <tr>
    <td></td>
    <td>64</td><td>93.9</td><td>90.2</td><td>92.1</td><td>82.3</td><td>80.0</td><td>54.7</td><td>67.9</td><td>48.6</td><td>40.8</td><td>72.3</td>
  </tr>
  <tr>
    <td>AceCodeRM-7B</td>
    <td>16</td><td>77.4</td><td>70.7</td><td>76.5</td><td>64.3</td><td>45.8</td><td>20.3</td><td>36.4</td><td>12.2</td><td>26.1</td><td>47.7</td>
  </tr>
  <tr>
    <td></td>
    <td>32</td><td>79.9</td><td>72.6</td><td>76.2</td><td>62.4</td><td>47.6</td><td>23.0</td><td>37.3</td><td>13.5</td><td>27.3</td><td>48.9</td>
  </tr>
  <tr>
    <td></td>
    <td>64</td><td>81.7</td><td>74.4</td><td>74.6</td><td>61.9</td><td>47.8</td><td>23.6</td><td>38.1</td><td>13.5</td><td>27.6</td><td>49.3</td>
  </tr>
  <tr>
    <td>Δ (RM-greedy)</td>
    <td>-</td><td>+12.8</td><td>+12.2</td><td>+9.3</td><td>+9.5</td><td>+9.3</td><td>+10.8</td><td>+6.2</td><td>0.0</td><td>+9.6</td><td>+8.4</td>
  </tr>
  <tr>
    <td>AceCodeRM-32B</td>
    <td>16</td><td>82.3</td><td>74.4</td><td>72.8</td><td>60.6</td><td>49.8</td><td>20.3</td><td>38.4</td><td>13.5</td><td>27.5</td><td>48.8</td>
  </tr>
  <tr>
    <td></td>
    <td>32</td><td>81.7</td><td>76.2</td><td>72.8</td><td>60.6</td><td>50.4</td><td>22.3</td><td>39.1</td><td>13.5</td><td>30.3</td><td>49.6</td>
  </tr>
  <tr>
    <td></td>
    <td>64</td><td>85.4</td><td>79.3</td><td>72.0</td><td>59.0</td><td>48.5</td><td>19.6</td><td>40.0</td><td>13.5</td><td>31.0</td><td>49.8</td>
  </tr>
  <tr>
    <td>Δ (RM-greedy)</td>
    <td>-</td><td>+16.5</td><td>+17.1</td><td>+9.3</td><td>+9.5</td><td>+11.8</td><td>+10.8</td><td>+8.2</td><td>+0.0</td><td>+13.0</td><td>+9.0</td>
  </tr>
</table>

Inference Model = Qwen2.5-Coder-7B-Instruct

<table>
  <tr>
    <th></th>
    <th>Greedy</th>
    <th>Average</th>
    <th>Oracle</th>
  </tr>
  <tr>
    <td></td>
    <td>1</td><td>91.5</td><td>86.0</td><td>82.8</td><td>71.4</td><td>49.5</td><td>19.6</td><td>41.8</td><td>20.3</td><td>34.2</td><td>55.2</td>
  </tr>
  <tr>
    <td></td>
    <td>64</td><td>86.0</td><td>80.1</td><td>77.9</td><td>65.6</td><td>45.3</td><td>18.6</td><td>37.3</td><td>16.2</td><td>31.8</td><td>51.0</td>
  </tr>
  <tr>
    <td></td>
    <td>64</td><td>98.2</td><td>95.7</td><td>97.4</td><td>90.7</td><td>80.9</td><td>62.8</td><td>73.5</td><td>53.4</td><td>57.4</td><td>78.9</td>
  </tr>
  <tr>
    <td>AceCodeRM-7B</td>
    <td>16</td><td>90.2</td><td>82.9</td><td>88.6</td><td>74.9</td><td>53.8</td><td>20.9</td><td>45.0</td><td>21.6</td><td>40.1</td><td>57.6</td>
  </tr>
  <tr>
    <td></td>
    <td>32</td><td>90.9</td><td>86.0</td><td>87.8</td><td>74.1</td><td>53.4</td><td>25.0</td><td>43.9</td><td>19.6</td><td>39.8</td><td>57.8</td>
  </tr>
  <tr>
    <td></td>
    <td>64</td><td>90.9</td><td>85.4</td><td>87.6</td><td>73.8</td><td>52.9</td><td>24.3</td><td>43.5</td><td>21.6</td><td>40.1</td><td>57.8</td>
  </tr>
  <tr>
    <td>Δ (RM-greedy)</td>
    <td>-</td><td>-0.6</td><td>0.0</td><td>+5.8</td><td>+3.4</td><td>+4.3</td><td>+5.4</td><td>+3.2</td><td>+1.4</td><td>+5.9</td><td>+2.6</td>
  </tr>
  <tr>
    <td>AceCodeRM-32B</td>
    <td>16</td><td>90.2</td><td>86.6</td><td>88.4</td><td>74.9</td><td>53.9</td><td>25.0</td><td>45.4</td><td>19.6</td><td>44.0</td><td>58.7</td>
  </tr>
  <tr>
    <td></td>
    <td>32</td><td>90.2</td><td>86.6</td><td>88.4</td><td>75.4</td><td>55.4</td><td>29.7</td><td>45.6</td><td>21.6</td><td>43.5</td><td>59.6</td>
  </tr>
  <tr>
    <td></td>
    <td>64</td><td>89.6</td><td>86.0</td><td>87.8</td><td>75.1</td><td>55.0</td><td>26.4</td><td>46.1</td><td>22.3</td><td>44.5</td><td>59.2</td>
  </tr>
  <tr>
    <td>Δ (RM-greedy)</td>
    <td>-</td><td>-0.6</td><td>+0.6</td><td>+5.8</td><td>+4.0</td><td>+6.0</td><td>+10.1</td><td>+4.3</td><td>+2.0</td><td>+10.3</td><td>+4.4</td>
  </tr>
</table>

Table 3: AceCode-RM’s best-of-n results on several benchmarks. Specifically, -C means completion split and -I means instruct split of BigCodeBench. The Δ might be off by 0.1 due to rounding.

3.2 Reinforcement Learning Setup

We perform RL training from three policy models: Qwen2.5-7B-Instruct 6, Qwen2.5-Coder-7B-Base 7, and Qwen2.5-Coder-7B-Instruct. Two types of reward can be used, i.e. the trained reward model AceCode-RM-7B and the rule-based reward, i.e. pass rate over the test cases in AceCode-87K. During training, we set the pass rate to be a binary reward, which is 1.0 when all test cases pass, otherwise 0. This is similar to the verifiable reward used in Tulu3 (Lambert et al., 2024a) and DeepSeek-R1 (Guo et al., 2025). Similar to DeepSeek-R1 (Guo et al., 2025), we also experiment with RL from the base model because SFT may cause the search space of the model to be stuck in a local minimum. Since coding is also a highly verifiable task like math, we include the Qwen2.5-Coder-7B-Base in our experiments.

We have trained different policy model backbones with different rewards, resulting in 6 RL models in total. All the RL-tuning is based on OpenRLHF (Hu et al., 2024). We adopt the Reinforcement++ (Hu, 2025) algorithm instead of PPO to improve the training efficiency without training the value model. It’s also proved to be more stable than PPO and GRPO. We train our model on a subsampled hard version of AceCode-87K, where we keep the 25% of the questions with lower aver-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">HumanEval</th>
    <th colspan="2">MBPP</th>
    <th colspan="2">BigCodeBench (C)</th>
    <th colspan="2">BigCodeBench (I)</th>
    <th colspan="2">LiveCodeBench V4</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>-</th><th>Plus</th><th>-</th><th>Plus</th><th>-</th><th>Full</th><th>Hard</th><th>Full</th><th>Hard</th><th>V4</th>
  </tr>
  <tr>
    <td>RLEF-8B</td><td>-</td><td>67.5</td><td>-</td><td>57.0</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>RLEF-70B</td><td>-</td><td>78.5</td><td>-</td><td>67.6</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>PPOCoder-7B</td><td>78.7</td><td>-</td><td>67.0</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>StepCoder-7B</td><td>76.8</td><td>-</td><td>63.8</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>CodeGemma-7B</td><td>60.5</td><td>-</td><td>55.2</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DSTC-33B</td><td>79.9</td><td>72.0</td><td>82.5</td><td>70.4</td><td>51.6</td><td>22.3</td><td>41.0</td><td>18.2</td><td>-</td><td>-</td><td>-</td>
  </tr>
</table>

Baseline = Qwen2.5-7B-Instruct

<table>
  <tr>
    <th></th>
    <th>Baseline</th>
    <th>AceCoder<sub>RM</sub></th>
    <th>AceCoder<sub>Rule</sub></th>
    <th>Δ (RL-baseline)</th>
  </tr>
  <tr>
    <td>HumanEval</td>
    <td>81.7</td><td>83.5</td><td>84.1</td><td>+2.4</td>
  </tr>
  <tr>
    <td>MBPP</td>
    <td>73.2</td><td>77.4</td><td>78.0</td><td>+4.3</td>
  </tr>
  <tr>
    <td>BigCodeBench (C)</td>
    <td>79.4</td><td>83.1</td><td>82.3</td><td>+3.7</td>
  </tr>
  <tr>
    <td>BigCodeBench (I)</td>
    <td>67.7</td><td>71.2</td><td>69.3</td><td>+3.4</td>
  </tr>
  <tr>
    <td>LiveCodeBench V4</td>
    <td>45.6</td><td>46.8</td><td>48.6</td><td>+1.2</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>16.9</td><td>19.9</td><td>18.2</td><td>+1.2</td>
  </tr>
</table>

Baseline = Qwen2.5-Coder-7B-Base

<table>
  <tr>
    <th></th>
    <th>Baseline</th>
    <th>AceCoder<sub>RM</sub></th>
    <th>AceCoder<sub>Rule</sub></th>
    <th>Δ (RL-baseline)</th>
  </tr>
  <tr>
    <td>HumanEval</td>
    <td>61.6</td><td>83.5</td><td>84.1</td><td>+22.5</td>
  </tr>
  <tr>
    <td>MBPP</td>
    <td>53.0</td><td>75.6</td><td>78.0</td><td>+25.0</td>
  </tr>
  <tr>
    <td>BigCodeBench (C)</td>
    <td>76.9</td><td>80.2</td><td>82.3</td><td>+5.4</td>
  </tr>
  <tr>
    <td>BigCodeBench (I)</td>
    <td>62.9</td><td>67.2</td><td>69.3</td><td>+6.4</td>
  </tr>
  <tr>
    <td>LiveCodeBench V4</td>
    <td>45.8</td><td>41.9</td><td>48.6</td><td>+2.8</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>16.2</td><td>14.9</td><td>18.2</td><td>+2.0</td>
  </tr>
</table>

Baseline = Qwen2.5-Coder-7B-Instruct

<table>
  <tr>
    <th></th>
    <th>Baseline</th>
    <th>AceCoder<sub>RM</sub></th>
    <th>AceCoder<sub>Rule</sub></th>
    <th>Δ (RL-baseline)</th>
  </tr>
  <tr>
    <td>HumanEval</td>
    <td>91.5</td><td>89.0</td><td>90.9</td><td>-0.6</td>
  </tr>
  <tr>
    <td>MBPP</td>
    <td>86.0</td><td>84.1</td><td>84.8</td><td>-1.2</td>
  </tr>
  <tr>
    <td>BigCodeBench (C)</td>
    <td>82.8</td><td>86.0</td><td>84.1</td><td>+3.2</td>
  </tr>
  <tr>
    <td>BigCodeBench (I)</td>
    <td>71.4</td><td>72.8</td><td>71.7</td><td>+1.3</td>
  </tr>
  <tr>
    <td>LiveCodeBench V4</td>
    <td>49.5</td><td>50.4</td><td>50.9</td><td>+1.4</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>19.6</td><td>18.9</td><td>23.0</td><td>+3.4</td>
  </tr>
</table>

Table 4: ACECODER’s Performance after RL tuning using Reinforcement++ algorithm. We start with 3 different initial policy models and 2 kinds of reward types, where RM means using our trained ACECODE-RM and Rule means using the binary pass rate. Results show consistent improvement across various benchmarks.

age pass rates and higher variance. This is to ensure the question is hard and that the sampled programs are diverse enough. For the training hyperparameters, we set the rollout batch size to 256, and 8 programs are sampled per question. The training batch size is 128 with a learning rate of 5e-7. All the models are trained for 1 episode and finished in 6 hours on 8 x H100 GPUs.

3.3 Evaluation Setup

We evaluate our method on four established code-focused benchmarks: HumanEval(+) (Chen et al., 2021; Liu et al., 2023), MBPP(+) (Austin et al., 2021; Liu et al., 2023), BigCodeBench (Zhuo et al., 2024) and LiveCodeBench (V4) (Jain et al., 2024). These benchmarks collectively cover a diverse array of coding tasks, enabling us to assess both the correctness and quality of generated code. For Best-of-N sampling, we adopt top-p sampling with a temperature of 1.0 to generate multiple (16/32/64) candidate solutions per question and then select the response with the highest reward for evaluation. For RL experiments, we use each benchmark’s default setting, which is greedy sampling most of the time.

3.4 Main Results

Here we show the experimental results of the reward models and RL-trained models.

RM Results We conduct Best-of-N experiments on 3 inference models, specifically Mistral-Instruct-V0.3-7B(AI, 2023), Llama-3.1-Instruct-8B (Grattafiori et al., 2024), and Qwen2.5-Coder-7B-Insruct (Hui et al., 2024b; Yang et al., 2024a). We additionally report the average score across all generated samples and also the oracle score (pass@N) for better comparison.

According to Table 3, ACECODE-RM can consistently boost the performance of inference models by a large margin compared to the greedy decoding results. On weaker models like Mistral (AI, 2023) and Llama-3.1 (Zheng et al., 2024b), the overall improvements are greater than 10 points. These improvements can be attributed to our reward model’s ability to identify high-quality completions among multiple candidates, thereby reducing the impact of suboptimal sampling on the final output. Notably, these gains become more pronounced on benchmarks where the gap between greedy decoding and oracle performance (i.e., the best possible completion among all samples) is larger. In such cases, the variance among sampled completions is relatively high, providing greater opportunities for the reward
model to pinpoint and elevate top-tier responses.

Greedy decoding systematically outperforms the average sampled performance, reflecting the strong code generation capability of these inference models. Consequently, while most reward models achieve best-of-N results above the average, we consider a reward model effective only if it surpasses the performance of greedy decoding.

RL Results We perform RL training over 3 different initial policy models in Table 4 with model-based and rule-based rewards. When starting from Qwen2.5-Instruct-7B, we can see that RL tuning can consistently improve performance, especially for HumanEval and MBPP. Even for the Plus version with more and harder test cases, the RL-tuned model also has more than 3 points of improvement.

When starting from the Qwen2.5-Coder-Instruct-7B itself, we can still observe improvements, especially when using the rule-based reward. For example, we get more than 3.4 improvement on BigCodeBench-Full-Hard. Using the reward model for RL can also bring a 3.2 improvement on MBPP. This highlights the charm of self-improvement given the reward model backbone is the same as the initial policy model. We compare our method with other RL-based models like RLEF (Chen et al., 2024), PPOCoder (Shojaee et al., 2023a), StepCoder (Dou et al., 2024b), DSTC (Liu et al., 2024d), etc. We show that our 7B model is able to beat these competitors across the evaluation benchmarks.

Another experiment we conduct is to perform RL training directly from the base model Qwen2.5-Coder-7B-base. We show significant improvement, especially through test-case pass rewards on HumanEval, MBPP, and BigCodeBench-I. These results are achieved by only training for 80 steps. We believe further scaling up the training will lead to much larger gains.

Comparison with Other RMs We compare our ACECODE-RM with 3 top-ranked RM on the RewardBench, including InternLM2-RM-8B (Cai et al., 2024), Skywork-Llama-3.1-8B, and Skywork-Gemma-27B (Liu et al., 2024a), where results are reported in Table 5. We can see that these general-purpose RM can hardly improve and sometimes decrease the performance through Best-of-N sampling compared to greedy sampling, showcasing the incapability in identifying the correct generated programs. On the other hand, our ACECODE-RM surpasses all other publicly released reward models in our evaluation and consistently gets positive gains. These findings further underscore our assumption that previous RM training lacks reliable signals for codes and prove that our RMs can generate reliable and state-of-the-art reward signals in code generation tasks.

Moreover, we have also evaluated our ACECODE-RM against other reward models on RM-Bench in Table 6, a benchmark designed to assess the reward model’s capabilities across code synthesis, mathematical reasoning, and other tasks (Liu et al., 2024c). In this head-to-head comparison, ACECODE-RM emerges as the clear state-of-the-art in coding, hard tasks, and overall average. Remarkably, our 7 billion-parameter variant, ACECODE-RM-7B, outperforms NVIDIA-Nemotron-340B-Reward(Nvidia et al., 2024) by 7.50 points on the coding benchmark, proving that a more compact model can deliver superior reward estimates for code generation. Beyond code, ACECODE-RM also generalizes well: it beats all other reward models on the average score, underscoring its robust reasoning and dialogue capabilities.

3.5 Ablation Studies

Test Case Quality Matters We also conduct experiments to investigate how filtering the test cases with a proxy model can affect the results. As shown in Table 7, training RM on data after the filtering improves the performance significantly, especially for those hard code questions like MBPP-Plus and BigCodeBench-Hard (C/I). We believe this is because the test case filtering can ensure the remaining ones are consistent with each other and thus point to the same implicit program, which improves the quality of the rewards.

RM Backbone Matters Our results in Table 8 clearly show that changing the backbone of the reward model from Llama-3.1 to Qwen2.5 can significantly improve the Best-of-16 performance. This is because the Qwen2.5-Coder models have been pre-trained on way more code-related data compared to the Llama-3.1 models, and thus more knowledgeable when tuning it into a reward model.

Does RL-style Tuning Work? Inspired by the recent DeepSeek-R1 (Guo et al., 2025), we also conduct the RL directly from the base model without any SFT. It turns out we get huge improvements when using rule-based rewards. For example, we get 25.0 points of improvements
<table>
  <tr>
    <th>Method & RM</th>
    <th colspan="2">HumanEval</th>
    <th colspan="2">MBPP</th>
    <th colspan="2">BigCodeBench-C</th>
    <th colspan="2">BigCodeBench-I</th>
    <th colspan="2">LiveCodeBench V4</th>
    <th>Average</th>
  </tr>
  <tr>
    <td></td>
    <td>-</td>
    <td>Plus</td>
    <td>-</td>
    <td>Plus</td>
    <td>Full</td>
    <td>Hard</td>
    <td>Full</td>
    <td>Hard</td>
    <td>Full</td>
    <td>Hard</td>
    <td></td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>68.9</td>
    <td>62.2</td>
    <td>67.2</td>
    <td>54.8</td>
    <td>38.5</td>
    <td>12.8</td>
    <td>31.8</td>
    <td><b>13.5</b></td>
    <td>18.0</td>
    <td>40.9</td>
    <td></td>
  </tr>
  <tr>
    <td>Average</td>
    <td>50.1</td>
    <td>42.2</td>
    <td>57.9</td>
    <td>47.2</td>
    <td>22.0</td>
    <td>10.6</td>
    <td>18.2</td>
    <td>12.0</td>
    <td>14.9</td>
    <td>30.6</td>
    <td></td>
  </tr>
  <tr>
    <td>InternLM-2-RM-8B</td>
    <td>57.9</td>
    <td>55.5</td>
    <td>66.7</td>
    <td>54.0</td>
    <td>38.7</td>
    <td>8.8</td>
    <td>29.8</td>
    <td>8.8</td>
    <td>15.1</td>
    <td>37.3</td>
    <td></td>
  </tr>
  <tr>
    <td>Skywork-Gemma-27B</td>
    <td>73.8</td>
    <td>67.1</td>
    <td>64.3</td>
    <td>53.4</td>
    <td>40.1</td>
    <td>14.9</td>
    <td>32.5</td>
    <td>12.8</td>
    <td>23.6</td>
    <td>42.5</td>
    <td></td>
  </tr>
  <tr>
    <td>Skywork-Llama-3.1-8B</td>
    <td>67.7</td>
    <td>61.6</td>
    <td>69.6</td>
    <td>56.9</td>
    <td>40.6</td>
    <td>10.8</td>
    <td>31.8</td>
    <td>12.2</td>
    <td>18.8</td>
    <td>41.1</td>
    <td></td>
  </tr>
  <tr>
    <td>Δ (max(other RM)-greedy)</td>
    <td>+4.9</td>
    <td>+4.9</td>
    <td>+2.4</td>
    <td>+2.1</td>
    <td>+2.1</td>
    <td>+2.0</td>
    <td>+0.6</td>
    <td>-0.7</td>
    <td>+5.6</td>
    <td>+2.6</td>
    <td></td>
  </tr>
  <tr>
    <td>ACECODE-RM-7B</td>
    <td><b>77.4</b></td>
    <td><b>70.7</b></td>
    <td><b>76.5</b></td>
    <td><b>64.3</b></td>
    <td><b>45.8</b></td>
    <td><b>20.3</b></td>
    <td><b>36.4</b></td>
    <td><b>12.2</b></td>
    <td><b>26.1</b></td>
    <td><b>47.7</b></td>
    <td></td>
  </tr>
  <tr>
    <td>Δ (RM-greedy)</td>
    <td>+8.5</td>
    <td>+8.5</td>
    <td>+9.3</td>
    <td>+9.5</td>
    <td>+7.3</td>
    <td>+7.4</td>
    <td>+4.6</td>
    <td>-1.4</td>
    <td>+8.1</td>
    <td>+6.8</td>
    <td></td>
  </tr>
</table>

Table 5: ACECODE-RM’s performance against other open-sourced reward models in terms of Best-of-16 sampling for Llama-3.1-8B-Inst. We can see the top-ranked RM on Reward Bench get little improvements compared to ours.

<table>
  <tr>
    <th>Model</th>
    <th>Code</th>
    <th>Chat</th>
    <th>Math</th>
    <th>Safety</th>
    <th>Easy</th>
    <th>Normal</th>
    <th>Hard</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>Skywork/Skywork-Reward-Llama-3.1-8B</td>
    <td>54.5</td>
    <td>69.5</td>
    <td>60.6</td>
    <td><b>95.7</b></td>
    <td><b>89</b></td>
    <td>74.7</td>
    <td>46.6</td>
    <td>70.1</td>
  </tr>
  <tr>
    <td>NVIDIA/Nemotron-340B-Reward</td>
    <td>59.4</td>
    <td>71.2</td>
    <td>59.8</td>
    <td>87.5</td>
    <td>81</td>
    <td>71.4</td>
    <td>56.1</td>
    <td>69.5</td>
  </tr>
  <tr>
    <td>internlm/internlm2-2b0-reward</td>
    <td>56.7</td>
    <td>63.1</td>
    <td>66.8</td>
    <td>86.5</td>
    <td>82.6</td>
    <td>71.6</td>
    <td>50.7</td>
    <td>68.3</td>
  </tr>
  <tr>
    <td>internlm/internlm2-7b-reward</td>
    <td>49.7</td>
    <td>61.7</td>
    <td><b>71.4</b></td>
    <td>85.5</td>
    <td>85.4</td>
    <td>70.7</td>
    <td>45.1</td>
    <td>67.1</td>
  </tr>
  <tr>
    <td>Skywork-Reward-Llama-3.1-8B-v0.2<sup>1</sup></td>
    <td>53.4</td>
    <td>69.2</td>
    <td>62.1</td>
    <td>96</td>
    <td>88.5</td>
    <td>74</td>
    <td>47.9</td>
    <td>70.1</td>
  </tr>
  <tr>
    <td>Skywork-Reward-Gemma-2-27B-v0.2<sup>1</sup></td>
    <td>45.8</td>
    <td>49.4</td>
    <td>50.7</td>
    <td>48.2</td>
    <td>50.3</td>
    <td>48.2</td>
    <td>47</td>
    <td>48.5</td>
  </tr>
  <tr>
    <td>AceCodeRM-7B</td>
    <td>66.9</td>
    <td>66.7</td>
    <td>65.3</td>
    <td>89.9</td>
    <td>79.9</td>
    <td>74.4</td>
    <td>62.2</td>
    <td>72.2</td>
  </tr>
  <tr>
    <td>AceCodeRM-32B</td>
    <td><b>72.1</b></td>
    <td><b>73.7</b></td>
    <td>70.5</td>
    <td>88</td>
    <td>84.5</td>
    <td><b>78.3</b></td>
    <td><b>65.5</b></td>
    <td><b>76.1</b></td>
  </tr>
  <tr>
    <td>Δ (AceCodeRM-7B - max(other RM))</td>
    <td>7.5</td>
    <td>-4.5</td>
    <td>-6.1</td>
    <td>-6.1</td>
    <td>-9.1</td>
    <td>-0.3</td>
    <td>6.1</td>
    <td>2.1</td>
  </tr>
  <tr>
    <td>Δ (AceCodeRM-32B - max(other RM))</td>
    <td>12.7</td>
    <td>2.5</td>
    <td>-0.9</td>
    <td>-8</td>
    <td>-4.5</td>
    <td>3.6</td>
    <td>9.4</td>
    <td>6</td>
  </tr>
</table>

<sup>1</sup> There is no official result for this model; however, the authors made best efforts to extend the original RM Bench code base to adapt to this new model.

Table 6: ACECODE-RM’s Performance on RM Bench against various other reward models. We can see that ACECODE-RM-32B model performed best in Coding, Chat, Normal, Hard, and Average scores against all other reward models.

<table>
  <tr>
    <th>Method</th>
    <th colspan="2">HumanEval</th>
    <th colspan="2">MBPP</th>
    <th colspan="2">BigCodeBench-C</th>
    <th colspan="2">BigCodeBench-I</th>
    <th colspan="2">LiveCodeBench</th>
    <th>Average</th>
  </tr>
  <tr>
    <td></td>
    <td>-</td>
    <td>Plus</td>
    <td>-</td>
    <td>Plus</td>
    <td>Full</td>
    <td>Hard</td>
    <td>Full</td>
    <td>Hard</td>
    <td>Full</td>
    <td>Hard</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="13">Inference Model = Llama-3.1-8B-Instruct</th>
  </tr>
  <tr>
    <td>RM w/o Test Case Filter</td>
    <td>73.8</td>
    <td>65.9</td>
    <td>73.3</td>
    <td>61.4</td>
    <td>44.6</td>
    <td>17.6</td>
    <td>35.5</td>
    <td>9.5</td>
    <td>25.1</td>
    <td>45.2</td>
    <td></td>
  </tr>
  <tr>
    <td>RM w/ Test Filter</td>
    <td><b>77.4</b></td>
    <td><b>70.7</b></td>
    <td><b>76.5</b></td>
    <td><b>64.3</b></td>
    <td><b>45.8</b></td>
    <td><b>20.3</b></td>
    <td><b>36.4</b></td>
    <td><b>12.2</b></td>
    <td><b>26.1</b></td>
    <td><b>47.7</b></td>
    <td></td>
  </tr>
  <tr>
    <td>Δ (w/ Filter - w/o Filter)</td>
    <td>+3.7</td>
    <td>+4.9</td>
    <td>+3.2</td>
    <td>+2.9</td>
    <td>+1.2</td>
    <td>+2.7</td>
    <td>+0.9</td>
    <td>+2.7</td>
    <td>+1.0</td>
    <td>+2.5</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="13">Inference Model = Qwen2.5-Coder-7B-Instruct</th>
  </tr>
  <tr>
    <td>RM w/o Test Case Filter</td>
    <td><b>91.5</b></td>
    <td><b>86.0</b></td>
    <td>86.0</td>
    <td>72.2</td>
    <td>52.5</td>
    <td><b>21.6</b></td>
    <td>43.4</td>
    <td>19.6</td>
    <td>36.9</td>
    <td>56.6</td>
    <td></td>
  </tr>
  <tr>
    <td>RM w/ Test Filter</td>
    <td>90.2</td>
    <td>82.9</td>
    <td><b>88.6</b></td>
    <td><b>74.9</b></td>
    <td><b>53.8</b></td>
    <td>20.9</td>
    <td><b>45.0</b></td>
    <td><b>21.6</b></td>
    <td><b>40.1</b></td>
    <td><b>57.6</b></td>
    <td></td>
  </tr>
  <tr>
    <td>Δ (w/ Filter - w/o Filter)</td>
    <td>-1.2</td>
    <td>-3.0</td>
    <td>+2.6</td>
    <td>+2.6</td>
    <td>+1.3</td>
    <td>-0.7</td>
    <td>+1.6</td>
    <td>+2.0</td>
    <td>+3.2</td>
    <td>+1.0</td>
    <td></td>
  </tr>
</table>

Table 7: Ablation study on test-case filtering. Results are Best-of-16 sampling performance.

<table>
  <tr>
    <th>Method</th>
    <th colspan="2">HumanEval</th>
    <th colspan="2">MBPP</th>
    <th colspan="2">BigCodeBench-C</th>
    <th colspan="2">BigCodeBench-I</th>
    <th colspan="2">LiveCodeBench</th>
    <th>Average</th>
  </tr>
  <tr>
    <td></td>
    <td>-</td>
    <td>Plus</td>
    <td>-</td>
    <td>Plus</td>
    <td>Full</td>
    <td>Hard</td>
    <td>Full</td>
    <td>Hard</td>
    <td>Full</td>
    <td>Hard</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="13">Inference Model = Llama-3.1-8B-Instruct</th>
  </tr>
  <tr>
    <td>ACECODE-RM (LLama)</td>
    <td>65.9</td>
    <td>59.1</td>
    <td>69.6</td>
    <td>57.9</td>
    <td>42.7</td>
    <td>12.8</td>
    <td>32.9</td>
    <td><b>13.5</b></td>
    <td>19.9</td>
    <td>41.6</td>
    <td></td>
  </tr>
  <tr>
    <td>ACECODE-RM (Qwen)</td>
    <td><b>77.4</b></td>
    <td><b>70.7</b></td>
    <td><b>76.5</b></td>
    <td><b>64.3</b></td>
    <td><b>45.8</b></td>
    <td><b>20.3</b></td>
    <td><b>36.4</b></td>
    <td><b>12.2</b></td>
    <td><b>26.1</b></td>
    <td><b>47.7</b></td>
    <td></td>
  </tr>
  <tr>
    <td>Δ (Qwen-Llama)</td>
    <td>+11.6</td>
    <td>+11.6</td>
    <td>+6.9</td>
    <td>+6.3</td>
    <td>+3.1</td>
    <td>+7.4</td>
    <td>+3.5</td>
    <td>-1.4</td>
    <td>+6.2</td>
    <td>+6.1</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="13">Inference Model = Qwen2.5-Coder-7B-Instruct</th>
  </tr>
  <tr>
    <td>ACECODE-RM (LLama)</td>
    <td>87.8</td>
    <td>81.7</td>
    <td>82.0</td>
    <td>67.7</td>
    <td>50.5</td>
    <td><b>25.0</b></td>
    <td>39.0</td>
    <td>19.6</td>
    <td>32.4</td>
    <td>54.0</td>
    <td></td>
  </tr>
  <tr>
    <td>ACECODE-RM (Qwen)</td>
    <td><b>90.2</b></td>
    <td><b>82.9</b></td>
    <td><b>88.6</b></td>
    <td><b>74.9</b></td>
    <td><b>53.8</b></td>
    <td>20.9</td>
    <td><b>45.0</b></td>
    <td><b>21.6</b></td>
    <td><b>40.1</b></td>
    <td><b>57.6</b></td>
    <td></td>
  </tr>
  <tr>
    <td>Δ (Qwen-Llama)</td>
    <td>+2.4</td>
    <td>+1.2</td>
    <td>+6.6</td>
    <td>+7.1</td>
    <td>+3.2</td>
    <td>-4.1</td>
    <td>+6.0</td>
    <td>+2.0</td>
    <td>+7.7</td>
    <td>+2.4</td>
    <td></td>
  </tr>
</table>

Table 8: Comparison of ACECODE-RM’s performance trained on different base model, where ACECODE-RM (LLama) is based on Llama-3.1-Inst-8B and ACECODE-RM (Qwen) is based on Qwen-Coder-2.5-7B-Inst. Results are Best-of-16 sampling performance.
on HumanEval-Plus after training only 6 hours from the Base Model, which is way more efficient than large-scale SFT. What’s more, the ACECODER Rule improve the BigCodeBench-Instruct-Full’s performance from 40.2 to 43.2, nearly the same performance with DeepSeek-R1-Distill-Qwen-32B (43.9) which was directly distilled from the DeepSeek-R1 Model. This further consolidates the finding of DeepSeek-Zero. However, we do find that using reward models for RL tuning can lead to worse results. We attribute this to the potential reward hacking during the tuning process.

4 Related Works

4.1 Synthesizing Test Cases

Automatic test generation is a widely used approach for verifying the correctness of LLM-generated programs. Prior work has commonly employed the same LLM that generates the programs to also generate test cases, selecting the most consistent program from multiple sampled outputs in a self-consistency manner (Chen et al., 2022; Huang et al., 2023; Jiao et al., 2024). However, these generated test cases often suffer from significant hallucinations. To address this issue, Algo (Zhang et al., 2023) introduced the use of an oracle program solution to improve test case quality. While similar in spirit to our test case filtering approach, Algo constructs its oracle solution by exhaustively enumerating all possible combinations of relevant variables, whereas we leverage a stronger coder LLM to generate the oracle solution. Beyond using test cases as verification signals, Clover (Sun et al., 2023) enhances program verification by performing consistency checks between code, docstrings, and formal annotations, incorporating formal verification tools alongside LLMs.

4.2 Reinforcement Learning for LLM

Reinforcement Learning from Human Feedback (RLHF)(Ouyang et al., 2022b) has been widely adopted to enhance the capabilities of large language models (LLMs) in various tasks, including conversational interactions and mathematical reasoning(Yang et al., 2024b). Reinforcement learning (RL) algorithms such as PPO(Schulman et al., 2017), GRPO(Shao et al., 2024), and Reinforcement++(Hu, 2025) have been employed to fine-tune models using reward signals derived from either learned reward models(Shao et al., 2024) or predefined rule-based heuristics (Guo et al., 2025; Wang et al., 2025).

Given that coding is an inherently verifiable task, recent studies have explored RL techniques that leverage direct execution accuracy as a reward signal. PPOCoder (Shojaee et al., 2023b) and CodeRL (Le et al., 2022) demonstrated the effectiveness of PPO-based RL for coding tasks, while RLEF (Gehring et al., 2024) extended this approach to multi-turn settings by incorporating execution feedback at each step. StepCoder (Dou et al., 2024a) refined the reward mechanism by assigning rewards at a more granular level, considering only successfully executed lines of code. Additionally, DSTC (Liu et al., 2024e) explored the application of Direct Preference Optimization (DPO) to code generation by using self-generated test cases and programs.

Despite these advancements, most prior RL-based approaches for coding have been constrained by the use of pre-annotated datasets such as APPS (Hendrycks et al., 2021), which consists of only 5,000 examples, with most problems having a single test case. This limited data availability poses challenges to scalable RL training. Furthermore, the potential of reward models for coding remains largely unexplored. In this work, we address these limitations by automatically synthesizing test cases and leveraging trained reward models for reinforcement learning, demonstrating the scalability and effectiveness of our approach.

5 Conclusion

We introduced AceCoder as the first approach to reward model training and RL tuning for code generation using large-scale, reliable test case synthesis. Our data pipeline produces high-quality verifiable code without relying on the most advanced models, enabling effective reward model training and reinforcement learning. Our method significantly improves Best-of-N performance. However, RL training gains are less pronounced, leaving it as a future work to enhance.

Limitations

Test Case Synthesis Despite our efforts to enhance the reliability of synthesized test cases through prompt engineering and filtering with a reference solution, inaccuracies can still arise. These errors may stem from an incorrect reference solution or test cases that are too simple, failing to capture challenging edge cases. Consequently, passing
all test cases does not necessarily guarantee a program’s correctness, leading to noise in the reward model training and reinforcement learning (RL) tuning signals. To address this, future work can leverage stronger large language models (LLMs) to synthesize more rigorous test cases, ensuring the inclusion of harder corner cases. Additionally, using more advanced coding LLMs to generate reference solutions could further improve test case filtering, preserving only high-quality examples.

Reinforcement Learning for Coding In this paper, we explored RL tuning using three models and two types of rewards: RM-based and rule-based. While significant improvements are observed when tuning Qwen2.5-7B-Instruct and Qwen2.5-Coder-7B-Base, tuning on Qwen2.5-Coder-7B-Instruct exhibited less pronounced gains due to its strong ability originally. This suggests that the current reward signals may still contain noise. Furthermore, there remains considerable room for improvement, particularly in tuning the Qwen2.5-Coder-7B-Base. Given recent advancements in models such as DeepSeek-R1, future work could further refine RL tuning strategies to achieve better performance with a more fine-grained reward design.

Ethical Statements

This work fully complies with the ACL Ethics Policy. We declare that there are no ethical issues in this paper, to the best of our knowledge.

References

Mistral AI. 2023. Mistral-7b-instruct-v0.3. https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

Ralph Allan Bradley and Milton E. Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324.

Zheng Cai, Maosong Cao, Haojing Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. 2024. Interlm2 technical report. Preprint, arXiv:2403.17297.

Angelica Chen, Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Samuel R Bowman, Kyunghyun Cho, and Ethan Perez. 2024. Learning from natural language feedback. Transactions on Machine Learning Research.

Bei Chen, Fengji Zhang, A. Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. ArXiv, abs/2207.10397.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidi Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. Preprint, arXiv:2107.03374.

Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Junjie Shan, Caishuang Huang, Wei Shen, Xiaoran Fan, Zhiheng Xi, Yuahou Zhou, Tao Ji, Rui Zheng, Qi Zhang, Xuanjing Huang, and Tao Gui. 2024a. Stepcode: Improve code generation with reinforcement learning from compiler feedback. ArXiv, abs/2402.01391.

Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang
Huang, Xiao Wang, Xiaoran Fan, et al. 2024b. Step-coder: Improve code generation with reinforcement learning from compiler feedback. arXiv preprint arXiv:2402.01391.

Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, and Gabriele Synnaeve. 2024. Rlef: Grounding code llms in execution feedback with reinforcement learning. ArXiv, abs/2410.02089.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and etc. 2024. The llama 3 herd of models. Preprint. arXiv:2407.21783.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shiron Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024a. Deepseek-coder: When the large language model meets programming - the rise of code intelligence. ArXiv, abs/2401.14196.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024b. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196.

Dan Hendrycks, Steven Basart, Saurav Kadavath, Man-tas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Measuring coding challenge competence with apps. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

Jian Hu. 2025. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262.

Jian Hu, Xibin Wu, Weixun Wang, Dehao Zhang, Yu Cao, OpenLLMAI Team, NetEase Fuxi, AI Lab, and Alibaba Group. 2024. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. ArXiv, abs/2405.11143.

Baizhou Huang, Shuai Lu, Weizhu Chen, Xiaojun Wan, and Nan Duan. 2023. Enhancing large language models in coding through multi-perspective self-consistency. In Annual Meeting of the Association for Computational Linguistics.

Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaren Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chencheng Zhang, Linzheng Chai, et al. 2024. Open-coder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:2411.04905.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Shanghaoran Quan, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyong Lin. 2024a. Qwen2.5-coder technical report. ArXiv, abs/2409.12186.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024b. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, Al Os-trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjin Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Live-codebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.

Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F. Chen, Shafiq Joty, and Furu Wei. 2024. Preference optimization for reasoning with pseudo feedback. ArXiv, abs/2411.16345.

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. 2024a. Tvl" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124.

Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. 2024b. Rewardbench: Evaluating reward models for language modeling. ArXiv, abs/2403.13787.

Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. ArXiv, abs/2207.01780.

Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. 2023. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852.
Chris Yuhao Liu, Liang Zeng, Jiaci Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024a. Skywork-reward: Bag of tricks for reward modeling in lms. arXiv preprint arXiv:2410.18451.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chat-GPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems.

Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024b. Evaluating language models for efficient code generation. In First Conference on Language Modeling.

Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. 2024c. Rm-bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint arXiv:2410.16184.

Zhihan Liu, Shenao Zhang, Yongfei Liu, Boyi Liu, Yingxiang Yang, and Zhaoran Wang. 2024d. Dstc: Direct preference learning with only self-generated tests and code to improve code lms. arXiv preprint arXiv:2411.13611.

Zhihan Liu, Shenao Zhang, Yongfei Liu, Boyi Liu, Yingxiang Yang, and Zhaoran Wang. 2024e. Dstc: Direct preference learning with only self-generated tests and code to improve code lms. ArXiv, abs/2411.13611.

Anton Lozhkov, Raymond Li, Loumba Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtiar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173.

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcode: Empowering code large language models with evol-instruct.

Nvidia. .; Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaei, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Mahabaleshwarkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostafa Patwary, Krzysztof Pawelec, Wei Ping, Shrimai Prabhumoyle, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Ji- aqi Zeng, Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, and Chen Zhu. 2024. Nemotron-4 340b technical report. Preprint, arXiv:2406.11704.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022a. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022b. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artym Kozechnikov, I. Evtimov, Joanna Bitton, Manish P Bhatt, Cristian Cantón Ferrer, Aaron Gratiafiori, Wenhao Xiong, Alexandre D’efossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code llama: Open foundation models for code. ArXiv, abs/2308.12950.

John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and P. Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. CoRR, abs/1506.02438.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. ArXiv, abs/1707.06347.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, JunMei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, abs/2402.03300.

Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K Reddy. 2023a. Execution-based code generation using deep reinforcement learning. Transactions on Machine Learning Research.

Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. 2023b. Execution-based code generation using deep reinforcement learning. ArXiv, abs/2301.13816.

Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert
Bastani, and Amir Yazdanbakhsh. 2023. Learning performance-improving code edits. arXiv preprint arXiv:2302.07867.

Benjamin Steenhock, Michele Tufano, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. Reinforcement learning from automatic feedback for high-quality unit test generation. arXiv preprint arXiv:2310.02368.

Chuyue Sun, Ying Sheng, Oded Padon, and Clark W. Barrett. 2023. Clover: Closed-loop verifiable code generation. In SAIV.

Haozhe Wang, Long Li, Chao Qu, Fengming Zhu, Weidi Xu, Wei Chu, and Fangzhen Lin. 2025. Learning autonomous code integration for math language models. ArXiv.

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Liming Zhang. 2024. Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. 2024a. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024b. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122.

Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921.

Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. 2023. Algo: Synthesizing algorithmic programs with generated oracle verifiers. ArXiv, abs/2305.14591.

Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301.

Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024a. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyuan Luo, Zhangchi Feng, and Yongqiang Ma. 2024b. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand. Association for Computational Linguistics.

Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadiira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877.
A Appendix

A.1 More related works

LLM for Code Generation  Large language models (LLMs) have demonstrated significant potential in code generation. Due to the unique nature of coding tasks, specialized coding models such as Code Llama (Rozière et al., 2023) and Qwen Coder (Hui et al., 2024b; Yang et al., 2024a) were developed shortly after the emergence of general-purpose LLMs. These models typically undergo a two-phase training process: pre-training and fine-tuning. During pre-training, they are exposed to extensive coding corpora sourced from various internet platforms, including raw text, GitHub repositories, and pull requests. This is followed by supervised fine-tuning, which enhances their instruction-following capabilities. To assess the performance of these models in code generation, several benchmarks have been established, including MBPP (Austin et al., 2021), HumanEval (Chen et al., 2021), EvalPlus (Liu et al., 2023, 2024b), Big Code Bench (Zhuo et al., 2024), and Live Code Bench (Jain et al., 2024). These benchmarks usually include a series of prompts or problems for the LLMs to solve, and they also contain test cases to assess the correctness of the generated code.

Reward Models  Reward models play a crucial role in aligning LLMs by assigning scalar values to response pairs based on specific evaluation criteria, such as human preference (Ouyang et al., 2022b) and accuracy (Zhang et al., 2025). They are widely used in reinforcement learning with human feedback (RLHF) to refine model behavior and in Best-of-N sampling to enhance test-time performance. However, while general-purpose reward models are effective for assessing human preference, they often struggle with specialized domains like mathematics and coding due to the complexity of these tasks. For instance, even top-ranked reward models from Reward Bench (Lambert et al., 2024b), such as Skywork-RM (Liu et al., 2024a), have difficulty providing reliable rewards for these domains. To address this issue, task-specific reward models have been developed, such as Qwen-2.5-Math-PRM (Zhang et al., 2025) for mathematical reasoning. However, coding reward models have remained largely absent due to the lack of reliable training signals—an issue that our proposed AceCODE-RM aims to address.

A.2 AceCODE-RM Model Breakdown

In this section, we present the different models involved in the training and evaluation process of AceCODE-RM in a concise table for further clarification.

<table>
  <tr>
    <th colspan="2">AceCodeRM-7B</th>
  </tr>
  <tr>
    <td>Backbone</td>
    <td>Qwen2.5-Coder-7B-Instruct</td>
  </tr>
  <tr>
    <td>Training Responses (Preference Pairs) Generation</td>
    <td>Qwen2.5-Coder-7B-Instruct</td>
  </tr>
  <tr>
    <td>Inference Model</td>
    <td>Any Model (Qwen2.5-Coder-7B-Instruct, Llama-3.1-8B-Instruct, etc.)</td>
  </tr>
  <tr>
    <th colspan="2">AceCodeRM-32B</th>
  </tr>
  <tr>
    <td>Backbone</td>
    <td>Qwen2.5-Coder-32B-Instruct</td>
  </tr>
  <tr>
    <td>Training Responses (Preference Pairs) Generation</td>
    <td>Qwen2.5-Coder-32B-Instruct</td>
  </tr>
  <tr>
    <td>Inference Model</td>
    <td>Any Model (Qwen2.5-Coder-7B-Instruct, Llama-3.1-8B-Instruct, etc.)</td>
  </tr>
</table>

Table 9: Models used during the training and evaluation process of AceCODE-RM.
A.3 Prompt Template

<table>
  <tr>
    <th>system:</th>
    <td>You are an AI assistant that helps people with python coding tasks.</td>
  </tr>
  <tr>
    <th>user:</th>
    <td>You are the latest and best bot aimed at transforming some code snippet into a leetcode style question. You will be provided with a prompt for writing code, along with a reference program that answers the question. Please complete the following for me:
      1. Come up with a leetcode style question which consists of a well-defined problem. The generated question should meet the following criteria:
        a. The question is clear and understandable, with enough details to describe what the input and output are.
        b. The question should be solvable by only implementing 1 function instead of multiple functions or a class. Therefore, please avoid questions which require complicated pipelines.
        c. The question itself should not require any access to external resource or database.
        d. Feel free to use part of the original question if necessary. Moreover, please do not ask for runtime and space complexity analysis or any test cases in your response.
      2. Based on the modified question that you generated in part 1, you need to create around 20 test cases for this modified question. Each test case should be independent assert clauses. The parameters and expected output of each test case should all be constants, **without accessing any external resources**.

      Here is the original question:
      {instruction}

      Here is the reference program that answers the question:
      ```python
      {program}
      ```

      Now give your modified question and generated test cases in the following json format:
      {"question": ..., "tests":["assert ...", "assert ..."]}.
    </td>
  </tr>
</table>

Table 10: Prompt Used for Converting Seed Code Dataset into LeetCode-style Questions and Test Cases
<table>
  <tr>
    <th>system:</th>
    <td>You are an AI assistant that helps people with python coding tasks.</td>
  </tr>
  <tr>
    <th>user:</th>
    <td>You are the latest and best bot aimed at transforming some code snippet into a leetcode style question. You will be provided with a reference program. Please complete the following for me:
      <ol>
        <li>Come up with a leetcode style question which consists of a well-defined problem. The generated question should meet the following criteria:
          <ul>
            <li>The question is clear and understandable, with enough details to describe what the input and output are.</li>
            <li>The question should be solvable by only implementing 1 function instead of multiple functions or a class. Therefore, please avoid questions which require complicated pipelines.</li>
            <li>The question itself should not require any access to external resource or database.</li>
            <li>Feel free to use part of the original question if necessary. Moreover, please do not ask for runtime and space complexity analysis or any test cases in your response.</li>
          </ul>
        </li>
        <li>Based on the modified question that you generated in part 1, you need to create around 20 test cases for this modified question. Each test case should be independent assert clauses. The parameters and expected output of each test case should all be constants, **without accessing any external resources**.</li>
      </ol>
      Here is the reference program:
      ```python
      {program}
      ```
    </td>
  </tr>
</table>

Table 11: Prompt Used for Converting Seed Code Dataset using only the reference program without instruction into LeetCode-style Questions and Test Cases
A.4 Case Studies on HumanEval

<table>
  <tr>
    <th>ID</th>
    <th>HumanEval/141</th>
    <th>Baseline Score</th>
    <th>Pass</th>
    <th>RL Score</th>
    <th>Fail</th>
  </tr>
  <tr>
    <td colspan="6">
      def file_name_check(file_name):
      # Create a function which takes a string representing a file's name, and returns 'Yes' if the file's name is valid, and returns 'No' otherwise.
      A file's name is considered to be valid if and only if all the following conditions are met:
      - There should not be more than three digits ('0'-9') in the file's name.
      - The file's name contains exactly one dot .
      - The substring before the dot should not be empty, and it starts with a letter from the latin alphabet ('a'-z' and 'A'-Z').
      - The substring after the dot should be one of these: ['txt', 'exe', 'dll']
      Examples:
      file_name_check("example.txt") # == 'Yes'
      file_name_check("example.dll") # == 'No' (the name should start with a latin alphabet letter)
      ***
    </td>
  </tr>
  <tr>
    <th colspan="2">Baseline's Response</th>
    <th colspan="4">RL's Response:</th>
  </tr>
  <tr>
    <td colspan="2">
      def file_name_check(file_name):
      # Create a function which takes a string representing a file's name, and returns 'Yes' if the file's name is valid, and returns 'No' otherwise.
      A file's name is considered to be valid if and only if all the following conditions are met:
      - There should not be more than three digits ('0'-9') in the file's name.
      - The file's name contains exactly one dot .
      - The substring before the dot should not be empty, and it starts with a letter from the latin alphabet ('a'-z' and 'A'-Z').
      - The substring after the dot should be one of these: ['txt', 'exe', 'dll']
      Examples:
      file_name_check("example.txt") # == 'Yes'
      file_name_check("example.dll") # == 'No' (the name should start with a latin alphabet letter)
      ***
      # Split the file name by the dot
      parts = file_name.split('.')
      # Check if there is exactly one dot
      if len(parts) != 2:
        return 'No'
      # Check if the substring before the dot is not empty and starts with a letter
      if not parts[0] or not parts[0][0].isalpha():
        return 'No'
      # Check if the substring after the dot is one of the allowed extensions
      if parts[1] not in ['txt', 'exe', 'dll']:
        return 'No'
      # Count the number of digits in the file name
      digit_count = sum(c.isdigit() for c in file_name)
      # Check if there are more than three digits
      if digit_count > 3:
        return 'No'
      # If all conditions are met, the file name is valid
      return 'Yes'
    </td>
    <td colspan="4">
      import re
      def file_name_check(file_name):
      # Create a function which takes a string representing a file's name, and returns 'Yes' if the file's name is valid, and returns 'No' otherwise.
      A file's name is considered to be valid if and only if all the following conditions are met:
      - There should not be more than three digits ('0'-9') in the file's name.
      - The file's name contains exactly one dot .
      - The substring before the dot should not be empty, and it starts with a letter from the latin alphabet ('a'-z' and 'A'-Z').
      - The substring after the dot should be one of these: ['txt', 'exe', 'dll']
      Examples:
      file_name_check("example.txt") # == 'Yes'
      file_name_check("example.dll") # == 'No' (the name should start with a latin alphabet letter)
      *****
      # Regular expression to match the file name criteria
      pattern = r"^[a-zA-Z][a-zA-Z0-9][0-9]{0,3}\.[txt|exe|dll]$"
      # Check if the file name matches the pattern
      if re.match(pattern, file_name):
        return "Yes"
      else:
        return "No"
    </td>
  </tr>
</table>

Figure 2: In this example, the RL model took a more advanced approach and attempted to use regular expression matching. However, the regular expression it came up is not correct as it did not include whitespace and did not handle the constraint "there should not be more than three digits ('0'-'9') in the file's name" correctly.
<table>
  <tr>
    <th>ID</th>
    <th>HumanEval/10</th>
    <th>Baseline Score</th>
    <th>Fail</th>
    <th>RL Score</th>
    <th>Pass</th>
  </tr>
  <tr>
    <td colspan="6">
      def is_palindrome(string: str) -> bool:
      *** Test if given string is a palindrome ***
      return string == string[::-1]

      def make_palindrome(string: str) -> str:
      # Find the shortest palindrome that begins with a supplied string.
      Algorithm idea is simple:
      - Find the longest prefix of supplied string that is a palindrome.
      - Append to the end of the string reverse of a string prefix that comes before the palindromic suffix.
      >>> make_palindrome("")
      ''
      >>> make_palindrome('cat')
      'catac'
      >>> make_palindrome('catra')
      'catracat'

      Baseline's Response:
      def is_palindrome(string: str) -> bool:
      *** Test if given string is a palindrome ***
      return string == string[::-1]
      def make_palindrome(string: str) -> str:
      # Find the shortest palindrome that begins with a supplied string.
      Algorithm idea is simple:
      - Find the longest prefix of supplied string that is a palindrome.
      - Append to the end of the string reverse of a string prefix that comes before the palindromic suffix.
      >>> make_palindrome("")
      ''
      >>> make_palindrome('cat')
      'catac'
      >>> make_palindrome('catra')
      'catracat'

      if is_palindrome(string):
        return string

      for i in range(len(string), -1, -1):
        if is_palindrome(string[:i]):
          return string + string[i:-1]

      RL's Response:
      def is_palindrome(string: str) -> bool:
      *** Test if given string is a palindrome ***
      return string == string[::-1]
      def make_palindrome(string: str) -> str:
      # Find the shortest palindrome that begins with a supplied string.
      Algorithm idea is simple:
      - Find the longest prefix of supplied string that is a palindrome.
      - Append to the end of the string reverse of a string prefix that comes before the palindromic suffix.
      >>> make_palindrome("")
      ''
      >>> make_palindrome('cat')
      'catac'
      >>> make_palindrome('catra')
      'catracat'

      if is_palindrome(string):
        return string

      for i in range(len(string)):
        if is_palindrome(string[:i]):
          return string + string[i:-1]
        return string + string[:1]
    </td>
  </tr>
</table>

Figure 3: In this example, the RL model correctly caught the error where the baseline did not consider the case where the whole string has to be repeated in order to create a palindrome.

<table>
  <tr>
    <th>ID</th>
    <th>HumanEval/106</th>
    <th>Baseline Score</th>
    <th>Fail</th>
    <th>RL Score</th>
    <th>Pass</th>
  </tr>
  <tr>
    <td colspan="6">
      def fn(i):
      *** Implement the function f that takes n as a parameter,
      and returns a list of size n such that the value of the element at index i is the factorial of i if i is even
      and sum of numbers from 1 to i otherwise.
      i starts from 1
      the factorial of i is the multiplication of the numbers from 1 to i (1 * 2 * ... * i).
      Example:
      f(5) == [1, 2, 6, 24, 15]

      Baseline's Response:
      def factorial(num):
      ***Helper function to calculate factorial of a number.***
      if num == 0 or num == 1:
        return 1
      else:
        return num * factorial(num - 1)

      def sum_of_numbers(num):
      ***Helper function to calculate sum of numbers from 1 to num.***
      return sum(range(1, num + 1))

      def fn(i):
      ***Function to generate a list based on the problem description.***
      result = []
      for i in range(1, n + 1):
        if i % 2 == 0:
          result.append(factorial(i))
        else:
          result.append(sum_of_numbers(i))
      return result

      RL's Response:
      def factorial(num):
      ***Helper function to calculate factorial of a number.***
      if num == 0 or num == 1:
        return 1
      result = 1
      for i in range(2, num + 1):
        result *= i
        return result

      def sum_of_numbers(num):
      ***Helper function to calculate sum of numbers from 1 to num.***
      return sum(range(1, num + 1))

      def fn(i):
      ***Function to generate a list based on the problem description.***
      result = []
      for i in range(1, n + 1):
        if i % 2 == 0:
          result.append(factorial(i))
        else:
          result.append(sum_of_numbers(i))
      return result
    </td>
  </tr>
</table>

Figure 4: In this example, while both codes would have identical output, the baseline’s output is slow due to the recursive calls. By using a for loop instead, the RL model’s code’s runtime is half of that of the baseline’s. Therefore, it passed the test whereas the baseline’s code got a time-out.

Reviewer paper summaries: This paper presents a reinforcement learning (RL) framework for code generation models, which is enhanced by automated test-case synthesis. The authors introduce AceCode-87K, a dataset consisting of 87K coding problems and 1.38M test cases, generated using GPT-4o-mini and filtered with Qwen2.5-Coder-32B. They train reward models (AceCode-RM) using Bradley-Terry loss on preference pairs based on test-case pass rates. The RL process employs the Reinforce++ algorithm with the reward models, as well as R1-style training with test-case pass rewards. The trained 7B models  (Qwen2.5-7B-Instruct, Qwen2.5-Coder-7B-Base, and Qwen2.5-Coder-7B-Instruct) demonstrate improvements across code generation benchmarks. Notably, R1-style training from Qwen2.5-Coder-7B-Base achieves significant improvements with minimal training steps.
Summary of strengths: Innovative Pipeline: The authors present their work as the first approach to reward model training and RL tuning for code generation using large-scale, reliable test-case synthesis.

Comprehensive Experiments: The paper conducts extensive experiments, validating the effectiveness of the reward model and RL process across multiple models and settings.

Promising Evaluation Results: The trained reward models demonstrate effectiveness in various best-of-n settings. Additionally, the trained policy model achieves consistent performance improvements across multiple code generation benchmarks.
Summary of weaknesses: Test-Case Quality Dependency: The approach relies heavily on GPT-4o-mini and Qwen2.5-Coder-32B for test-case generation and filtering. The potential hallucinations or biases may propagate, as the effectiveness of the RL process is closely tied to the quality of the generated test cases.

Limited Improvement for Strong Models: The RL process shows diminishing returns when starting with already strong models (e.g., Qwen2.5-Coder-7B-Instruct). And a similar trend could be observed in the ACECode-RM's Best-of-N results.
Comments, suggestions and typos: This paper conducts RL by leveraging data from automated test-case synthesis. A key concern is the quality of the test cases. While the authors acknowledge the limitations, the approach relies heavily on GPT-4o-mini and Qwen2.5-Coder-32B, which may limit the effectiveness of the following RL process. It would be beneficial to validate the test cases through more diverse evaluations or human annotation to ensure their quality.

The improvements for strong models are limited. In particular, the RL process shows diminishing returns when starting with already strong models (e.g., Qwen2.5-Coder-7B-Instruct). And a similar trend could be observed in the ACECode-RM's Best-of-N results. 

Another issue is the lack of discussion regarding the chosen models. The paper involves different models, such as models for automated test-case synthesis, inference models for reward model (RM) evaluation and policy models for RL evaluation. It would strengthen the paper’s soundness to provide a discussion on the rationale behind these choices.

Additionally, the authors should explicitly commit to releasing AceCode-87K, the code, and the trained models to enhance reproducibility.

Moreover, the existing models used for sampling programs in preference pairs construction are not clearly stated in line 249.

Lastly, there is a misleading paper structure: At the beginning of the Methodology section (line 123), the authors state, "In this section, we will introduce the overall methodology of ACECODER…begin with formulation …then…Finally…", yet only the Problem Formulation is included in this section.
Overall assessment scores: 3
Metareview score: 3.5
Metareview: This paper presents AceCoder, a reinforcement learning framework aimed at improving code generation models via automated test-case synthesis and reward model training. To address the lack of reliable reward signals and large-scale test data, the authors construct AceCode-87k, a dataset containing 87k coding problems and 1.38M test cases, generated and filtered using large language models. A reward model is trained using the Bradley-Terry loss on preference pairs based on test case pass rates. The RL process employs Reinforce++ algorithms with this reward model, and R1-style training with test case pass rates. Experimental results on several models show performance gains on standard code generation benchmarks.
Summary of reasons to publish: The authors present this work as the first approach that employs RL training to increase the code generation performance of LLMs, using automatically synthesized test cases.

The experimental results show that AceCoder-RM increases several models' performance on the BoN setting. AceCoder's performance after RL training outperforms baselines on several benchmarks.

The paper is well-written and well-structured.