CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning

Authors: Huimu Yu, Xing W, Weidong Yin, Songlin Hu
Primary area: foundation or frontier models, including LLMs
Keywords: Preference Model Pretraining

Abstract:
Large language models (LLMs) have achieved notable advancements in natural language understanding and generation, driven by scalable pretraining and advanced finetuning techniques. However, improving reasoning abilities in LLMs, particularly through reinforcement learning from human feedback (RLHF), remains a challenge due to the scarcity of high-quality preference data, which is often labor-intensive to annotate and essential for reward model (RM) finetuning. To alleviate this issue, we introduce CodePMP, a scalable preference model pretraining (PMP) pipeline that leverages vast amounts of code-preference pairs synthesized from publicly available, high-quality source code. CodePMP improves the sample efficiency of RM finetuning by sufficiently pre-training preference models on synthesized code-preference pairs. In addition to validating CodePMP on widely used mathematical reasoning tasks (GSM8K, MATH), we also demonstrate its effectiveness on logical reasoning benchmarks (ReClor, LogiQA). The results consistently indicate that CodePMP significantly improves the reasoning performance of large language models (LLMs). Furthermore, our findings underscore the critical role of scalable preference model pretraining (PMP) in achieving efficient reward modeling.

Decision: reject
Reviewer scores: 3, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730077763706

thread_type:
official_review

<thread_object>
creator: reviewer_m96b

<content>
summary:
This paper presents CodePMP, a novel pipeline for preference model pretraining aimed at enhancing the reasoning capabilities of large language models (LLMs). The primary challenge addressed is the scarcity of high-quality preference data, which is crucial for reinforcement learning from human feedback (RLHF) but costly to annotate. The authors propose a solution by leveraging code-preference pairs synthesized from publicly available high-quality source code, such as from GitHub, to improve reward model finetuning.

The CodePMP pipeline generates millions of code-preference pairs by using two different code models (a stronger model and a weaker model) to create a “chosen” and “rejected” code response for a given code prompt. These pairs are then used to pretrain the preference models, leading to better sample efficiency and improved reasoning performance in downstream tasks like mathematical reasoning (GSM8K, MATH) and logical reasoning (ReClor, LogiQA2.0).

strengths:
**1. Efficiency in Reward Model Fine-Tuning**


CodePMP significantly enhances the sample efficiency of reward model (RM) fine-tuning, as the experiments indicate that fewer samples are required to achieve high accuracy. The paper shows that RMs initialized with CodePMP outperform directly fine-tuned models, even with smaller amounts of data. This efficiency makes the method appealing for real-world applications where data collection can be expensive or time-consuming.

**2. Scalable Data Generation**


One of the major strengths of CodePMP is its ability to generate large-scale preference data using code-preference pairs. By synthesizing data from publicly available source code on platforms like GitHub, the authors address the scarcity of high-quality preference data. This reduces the reliance on costly human annotation, making the pretraining process significantly more scalable and potentially more cost-effective than traditional methods relying solely on human feedback.

**3. Leveraging Code for Reasoning Tasks**

The paper introduces an innovative use of GitHub-sourced code to enhance reasoning in LLMs. By leveraging the logical structure of code, the authors create code-preference pairs, where stronger models generate "accepted" responses and weaker models generate "rejected" ones. This approach capitalizes on the abundance of publicly available code on GitHub, using it to simulate reasoning tasks. The structured nature of code makes it an ideal dataset for pretraining models to improve problem-solving and logical reasoning capabilities, aligning with recent research showing that training on code boosts reasoning performance in LLMs.

weaknesses:
**1. Lack of Examples** 

The paper lacks clear and detailed examples that illustrate how CodePMP actually solves reasoning tasks using code-preference pairs. While the overall methodology is explained, it's hard for the reader to grasp how reasoning is specifically improved through the use of code data. The absence of step-by-step examples of how the code-preference pairs are used to train the reward model leaves an important gap. For instance, showing a few real examples of code, its corresponding “chosen” and “rejected” responses, and how these responses contribute to solving a reasoning task would be highly valuable. The appendix, while providing some additional technical details, does not adequately fill this gap. 


**2. Limited Baselines**

The paper’s comparison to **alternative methods** is quite limited. It focuses primarily on showing the improvements over direct RM fine-tuning without pretraining, but fails to benchmark CodePMP against other advanced reasoning techniques such as:

Chain-of-Thought (CoT) prompting, which has been shown to significantly improve reasoning performance by breaking down tasks into intermediate steps.
Search-based methods like Monte Carlo Tree Search (MCTS), which have been effective in enhancing reasoning capabilities in language models through external computation and verification steps.
A reader would expect a thorough comparison of CodePMP against these methods, especially since these are widely used in the current state-of-the-art LLM systems. Additionally, the paper should clarify:

How does CodePMP stack up against CoT and search-based approaches in the same experimental setup?
What are the strengths and weaknesses of CodePMP relative to these techniques?

**3. Challenges in Maintaining Data Quality with Evolving LLM Capabilities**

A key limitation of the CodePMP methodology is its reliance on two distinct models—one to generate "accepted" (high-quality) responses and another to produce "rejected" (low-quality) responses. While this setup works for synthesizing large quantities of training data, there is a valid concern that as LLMs improve, the gap between the two models could narrow, leading to mismatches in the data quality. For example, there is a risk of **data distribution drift:** As the strong model improves, the generated "accepted" responses may no longer reflect the kind of reasoning tasks the preference model was initially trained on, leading to a mismatch between the training data and the actual data seen in practice. The system might overfit to synthetic training data that does not generalize well to the actual tasks faced by stronger LLMs.

questions:
Given that CodePMP relies heavily on code data, how well do you expect it to generalize to reasoning tasks that do not have a structured, logical basis like code (e.g., common-sense reasoning, social interactions, or natural language inference)? Have you explored tasks beyond mathematical and logical reasoning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730584146320

thread_type:
official_review

<thread_object>
creator: reviewer_zxhn

<content>
summary:
This work introduce CodePMP, a pipeline that create synthesize code-preference pairs in order to have more high quality preference data. The authors show that a better reward model can be obtained with these preference pair data. And it also shows the improvements in several reasoning tasks (GSM8K, MATH, ReClor, LogiQA2.0) with these synthesize code-preference pairs data.

strengths:
1. A pipeline that create synthesize code-preference pairs is introduce in this work. It can help to solve the scarcity of high-quality preference data if it is working well.

2. Large improvements are achieved on several reasoning tasks (GSM8K, MATH, ReClor, LogiQA2.0) with these synthesize code-preference pairs data.

3. Some details of CodePMP are shown. These can be helpful to the community.

weaknesses:
1. The experiments details are missing or confused. It is better to clarify in the next version. Please check the following Questions section for more details. 

2. In the experiments, only two Qwen2 models are used for the evaluation. Other model family results can be used for the verification of the methods.

questions:
1. In the Data Construction, a description summarizer is used to generate prompts that describe the code. There is no any detail on this part. And what is quality of constructed data pair? There is no any related analysis in this work. It is not a human-annotated dataset. So it is important to show the data quality.  

2. In Section 4.2.2, how the different number of N candidate responses are generated? Is Qwen2 model used to sample these candidate response? There is no details about this.

3. In Figure 4, the author claims that: RM with Code PMP initialization consistently outperform. However, it seems not right. Figure 4 (g) shown a counter part case. RM with Code PMP initialization is consistently  worse than RM without Code PMP initialization. Please check in detail.

4. As shown in Figure 5, increasing the number of code-preference pairs consistently improve accuracy. Look like there is still some improvement space. Do you think more preference data can be obtained without too much work?

Other questions: is the PMP data planed to be released? how many code-preference pairs? (only file number and token number are shown, e.g, Python files, with 20 million files and 13.1 billion tokens,)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713034536

thread_type:
official_review

<thread_object>
creator: reviewer_2fuy

<content>
summary:
The paper presents codepmp, a scalable pretraining method designed to improve the reward model by leveraging synthesized code-preference pairs. CodePMP collects a vast repository of publicly available source code from GitHub and leverages a strong and a weak model to generate synthetic preference pairs to pretrain a reward model. Utilizing this pretrained reward model, LLM's ability to solve reasoning tasks is largely improved by best of N sampling.

strengths:
1. This work proposes an interesting idea of preference pre-training.
2. This method is automated, reducing the dependency on manually annotated preference data.
3. With best-of-N strategy, CodePMP could improve LLM's reasoning performance.

weaknesses:
1. Unfortunately, the effectiveness of the RM was not fully verified, e.g. by using the RM for RFT or PPO training.
2. It remains unclear whether and how code training could help reasoning tasks in natural language. It would be great if the authors could have explored more on relationship between coding and reasoning tasks for model training.
3. No mention of whether the dataset will be open sourced.
4. The models used in data construction are limited. It would be helpful to verify the generalization with more number of generators, at least the ones used to construct the training data.
5. Evalution tasks are limited. Regular coding benchmarks like humaneval and mbpp should be relevant.

questions:
1. Figure 4 and 5 are too small to watch.
2. Why CodePMP can help reward modeling in mathematical tasks? Please give more analysis in the paper.
3. Reference formatting issues, such as line 429.
4. Would you consider using a larger reward model to verify the scalability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730740406609

thread_type:
official_review

<thread_object>
creator: reviewer_yons

<content>
summary:
The paper presents CodePMP, a novel and scalable preference model pretraining (PMP) pipeline aimed at enhancing the reasoning capabilities of Large Language Models (LLMs). 

CodePMP builds its dataset in two stages: 1) generating instructions from GitHub files using a CodeLLM, and 2) generating two responses for each instruction from a strong CodeLLM and a weak CodeLLM, which are the preferred and rejected response pairs. 28 million such response pairs are constructed.

After pre-training an LLM on these 28 million pairs, CodePMP improves RM finetuning efficiency. The resulting reward model built upon CodePMP can then be leveraged to improve performance on reasoning tasks, including solving mathematical and logical problems.

strengths:
- The proposed method is highly scalable, allowing it to create 28 million preferred and rejected response pairs.

- The reward model can improve two different reasoning tasks (math reasoning and logic reasoning).

weaknesses:
- CodePMP creates preference pairs for coding tasks, but coding tasks are not evaluated in experiments. 

- The CodePMP data is constructed by deepseek-coder-instruct. It would be interesting to see whether CodePMP can further improve deepseek-coder-instruct on coding tasks.

- The reward model is initialized using Qwen2 models (Qwen2-1.5B and Qwen2-7B), which are more capable in math reasoning than the math generator MetaMath-Mistral-7B in Section 4.1.3. A more meaningful setting would be to examine whether the Qwen2 based reward model can further improve the performance of Qwen2 or Qwen2-Math on math reasoning tasks.

- Majority voting should be included as a baseline in Figure 3.

- Further analysis is needed to understand why the coding-based CodePMP contributes to improvements in math and logical reasoning.

questions:
- Is Majority voting@K comparable with reranking top K with a RM in Figure 3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732545937394

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Author Response

comment:
We sincerely appreciate the thorough and thoughtful feedback from all reviewers! Some reviewers expressed concerns about the generality and scalability of the CodePMP approach. We summarize these concerns into three categories:
- Using models from different families as the PMP/RM backbone.
- Employing a stronger Qwen model as the PMP/RM backbone.
- Using a more powerful Generator.
To address these concerns, we have designed a set of supplementary experiments for each category.

**1. Using different families of models as the PMP/RM backbone**
In our paper, we have already used Qwen and **Gemma2**(refer to Section 5.4 and Figure 8) as PMP/RM backbones. Here, we supplement these results by using the recently released Llama3.2 as the PMP/RM backbone. The results are shown in the table below:

- a. RM initialized with CodePMP consistently outperforms directly trained RM in reasoning tasks, demonstrating obvious performance improvements.
- b. RM initialized with CodePMP exhibits robustness when N increases, with a delayed performance drop compared to directly trained RM. 

Results on Gemma2 and Llama3.2 highlight the general applicability of CodePMP, significantly enhancing models from other families as well.

| BoN  | w/o pmp (GSM8K) | **w/ pmp (GSM8K)** | w/o pmp (MATH) | **w/ pmp (MATH)** | w/o pmp (Reclor) | **w/ pmp (Reclor)** | w/o pmp (Logiqa-v2) | **w/ pmp (Logiqa-v2)** |
|---------|-----------------|-----------------|----------------|-----------------|------------------|-----------------|-----------------|-------------------|
| N=1  | 0.7718  | 0.7718  | 0.298  | 0.298  |  |  |  |  |
| N=4  | 0.7953  | 0.8218  | 0.332  | 0.35  |  0.574  |  **0.664**  |  0.5325  |  **0.5804**  |
| N=32  | 0.7612  | **0.8302**  | 0.3  | **0.354**  |  |  |  |  |
| N=256  | 0.721  | 0.8074  | 0.252  | 0.328  |  |  |  |  |

**2. Using a stronger Qwen model as the PMP/RM backbone**
To evaluate the effects of a stronger Qwen model as the PMP/RM backbone, we conducted experiments using the largest model in the Qwen2 series, Qwen2-72B. The results are summarized in the table below:

- a. RM initialized with CodePMP consistently outperforms directly trained RM across all N values in reasoning tasks.
- b. Stronger backbone models exhibit greater robustness. For example, in the MATH dataset, RM initialized with CodePMP maintains the upward trend even when N=256.

This demonstrates the scalability of CodePMP, as it enhances a larger and more powerful model.

| BoN  | w/o pmp (GSM8K) | **w/ pmp (GSM8K)** | w/o pmp (MATH) | **w/ pmp (MATH)** | w/o pmp (Reclor) | **w/ pmp (Reclor)** | w/o pmp (Logiqa-v2) | **w/ pmp (Logiqa-v2)** |
|-------|-----------------|--------------------|----------------|-------------------|------------------|-----------------|-----------------|--------------------|
|  |  |  |  |  |  |  |  |  |
| N=1  | 0.7718  | 0.7718  | 0.298  | 0.298  |  |  |  |  |
| N=4  | 0.8453  | 0.8453  | 0.424  | 0.424  |  0.894  |  **0.918**  |  0.7117  |  **0.7927**  |
| N=32  | 0.8529  | **0.8628**  | 0.488  | 0.5  |  |  |  |  |
| N=256 | 0.8249  | 0.84  | 0.506  | **0.514**  |  |  |  |  |

**3. Using a more powerful Generator**

To test whether CodePMP can improve more powerful Generators, we chose Qwen2-Math-7B, a model specifically optimized for mathematics, as the Generator. (Note: Since Reclor and Logiqa-v2 candidate responses are not generated by the Generator, we evaluate on the GSM8K and MATH datasets.) The results are as follows:

- RM initialized with CodePMP consistently outperforms directly trained RM across all N values in mathematical reasoning tasks. 
This shows that CodePMP can even bring significant improvements to stronger generator optimized specifically for mathematics.

| BoN  | GSM8K w/o PMP | GSM8K **w/ PMP** | MATH w/o PMP | MATH **w/ PMP** |
|--------|---------------|--------------|--------------|-------------|
| N=1  | 0.8635  | 0.8635  | 0.698  | 0.698  |
| N=4  | 0.8544  | **0.8931**  | 0.690  | **0.724**  |
| N=32 | 0.8446  | 0.8795  | 0.643  | 0.698  |
| N=256| 0.8256  | 0.8590  | 0.614  | 0.690  |


**Summary**
Across these three supplementary experiments, CodePMP demonstrates excellent scalability and generality. It provides consistent improvements in reasoning tasks for models of varying capabilities and sizes. We will include the detailed experiment setups in future versions of the paper.
We hope these additional experiments, combined with the extensive experiments (especially the resource-intensive extended experiments) in the paper, effectively address the reviewers' concerns. We also hope that CodePMP will provide new impetus for RLHF research and inspire our peers from a scalable perspective.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces CodePMP, a scalable preference model pretraining pipeline that leverages synthesized code-preference pairs from publicly available, high-quality source code to improve reward model fine-tuning. I appreciate the authors added additional results during the rebuttal period. However, there are some concerns still in the current work: (1)  the evaluation does not demonstrate the results of reward models in model optimization (e.g. DPO), which should be the end goal; (2) Scalability Concerns: While CodePMP is designed to be scalable, LLM models often evolve and would need continuous iteration of preference data to adapt to new LLM capabilities.